模型:
voidful/dpr-ctx_encoder-bert-base-multilingual
多语言DPR模型基于bert-base-multilingual-cased。 DPR model DPR repo
训练用的问题对数:644,217 验证用的问题对数:73,710
*DRCD和MLQA使用haystack的脚本进行转换 squad_to_dpr.py
我使用来自 haystack 的脚本
from transformers import DPRContextEncoder, DPRContextEncoderTokenizer tokenizer = DPRContextEncoderTokenizer.from_pretrained('voidful/dpr-ctx_encoder-bert-base-multilingual') model = DPRContextEncoder.from_pretrained('voidful/dpr-ctx_encoder-bert-base-multilingual') input_ids = tokenizer("Hello, is my dog cute ?", return_tensors='pt')["input_ids"] embeddings = model(input_ids).pooler_output
按照haystack的教程进行操作: Better Retrievers via "Dense Passage Retrieval"
from haystack.retriever.dense import DensePassageRetriever retriever = DensePassageRetriever(document_store=document_store, query_embedding_model="voidful/dpr-question_encoder-bert-base-multilingual", passage_embedding_model="voidful/dpr-ctx_encoder-bert-base-multilingual", max_seq_len_query=64, max_seq_len_passage=256, batch_size=16, use_gpu=True, embed_title=True, use_fast_tokenizers=True)