模型:
voidful/dpr-question_encoder-bert-base-multilingual
多语言DPR模型基于bert-base-multilingual-cased。 DPR model DPR repo
训练的问题对数量:644,217 开发集的问题对数量:73,710
*DRCD和MLQA使用来自haystack的脚本进行转换 squad_to_dpr.py
我使用来自 haystack 的脚本
from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer tokenizer = DPRQuestionEncoderTokenizer.from_pretrained('voidful/dpr-question_encoder-bert-base-multilingual') model = DPRQuestionEncoder.from_pretrained('voidful/dpr-question_encoder-bert-base-multilingual') input_ids = tokenizer("Hello, is my dog cute ?", return_tensors='pt')["input_ids"] embeddings = model(input_ids).pooler_output
按照haystack的教程进行操作 Better Retrievers via "Dense Passage Retrieval"
from haystack.retriever.dense import DensePassageRetriever retriever = DensePassageRetriever(document_store=document_store, query_embedding_model="voidful/dpr-question_encoder-bert-base-multilingual", passage_embedding_model="voidful/dpr-ctx_encoder-bert-base-multilingual", max_seq_len_query=64, max_seq_len_passage=256, batch_size=16, use_gpu=True, embed_title=True, use_fast_tokenizers=True)