Language model: gbert-base-germandpr-reranking Language: German Training data: GermanDPR train set (~ 56MB) Eval data: GermanDPR test set (~ 6MB) Infrastructure : 1x V100 GPU Published : June 3rd, 2021
batch_size = 16 n_epochs = 2 max_seq_len = 512 tokens for question and passage concatenated learning_rate = 2e-5 lr_schedule = LinearWarmup embeds_dropout_prob = 0.1
We use the GermanDPR test dataset as ground truth labels and run two experiments to compare how a BM25 retriever performs with or without reranking with our model. The first experiment runs retrieval on the full German Wikipedia (more than 2 million passages) and second experiment runs retrieval on the GermanDPR dataset only (not more than 5000 passages). Both experiments use 1025 queries. Note that the second experiment is evaluating on a much simpler task because of the smaller dataset size, which explains strong BM25 retrieval performance.
BM25 Retriever without Reranking
BM25 Retriever with Reranking Top 10 Documents
BM25 Retriever without Reranking
BM25 Retriever with Reranking Top 10 Documents
You can load the model in haystack for reranking the documents returned by a Retriever:
... retriever = ElasticsearchRetriever(document_store=document_store) ranker = FARMRanker(model_name_or_path="deepset/gbert-base-germandpr-reranking") ... p = Pipeline() p.add_node(component=retriever, name="ESRetriever", inputs=["Query"]) p.add_node(component=ranker, name="Ranker", inputs=["ESRetriever"]) )
We bring NLP to the industry via open source! Our focus: Industry specific language models & large scale QA systems.
Some of our work:
Get in touch: Twitter | LinkedIn | Website
By the way: we're hiring!