数据集:
castorini/msmarco_v2_doc_segmented_doc2query-t5_expansions
该存储库提供了针对MS MARCO v2文档分割语料库使用docTTTTTquery(有时写作docT5query或doc2query-T5)生成的查询的数据。docTTTTTquery是doc2query系列文档扩展模型的最新版本。其基本思想是训练一个模型,当给定输入文档时,生成可能回答该文档的问题(或更广泛地说,该文档可能相关的查询)。然后,将这些预测的问题(或查询)附加到原始文档中,然后像以前一样对其进行索引。docTTTTTquery模型的命名来自于使用T5作为扩展模型的用法。
所有三个数据集(训练集、开发集和测试集)共享同一个语料库。一个示例的数据条目如下所示:
{ 'docid': '25#0', 'title': 'Autism', 'text': 'Autism is a developmental disorder characterized by difficulties with social interaction and communication, ...' }
加载数据集的示例:
dataset = load_dataset('castorini/msmarco_v2_doc_segmented_doc2query-t5_expansions', data_files='d2q/d2q.jsonl???.gz')
@article{docTTTTTquery, title={From doc2query to {docTTTTTquery}}, author={Nogueira, Rodrigo and Lin, Jimmy}, year={2019} } @article{emdt5, author = "Ronak Pradeep and Rodrigo Nogueira and Jimmy Lin", title = "The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models", journal = "arXiv:2101.05667", year = 2021, }