问题回答上下文数据集卡片

数据集摘要

问题回答上下文是一个用于建模、理解和参与信息搜索对话的数据集。数据实例由两名众包工人之间的互动对话组成：（1）一个学生通过提出一系列自由形式的问题来尽可能多地了解隐藏的维基百科文字，（2）一个教师通过提供来自文本的短摘录（段落）来回答问题。QuAC带来了现有机器理解数据集中没有的挑战：其问题通常更加开放，无法回答，或者只有在对话上下文中才有意义。

支持的任务和排行榜

核心问题涉及预测用于回答关于维基百科章节的问题的文本片段（抽取式问题回答）。由于QuAC的问题包括对话组成部分，每个实例都包括给定问题之前对话中提出的问题和回答的“对话历史记录”，以及一些额外的元数据。

作者提供了 an official evaluation script 条数据用于评估。

语言

数据集中的文本为英文。相关的BCP-47代码是en。

数据集结构

数据实例

验证示例如下（每个对话一个条目）：

{
  'dialogue_id': 'C_6abd2040a75d47168a9e4cca9ca3fed5_0',

  'wikipedia_page_title': 'Satchel Paige',

  'background': 'Leroy Robert "Satchel" Paige (July 7, 1906 - June 8, 1982) was an American Negro league baseball and Major League Baseball (MLB) pitcher who became a legend in his own lifetime by being known as perhaps the best pitcher in baseball history, by his longevity in the game, and by attracting record crowds wherever he pitched. Paige was a right-handed pitcher, and at age 42 in 1948, he was the oldest major league rookie while playing for the Cleveland Indians. He played with the St. Louis Browns until age 47, and represented them in the All-Star Game in 1952 and 1953.',

  'section_title': 'Chattanooga and Birmingham: 1926-29',

  'context': 'A former friend from the Mobile slums, Alex Herman, was the player/manager for the Chattanooga White Sox of the minor Negro Southern League. In 1926 he discovered Paige and offered to pay him $250 per month, of which Paige would collect $50 with the rest going to his mother. He also agreed to pay Lula Paige a $200 advance, and she agreed to the contract. The local newspapers--the Chattanooga News and Chattanooga Times--recognized from the beginning that Paige was special. In April 1926, shortly after his arrival, he recorded nine strikeouts over six innings against the Atlanta Black Crackers. Part way through the 1927 season, Paige\'s contract was sold to the Birmingham Black Barons of the major Negro National League (NNL). According to Paige\'s first memoir, his contract was for $450 per month, but in his second he said it was for $275. Pitching for the Black Barons, Paige threw hard but was wild and awkward. In his first big game in late June 1927, against the St. Louis Stars, Paige incited a brawl when his fastball hit the hand of St. Louis catcher Mitchell Murray. Murray then charged the mound and Paige raced for the dugout, but Murray flung his bat and struck Paige above the hip. The police were summoned, and the headline of the Birmingham Reporter proclaimed a "Near Riot." Paige improved and matured as a pitcher with help from his teammates, Sam Streeter and Harry Salmon, and his manager, Bill Gatewood. He finished the 1927 season 7-1 with 69 strikeouts and 26 walks in 89 1/3 innings. Over the next two seasons, Paige went 12-5 and 10-9 while recording 176 strikeouts in 1929. (Several sources credit his 1929 strikeout total as the all-time single-season record for the Negro leagues, though there is variation among the sources about the exact number of strikeouts.) On April 29 of that season he recorded 17 strikeouts in a game against the Cuban Stars, which exceeded what was then the major league record of 16 held by Noodles Hahn and Rube Waddell. Six days later he struck out 18 Nashville Elite Giants, a number that was tied in the white majors by Bob Feller in 1938. Due to his increased earning potential, Barons owner R. T. Jackson would "rent" Paige out to other ball clubs for a game or two to draw a decent crowd, with both Jackson and Paige taking a cut. CANNOTANSWER',

  'turn_ids': ['C_6abd2040a75d47168a9e4cca9ca3fed5_0_q#0', 'C_6abd2040a75d47168a9e4cca9ca3fed5_0_q#1', 'C_6abd2040a75d47168a9e4cca9ca3fed5_0_q#2', 'C_6abd2040a75d47168a9e4cca9ca3fed5_0_q#3', 'C_6abd2040a75d47168a9e4cca9ca3fed5_0_q#4', 'C_6abd2040a75d47168a9e4cca9ca3fed5_0_q#5', 'C_6abd2040a75d47168a9e4cca9ca3fed5_0_q#6', 'C_6abd2040a75d47168a9e4cca9ca3fed5_0_q#7'],

  'questions': ['what did he do in Chattanooga', 'how did he discover him', 'what position did he play', 'how did they help him', 'when did he go to Birmingham', 'how did he feel about this', 'how did he do with this team', 'What made him leave the team'],

  'followups': [0, 2, 0, 1, 0, 1, 0, 1],

  'yesnos': [2, 2, 2, 2, 2, 2, 2, 2]

  'answers': {
    'answer_starts': [
      [480, 39, 0, 67, 39],
      [2300, 2300, 2300],
      [848, 1023, 848, 848, 1298],
      [2300, 2300, 2300, 2300, 2300],
      [600, 600, 600, 634, 600],
      [2300, 2300, 2300],
      [939, 1431, 848, 848, 1514],
      [2106, 2106, 2165]
    ],
    'texts': [
      ['April 1926, shortly after his arrival, he recorded nine strikeouts over six innings against the Atlanta Black Crackers.', 'Alex Herman, was the player/manager for the Chattanooga White Sox of the minor Negro Southern League. In 1926 he discovered Paige', 'A former friend from the Mobile slums, Alex Herman, was the player/manager for the Chattanooga White Sox of the minor Negro Southern League.', 'manager for the Chattanooga White Sox of the minor Negro Southern League. In 1926 he discovered Paige and offered to pay him $250 per month,', 'Alex Herman, was the player/manager for the Chattanooga White Sox of the minor Negro Southern League. In 1926 he discovered Paige and offered to pay him $250 per month,'],
      ['CANNOTANSWER', 'CANNOTANSWER', 'CANNOTANSWER'],
      ['Pitching for the Black Barons,', 'fastball', 'Pitching for', 'Pitching', 'Paige improved and matured as a pitcher with help from his teammates,'], ['CANNOTANSWER', 'CANNOTANSWER', 'CANNOTANSWER', 'CANNOTANSWER', 'CANNOTANSWER'],
      ["Part way through the 1927 season, Paige's contract was sold to the Birmingham Black Barons", "Part way through the 1927 season, Paige's contract was sold to the Birmingham Black Barons", "Part way through the 1927 season, Paige's contract was sold to the Birmingham Black Barons", "Paige's contract was sold to the Birmingham Black Barons of the major Negro National League (NNL", "Part way through the 1927 season, Paige's contract was sold to the Birmingham Black Barons"], ['CANNOTANSWER', 'CANNOTANSWER', 'CANNOTANSWER'],
      ['game in late June 1927, against the St. Louis Stars, Paige incited a brawl when his fastball hit the hand of St. Louis catcher Mitchell Murray.', 'He finished the 1927 season 7-1 with 69 strikeouts and 26 walks in 89 1/3 innings.', 'Pitching for the Black Barons, Paige threw hard but was wild and awkward.', 'Pitching for the Black Barons, Paige threw hard but was wild and awkward.', 'Over the next two seasons, Paige went 12-5 and 10-9 while recording 176 strikeouts in 1929. ('],
      ['Due to his increased earning potential, Barons owner R. T. Jackson would "rent" Paige out to other ball clubs', 'Due to his increased earning potential, Barons owner R. T. Jackson would "rent" Paige out to other ball clubs for a game or two to draw a decent crowd,', 'Jackson would "rent" Paige out to other ball clubs for a game or two to draw a decent crowd, with both Jackson and Paige taking a cut.']
    ]
  },

  'orig_answers': {
    'answer_starts': [39, 2300, 1298, 2300, 600, 2300, 1514, 2165],
    'texts': ['Alex Herman, was the player/manager for the Chattanooga White Sox of the minor Negro Southern League. In 1926 he discovered Paige and offered to pay him $250 per month,', 'CANNOTANSWER', 'Paige improved and matured as a pitcher with help from his teammates,', 'CANNOTANSWER', "Part way through the 1927 season, Paige's contract was sold to the Birmingham Black Barons", 'CANNOTANSWER', 'Over the next two seasons, Paige went 12-5 and 10-9 while recording 176 strikeouts in 1929. (', 'Jackson would "rent" Paige out to other ball clubs for a game or two to draw a decent crowd, with both Jackson and Paige taking a cut.']
  },
}

数据字段

• dialogue_id：对话的ID。
• wikipedia_page_title：维基百科页面的标题。
• background：主维基百科文章的第一段。
• section_tile：维基百科章节标题。
• context：维基百科章节文本。
• turn_ids：对话转弯的标识列表。每个对话一个ID列表。
• questions：对话中的问题列表。每个对话一个问题列表。
• followups：对话中的后续行动列表。每个对话一个后续行动列表。 y：跟进，m：可能跟进，n：不跟进。
• yesnos：对话中的是/否列表。每个对话一个是/否列表。y：是，n：否，x：都不是。
• answers：问题的答案字典（数据收集的验证步骤）
  • answer_starts：开始偏移的列表列表。对于训练，每个问题一个元素的列表（每个问题一个答案）。
  • texts：回答问题的跨度文本的列表列表。对于训练，每个问题一个元素的列表（每个问题一个答案）。
• orig_answers：原始答案的字典（由对话中的教师提供）
  • answer_starts：开始偏移的列表。
  • texts：回答问题的跨度文本的列表。

数据拆分

QuAC包含来自13,594个对话的98,407个问答对。对话是在3,611个唯一的维基百科文章的8,854个唯一章节上进行的，每个对话包含四到十二个问题。

数据集提供了一个训练/开发集拆分，以保证拆分之间没有章节重叠。此外，开发集和测试集仅包括每个章节的一个对话，而训练集可以在每个章节中有多个对话。开发集和测试实例相比训练集来说，每个实例都有五个参考答案，而不仅仅是一个；我们获取额外的参考答案是为了提高评估的可靠性，因为问题可以有多个有效的答案跨度。测试集不公开提供；相反，研究人员必须将他们的模型提交给 leaderboard ，后者将在我们的隐藏测试集上运行模型。

训练集包含83,568个问题（11,567个对话），而验证集和测试集分别保留了7,354个（1,000个）和7,353个（1,002个）独立的问题。

数据集创建

策划理由

请参考数据集作者的 Datasheet 。

源数据

请参考数据集作者的 Datasheet 。

请参考数据集作者的 Datasheet 。

请参考数据集作者的 Datasheet 。

注释

请参考数据集作者的 Datasheet 。

请参考数据集作者的 Datasheet 。

请参考数据集作者的 Datasheet 。

个人和敏感信息

请参考数据集作者的 Datasheet 。

使用数据的考虑事项

数据的社会影响

请参考数据集作者的 Datasheet 。

讨论偏见

请参考数据集作者的 Datasheet 。

其他已知限制

请参考数据集作者的 Datasheet 。

其他信息

数据集策划者

请参考数据集作者的 Datasheet 。

许可信息

该数据集使用MIT许可证分发。

引用信息

为数据集提供 BibTex 格式的引用。例如：

@inproceedings{choi-etal-2018-quac,
    title = "{Q}u{AC}: Question Answering in Context",
    author = "Choi, Eunsol  and
      He, He  and
      Iyyer, Mohit  and
      Yatskar, Mark  and
      Yih, Wen-tau  and
      Choi, Yejin  and
      Liang, Percy  and
      Zettlemoyer, Luke",
    booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing",
    month = oct # "-" # nov,
    year = "2018",
    address = "Brussels, Belgium",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/D18-1241",
    doi = "10.18653/v1/D18-1241",
    pages = "2174--2184",
    abstract = "We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total). The dialogs involve two crowd workers: (1) a student who poses a sequence of freeform questions to learn as much as possible about a hidden Wikipedia text, and (2) a teacher who answers the questions by providing short excerpts from the text. QuAC introduces challenges not found in existing machine comprehension datasets: its questions are often more open-ended, unanswerable, or only meaningful within the dialog context, as we show in a detailed qualitative evaluation. We also report results for a number of reference models, including a recently state-of-the-art reading comprehension architecture extended to model dialog context. Our best model underperforms humans by 20 F1, suggesting that there is significant room for future work on this data. Dataset, baseline, and leaderboard available at \url{http://quac.ai}.",
}

贡献

感谢 @VictorSanh 添加了这个数据集。