BanglaNMT 数据集卡片

数据集摘要

这是最大的孟加拉语-英语机器翻译数据集，利用引入的新的句子对齐方法进行策划。

注意：这是作者用于NMT训练的原始数据集的筛选版本。有关完整集，请参阅官方链接。

支持的任务和排行榜

More information needed

语言

• 孟加拉语
• 英语

用途

from datasets import load_dataset
dataset = load_dataset("csebuetnlp/BanglaNMT")

数据集结构

数据示例

下面以JSON格式给出了数据集中的一个示例。

{
  'bn': 'বিমানবন্দরে যুক্তরাজ্যে নিযুক্ত বাংলাদেশ হাইকমিশনার সাঈদা মুনা তাসনীম ও লন্ডনে বাংলাদেশ মিশনের জ্যেষ্ঠ কর্মকর্তারা তাকে বিদায় জানান।',
  'en': 'Bangladesh High Commissioner to the United Kingdom Saida Muna Tasneen and senior officials of Bangladesh Mission in London saw him off at the airport.'
}

数据字段

数据字段如下：

• bn ：表示孟加拉语句子的字符串特征。
• en ：表示英语翻译的字符串特征。

数据划分

数据集创建

More information needed

策划原理

More information needed

源数据

More information needed

More information needed

More information needed

注释

More information needed

More information needed

More information needed

个人和敏感信息

More information needed

使用数据的注意事项

数据的社会影响

More information needed

偏见讨论

More information needed

其他已知限制

More information needed

附加信息

数据集策划者

More information needed

许可信息

该存储库的内容仅限于非商业研究目的。数据集内容的版权归原始版权持有人所有。

引用信息

如果您使用此数据集，请引用以下论文：

@inproceedings{hasan-etal-2020-low,
    title = "Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for {B}engali-{E}nglish Machine Translation",
    author = "Hasan, Tahmid  and
      Bhattacharjee, Abhik  and
      Samin, Kazi  and
      Hasan, Masum  and
      Basak, Madhusudan  and
      Rahman, M. Sohel  and
      Shahriyar, Rifat",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.emnlp-main.207",
    doi = "10.18653/v1/2020.emnlp-main.207",
    pages = "2612--2623",
    abstract = "Despite being the seventh most widely spoken language in the world, Bengali has received much less attention in machine translation literature due to being low in resources. Most publicly available parallel corpora for Bengali are not large enough; and have rather poor quality, mostly because of incorrect sentence alignments resulting from erroneous sentence segmentation, and also because of a high volume of noise present in them. In this work, we build a customized sentence segmenter for Bengali and propose two novel methods for parallel corpus creation on low-resource setups: aligner ensembling and batch filtering. With the segmenter and the two methods combined, we compile a high-quality Bengali-English parallel corpus comprising of 2.75 million sentence pairs, more than 2 million of which were not available before. Training on neural models, we achieve an improvement of more than 9 BLEU score over previous approaches to Bengali-English machine translation. We also evaluate on a new test set of 1000 pairs made with extensive quality control. We release the segmenter, parallel corpus, and the evaluation set, thus elevating Bengali from its low-resource status. To the best of our knowledge, this is the first ever large scale study on Bengali-English machine translation. We believe our study will pave the way for future research on Bengali-English machine translation as well as other low-resource languages. Our data and code are available at https://github.com/csebuetnlp/banglanmt.",
}

贡献

感谢 @abhik1505040 和 @Tahmid 添加了这个数据集。