模型:
csebuetnlp/banglishbert
此存储库包含BanglishBERT模型的预训练鉴别器检查点。这是一个 ELECTRA 与Replace Token Detection (RTD)目标在大量孟加拉语和英语语料库上预训练的鉴别器模型。BanglishBERT在许多孟加拉语的NLP任务中取得了最先进的零射击跨语言转移结果。
对于在不同的下游任务上进行微调,如情感分类、命名实体识别、自然语言推理等,请参考官方GitHub上的脚本 repository 。
注意:此模型是使用特定的规范化流程进行预训练的,该流程在 here 中可用。官方GitHub存储库中的所有微调脚本默认使用此规范化。如果您需要为不同任务调整预训练模型,请确保在标记化之前使用此流程对文本单元进行标准化以获得最佳结果。以下是一个基本示例:
from transformers import AutoModelForPreTraining, AutoTokenizer from normalizer import normalize # pip install git+https://github.com/csebuetnlp/normalizer import torch model = AutoModelForPreTraining.from_pretrained("csebuetnlp/banglishbert") tokenizer = AutoTokenizer.from_pretrained("csebuetnlp/banglishbert") original_sentence = "আমি কৃতজ্ঞ কারণ আপনি আমার জন্য অনেক কিছু করেছেন।" fake_sentence = "আমি হতাশ কারণ আপনি আমার জন্য অনেক কিছু করেছেন।" fake_sentence = normalize(fake_sentence) # this normalization step is required before tokenizing the text fake_tokens = tokenizer.tokenize(fake_sentence) fake_inputs = tokenizer.encode(fake_sentence, return_tensors="pt") discriminator_outputs = model(fake_inputs).logits predictions = torch.round((torch.sign(discriminator_outputs) + 1) / 2) [print("%7s" % token, end="") for token in fake_tokens] print("\n" + "-" * 50) [print("%7s" % int(prediction), end="") for prediction in predictions.squeeze().tolist()[1:-1]] print("\n" + "-" * 50)
Model | Params | SC (macro-F1) | NLI (accuracy) | NER (micro-F1) | QA (EM/F1) | BangLUE score |
---|---|---|---|---|---|---|
1236321 | 180M | 27.05 | 62.22 | 39.27 | 59.01/64.18 | 50.35 |
1237321 | 270M | 42.03 | 72.18 | 45.37 | 55.03/61.83 | 55.29 |
1238321 | 550M | 49.49 | 78.13 | 56.48 | 71.13/77.70 | 66.59 |
1239321 | 110M | 48.39 | 75.26 | 55.56 | 72.87/78.63 | 66.14 |
Model | Params | SC (macro-F1) | NLI (accuracy) | NER (micro-F1) | QA (EM/F1) | BangLUE score |
---|---|---|---|---|---|---|
1236321 | 180M | 67.59 | 75.13 | 68.97 | 67.12/72.64 | 70.29 |
1237321 | 270M | 69.54 | 78.46 | 73.32 | 68.09/74.27 | 72.82 |
1238321 | 550M | 70.97 | 82.40 | 78.39 | 73.15/79.06 | 76.79 |
12313321 | 18M | 71.12 | 76.92 | 70.94 | 65.48/70.69 | 71.03 |
1239321 | 110M | 70.61 | 80.95 | 76.28 | 72.43/78.40 | 75.73 |
12315321 | 110M | 72.89 | 82.80 | 77.78 | 72.63/79.34 | 77.09 |
基准测试数据集如下:
如果您使用此模型,请引用以下论文:
@inproceedings{bhattacharjee-etal-2022-banglabert, title = {BanglaBERT: Lagnuage Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla}, author = "Bhattacharjee, Abhik and Hasan, Tahmid and Mubasshir, Kazi and Islam, Md. Saiful and Uddin, Wasi Ahmad and Iqbal, Anindya and Rahman, M. Sohel and Shahriyar, Rifat", booktitle = "Findings of the North American Chapter of the Association for Computational Linguistics: NAACL 2022", month = july, year = {2022}, url = {https://arxiv.org/abs/2101.00204}, eprinttype = {arXiv}, eprint = {2101.00204} }
如果使用规范化模块,请引用以下论文:
@inproceedings{hasan-etal-2020-low, title = "Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for {B}engali-{E}nglish Machine Translation", author = "Hasan, Tahmid and Bhattacharjee, Abhik and Samin, Kazi and Hasan, Masum and Basak, Madhusudan and Rahman, M. Sohel and Shahriyar, Rifat", booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)", month = nov, year = "2020", address = "Online", publisher = "Association for Computational Linguistics", url = "https://www.aclweb.org/anthology/2020.emnlp-main.207", doi = "10.18653/v1/2020.emnlp-main.207", pages = "2612--2623", abstract = "Despite being the seventh most widely spoken language in the world, Bengali has received much less attention in machine translation literature due to being low in resources. Most publicly available parallel corpora for Bengali are not large enough; and have rather poor quality, mostly because of incorrect sentence alignments resulting from erroneous sentence segmentation, and also because of a high volume of noise present in them. In this work, we build a customized sentence segmenter for Bengali and propose two novel methods for parallel corpus creation on low-resource setups: aligner ensembling and batch filtering. With the segmenter and the two methods combined, we compile a high-quality Bengali-English parallel corpus comprising of 2.75 million sentence pairs, more than 2 million of which were not available before. Training on neural models, we achieve an improvement of more than 9 BLEU score over previous approaches to Bengali-English machine translation. We also evaluate on a new test set of 1000 pairs made with extensive quality control. We release the segmenter, parallel corpus, and the evaluation set, thus elevating Bengali from its low-resource status. To the best of our knowledge, this is the first ever large scale study on Bengali-English machine translation. We believe our study will pave the way for future research on Bengali-English machine translation as well as other low-resource languages. Our data and code are available at https://github.com/csebuetnlp/banglanmt.", }