神经机器翻译模型,用于将英语(en)翻译为保加利亚语(bg)。
该模型是 OPUS-MT project 的一部分,旨在为世界上的许多语言提供广泛可用和可访问的神经机器翻译模型。所有模型最初都使用了炫人的 Marian NMT 框架进行训练,这是一个使用纯C ++编写的高效NMT实现。这些模型已经使用huggingface的transformers库将其转换为pyTorch。训练数据取自 OPUS ,训练流程使用了 OPUS-MT-train 的过程。
@inproceedings{tiedemann-thottingal-2020-opus, title = "{OPUS}-{MT} {--} Building open translation services for the World", author = {Tiedemann, J{\"o}rg and Thottingal, Santhosh}, booktitle = "Proceedings of the 22nd Annual Conference of the European Association for Machine Translation", month = nov, year = "2020", address = "Lisboa, Portugal", publisher = "European Association for Machine Translation", url = "https://aclanthology.org/2020.eamt-1.61", pages = "479--480", } @inproceedings{tiedemann-2020-tatoeba, title = "The Tatoeba Translation Challenge {--} Realistic Data Sets for Low Resource and Multilingual {MT}", author = {Tiedemann, J{\"o}rg}, booktitle = "Proceedings of the Fifth Conference on Machine Translation", month = nov, year = "2020", address = "Online", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2020.wmt-1.139", pages = "1174--1182", }
简短的示例代码:
from transformers import MarianMTModel, MarianTokenizer src_text = [ "2001 is the year when the 21st century begins.", "This is Copacabana!" ] model_name = "pytorch-models/opus-mt-tc-big-en-bg" tokenizer = MarianTokenizer.from_pretrained(model_name) model = MarianMTModel.from_pretrained(model_name) translated = model.generate(**tokenizer(src_text, return_tensors="pt", padding=True)) for t in translated: print( tokenizer.decode(t, skip_special_tokens=True) ) # expected output: # 2001 е годината, в която започва 21-ви век. # Това е Копакабана!
您也可以使用transformers管道来使用OPUS-MT模型,例如:
from transformers import pipeline pipe = pipeline("translation", model="Helsinki-NLP/opus-mt-tc-big-en-bg") print(pipe("2001 is the year when the 21st century begins.")) # expected output: 2001 е годината, в която започва 21-ви век.
langpair | testset | chr-F | BLEU | #sent | #words |
---|---|---|---|---|---|
eng-bul | tatoeba-test-v2021-08-07 | 0.68987 | 51.5 | 10000 | 69504 |
eng-bul | flores101-devtest | 0.69891 | 44.9 | 1012 | 24700 |
该工作得到 European Language Grid 的支持作为 pilot project 2866 ,由 FoTran project 资助,该项目受欧洲研究理事会(ERC)支持,根据欧洲联盟的Horizon 2020研究和创新计划(授予协议编号771113),以及 MeMAD project ,该项目受欧洲联盟Horizon 2020研究和创新计划的资助,授予协议编号780069。我们还感谢 CSC -- IT Center for Science 提供的慷慨计算资源和IT基础设施,芬兰。