用于从南斯拉夫语言(zls)翻译为英语(en)的神经机器翻译模型。
此模型是 OPUS-MT project 的一部分,旨在使神经机器翻译模型在世界上许多语言中广泛可用和易于访问。所有模型最初使用纯C++编写的令人惊叹的 Marian NMT 框架进行训练。使用 transformers library by huggingface 将模型转换为pyTorch。训练数据来自 OPUS ,并且训练流程使用 OPUS-MT-train 的程序。
@inproceedings{tiedemann-thottingal-2020-opus, title = "{OPUS}-{MT} {--} Building open translation services for the World", author = {Tiedemann, J{\"o}rg and Thottingal, Santhosh}, booktitle = "Proceedings of the 22nd Annual Conference of the European Association for Machine Translation", month = nov, year = "2020", address = "Lisboa, Portugal", publisher = "European Association for Machine Translation", url = "https://aclanthology.org/2020.eamt-1.61", pages = "479--480", } @inproceedings{tiedemann-2020-tatoeba, title = "The Tatoeba Translation Challenge {--} Realistic Data Sets for Low Resource and Multilingual {MT}", author = {Tiedemann, J{\"o}rg}, booktitle = "Proceedings of the Fifth Conference on Machine Translation", month = nov, year = "2020", address = "Online", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2020.wmt-1.139", pages = "1174--1182", }
简短的示例代码:
from transformers import MarianMTModel, MarianTokenizer src_text = [ "Да не би случайно Том да остави Мери да кара колата?", "Какво е времето днес?" ] model_name = "pytorch-models/opus-mt-tc-big-zls-en" tokenizer = MarianTokenizer.from_pretrained(model_name) model = MarianMTModel.from_pretrained(model_name) translated = model.generate(**tokenizer(src_text, return_tensors="pt", padding=True)) for t in translated: print( tokenizer.decode(t, skip_special_tokens=True) ) # expected output: # Did Tom just let Mary drive the car? # What's the weather like today?
您还可以使用transformers pipelines来使用OPUS-MT模型,例如:
from transformers import pipeline pipe = pipeline("translation", model="Helsinki-NLP/opus-mt-tc-big-zls-en") print(pipe("Да не би случайно Том да остави Мери да кара колата?")) # expected output: Did Tom just let Mary drive the car?
langpair | testset | chr-F | BLEU | #sent | #words |
---|---|---|---|---|---|
bos_Latn-eng | tatoeba-test-v2021-08-07 | 0.79339 | 66.5 | 301 | 1826 |
bul-eng | tatoeba-test-v2021-08-07 | 0.72656 | 59.3 | 10000 | 71872 |
hbs-eng | tatoeba-test-v2021-08-07 | 0.71783 | 57.3 | 10017 | 68934 |
hrv-eng | tatoeba-test-v2021-08-07 | 0.74066 | 59.2 | 1480 | 10620 |
mkd-eng | tatoeba-test-v2021-08-07 | 0.70043 | 57.4 | 10010 | 65667 |
slv-eng | tatoeba-test-v2021-08-07 | 0.39534 | 23.5 | 2495 | 16940 |
srp_Cyrl-eng | tatoeba-test-v2021-08-07 | 0.67628 | 47.0 | 1580 | 10181 |
srp_Latn-eng | tatoeba-test-v2021-08-07 | 0.71878 | 58.5 | 6656 | 46307 |
bul-eng | flores101-devtest | 0.67375 | 42.0 | 1012 | 24721 |
hrv-eng | flores101-devtest | 0.63914 | 37.1 | 1012 | 24721 |
mkd-eng | flores101-devtest | 0.67444 | 43.2 | 1012 | 24721 |
slv-eng | flores101-devtest | 0.62087 | 35.2 | 1012 | 24721 |
srp_Cyrl-eng | flores101-devtest | 0.67810 | 36.8 | 1012 | 24721 |
该工作得到 European Language Grid 提供的支持,如 pilot project 2866 ,以及欧洲研究理事会(ERC)根据欧洲联盟的“Horizon 2020”研究与创新计划(授予协议编号771113)和 MeMAD project 提供的支持,该计划是由欧洲联盟的“Horizon 2020”研究与创新计划在授权协议号780069下提供的。我们还感谢 CSC -- IT Center for Science 提供的慷慨的计算资源和IT基础设施,芬兰。