xprophetnet-large-wiki100-cased

跨语言版本 ProphetNet ，在 wiki100 xGLUE dataset 上进行了预训练。ProphetNet 是一种新的用于序列到序列学习的预训练语言模型，采用了一种称为未来 n-gram 预测的新型自监督目标。ProphetNet 可以通过 n-stream 解码器预测更多未来的标记。原始实现是在 github repo 上的 Fairseq 版本。

xProphetNet 还作为 xGLUE 跨语言自然语言生成任务的基线模型。对于 xGLUE 跨语言 NLG 任务，xProphetNet 在英文数据上进行微调，但可以在英文和其他零样本语言数据上进行推断。

使用方法

这个预训练模型可以在序列到序列任务上进行微调。可以如下训练该模型用于英文标题生成：

from transformers import XLMProphetNetForConditionalGeneration, XLMProphetNetTokenizer

model = XLMProphetNetForConditionalGeneration.from_pretrained("microsoft/xprophetnet-large-wiki100-cased")
tokenizer = XLMProphetNetTokenizer.from_pretrained("microsoft/xprophetnet-large-wiki100-cased")

input_str = "the us state department said wednesday it had received no formal word from bolivia that it was expelling the us ambassador there but said the charges made against him are `` baseless ."
target_str = "us rejects charges against its ambassador in bolivia"

input_ids = tokenizer(input_str, return_tensors="pt").input_ids
labels = tokenizer(target_str, return_tensors="pt").input_ids

loss = model(input_ids, labels=labels).loss

请注意，由于这个模型是多语言模型，因此可以在各种其他语言上进行微调。

引用

@article{yan2020prophetnet,
  title={Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training},
  author={Yan, Yu and Qi, Weizhen and Gong, Yeyun and Liu, Dayiheng and Duan, Nan and Chen, Jiusheng and Zhang, Ruofei and Zhou, Ming},
  journal={arXiv preprint arXiv:2001.04063},
  year={2020}
}