模型:
bigscience/mt0-xxl
我们提出了BLOOMZ& mT0模型系列,能够零-shot地遵循多种语言的人类指令。我们在跨语言任务混合(xP3)上微调BLOOM& mT5预训练的多语言模型,发现我们得到的模型能够在未见的任务和语言上实现跨语言泛化。
Multitask finetuned on 1239321 . Recommended for prompting in English. | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Parameters | 300M | 580M | 1.2B | 3.7B | 13B | 560M | 1.1B | 1.7B | 3B | 7.1B | 176B |
Finetuned Model | 12310321 | 12311321 | 12312321 | 12313321 | 12314321 | 12315321 | 12316321 | 12317321 | 12318321 | 12319321 | 12320321 |
Multitask finetuned on 12321321 . Recommended for prompting in non-English. | |||||||||||
Finetuned Model | 12322321 | 12323321 | 12324321 | ||||||||
Multitask finetuned on 12325321 . Released for research purposes only. Strictly inferior to above models! | |||||||||||
Finetuned Model | 12326321 | 12327321 | 12328321 | ||||||||
Original pretrained checkpoints. Not recommended. | |||||||||||
Pretrained Model | 12329321 | 12330321 | 12331321 | 12332321 | 12333321 | 12334321 | 12335321 | 12336321 | 12337321 | 12338321 | 12339321 |
我们建议使用该模型执行用自然语言表达的任务。例如,给定提示“翻译成英语:Je t’aime”,模型最可能的答案是“我爱你”。我们论文中的一些提示点子:
欢迎在社区选项卡中分享您的生成内容!
# pip install -q transformers from transformers import AutoModelForSeq2SeqLM, AutoTokenizer checkpoint = "bigscience/mt0-xxl" tokenizer = AutoTokenizer.from_pretrained(checkpoint) model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint) inputs = tokenizer.encode("Translate to English: Je t’aime.", return_tensors="pt") outputs = model.generate(inputs) print(tokenizer.decode(outputs[0]))
# pip install -q transformers accelerate from transformers import AutoModelForSeq2SeqLM, AutoTokenizer checkpoint = "bigscience/mt0-xxl" tokenizer = AutoTokenizer.from_pretrained(checkpoint) model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint, torch_dtype="auto", device_map="auto") inputs = tokenizer.encode("Translate to English: Je t’aime.", return_tensors="pt").to("cuda") outputs = model.generate(inputs) print(tokenizer.decode(outputs[0]))
# pip install -q transformers accelerate bitsandbytes from transformers import AutoModelForSeq2SeqLM, AutoTokenizer checkpoint = "bigscience/mt0-xxl" tokenizer = AutoTokenizer.from_pretrained(checkpoint) model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint, device_map="auto", load_in_8bit=True) inputs = tokenizer.encode("Translate to English: Je t’aime.", return_tensors="pt").to("cuda") outputs = model.generate(inputs) print(tokenizer.decode(outputs[0]))
提示工程:性能可能会根据提示的不同而有所变化。对于BLOOMZ模型,我们建议清楚地指示输入何时结束,以避免模型尝试继续输入。例如,提示“翻译成英语:Je t'aime”如果末尾没有句号(.),可能导致模型试图继续法语句子。更好的提示示例是“翻译成英语:Je t'aime。”、“翻译成英语:Je t'aime。翻译:”、““Je t'aime.” 在英语中是什么?”等,这样模型就清楚应该在何时回答了。此外,我们建议尽可能为模型提供更多上下文。例如,如果您想让它用泰卢固语回答,请告诉模型,例如“用泰卢固语的一句话解释神经网络中的反向传播是什么。”。
我们参考了我们论文的第7表,以及对于未见任务的零-shot结果的 paper 和 bigscience/evaluation-results 。侧边栏报告了每个数据集配置的最佳提示的零-shot性能。
@article{muennighoff2022crosslingual, title={Crosslingual generalization through multitask finetuning}, author={Muennighoff, Niklas and Wang, Thomas and Sutawika, Lintang and Roberts, Adam and Biderman, Stella and Scao, Teven Le and Bari, M Saiful and Shen, Sheng and Yong, Zheng-Xin and Schoelkopf, Hailey and others}, journal={arXiv preprint arXiv:2211.01786}, year={2022} }