Twitter-scratch-roBERTa-base

这是一个从头开始训练的 RoBERTa-base 模型，使用了大约 5800 万条推文进行训练，如 TweetEval benchmark (Findings of EMNLP 2020) 中描述和评估。要对该模型及其他语言模型在 Twitter 特定数据上进行评估，请参考 Tweeteval official repository 。

预处理文本

将用户名和链接替换为占位符："@user" 和 "http"。

def preprocess(text):
    new_text = []
    for t in text.split(" "):
        t = '@user' if t.startswith('@') and len(t) > 1 else t
        t = 'http' if t.startswith('http') else t
        new_text.append(t)
    return " ".join(new_text)

示例遮蔽语言模型

from transformers import pipeline, AutoTokenizer
import numpy as np

MODEL = "cardiffnlp/twitter-scratch-roberta-base"
fill_mask = pipeline("fill-mask", model=MODEL, tokenizer=MODEL)
tokenizer = AutoTokenizer.from_pretrained(MODEL)

def print_candidates():
    for i in range(5):
        token = tokenizer.decode(candidates[i]['token'])
        score = np.round(candidates[i]['score'], 4)
        print(f"{i+1}) {token} {score}")

texts = [
 "I am so <mask> ?",
 "I am so <mask> ?" 
]
for text in texts:
    t = preprocess(text)
    print(f"{'-'*30}\n{t}")
    candidates = fill_mask(t)
    print_candidates()

输出：

------------------------------
I am so <mask> ?
1)  happy 0.530
2)  grateful 0.083
3)  excited 0.078
4)  thankful 0.053
5)  blessed 0.041
------------------------------
I am so <mask> ?
1)  sad 0.439
2)  sorry 0.088
3)  tired 0.045
4)  hurt 0.026
5)  upset 0.026

BibTeX 条目和引用信息

如果您使用此模型，请引用 reference paper 。

@inproceedings{barbieri-etal-2020-tweeteval,
    title = "{T}weet{E}val: Unified Benchmark and Comparative Evaluation for Tweet Classification",
    author = "Barbieri, Francesco  and
      Camacho-Collados, Jose  and
      Espinosa Anke, Luis  and
      Neves, Leonardo",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2020",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2020.findings-emnlp.148",
    doi = "10.18653/v1/2020.findings-emnlp.148",
    pages = "1644--1650"
}