模型:
masakhane/afroxlmr-large-ner-masakhaner-1.0_2.0
masakhane/afroxlmr-large-ner-masakhaner-1.0_2.0 是一个用于21种非洲语言的命名实体识别(NER)模型。具体来说,该模型是在Davlan/afro-xlmr-large模型基础上,通过聚合了两个版本的MasakhaNER数据集(即 MasakhaNER 1.0 和 MasakhaNER 2.0 )的非洲语言数据进行微调的。覆盖的语言有:
它被训练用于识别四种类型的实体:日期和时间(DATE),位置(LOC),组织(ORG)和个人(PER)。
您可以使用Transformers管道进行NER模型的使用。
from transformers import AutoTokenizer, AutoModelForTokenClassification from transformers import pipeline tokenizer = AutoTokenizer.from_pretrained("masakhane/afroxlmr-large-ner-masakhaner-1.0_2.0") model = AutoModelForTokenClassification.from_pretrained("masakhane/afroxlmr-large-ner-masakhaner-1.0_2.0") nlp = pipeline("ner", model=model, tokenizer=tokenizer) example = "Emir of Kano turban Zhang wey don spend 18 years for Nigeria" ner_results = nlp(example) print(ner_results)
模型在MasakhaNER 1.0和MasakhaNER 2.0测试集上进行了评估
language | MasakhaNER 1.0 | MasakhaNER 2.0 |
---|---|---|
amh | 80.5 | |
bam | 83.1 | |
bbj | 76.6 | |
ewe | 89.6 | |
fon | 83.8 | |
hau | 90.3 | 87.5 |
ibo | 89.5 | 93.5 |
kin | 82.0 | 87.6 |
lug | 87.1 | 89.7 |
luo | 80.8 | 82.5 |
mos | 75.5 | |
nya | 92.7 | |
pcm | 91.1 | 90.9 |
sna | 96.5 | |
swa | 88.5 | 93.4 |
tsn | 90.3 | |
twi | 81.3 | |
wol | 72.7 | 87.3 |
xho | 90.0 | |
yor | 88.1 | 90.5 |
zul | 91.3 | |
avg | 85.1 | 87.7 |
该模型受其特定时间段的实体标注新闻文章训练数据集的限制。这可能不适用于不同领域中的所有用例的泛化。
该模型是在 MasakhaNER 1.0 和 MasakhaNER 2.0 数据集的聚合上进行微调的
训练数据集区分实体的开头和延续,因此如果有相同类型的连续实体,则模型可以输出第二个实体的起始位置。与数据集中一样,每个标记将被归类为以下类别之一:
Abbreviation | Description |
---|---|
O | Outside of a named entity |
B-DATE | Beginning of a DATE entity right after another DATE entity |
I-DATE | DATE entity |
B-PER | Beginning of a person’s name right after another person’s name |
I-PER | Person’s name |
B-ORG | Beginning of an organisation right after another organisation |
I-ORG | Organisation |
B-LOC | Beginning of a location right after another location |
I-LOC | Location |
该模型在一台NVIDIA V100 GPU上进行训练,使用了 original MasakhaNER paper 的推荐超参数,该超参数在MasakhaNER语料库上对模型进行了训练和评估。
@article{Adelani2022MasakhaNER2A, title={MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition}, author={David Ifeoluwa Adelani and Graham Neubig and Sebastian Ruder and Shruti Rijhwani and Michael Beukman and Chester Palen-Michel and Constantine Lignos and Jesujoba Oluwadara Alabi and Shamsuddeen Hassan Muhammad and Peter Nabende and Cheikh M. Bamba Dione and Andiswa Bukula and Rooweither Mabuya and Bonaventure F. P. Dossou and Blessing K. Sibanda and Happy Buzaaba and Jonathan Mukiibi and Godson Kalipe and Derguene Mbaye and Amelia Taylor and Fatoumata Kabore and Chris C. Emezue and Anuoluwapo Aremu and Perez Ogayo and Catherine W. Gitau and Edwin Munkoh-Buabeng and Victoire Memdjokam Koagne and Allahsera Auguste Tapo and Tebogo Macucwa and Vukosi Marivate and Elvis Mboning and Tajuddeen R. Gwadabe and Tosin P. Adewumi and Orevaoghene Ahia and Joyce Nakatumba-Nabende and Neo L. Mokono and Ignatius M Ezeani and Chiamaka Ijeoma Chukwuneke and Mofetoluwa Adeyemi and Gilles Hacheme and Idris Abdulmumin and Odunayo Ogundepo and Oreen Yousuf and Tatiana Moteu Ngoli and Dietrich Klakow}, journal={ArXiv}, year={2022}, volume={abs/2210.12391} }