中文

T5-Encoder(T5-large model) fine-tuned on very small dataset for token classification

Simple experimental model that was trained in 3 epochs on very small dataset

Usage

from transformers import AutoTokenizer, AutoModelForTokenClassification, NerPipeline

model = AutoModelForTokenClassification.from_pretrained("imvladikon/t5-english-ner", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("imvladikon/t5-english-ner", trust_remote_code=True)

pipe = NerPipeline(model=model, tokenizer=tokenizer, aggregation_strategy="max")
print(pipe("London is the capital city of England and the United Kingdom"))
"""
[{'entity_group': 'LOCATION',
  'score': 0.84536326,
  'word': 'London',
  'start': 0,
  'end': 6},
 {'entity_group': 'LOCATION',
  'score': 0.8957489,
  'word': 'England',
  'start': 30,
  'end': 37},
 {'entity_group': 'LOCATION',
  'score': 0.73186326,
  'word': 'UnitedKingdom',
  'start': 46,
  'end': 60}]
"""

Usage in spacy

pip install spacy transformers git+https://github.com/explosion/spacy-huggingface-pipelines -q
import spacy
from spacy import displacy

text = "My name is Sarah and I live in London"

nlp = spacy.blank("en")
nlp.add_pipe("hf_token_pipe", config={"model": "imvladikon/t5-english-ner", "kwargs": {"trust_remote_code":True}})
doc = nlp(text)
print(doc.ents)
# (Sarah, London)

This model is a fine-tuned version of t5-large on the private(en) dataset. It achieves the following results on the evaluation set:

  • Loss: 0.1956
  • Commercial Item Precision: 0.0
  • Commercial Item Recall: 0.0
  • Commercial Item F1: 0.0
  • Commercial Item Number: 1
  • Date Precision: 0.8125
  • Date Recall: 0.9286
  • Date F1: 0.8667
  • Date Number: 14
  • Location Precision: 0.7143
  • Location Recall: 0.75
  • Location F1: 0.7317
  • Location Number: 20
  • Organization Precision: 0.8588
  • Organization Recall: 0.9125
  • Organization F1: 0.8848
  • Organization Number: 80
  • Other Precision: 0.3684
  • Other Recall: 0.3333
  • Other F1: 0.35
  • Other Number: 21
  • Person Precision: 0.8182
  • Person Recall: 0.9310
  • Person F1: 0.8710
  • Person Number: 29
  • Quantity Precision: 0.8
  • Quantity Recall: 0.8571
  • Quantity F1: 0.8276
  • Quantity Number: 14
  • Title Precision: 0.0
  • Title Recall: 0.0
  • Title F1: 0.0
  • Title Number: 7
  • Overall Precision: 0.75
  • Overall Recall: 0.7903
  • Overall F1: 0.7696
  • Overall Accuracy: 0.9534

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-05
  • train_batch_size: 2
  • eval_batch_size: 2
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 3.0

Training results

Training Loss Epoch Step Validation Loss Commercial Item Precision Commercial Item Recall Commercial Item F1 Commercial Item Number Date Precision Date Recall Date F1 Date Number Location Precision Location Recall Location F1 Location Number Organization Precision Organization Recall Organization F1 Organization Number Other Precision Other Recall Other F1 Other Number Person Precision Person Recall Person F1 Person Number Quantity Precision Quantity Recall Quantity F1 Quantity Number Title Precision Title Recall Title F1 Title Number Overall Precision Overall Recall Overall F1 Overall Accuracy
0.8868 1.0 708 0.2725 0.0 0.0 0.0 1 0.8125 0.9286 0.8667 14 0.4167 0.75 0.5357 20 0.8272 0.8375 0.8323 80 1.0 0.0476 0.0909 21 0.8438 0.9310 0.8852 29 0.6667 0.7143 0.6897 14 0.0 0.0 0.0 7 0.7348 0.7151 0.7248 0.9446
0.2984 2.0 1416 0.2121 0.0 0.0 0.0 1 0.8667 0.9286 0.8966 14 0.5 0.8 0.6154 20 0.8375 0.8375 0.8375 80 0.3077 0.1905 0.2353 21 0.8182 0.9310 0.8710 29 0.7333 0.7857 0.7586 14 0.0 0.0 0.0 7 0.7077 0.7419 0.7244 0.9481
0.1729 3.0 2124 0.1956 0.0 0.0 0.0 1 0.8125 0.9286 0.8667 14 0.7143 0.75 0.7317 20 0.8588 0.9125 0.8848 80 0.3684 0.3333 0.35 21 0.8182 0.9310 0.8710 29 0.8 0.8571 0.8276 14 0.0 0.0 0.0 7 0.75 0.7903 0.7696 0.9534

Framework versions

  • Transformers 4.21.1
  • Pytorch 1.12.0+cu113
  • Datasets 2.4.0
  • Tokenizers 0.12.1

WANDB

training logs and reports