模型名称

SingBert - 适用于新加坡和马来西亚的新马英语（Singlish）和门牙语（Manglish）的Bert模型。

模型描述

BERT base uncased ，使用预训练在 singlish 和 manglish 数据上微调。

预期用途和限制

如何使用

>>> from transformers import pipeline
>>> nlp = pipeline('fill-mask', model='zanelim/singbert')
>>> nlp("kopi c siew [MASK]")

[{'sequence': '[CLS] kopi c siew dai [SEP]',
  'score': 0.5092713236808777,
  'token': 18765,
  'token_str': 'dai'},
 {'sequence': '[CLS] kopi c siew mai [SEP]',
  'score': 0.3515934646129608,
  'token': 14736,
  'token_str': 'mai'},
 {'sequence': '[CLS] kopi c siew bao [SEP]',
  'score': 0.05576375499367714,
  'token': 25945,
  'token_str': 'bao'},
 {'sequence': '[CLS] kopi c siew. [SEP]',
  'score': 0.006019321270287037,
  'token': 1012,
  'token_str': '.'},
 {'sequence': '[CLS] kopi c siew sai [SEP]',
  'score': 0.0038361591286957264,
  'token': 18952,
  'token_str': 'sai'}]

>>> nlp("one teh c siew dai, and one kopi [MASK].")

[{'sequence': '[CLS] one teh c siew dai, and one kopi c [SEP]',
  'score': 0.6176503300666809,
  'token': 1039,
  'token_str': 'c'},
 {'sequence': '[CLS] one teh c siew dai, and one kopi o [SEP]',
  'score': 0.21094971895217896,
  'token': 1051,
  'token_str': 'o'},
 {'sequence': '[CLS] one teh c siew dai, and one kopi. [SEP]',
  'score': 0.13027705252170563,
  'token': 1012,
  'token_str': '.'},
 {'sequence': '[CLS] one teh c siew dai, and one kopi! [SEP]',
  'score': 0.004680239595472813,
  'token': 999,
  'token_str': '!'},
 {'sequence': '[CLS] one teh c siew dai, and one kopi w [SEP]',
  'score': 0.002034128177911043,
  'token': 1059,
  'token_str': 'w'}]

>>> nlp("dont play [MASK] leh")

[{'sequence': '[CLS] dont play play leh [SEP]',
  'score': 0.9281464219093323,
  'token': 2377,
  'token_str': 'play'},
 {'sequence': '[CLS] dont play politics leh [SEP]',
  'score': 0.010990909300744534,
  'token': 4331,
  'token_str': 'politics'},
 {'sequence': '[CLS] dont play punk leh [SEP]',
  'score': 0.005583590362221003,
  'token': 7196,
  'token_str': 'punk'},
 {'sequence': '[CLS] dont play dirty leh [SEP]',
  'score': 0.0025784350000321865,
  'token': 6530,
  'token_str': 'dirty'},
 {'sequence': '[CLS] dont play cheat leh [SEP]',
  'score': 0.0025066907983273268,
  'token': 21910,
  'token_str': 'cheat'}]

>>> nlp("catch no [MASK]")

[{'sequence': '[CLS] catch no ball [SEP]',
  'score': 0.7922210693359375,
  'token': 3608,
  'token_str': 'ball'},
 {'sequence': '[CLS] catch no balls [SEP]',
  'score': 0.20503675937652588,
  'token': 7395,
  'token_str': 'balls'},
 {'sequence': '[CLS] catch no tail [SEP]',
  'score': 0.0006608376861549914,
  'token': 5725,
  'token_str': 'tail'},
 {'sequence': '[CLS] catch no talent [SEP]',
  'score': 0.0002158183924620971,
  'token': 5848,
  'token_str': 'talent'},
 {'sequence': '[CLS] catch no prisoners [SEP]',
  'score': 5.3481446229852736e-05,
  'token': 5895,
  'token_str': 'prisoners'}]

>>> nlp("confirm plus [MASK]")

[{'sequence': '[CLS] confirm plus chop [SEP]',
  'score': 0.992355227470398,
  'token': 24494,
  'token_str': 'chop'},
 {'sequence': '[CLS] confirm plus one [SEP]',
  'score': 0.0037301010452210903,
  'token': 2028,
  'token_str': 'one'},
 {'sequence': '[CLS] confirm plus minus [SEP]',
  'score': 0.0014284878270700574,
  'token': 15718,
  'token_str': 'minus'},
 {'sequence': '[CLS] confirm plus 1 [SEP]',
  'score': 0.0011354683665558696,
  'token': 1015,
  'token_str': '1'},
 {'sequence': '[CLS] confirm plus chopped [SEP]',
  'score': 0.0003804611915256828,
  'token': 24881,
  'token_str': 'chopped'}]

>>> nlp("die [MASK] must try")

[{'sequence': '[CLS] die die must try [SEP]',
  'score': 0.9552758932113647,
  'token': 3280,
  'token_str': 'die'},
 {'sequence': '[CLS] die also must try [SEP]',
  'score': 0.03644804656505585,
  'token': 2036,
  'token_str': 'also'},
 {'sequence': '[CLS] die liao must try [SEP]',
  'score': 0.003282855963334441,
  'token': 727,
  'token_str': 'liao'},
 {'sequence': '[CLS] die already must try [SEP]',
  'score': 0.0004937972989864647,
  'token': 2525,
  'token_str': 'already'},
 {'sequence': '[CLS] die hard must try [SEP]',
  'score': 0.0003659659414552152,
  'token': 2524,
  'token_str': 'hard'}]

：

以下是如何在PyTorch中使用此模型获取给定文本特征的方法：

from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('zanelim/singbert')
model = BertModel.from_pretrained("zanelim/singbert")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

以及在TensorFlow中：

from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained("zanelim/singbert")
model = TFBertModel.from_pretrained("zanelim/singbert")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)

限制和偏见

该模型是在口语化的新加坡和马来西亚语料库上进行微调的，因此最适合用于涉及主要语言-英语、华语、马来语的下游任务。此外，由于训练数据主要来自论坛，存在固有的偏见，请注意。

训练数据

口语化的新加坡英语和马来西亚英语语料库（它们都是英语、华语、泰米尔语、马来语以及其他方言如福建话、粤语或潮州话的混合体）。这些语料库是从subreddits- r/singapore 和 r/malaysia，以及像hardwarezone这样的论坛中收集的。

训练过程

使用 bert base uncased 个词汇和检查点（预训练权重）进行初始化。从训练数据中进一步提取了1000个自定义词汇标记（与原始bert词汇不重叠），并填充到原始bert词汇中未使用的标记中。

进一步使用以下超参数对训练数据进行了微调：

train_batch_size: 512
max_seq_length: 128
num_train_steps: 300000
num_warmup_steps: 5000
learning_rate: 2e-5
hardware: TPU v3-8

作者:

Zane Lim

数据集大小:

1.32 GB