模型名称

SingBert大型 - 用于新加坡式英语（SG）和马来式英语（MY）的Bert模型。

模型描述

类似于 SingBert 但是是大型版本，从 BERT large uncased (whole word masking) 中初始化，进行了 singlish 和 manglish 数据上的预训练微调。

预期用途和限制

如何使用

>>> from transformers import pipeline
>>> nlp = pipeline('fill-mask', model='zanelim/singbert-large-sg')
>>> nlp("kopi c siew [MASK]")

[{'sequence': '[CLS] kopi c siew dai [SEP]',
  'score': 0.9003700017929077,
  'token': 18765,
  'token_str': 'dai'},
 {'sequence': '[CLS] kopi c siew mai [SEP]',
  'score': 0.0779474675655365,
  'token': 14736,
  'token_str': 'mai'},
 {'sequence': '[CLS] kopi c siew. [SEP]',
  'score': 0.0032227332703769207,
  'token': 1012,
  'token_str': '.'},
 {'sequence': '[CLS] kopi c siew bao [SEP]',
  'score': 0.0017727474914863706,
  'token': 25945,
  'token_str': 'bao'},
 {'sequence': '[CLS] kopi c siew peng [SEP]',
  'score': 0.0012526646023616195,
  'token': 26473,
  'token_str': 'peng'}]

>>> nlp("one teh c siew dai, and one kopi [MASK]")

[{'sequence': '[CLS] one teh c siew dai, and one kopi. [SEP]',
  'score': 0.5249741077423096,
  'token': 1012,
  'token_str': '.'},
 {'sequence': '[CLS] one teh c siew dai, and one kopi o [SEP]',
  'score': 0.27349168062210083,
  'token': 1051,
  'token_str': 'o'},
 {'sequence': '[CLS] one teh c siew dai, and one kopi peng [SEP]',
  'score': 0.057190295308828354,
  'token': 26473,
  'token_str': 'peng'},
 {'sequence': '[CLS] one teh c siew dai, and one kopi c [SEP]',
  'score': 0.04022320732474327,
  'token': 1039,
  'token_str': 'c'},
 {'sequence': '[CLS] one teh c siew dai, and one kopi? [SEP]',
  'score': 0.01191170234233141,
  'token': 1029,
  'token_str': '?'}]

>>> nlp("die [MASK] must try")

[{'sequence': '[CLS] die die must try [SEP]',
  'score': 0.9921030402183533,
  'token': 3280,
  'token_str': 'die'},
 {'sequence': '[CLS] die also must try [SEP]',
  'score': 0.004993876442313194,
  'token': 2036,
  'token_str': 'also'},
 {'sequence': '[CLS] die liao must try [SEP]',
  'score': 0.000317625846946612,
  'token': 727,
  'token_str': 'liao'},
 {'sequence': '[CLS] die still must try [SEP]',
  'score': 0.0002260878391098231,
  'token': 2145,
  'token_str': 'still'},
 {'sequence': '[CLS] die i must try [SEP]',
  'score': 0.00016935862367972732,
  'token': 1045,
  'token_str': 'i'}]

>>> nlp("dont play [MASK] leh")

[{'sequence': '[CLS] dont play play leh [SEP]',
  'score': 0.9079819321632385,
  'token': 2377,
  'token_str': 'play'},
 {'sequence': '[CLS] dont play punk leh [SEP]',
  'score': 0.006846973206847906,
  'token': 7196,
  'token_str': 'punk'},
 {'sequence': '[CLS] dont play games leh [SEP]',
  'score': 0.004041737411171198,
  'token': 2399,
  'token_str': 'games'},
 {'sequence': '[CLS] dont play politics leh [SEP]',
  'score': 0.003728888463228941,
  'token': 4331,
  'token_str': 'politics'},
 {'sequence': '[CLS] dont play cheat leh [SEP]',
  'score': 0.0032805048394948244,
  'token': 21910,
  'token_str': 'cheat'}]

>>> nlp("confirm plus [MASK]")

{'sequence': '[CLS] confirm plus chop [SEP]',
  'score': 0.9749826192855835,
  'token': 24494,
  'token_str': 'chop'},
 {'sequence': '[CLS] confirm plus chopped [SEP]',
  'score': 0.017554156482219696,
  'token': 24881,
  'token_str': 'chopped'},
 {'sequence': '[CLS] confirm plus minus [SEP]',
  'score': 0.002725469646975398,
  'token': 15718,
  'token_str': 'minus'},
 {'sequence': '[CLS] confirm plus guarantee [SEP]',
  'score': 0.000900257145985961,
  'token': 11302,
  'token_str': 'guarantee'},
 {'sequence': '[CLS] confirm plus one [SEP]',
  'score': 0.0004384620988275856,
  'token': 2028,
  'token_str': 'one'}]

>>> nlp("catch no [MASK]")

[{'sequence': '[CLS] catch no ball [SEP]',
  'score': 0.9381157159805298,
  'token': 3608,
  'token_str': 'ball'},
 {'sequence': '[CLS] catch no balls [SEP]',
  'score': 0.060842301696538925,
  'token': 7395,
  'token_str': 'balls'},
 {'sequence': '[CLS] catch no fish [SEP]',
  'score': 0.00030917322146706283,
  'token': 3869,
  'token_str': 'fish'},
 {'sequence': '[CLS] catch no breath [SEP]',
  'score': 7.552534952992573e-05,
  'token': 3052,
  'token_str': 'breath'},
 {'sequence': '[CLS] catch no tail [SEP]',
  'score': 4.208395694149658e-05,
  'token': 5725,
  'token_str': 'tail'}]

这里是如何使用此模型在PyTorch中获取给定文本的特征的方法：

from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('zanelim/singbert-large-sg')
model = BertModel.from_pretrained("zanelim/singbert-large-sg")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

并且在TensorFlow中：

from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained("zanelim/singbert-large-sg")
model = TFBertModel.from_pretrained("zanelim/singbert-large-sg")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)

限制和偏见

此模型是在口语化的新加坡式英语和马来式英语语料库上进行微调的，因此最适用于涉及英语、普通话和马来语等主要语种的下游任务。此外，由于训练数据主要来自论坛，要注意现有的固有偏见。

训练数据

口语化的新加坡式英语和马来式英语语料库，两者都是英语、普通话、泰米尔语、马来语和其他地方方言（如福建话、广东话或潮州话）的混合体。该语料库是从subreddits- r/singapore 和 r/malaysia 以及论坛如hardwarezone 收集而来的。

训练过程

使用 bert large uncased (whole word masking) 词汇和检查点（预训练权重）进行初始化。从训练数据中进一步提取了前1000个自定义词汇标记（与原始bert词汇无重叠），并填充到原始bert词汇中未使用的标记中。

预训练通过以下超参数对训练数据进行进一步微调

train_batch_size: 512
max_seq_length: 128
num_train_steps: 300000
num_warmup_steps: 5000
learning_rate: 2e-5
hardware: TPU v3-8

作者:

Zane Lim

数据集大小:

3.88 GB