SEC-BERT
SEC-BERT是金融领域的BERT模型系列,旨在协助金融自然语言处理研究和金融科技应用。SEC-BERT包括以下模型:
-
SEC-BERT-BASE
:与BERT-BASE相同的架构,用于金融文件训练。
- SEC-BERT-NUM(此模型):与SEC-BERT-BASE相同,但我们用[NUM]伪token替换每个数字token,以统一处理所有数字表达式,禁止其分割)。
-
SEC-BERT-SHAPE
:与SEC-BERT-BASE相同,但我们用伪token代替数字,以代表数字的形状,因此数字表达式(已知形状)不再分割,例如,'53.2'变成'[XX.X]','40,200.5'变成'[XX,XXX.X]'。
预训练语料库
该模型在1993-2019年的260,773份10-K申报文件上进行了预训练,这些文件可以公开获取,链接如下:
预训练细节
- 我们通过使用预训练语料库自定义了一个由30k个子词组成的词汇表。
- 我们使用了官方提供的代码进行了BERT的训练。
- 然后,我们使用了Hugging Face的转换脚本将TF检查点转换为所需格式,以便能够以两行代码在PyTorch和TF2中加载模型。
- 我们发布了一个类似于英文BERT-BASE模型的模型(12层,768隐藏单元,12个头部,110M参数)。
- 我们选择了相同的训练设置:100万个训练步骤,每个批次256个长度为512的序列,初始学习率为1e-4。
- 我们能够免费使用Google提供的单个Google Cloud TPU v3-8,同时也利用了
GCP research credits
。
加载预训练模型
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("nlpaueb/sec-bert-num")
model = AutoModel.from_pretrained("nlpaueb/sec-bert-num")
预处理文本
要使用SEC-BERT-NUM,您需要对文本进行预处理,将每个数字标记替换为[NUM]伪token。下面是一个简单句子的预处理示例。这种方法非常简单,您可以根据需要自由修改。
import re
import spacy
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("nlpaueb/sec-bert-num")
spacy_tokenizer = spacy.load("en_core_web_sm")
sentence = "Total net sales decreased 2% or $5.4 billion during 2019 compared to 2018."
def sec_bert_num_preprocess(text):
tokens = [t.text for t in spacy_tokenizer(text)]
processed_text = []
for token in tokens:
if re.fullmatch(r"(\d+[\d,.]*)|([,.]\d+)", token):
processed_text.append('[NUM]')
else:
processed_text.append(token)
return ' '.join(processed_text)
tokenized_sentence = tokenizer.tokenize(sec_bert_num_preprocess(sentence))
print(tokenized_sentence)
"""
['total', 'net', 'sales', 'decreased', '[NUM]', '%', 'or', '$', '[NUM]', 'billion', 'during', '[NUM]', 'compared', 'to', '[NUM]', '.']
"""
使用SEC-BERT变体作为语言模型
Sample
|
Masked Token
|
Total net sales [MASK] 2% or $5.4 billion during 2019 compared to 2018.
|
decreased
|
Model
|
Predictions (Probability)
|
BERT-BASE-UNCASED
|
increased (0.221), were (0.131), are (0.103), rose (0.075), of (0.058)
|
SEC-BERT-BASE
|
increased (0.678), decreased (0.282), declined (0.017), grew (0.016), rose (0.004)
|
SEC-BERT-NUM
|
increased (0.753), decreased (0.211), grew (0.019), declined (0.010), rose (0.006)
|
SEC-BERT-SHAPE
|
increased (0.747), decreased (0.214), grew (0.021), declined (0.013), rose (0.002)
|
Sample
|
Masked Token
|
Total net sales decreased 2% or $5.4 [MASK] during 2019 compared to 2018.
|
billion
|
Model
|
Predictions (Probability)
|
BERT-BASE-UNCASED
|
billion (0.841), million (0.097), trillion (0.028), ##m (0.015), ##bn (0.006)
|
SEC-BERT-BASE
|
million (0.972), billion (0.028), millions (0.000), ##million (0.000), m (0.000)
|
SEC-BERT-NUM
|
million (0.974), billion (0.012), , (0.010), thousand (0.003), m (0.000)
|
SEC-BERT-SHAPE
|
million (0.978), billion (0.021), % (0.000), , (0.000), millions (0.000)
|
Sample
|
Masked Token
|
Total net sales decreased [MASK]% or $5.4 billion during 2019 compared to 2018.
|
2
|
Model
|
Predictions (Probability)
|
BERT-BASE-UNCASED
|
20 (0.031), 10 (0.030), 6 (0.029), 4 (0.027), 30 (0.027)
|
SEC-BERT-BASE
|
13 (0.045), 12 (0.040), 11 (0.040), 14 (0.035), 10 (0.035)
|
SEC-BERT-NUM
|
[NUM] (1.000), one (0.000), five (0.000), three (0.000), seven (0.000)
|
SEC-BERT-SHAPE
|
[XX] (0.316), [XX.X] (0.253), [X.X] (0.237), [X] (0.188), [X.XX] (0.002)
|
Sample
|
Masked Token
|
Total net sales decreased 2[MASK] or $5.4 billion during 2019 compared to 2018.
|
%
|
Model
|
Predictions (Probability)
|
BERT-BASE-UNCASED
|
% (0.795), percent (0.174), ##fold (0.009), billion (0.004), times (0.004)
|
SEC-BERT-BASE
|
% (0.924), percent (0.076), points (0.000), , (0.000), times (0.000)
|
SEC-BERT-NUM
|
% (0.882), percent (0.118), million (0.000), units (0.000), bps (0.000)
|
SEC-BERT-SHAPE
|
% (0.961), percent (0.039), bps (0.000), , (0.000), bcf (0.000)
|
Sample
|
Masked Token
|
Total net sales decreased 2% or $[MASK] billion during 2019 compared to 2018.
|
5.4
|
Model
|
Predictions (Probability)
|
BERT-BASE-UNCASED
|
1 (0.074), 4 (0.045), 3 (0.044), 2 (0.037), 5 (0.034)
|
SEC-BERT-BASE
|
1 (0.218), 2 (0.136), 3 (0.078), 4 (0.066), 5 (0.048)
|
SEC-BERT-NUM
|
[NUM] (1.000), l (0.000), 1 (0.000), - (0.000), 30 (0.000)
|
SEC-BERT-SHAPE
|
[X.X] (0.787), [X.XX] (0.095), [XX.X] (0.049), [X.XXX] (0.046), [X] (0.013)
|
Sample
|
Masked Token
|
Total net sales decreased 2% or $5.4 billion during [MASK] compared to 2018.
|
2019
|
Model
|
Predictions (Probability)
|
BERT-BASE-UNCASED
|
2017 (0.485), 2018 (0.169), 2016 (0.164), 2015 (0.070), 2014 (0.022)
|
SEC-BERT-BASE
|
2019 (0.990), 2017 (0.007), 2018 (0.003), 2020 (0.000), 2015 (0.000)
|
SEC-BERT-NUM
|
[NUM] (1.000), as (0.000), fiscal (0.000), year (0.000), when (0.000)
|
SEC-BERT-SHAPE
|
[XXXX] (1.000), as (0.000), year (0.000), periods (0.000), , (0.000)
|
Sample
|
Masked Token
|
Total net sales decreased 2% or $5.4 billion during 2019 compared to [MASK].
|
2018
|
Model
|
Predictions (Probability)
|
BERT-BASE-UNCASED
|
2017 (0.100), 2016 (0.097), above (0.054), inflation (0.050), previously (0.037)
|
SEC-BERT-BASE
|
2018 (0.999), 2019 (0.000), 2017 (0.000), 2016 (0.000), 2014 (0.000)
|
SEC-BERT-NUM
|
[NUM] (1.000), year (0.000), last (0.000), sales (0.000), fiscal (0.000)
|
SEC-BERT-SHAPE
|
[XXXX] (1.000), year (0.000), sales (0.000), prior (0.000), years (0.000)
|
Sample
|
Masked Token
|
During 2019, the Company [MASK] $67.1 billion of its common stock and paid dividend equivalents of $14.1 billion.
|
repurchased
|
Model
|
Predictions (Probability)
|
BERT-BASE-UNCASED
|
held (0.229), sold (0.192), acquired (0.172), owned (0.052), traded (0.033)
|
SEC-BERT-BASE
|
repurchased (0.913), issued (0.036), purchased (0.029), redeemed (0.010), sold (0.003)
|
SEC-BERT-NUM
|
repurchased (0.917), purchased (0.054), reacquired (0.013), issued (0.005), acquired (0.003)
|
SEC-BERT-SHAPE
|
repurchased (0.902), purchased (0.068), issued (0.010), reacquired (0.008), redeemed (0.006)
|
Sample
|
Masked Token
|
During 2019, the Company repurchased $67.1 billion of its common [MASK] and paid dividend equivalents of $14.1 billion.
|
stock
|
Model
|
Predictions (Probability)
|
BERT-BASE-UNCASED
|
stock (0.835), assets (0.039), equity (0.025), debt (0.021), bonds (0.017)
|
SEC-BERT-BASE
|
stock (0.857), shares (0.135), equity (0.004), units (0.002), securities (0.000)
|
SEC-BERT-NUM
|
stock (0.842), shares (0.157), equity (0.000), securities (0.000), units (0.000)
|
SEC-BERT-SHAPE
|
stock (0.888), shares (0.109), equity (0.001), securities (0.001), stocks (0.000)
|
Sample
|
Masked Token
|
During 2019, the Company repurchased $67.1 billion of its common stock and paid [MASK] equivalents of $14.1 billion.
|
dividend
|
Model
|
Predictions (Probability)
|
BERT-BASE-UNCASED
|
cash (0.276), net (0.128), annual (0.083), the (0.040), debt (0.027)
|
SEC-BERT-BASE
|
dividend (0.890), cash (0.018), dividends (0.016), share (0.013), tax (0.010)
|
SEC-BERT-NUM
|
dividend (0.735), cash (0.115), share (0.087), tax (0.025), stock (0.013)
|
SEC-BERT-SHAPE
|
dividend (0.655), cash (0.248), dividends (0.042), share (0.019), out (0.003)
|
Sample
|
Masked Token
|
During 2019, the Company repurchased $67.1 billion of its common stock and paid dividend [MASK] of $14.1 billion.
|
equivalents
|
Model
|
Predictions (Probability)
|
BERT-BASE-UNCASED
|
revenue (0.085), earnings (0.078), rates (0.065), amounts (0.064), proceeds (0.062)
|
SEC-BERT-BASE
|
payments (0.790), distributions (0.087), equivalents (0.068), cash (0.013), amounts (0.004)
|
SEC-BERT-NUM
|
payments (0.845), equivalents (0.097), distributions (0.024), increases (0.005), dividends (0.004)
|
SEC-BERT-SHAPE
|
payments (0.784), equivalents (0.093), distributions (0.043), dividends (0.015), requirements (0.009)
|
出版物
如果您使用此模型,请引用以下文章:
FiNER: Financial Numeric Entity Recognition for XBRL Tagging
Lefteris Loukas, Manos Fergadiotis, Ilias Chalkidis, Eirini Spyropoulou, Prodromos Malakasiotis, Ion Androutsopoulos和George Paliouras,发表于第60届计算语言学协会年会(ACL 2022)(长文),爱尔兰都柏林,2022年5月22日-27日
@inproceedings{loukas-etal-2022-finer,
title = {FiNER: Financial Numeric Entity Recognition for XBRL Tagging},
author = {Loukas, Lefteris and
Fergadiotis, Manos and
Chalkidis, Ilias and
Spyropoulou, Eirini and
Malakasiotis, Prodromos and
Androutsopoulos, Ion and
Paliouras George},
booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022)},
publisher = {Association for Computational Linguistics},
location = {Dublin, Republic of Ireland},
year = {2022},
url = {https://arxiv.org/abs/2203.06482}
}
关于我们
AUEB's Natural Language Processing Group
开发算法、模型和系统,使计算机能够处理和生成自然语言文本。
该小组目前的研究兴趣包括:
- 用于数据库、本体、文档集和Web的问答系统,特别是生物医学问答;
- 从数据库和本体生成自然语言,特别是语义Web本体;对文本进行分类,包括过滤垃圾邮件和辱骂内容;
- 信息提取和意见挖掘,包括法律文本分析和情感分析;
- 希腊语的自然语言处理工具,例如解析器和命名实体识别器;自然语言处理中的机器学习,特别是深度学习。
该小组是雅典经济与商业大学信息处理实验室的一部分。
Manos Fergadiotis
代表
AUEB's Natural Language Processing Group