LEGAL-BERT：脱离法学院的Muppets

LEGAL-BERT是用于法律领域的BERT模型的系列，旨在辅助法律自然语言处理研究、计算法律和法律技术应用。为了对不同变种的LEGAL-BERT进行预训练，我们从公开的资源中收集了12GB的多样化的英文法律文本，涵盖了多个领域（例如法规、法院案例、合同）。与直接使用BERT相比，子领域的变种（CONTRACTS-、EURLEX-、ECHR-）和/或通用LEGAL-BERT在特定领域任务上表现更好。此外，还提供了一个从头开始预训练的轻量级模型（仅为BERT-BASE大小的33%），性能与大型模型相当。

I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras and I. Androutsopoulos. "LEGAL-BERT：脱离法学院的Muppets". 在Empirical Methods in Natural Language Processing (EMNLP 2020) (Short Papers)会议上，线上举办，2020年。（ https://aclanthology.org/2020.findings-emnlp.261 ）。

预训练语料

LEGAL-BERT的预训练语料包括：

116,062份欧盟法规文件，可从欧盟出版社的EURLEX（ http://eur-lex.europa.eu ）获取，这是欧盟法律库。
61,826份英国立法文件，可从英国立法门户网站（ http://www.legislation.gov.uk ）获取。
19,867份欧洲法院（ECJ）的案例，也可从EURLEX获取。
12,554份欧洲人权法院（ECHR）的案例，可从欧洲人权法院的HUDOC库（ http://hudoc.echr.coe.int/eng ）获取。
164,141份来自美国各地法院的案例，托管在Case Law Access Project门户网站（ https://case.law ）。
76,366份美国证券交易委员会（SECOM）的合同，可从美国证券交易委员会的EDGAR数据库（ https://www.sec.gov/edgar.shtml ）获取。

预训练细节

我们使用Google BERT在GitHub上提供的官方代码进行BERT训练（ https://github.com/google-research/bert ）。
发布了类似英文BERT-BASE模型的模型（12层，768隐藏层，12个注意头，110M参数）。
我们选择遵循相同的训练设置：100万个训练步骤，batch大小为256，序列长度为512，初始学习率为1e-4。
我们能够使用免费提供的Google Cloud TPU v3-8（ TensorFlow Research Cloud (TFRC) ），同时也利用了（ GCP research credits ）。特此感谢这两个Google项目对我们的支持！
LEGAL-BERT的一部分是一种基于法律数据从头开始预训练的轻量级模型，性能可与更大的模型相媲美，同时更高效（速度大约快4倍），对环境的影响更小。

模型列表

Model name	Model Path	Training corpora
CONTRACTS-BERT-BASE	nlpaueb/bert-base-uncased-contracts	US contracts
EURLEX-BERT-BASE	nlpaueb/bert-base-uncased-eurlex	EU legislation
ECHR-BERT-BASE	nlpaueb/bert-base-uncased-echr	ECHR cases
LEGAL-BERT-BASE *	nlpaueb/legal-bert-base-uncased	All
LEGAL-BERT-SMALL	nlpaueb/legal-bert-small-uncased	All

*LEGAL-BERT-BASE是Chalkidis等人（2020）中所提到的LEGAL-BERT-SC模型的名称，它是使用法律语料库从头开始训练的模型，使用了由句子分段分词器在同一语料库上训练得到的新词汇表。

**鉴于您中的许多人对LEGAL-BERT-FP模型（依赖原始BERT-BASE检查点的模型）表示了兴趣，这些模型已在Archive.org（ https://archive.org/details/legal_bert_fp ）上发布，因为这些模型是辅助性的，可能只对那些想要深入研究Chalkidis等人（2020）的问题的人感兴趣。

加载预训练模型

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("nlpaueb/legal-bert-base-uncased")
model = AutoModel.from_pretrained("nlpaueb/legal-bert-base-uncased")

使用LEGAL-BERT变体作为语言模型

Corpus	Model	Masked token	Predictions
BERT-BASE-UNCASED
(Contracts)	This [MASK] Agreement is between General Motors and John Murray .	employment	('new', '0.09'), ('current', '0.04'), ('proposed', '0.03'), ('marketing', '0.03'), ('joint', '0.02')
(ECHR)	The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate	torture	('torture', '0.32'), ('rape', '0.22'), ('abuse', '0.14'), ('death', '0.04'), ('violence', '0.03')
(EURLEX)	Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products .	bovine	('farm', '0.25'), ('livestock', '0.08'), ('draft', '0.06'), ('domestic', '0.05'), ('wild', '0.05')
CONTRACTS-BERT-BASE
(Contracts)	This [MASK] Agreement is between General Motors and John Murray .	employment	('letter', '0.38'), ('dealer', '0.04'), ('employment', '0.03'), ('award', '0.03'), ('contribution', '0.02')
(ECHR)	The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate	torture	('death', '0.39'), ('imprisonment', '0.07'), ('contempt', '0.05'), ('being', '0.03'), ('crime', '0.02')
(EURLEX)	Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products .	bovine	(('domestic', '0.18'), ('laboratory', '0.07'), ('household', '0.06'), ('personal', '0.06'), ('the', '0.04')
EURLEX-BERT-BASE
(Contracts)	This [MASK] Agreement is between General Motors and John Murray .	employment	('supply', '0.11'), ('cooperation', '0.08'), ('service', '0.07'), ('licence', '0.07'), ('distribution', '0.05')
(ECHR)	The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate	torture	('torture', '0.66'), ('death', '0.07'), ('imprisonment', '0.07'), ('murder', '0.04'), ('rape', '0.02')
(EURLEX)	Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products .	bovine	('live', '0.43'), ('pet', '0.28'), ('certain', '0.05'), ('fur', '0.03'), ('the', '0.02')
ECHR-BERT-BASE
(Contracts)	This [MASK] Agreement is between General Motors and John Murray .	employment	('second', '0.24'), ('latter', '0.10'), ('draft', '0.05'), ('bilateral', '0.05'), ('arbitration', '0.04')
(ECHR)	The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate	torture	('torture', '0.99'), ('death', '0.01'), ('inhuman', '0.00'), ('beating', '0.00'), ('rape', '0.00')
(EURLEX)	Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products .	bovine	('pet', '0.17'), ('all', '0.12'), ('slaughtered', '0.10'), ('domestic', '0.07'), ('individual', '0.05')
LEGAL-BERT-BASE
(Contracts)	This [MASK] Agreement is between General Motors and John Murray .	employment	('settlement', '0.26'), ('letter', '0.23'), ('dealer', '0.04'), ('master', '0.02'), ('supplemental', '0.02')
(ECHR)	The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate	torture	('torture', '1.00'), ('detention', '0.00'), ('arrest', '0.00'), ('rape', '0.00'), ('death', '0.00')
(EURLEX)	Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products .	bovine	('live', '0.67'), ('beef', '0.17'), ('farm', '0.03'), ('pet', '0.02'), ('dairy', '0.01')
LEGAL-BERT-SMALL
(Contracts)	This [MASK] Agreement is between General Motors and John Murray .	employment	('license', '0.09'), ('transition', '0.08'), ('settlement', '0.04'), ('consent', '0.03'), ('letter', '0.03')
(ECHR)	The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate	torture	('torture', '0.59'), ('pain', '0.05'), ('ptsd', '0.05'), ('death', '0.02'), ('tuberculosis', '0.02')
(EURLEX)	Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products .	bovine	('all', '0.08'), ('live', '0.07'), ('certain', '0.07'), ('the', '0.07'), ('farm', '0.05')

下游任务评估

请参阅文章“LEGAL-BERT：脱离法学院的Muppets”。Chalkidis等人，2020年（ https://aclanthology.org/2020.findings-emnlp.261 ）。

作者-出版物

@inproceedings{chalkidis-etal-2020-legal,
    title = "{LEGAL}-{BERT}: The Muppets straight out of Law School",
    author = "Chalkidis, Ilias  and
      Fergadiotis, Manos  and
      Malakasiotis, Prodromos  and
      Aletras, Nikolaos  and
      Androutsopoulos, Ion",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2020",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    doi = "10.18653/v1/2020.findings-emnlp.261",
    pages = "2898--2904"
}

关于我们

AUEB's Natural Language Processing Group 开发算法、模型和系统，使计算机能够处理和生成自然语言文本。

该团队目前的研究兴趣包括：

对数据库、本体、文档集合和Web等进行问题回答系统，尤其是生物医学问题回答；
从数据库和本体生成自然语言，尤其是语义Web本体；文本分类，包括垃圾邮件和滥用内容过滤；
信息抽取和意见挖掘，包括法律文本分析和情感分析；
用于希腊语的自然语言处理工具，例如解析器和命名实体识别器；自然语言处理中的机器学习，特别是深度学习。

该团队是希腊经济与商业雅典经济与商业学院信息处理实验室的一部分。

Ilias Chalkidis 代表 AUEB's Natural Language Processing Group

| Github： @ilias.chalkidis | Twitter： @KiddoThe2B |

作者:

AUEB NLP Group

数据集大小:

1.32 GB