模型:

nlpaueb/bert-base-uncased-contracts

英文

LEGAL-BERT:脱离法学院的木偶

LEGAL-BERT是用于法律领域的BERT模型系列,旨在辅助法律自然语言处理研究、计算法律和法律科技应用。为了预训练不同变体的LEGAL-BERT,我们从公开可用资源中收集了来自多个领域(包括立法、法庭案例和合同)的12GB多样化的英文法律文本。子领域变体(CONTRACTS-、EURLEX-、ECHR-)和/或通用的LEGAL-BERT在特定领域的任务上表现比直接使用BERT更好。这是在美国合同上预训练的子领域变体。

I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras和I. Androutsopoulos. "LEGAL-BERT:脱离法学院的木偶"。在Empirical Methods in Natural Language Processing (EMNLP 2020) (Short Papers)上发表,将在线举行,2020年。( https://aclanthology.org/2020.findings-emnlp.261 )

预训练语料库

LEGAL-BERT的预训练语料库包括:

预训练细节

  • 我们使用Google BERT的GitHub存储库中提供的官方代码训练了BERT( https://github.com/google-research/bert )。
  • 我们发布了一个与英文BERT-BASE模型相似的模型(12层,768隐藏层,12个头,110M参数)。
  • 我们选择遵循相同的训练设置:100万个训练步骤,每批256个长度为512的序列,初始学习率为1e-4。
  • 我们能够免费使用一个Google Cloud TPU v3-8( TensorFlow Research Cloud (TFRC) ),同时利用( GCP research credits )。非常感谢这两个Google项目的支持!

模型列表

Model name Model Path Training corpora
CONTRACTS-BERT-BASE nlpaueb/bert-base-uncased-contracts US contracts
EURLEX-BERT-BASE nlpaueb/bert-base-uncased-eurlex EU legislation
ECHR-BERT-BASE nlpaueb/bert-base-uncased-echr ECHR cases
LEGAL-BERT-BASE * nlpaueb/legal-bert-base-uncased All
LEGAL-BERT-SMALL nlpaueb/legal-bert-small-uncased All

* LEGAL-BERT-BASE是Chalkidis等人(2020)中提到的LEGAL-BERT-SC模型;该模型是根据下面提到的法律语料库从头开始训练的模型,使用相同语料库训练的句子分隔符分词器创建了一个新的词汇表。

** 由于许多人对LEGAL-BERT-FP模型(依赖于原始BERT-BASE检查点)表示了兴趣,它们已在Archive.org( https://archive.org/details/legal_bert_fp )上发布,因为这些模型是次要的,并且可能只对那些有意深入研究Chalkidis等人(2020)中的问题的人有趣。

加载预训练模型

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("nlpaueb/bert-base-uncased-contracts")
model = AutoModel.from_pretrained("nlpaueb/bert-base-uncased-contracts")

将LEGAL-BERT变体用作语言模型

Corpus Model Masked token Predictions
BERT-BASE-UNCASED
(Contracts) This [MASK] Agreement is between General Motors and John Murray . employment ('new', '0.09'), ('current', '0.04'), ('proposed', '0.03'), ('marketing', '0.03'), ('joint', '0.02')
(ECHR) The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate torture ('torture', '0.32'), ('rape', '0.22'), ('abuse', '0.14'), ('death', '0.04'), ('violence', '0.03')
(EURLEX) Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products . bovine ('farm', '0.25'), ('livestock', '0.08'), ('draft', '0.06'), ('domestic', '0.05'), ('wild', '0.05')
CONTRACTS-BERT-BASE
(Contracts) This [MASK] Agreement is between General Motors and John Murray . employment ('letter', '0.38'), ('dealer', '0.04'), ('employment', '0.03'), ('award', '0.03'), ('contribution', '0.02')
(ECHR) The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate torture ('death', '0.39'), ('imprisonment', '0.07'), ('contempt', '0.05'), ('being', '0.03'), ('crime', '0.02')
(EURLEX) Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products . bovine (('domestic', '0.18'), ('laboratory', '0.07'), ('household', '0.06'), ('personal', '0.06'), ('the', '0.04')
EURLEX-BERT-BASE
(Contracts) This [MASK] Agreement is between General Motors and John Murray . employment ('supply', '0.11'), ('cooperation', '0.08'), ('service', '0.07'), ('licence', '0.07'), ('distribution', '0.05')
(ECHR) The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate torture ('torture', '0.66'), ('death', '0.07'), ('imprisonment', '0.07'), ('murder', '0.04'), ('rape', '0.02')
(EURLEX) Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products . bovine ('live', '0.43'), ('pet', '0.28'), ('certain', '0.05'), ('fur', '0.03'), ('the', '0.02')
ECHR-BERT-BASE
(Contracts) This [MASK] Agreement is between General Motors and John Murray . employment ('second', '0.24'), ('latter', '0.10'), ('draft', '0.05'), ('bilateral', '0.05'), ('arbitration', '0.04')
(ECHR) The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate torture ('torture', '0.99'), ('death', '0.01'), ('inhuman', '0.00'), ('beating', '0.00'), ('rape', '0.00')
(EURLEX) Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products . bovine ('pet', '0.17'), ('all', '0.12'), ('slaughtered', '0.10'), ('domestic', '0.07'), ('individual', '0.05')
LEGAL-BERT-BASE
(Contracts) This [MASK] Agreement is between General Motors and John Murray . employment ('settlement', '0.26'), ('letter', '0.23'), ('dealer', '0.04'), ('master', '0.02'), ('supplemental', '0.02')
(ECHR) The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate torture ('torture', '1.00'), ('detention', '0.00'), ('arrest', '0.00'), ('rape', '0.00'), ('death', '0.00')
(EURLEX) Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products . bovine ('live', '0.67'), ('beef', '0.17'), ('farm', '0.03'), ('pet', '0.02'), ('dairy', '0.01')
LEGAL-BERT-SMALL
(Contracts) This [MASK] Agreement is between General Motors and John Murray . employment ('license', '0.09'), ('transition', '0.08'), ('settlement', '0.04'), ('consent', '0.03'), ('letter', '0.03')
(ECHR) The applicant submitted that her husband was subjected to treatment amounting to [MASK] whilst in the custody of Adana Security Directorate torture ('torture', '0.59'), ('pain', '0.05'), ('ptsd', '0.05'), ('death', '0.02'), ('tuberculosis', '0.02')
(EURLEX) Establishing a system for the identification and registration of [MASK] animals and regarding the labelling of beef and beef products . bovine ('all', '0.08'), ('live', '0.07'), ('certain', '0.07'), ('the', '0.07'), ('farm', '0.05')

在下游任务上的评估

请参考文章"LEGAL-BERT:脱离法学院的木偶"。Chalkidis等人,2020年,( https://aclanthology.org/2020.findings-emnlp.261

作者-出版物

@inproceedings{chalkidis-etal-2020-legal,
    title = "{LEGAL}-{BERT}: The Muppets straight out of Law School",
    author = "Chalkidis, Ilias  and
      Fergadiotis, Manos  and
      Malakasiotis, Prodromos  and
      Aletras, Nikolaos  and
      Androutsopoulos, Ion",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2020",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    doi = "10.18653/v1/2020.findings-emnlp.261",
    pages = "2898--2904"
}

关于我们

AUEB's Natural Language Processing Group 开发了能够处理和生成自然语言文本的算法、模型和系统。

该小组目前的研究兴趣包括:

  • 问题回答系统,涉及数据库、本体、文档集合和网络,特别是生物医学问题回答,
  • 从数据库和本体中生成自然语言,特别是语义Web本体,文本分类,包括过滤垃圾邮件和滥用内容,
  • 信息提取和观点挖掘,包括法律文本分析和情感分析,
  • 希腊语的自然语言处理工具,例如解析器和命名实体识别器,自然语言处理中的机器学习,特别是深度学习。

该小组是雅典经济与商业大学信息处理实验室的一部分。

Ilias Chalkidis 代表 AUEB's Natural Language Processing Group

| Github: @ilias.chalkidis | Twitter: @KiddoThe2B |