数据集:

joelito/MultiLegalPileWikipediaFiltered

计算机处理:

multilingual

大小:

10M<n<100M

语言创建人:

found

批注创建人:

other

源数据集:

original

许可:

cc-by-4.0
中文

Dataset Card for MultiLegalPileWikipediaFiltered: A filtered version of the MultiLegalPile dataset, together with wikipedia articles

Dataset Summary

The Multi_Legal_Pile is a large-scale multilingual legal dataset suited for pretraining language models. It spans over 24 languages and four legal text types.

Supported Tasks and Leaderboards

The dataset supports the tasks of fill-mask.

Languages

The following languages are supported: bg, cs, da, de, el, en, es, et, fi, fr, ga, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv

Dataset Structure

It is structured in the following format: {language} {text_type} {shard}.jsonl.xz

text_type is one of the following:

  • caselaw
  • contracts
  • legislation
  • other
  • wikipedia

Use the dataset like this:

from datasets import load_dataset

config = 'en_contracts' # {language}_{text_type}
dataset = load_dataset('joelito/Multi_Legal_Pile', config, split='train', streaming=True)

'config' is a combination of language and text_type, e.g. 'en_contracts' or 'de_caselaw'. To load all the languages or all the text_types, use 'all' instead of the language or text_type (e.g., ' all_legislation').

Data Instances

The file format is jsonl.xz and there is a train and validation split available. Since some configurations are very small or non-existent, they might not contain a train split or not be present at all.

The complete dataset consists of five large subsets:

Language Source Size (MB) Words Documents Words/Document
all all 1.29761e+06 81214262514 57305071 1417
all caselaw 695837 44372248995 30085886 1474
all contracts 122599 7964531030 1785686 4460
all legislation 189135 10879386581 3601518 3020
all other 126570 8780080882 3358073 2614
all wikipedia 163468 9218015026 18473908 498
bg all 14028 535256525 355650 1505
bg caselaw 2897 109634090 52648 2082
bg contracts 748 31292877 7107 4403
bg legislation 8015 308946116 82777 3732
bg other 0 0 0 0
bg wikipedia 2368 85383442 213118 400
cs all 21818 1123000335 839914 1337
cs caselaw 11151 574336489 296652 1936
cs contracts 492 28106428 7383 3806
cs legislation 6288 333850509 88731 3762
cs other 0 0 0 0
cs wikipedia 3887 186706909 447148 417
da all 16024 970954498 576256 1684
da caselaw 3469 210730560 89702 2349
da contracts 559 35592407 10827 3287
da legislation 10736 653153146 265868 2456
da other 0 0 0 0
da wikipedia 1259 71478385 209859 340
de all 63887 3512253170 3216030 1092
de caselaw 31527 1785439383 596800 2991
de contracts 614 36786772 11041 3331
de legislation 8934 512840663 276034 1857
de other 0 0 0 0
de wikipedia 22812 1177186352 2332155 504
el all 23167 800722723 457553 1750
el caselaw 6007 203770918 85496 2383
el contracts 1050 38963772 10266 3795
el legislation 12906 455240770 171356 2656
el other 0 0 0 0
el wikipedia 3204 102747263 190435 539
en all 712173 47279626514 21112650 2239
en caselaw 380976 25561971376 10240724 2496
en contracts 71360 7260323438 1594942 4552
en legislation 36587 2537696894 657805 3857
en other 126570 8780080882 3358073 2614
en wikipedia 51053 3139553924 5261106 596
es all 23657 1515689548 1567527 966
es caselaw 3299 220506573 83872 2629
es contracts 594 41840328 10048 4164
es legislation 6837 462661276 149368 3097
es other 0 0 0 0
es wikipedia 12928 790681371 1324239 597
et all 7446 372896353 261641 1425
et caselaw 1835 92951578 58736 1582
et contracts 433 24017402 7371 3258
et legislation 4200 210952455 63922 3300
et other 0 0 0 0
et wikipedia 978 44974918 131612 341
fi all 11501 513990484 592986 866
fi caselaw 2854 126368889 77882 1622
fi contracts 504 25386705 8894 2854
fi legislation 5532 252344531 103907 2428
fi other 0 0 0 0
fi wikipedia 2610 109890359 402303 273
fr all 47186 2936056985 2734954 1073
fr caselaw 18313 1170335690 435569 2686
fr contracts 633 41983091 11071 3792
fr legislation 9297 600170792 243313 2466
fr other 0 0 0 0
fr wikipedia 18942 1123567412 2045001 549
ga all 1209 72041312 30064 2396
ga caselaw 11 676795 835 810
ga contracts 29 1820765 365 4988
ga legislation 1048 62513018 5983 10448
ga other 0 0 0 0
ga wikipedia 122 7030734 22881 307
hr all 5377 315295665 211151 1493
hr caselaw 1026 62358456 31322 1990
hr contracts 395 24957774 6552 3809
hr legislation 2906 171415656 36365 4713
hr other 0 0 0 0
hr wikipedia 1050 56563779 136912 413
hu all 12351 564082537 495822 1137
hu caselaw 2376 110034426 59074 1862
hu contracts 534 27258352 7385 3691
hu legislation 5744 264572303 86862 3045
hu other 0 0 0 0
hu wikipedia 3697 162217456 342501 473
it all 26744 1658638775 1615301 1026
it caselaw 6483 406520336 156630 2595
it contracts 597 40131223 10985 3653
it legislation 8332 542579039 227968 2380
it other 0 0 0 0
it wikipedia 11332 669408177 1219718 548
lt all 7772 399310081 264537 1509
lt caselaw 1992 101672069 59485 1709
lt contracts 475 27009922 7473 3614
lt legislation 4550 235543873 64106 3674
lt other 0 0 0 0
lt wikipedia 755 35084217 133473 262
lv all 7701 386833125 211244 1831
lv caselaw 2082 103311512 58992 1751
lv contracts 481 26692972 7429 3593
lv legislation 4621 233088284 64087 3637
lv other 0 0 0 0
lv wikipedia 518 23740357 80736 294
mt all 7180 370558634 122056 3035
mt caselaw 2016 100309542 52942 1894
mt contracts 486 27701852 6937 3993
mt legislation 4620 239708644 57979 4134
mt other 0 0 0 0
mt wikipedia 58 2838596 4198 676
nl all 17674 1112460059 1200534 926
nl caselaw 3227 206147113 87170 2364
nl contracts 604 40245662 11027 3649
nl legislation 8484 550788527 232204 2372
nl other 0 0 0 0
nl wikipedia 5360 315278757 870133 362
pl all 14762 773692198 1160849 666
pl caselaw 2141 115695709 59649 1939
pl contracts 489 28543526 7478 3817
pl legislation 5459 299334705 89264 3353
pl other 0 0 0 0
pl wikipedia 6672 330118258 1004458 328
pt all 210656 13466463586 18173061 741
pt caselaw 196919 12611760973 17251236 731
pt contracts 571 37997495 9897 3839
pt legislation 6853 439066783 148176 2963
pt other 0 0 0 0
pt wikipedia 6313 377638335 763752 494
ro all 14794 808799454 481763 1678
ro caselaw 1960 114665535 53092 2159
ro contracts 495 31496978 7202 4373
ro legislation 10464 559092153 215694 2592
ro other 0 0 0 0
ro wikipedia 1874 103544788 205775 503
sk all 8700 463447112 262638 1764
sk caselaw 2072 109996398 59383 1852
sk contracts 489 28298113 7470 3788
sk legislation 5208 280182047 76760 3650
sk other 0 0 0 0
sk wikipedia 931 44970554 119025 377
sl all 9345 561775614 277497 2024
sl caselaw 1816 111097741 59193 1876
sl contracts 432 28238938 7475 3777
sl legislation 6057 365513763 88651 4123
sl other 0 0 0 0
sl wikipedia 1041 56925172 122178 465
sv all 12457 700417227 1083393 646
sv caselaw 2806 161956844 78802 2055
sv contracts 491 29844238 9061 3293
sv legislation 5456 308130634 104338 2953
sv other 0 0 0 0
sv wikipedia 3704 200485511 891192 224

Data Fields

[More Information Needed]

Data Splits

There are two splits: train and validation. The validation split contains 1000 examples and the training split contains the rest of the data.

Data Size
$ xz --list data/*.xz
Strms  Blocks   Compressed Uncompressed  Ratio  Check   Filename
    1       1    167.6 MiB  3’276.3 MiB  0.051  CRC64   data/bg_caselaw_train.0.jsonl.xz
    1       1    502.3 KiB  9’398.0 KiB  0.053  CRC64   data/bg_caselaw_validation.0.jsonl.xz
    1       1     33.4 MiB    700.3 MiB  0.048  CRC64   data/bg_contracts_train.0.jsonl.xz
    1       1  5’989.6 KiB    123.0 MiB  0.048  CRC64   data/bg_contracts_validation.0.jsonl.xz
    1       1    418.5 MiB  8’931.0 MiB  0.047  CRC64   data/bg_legislation_train.0.jsonl.xz
    1       1  5’029.4 KiB    103.1 MiB  0.048  CRC64   data/bg_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/bg_other_validation.0.jsonl.xz
    1       1    192.2 MiB  2’488.6 MiB  0.077  CRC64   data/bg_wikipedia_train.0.jsonl.xz
    1       1  1’757.8 KiB     22.9 MiB  0.075  CRC64   data/bg_wikipedia_validation.0.jsonl.xz
    1       1    476.9 MiB  4’126.1 MiB  0.116  CRC64   data/cs_caselaw_train.0.jsonl.xz
    1       1    259.8 MiB  2’556.9 MiB  0.102  CRC64   data/cs_caselaw_train.1.jsonl.xz
    1       1    420.1 KiB  3’370.3 KiB  0.125  CRC64   data/cs_caselaw_validation.0.jsonl.xz
    1       1     24.9 MiB    237.9 MiB  0.105  CRC64   data/cs_contracts_train.0.jsonl.xz
    1       1  4’412.1 KiB     41.7 MiB  0.103  CRC64   data/cs_contracts_validation.0.jsonl.xz
    1       1    361.2 MiB  3’488.9 MiB  0.104  CRC64   data/cs_legislation_train.0.jsonl.xz
    1       1     10.3 MiB     91.6 MiB  0.112  CRC64   data/cs_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/cs_other_validation.0.jsonl.xz
    1       1    390.6 MiB  1’939.4 MiB  0.201  CRC64   data/cs_wikipedia_train.0.jsonl.xz
    1       1  2’604.7 KiB     12.2 MiB  0.209  CRC64   data/cs_wikipedia_validation.0.jsonl.xz
    1       1    252.5 MiB  1’529.7 MiB  0.165  CRC64   data/da_caselaw_train.0.jsonl.xz
    1       1    555.9 KiB  3’227.1 KiB  0.172  CRC64   data/da_caselaw_validation.0.jsonl.xz
    1       1     30.1 MiB    233.9 MiB  0.129  CRC64   data/da_contracts_train.0.jsonl.xz
    1       1  2’897.6 KiB     23.6 MiB  0.120  CRC64   data/da_contracts_validation.0.jsonl.xz
    1       1    476.9 MiB  3’325.8 MiB  0.143  CRC64   data/da_legislation_train.0.jsonl.xz
    1       1    237.3 MiB  1’444.5 MiB  0.164  CRC64   data/da_legislation_train.1.jsonl.xz
    1       1  3’232.5 KiB     60.6 MiB  0.052  CRC64   data/da_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/da_other_validation.0.jsonl.xz
    1       1    128.8 MiB    512.1 MiB  0.252  CRC64   data/da_wikipedia_train.0.jsonl.xz
    1       1  1’514.1 KiB  5’476.3 KiB  0.276  CRC64   data/da_wikipedia_validation.0.jsonl.xz
    1       1    476.9 MiB  2’803.8 MiB  0.170  CRC64   data/de_caselaw_train.0.jsonl.xz
    1       1    476.9 MiB  2’821.4 MiB  0.169  CRC64   data/de_caselaw_train.1.jsonl.xz
    1       1    476.9 MiB  2’720.2 MiB  0.175  CRC64   data/de_caselaw_train.2.jsonl.xz
    1       1    476.9 MiB  2’704.1 MiB  0.176  CRC64   data/de_caselaw_train.3.jsonl.xz
    1       1    460.5 MiB  2’504.5 MiB  0.184  CRC64   data/de_caselaw_train.4.jsonl.xz
    1       1    594.0 KiB  3’416.4 KiB  0.174  CRC64   data/de_caselaw_validation.0.jsonl.xz
    1       1     32.0 MiB    255.8 MiB  0.125  CRC64   data/de_contracts_train.0.jsonl.xz
    1       1  3’037.7 KiB     24.7 MiB  0.120  CRC64   data/de_contracts_validation.0.jsonl.xz
    1       1    476.9 MiB  3’386.0 MiB  0.141  CRC64   data/de_legislation_train.0.jsonl.xz
    1       1     93.3 MiB    592.3 MiB  0.158  CRC64   data/de_legislation_train.1.jsonl.xz
    1       1  3’265.9 KiB     20.5 MiB  0.156  CRC64   data/de_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/de_other_validation.0.jsonl.xz
    1       1    476.9 MiB  1’883.7 MiB  0.253  CRC64   data/de_wikipedia_train.0.jsonl.xz
    1       1    476.9 MiB  1’891.6 MiB  0.252  CRC64   data/de_wikipedia_train.1.jsonl.xz
    1       1    476.9 MiB  1’893.7 MiB  0.252  CRC64   data/de_wikipedia_train.2.jsonl.xz
    1       1    476.9 MiB  1’894.1 MiB  0.252  CRC64   data/de_wikipedia_train.3.jsonl.xz
    1       1    407.9 MiB  1’622.0 MiB  0.251  CRC64   data/de_wikipedia_train.4.jsonl.xz
    1       1  1’172.5 KiB  4’210.2 KiB  0.278  CRC64   data/de_wikipedia_validation.0.jsonl.xz
    1       1    344.7 MiB  6’908.3 MiB  0.050  CRC64   data/el_caselaw_train.0.jsonl.xz
    1       1    870.4 KiB     14.3 MiB  0.060  CRC64   data/el_caselaw_validation.0.jsonl.xz
    1       1     49.7 MiB  1’083.8 MiB  0.046  CRC64   data/el_contracts_train.0.jsonl.xz
    1       1  4’701.3 KiB    101.6 MiB  0.045  CRC64   data/el_contracts_validation.0.jsonl.xz
    1       1    476.9 MiB     10.2 GiB  0.046  CRC64   data/el_legislation_train.0.jsonl.xz
    1       1    203.0 MiB  3’994.0 MiB  0.051  CRC64   data/el_legislation_train.1.jsonl.xz
    1       1  9’744.3 KiB    186.6 MiB  0.051  CRC64   data/el_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/el_other_validation.0.jsonl.xz
    1       1    246.4 MiB  3’465.7 MiB  0.071  CRC64   data/el_wikipedia_train.0.jsonl.xz
    1       1  2’591.7 KiB     35.6 MiB  0.071  CRC64   data/el_wikipedia_validation.0.jsonl.xz
    1       1    476.9 MiB  2’188.6 MiB  0.218  CRC64   data/en_caselaw_train.0.jsonl.xz
    1       1    476.9 MiB  2’416.1 MiB  0.197  CRC64   data/en_caselaw_train.10.jsonl.xz
    1       1    477.2 MiB  2’688.1 MiB  0.178  CRC64   data/en_caselaw_train.11.jsonl.xz
    1       1    476.9 MiB  2’865.9 MiB  0.166  CRC64   data/en_caselaw_train.12.jsonl.xz
    1       1    476.9 MiB  2’494.1 MiB  0.191  CRC64   data/en_caselaw_train.13.jsonl.xz
    1       1    476.9 MiB  2’126.6 MiB  0.224  CRC64   data/en_caselaw_train.14.jsonl.xz
    1       1    476.9 MiB  2’440.9 MiB  0.195  CRC64   data/en_caselaw_train.15.jsonl.xz
    1       1    476.9 MiB  3’822.2 MiB  0.125  CRC64   data/en_caselaw_train.16.jsonl.xz
    1       1    476.9 MiB  3’831.4 MiB  0.124  CRC64   data/en_caselaw_train.17.jsonl.xz
    1       1    476.9 MiB  3’812.2 MiB  0.125  CRC64   data/en_caselaw_train.18.jsonl.xz
    1       1    476.9 MiB  2’233.5 MiB  0.214  CRC64   data/en_caselaw_train.19.jsonl.xz
    1       1    476.9 MiB  2’195.9 MiB  0.217  CRC64   data/en_caselaw_train.1.jsonl.xz
    1       1    476.9 MiB  2’185.8 MiB  0.218  CRC64   data/en_caselaw_train.20.jsonl.xz
    1       1    476.9 MiB  2’634.9 MiB  0.181  CRC64   data/en_caselaw_train.21.jsonl.xz
    1       1    476.9 MiB  2’670.8 MiB  0.179  CRC64   data/en_caselaw_train.22.jsonl.xz
    1       1    476.9 MiB  2’762.0 MiB  0.173  CRC64   data/en_caselaw_train.23.jsonl.xz
    1       1    476.9 MiB  2’153.6 MiB  0.221  CRC64   data/en_caselaw_train.24.jsonl.xz
    1       1    476.9 MiB  2’152.0 MiB  0.222  CRC64   data/en_caselaw_train.25.jsonl.xz
    1       1    476.9 MiB  2’205.0 MiB  0.216  CRC64   data/en_caselaw_train.26.jsonl.xz
    1       1    476.9 MiB  2’141.0 MiB  0.223  CRC64   data/en_caselaw_train.27.jsonl.xz
    1       1    476.9 MiB  2’145.1 MiB  0.222  CRC64   data/en_caselaw_train.28.jsonl.xz
    1       1    476.9 MiB  2’137.9 MiB  0.223  CRC64   data/en_caselaw_train.29.jsonl.xz
    1       1    476.9 MiB  2’189.0 MiB  0.218  CRC64   data/en_caselaw_train.2.jsonl.xz
    1       1    476.9 MiB  2’150.9 MiB  0.222  CRC64   data/en_caselaw_train.30.jsonl.xz
    1       1    476.9 MiB  2’142.7 MiB  0.223  CRC64   data/en_caselaw_train.31.jsonl.xz
    1       1    476.9 MiB  2’203.4 MiB  0.216  CRC64   data/en_caselaw_train.32.jsonl.xz
    1       1    476.9 MiB  2’205.4 MiB  0.216  CRC64   data/en_caselaw_train.33.jsonl.xz
    1       1    476.9 MiB  2’206.0 MiB  0.216  CRC64   data/en_caselaw_train.34.jsonl.xz
    1       1    476.9 MiB  2’164.9 MiB  0.220  CRC64   data/en_caselaw_train.35.jsonl.xz
    1       1    476.9 MiB  2’810.3 MiB  0.170  CRC64   data/en_caselaw_train.36.jsonl.xz
    1       1    476.9 MiB  2’854.1 MiB  0.167  CRC64   data/en_caselaw_train.37.jsonl.xz
    1       1    476.9 MiB  3’109.2 MiB  0.153  CRC64   data/en_caselaw_train.38.jsonl.xz
    1       1    476.9 MiB  3’323.6 MiB  0.143  CRC64   data/en_caselaw_train.39.jsonl.xz
    1       1    476.9 MiB  2’155.3 MiB  0.221  CRC64   data/en_caselaw_train.3.jsonl.xz
    1       1    476.9 MiB  2’881.5 MiB  0.165  CRC64   data/en_caselaw_train.40.jsonl.xz
    1       1    476.9 MiB  2’157.1 MiB  0.221  CRC64   data/en_caselaw_train.41.jsonl.xz
    1       1    477.0 MiB  2’530.2 MiB  0.189  CRC64   data/en_caselaw_train.42.jsonl.xz
    1       1    476.8 MiB  2’540.1 MiB  0.188  CRC64   data/en_caselaw_train.43.jsonl.xz
    1       1    476.9 MiB  2’182.2 MiB  0.219  CRC64   data/en_caselaw_train.44.jsonl.xz
    1       1    476.9 MiB  2’163.2 MiB  0.220  CRC64   data/en_caselaw_train.45.jsonl.xz
    1       1    476.9 MiB  2’213.3 MiB  0.215  CRC64   data/en_caselaw_train.46.jsonl.xz
    1       1    476.9 MiB  2’241.5 MiB  0.213  CRC64   data/en_caselaw_train.47.jsonl.xz
    1       1    476.9 MiB  2’203.6 MiB  0.216  CRC64   data/en_caselaw_train.48.jsonl.xz
    1       1    476.9 MiB  2’480.6 MiB  0.192  CRC64   data/en_caselaw_train.49.jsonl.xz
    1       1    476.9 MiB  2’176.7 MiB  0.219  CRC64   data/en_caselaw_train.4.jsonl.xz
    1       1    476.9 MiB  2’214.7 MiB  0.215  CRC64   data/en_caselaw_train.50.jsonl.xz
    1       1    476.9 MiB  2’128.0 MiB  0.224  CRC64   data/en_caselaw_train.51.jsonl.xz
    1       1    476.9 MiB  2’151.0 MiB  0.222  CRC64   data/en_caselaw_train.52.jsonl.xz
    1       1    476.9 MiB  2’173.6 MiB  0.219  CRC64   data/en_caselaw_train.53.jsonl.xz
    1       1    476.9 MiB  2’773.8 MiB  0.172  CRC64   data/en_caselaw_train.54.jsonl.xz
    1       1    476.9 MiB  2’806.2 MiB  0.170  CRC64   data/en_caselaw_train.55.jsonl.xz
    1       1    476.9 MiB  3’920.9 MiB  0.122  CRC64   data/en_caselaw_train.56.jsonl.xz
    1       1    476.9 MiB  2’517.2 MiB  0.189  CRC64   data/en_caselaw_train.57.jsonl.xz
    1       1    477.5 MiB  2’844.0 MiB  0.168  CRC64   data/en_caselaw_train.58.jsonl.xz
    1       1    476.9 MiB  2’810.7 MiB  0.170  CRC64   data/en_caselaw_train.59.jsonl.xz
    1       1    476.9 MiB  2’160.4 MiB  0.221  CRC64   data/en_caselaw_train.5.jsonl.xz
    1       1    476.9 MiB  3’033.0 MiB  0.157  CRC64   data/en_caselaw_train.60.jsonl.xz
    1       1    476.9 MiB  2’255.1 MiB  0.211  CRC64   data/en_caselaw_train.61.jsonl.xz
    1       1    476.9 MiB  2’110.1 MiB  0.226  CRC64   data/en_caselaw_train.62.jsonl.xz
    1       1    476.9 MiB  2’130.3 MiB  0.224  CRC64   data/en_caselaw_train.63.jsonl.xz
    1       1    476.9 MiB  2’133.2 MiB  0.224  CRC64   data/en_caselaw_train.64.jsonl.xz
    1       1     44.8 MiB    199.6 MiB  0.225  CRC64   data/en_caselaw_train.65.jsonl.xz
    1       1    476.9 MiB  2’153.3 MiB  0.221  CRC64   data/en_caselaw_train.6.jsonl.xz
    1       1    476.9 MiB  2’130.8 MiB  0.224  CRC64   data/en_caselaw_train.7.jsonl.xz
    1       1    476.9 MiB  2’152.2 MiB  0.222  CRC64   data/en_caselaw_train.8.jsonl.xz
    1       1    476.9 MiB  2’173.3 MiB  0.219  CRC64   data/en_caselaw_train.9.jsonl.xz
    1       1  2’977.4 KiB     12.9 MiB  0.226  CRC64   data/en_caselaw_validation.0.jsonl.xz
    1       1    476.9 MiB  3’016.6 MiB  0.158  CRC64   data/en_contracts_train.0.jsonl.xz
    1       1    476.9 MiB  3’015.3 MiB  0.158  CRC64   data/en_contracts_train.10.jsonl.xz
    1       1    476.9 MiB  3’012.5 MiB  0.158  CRC64   data/en_contracts_train.11.jsonl.xz
    1       1    477.0 MiB  3’002.5 MiB  0.159  CRC64   data/en_contracts_train.12.jsonl.xz
    1       1    476.9 MiB  2’962.4 MiB  0.161  CRC64   data/en_contracts_train.13.jsonl.xz
    1       1    476.9 MiB  3’019.4 MiB  0.158  CRC64   data/en_contracts_train.14.jsonl.xz
    1       1    124.1 MiB    781.2 MiB  0.159  CRC64   data/en_contracts_train.15.jsonl.xz
    1       1    476.9 MiB  2’994.0 MiB  0.159  CRC64   data/en_contracts_train.1.jsonl.xz
    1       1    476.8 MiB  3’084.9 MiB  0.155  CRC64   data/en_contracts_train.2.jsonl.xz
    1       1    476.9 MiB  3’123.4 MiB  0.153  CRC64   data/en_contracts_train.3.jsonl.xz
    1       1    476.9 MiB  3’120.7 MiB  0.153  CRC64   data/en_contracts_train.4.jsonl.xz
    1       1    477.0 MiB  3’094.2 MiB  0.154  CRC64   data/en_contracts_train.5.jsonl.xz
    1       1    476.9 MiB  3’010.9 MiB  0.158  CRC64   data/en_contracts_train.6.jsonl.xz
    1       1    476.9 MiB  3’015.0 MiB  0.158  CRC64   data/en_contracts_train.7.jsonl.xz
    1       1    476.9 MiB  2’995.7 MiB  0.159  CRC64   data/en_contracts_train.8.jsonl.xz
    1       1    476.9 MiB  3’017.9 MiB  0.158  CRC64   data/en_contracts_train.9.jsonl.xz
    1       1  9’980.4 KiB     63.7 MiB  0.153  CRC64   data/en_contracts_validation.0.jsonl.xz
    1       1    476.9 MiB  3’040.8 MiB  0.157  CRC64   data/en_legislation_train.0.jsonl.xz
    1       1    476.9 MiB  3’047.3 MiB  0.156  CRC64   data/en_legislation_train.1.jsonl.xz
    1       1    476.9 MiB  3’351.5 MiB  0.142  CRC64   data/en_legislation_train.2.jsonl.xz
    1       1    478.7 MiB  3’408.4 MiB  0.140  CRC64   data/en_legislation_train.3.jsonl.xz
    1       1    372.5 MiB  2’620.0 MiB  0.142  CRC64   data/en_legislation_train.4.jsonl.xz
    1       1  2’733.5 KiB     13.8 MiB  0.193  CRC64   data/en_legislation_validation.0.jsonl.xz
    1       1    476.9 MiB  4’782.4 MiB  0.100  CRC64   data/en_other_train.0.jsonl.xz
    1       1    476.9 MiB  4’347.1 MiB  0.110  CRC64   data/en_other_train.10.jsonl.xz
    1       1    477.1 MiB  3’044.6 MiB  0.157  CRC64   data/en_other_train.11.jsonl.xz
    1       1    477.1 MiB  2’147.8 MiB  0.222  CRC64   data/en_other_train.12.jsonl.xz
    1       1    477.0 MiB  2’182.8 MiB  0.219  CRC64   data/en_other_train.13.jsonl.xz
    1       1     33.3 MiB    151.7 MiB  0.219  CRC64   data/en_other_train.14.jsonl.xz
    1       1    476.9 MiB  4’883.8 MiB  0.098  CRC64   data/en_other_train.1.jsonl.xz
    1       1    476.9 MiB  4’646.7 MiB  0.103  CRC64   data/en_other_train.2.jsonl.xz
    1       1    476.9 MiB  4’542.8 MiB  0.105  CRC64   data/en_other_train.3.jsonl.xz
    1       1    476.9 MiB  4’574.8 MiB  0.104  CRC64   data/en_other_train.4.jsonl.xz
    1       1    476.9 MiB  4’622.5 MiB  0.103  CRC64   data/en_other_train.5.jsonl.xz
    1       1    476.9 MiB  4’520.7 MiB  0.105  CRC64   data/en_other_train.6.jsonl.xz
    1       1    476.9 MiB  2’942.4 MiB  0.162  CRC64   data/en_other_train.7.jsonl.xz
    1       1    476.9 MiB  2’544.0 MiB  0.187  CRC64   data/en_other_train.8.jsonl.xz
    1       1    476.9 MiB  4’515.4 MiB  0.106  CRC64   data/en_other_train.9.jsonl.xz
    1       1  2’165.8 KiB     19.6 MiB  0.108  CRC64   data/en_other_validation.0.jsonl.xz
    1       1    476.9 MiB  1’803.2 MiB  0.264  CRC64   data/en_wikipedia_train.0.jsonl.xz
    1       1    441.1 MiB  1’670.5 MiB  0.264  CRC64   data/en_wikipedia_train.10.jsonl.xz
    1       1    476.9 MiB  1’803.6 MiB  0.264  CRC64   data/en_wikipedia_train.1.jsonl.xz
    1       1    476.9 MiB  1’802.5 MiB  0.265  CRC64   data/en_wikipedia_train.2.jsonl.xz
    1       1    476.9 MiB  1’805.0 MiB  0.264  CRC64   data/en_wikipedia_train.3.jsonl.xz
    1       1    476.9 MiB  1’804.3 MiB  0.264  CRC64   data/en_wikipedia_train.4.jsonl.xz
    1       1    476.9 MiB  1’804.0 MiB  0.264  CRC64   data/en_wikipedia_train.5.jsonl.xz
    1       1    476.9 MiB  1’804.1 MiB  0.264  CRC64   data/en_wikipedia_train.6.jsonl.xz
    1       1    476.9 MiB  1’803.6 MiB  0.264  CRC64   data/en_wikipedia_train.7.jsonl.xz
    1       1    476.9 MiB  1’805.2 MiB  0.264  CRC64   data/en_wikipedia_train.8.jsonl.xz
    1       1    476.9 MiB  1’804.3 MiB  0.264  CRC64   data/en_wikipedia_train.9.jsonl.xz
    1       1  1’004.9 KiB  3’492.7 KiB  0.288  CRC64   data/en_wikipedia_validation.0.jsonl.xz
    1       1    216.4 MiB  1’458.0 MiB  0.148  CRC64   data/es_caselaw_train.0.jsonl.xz
    1       1    586.4 KiB  3’537.8 KiB  0.166  CRC64   data/es_caselaw_validation.0.jsonl.xz
    1       1     29.0 MiB    244.0 MiB  0.119  CRC64   data/es_contracts_train.0.jsonl.xz
    1       1  3’826.2 KiB     31.2 MiB  0.120  CRC64   data/es_contracts_validation.0.jsonl.xz
    1       1    401.8 MiB  3’054.9 MiB  0.132  CRC64   data/es_legislation_train.0.jsonl.xz
    1       1  8’217.6 KiB     56.6 MiB  0.142  CRC64   data/es_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/es_other_validation.0.jsonl.xz
    1       1    476.9 MiB  2’017.9 MiB  0.236  CRC64   data/es_wikipedia_train.0.jsonl.xz
    1       1    476.9 MiB  2’025.0 MiB  0.235  CRC64   data/es_wikipedia_train.1.jsonl.xz
    1       1    308.8 MiB  1’305.6 MiB  0.237  CRC64   data/es_wikipedia_train.2.jsonl.xz
    1       1  1’339.7 KiB  5’265.5 KiB  0.254  CRC64   data/es_wikipedia_validation.0.jsonl.xz
    1       1    132.5 MiB    831.3 MiB  0.159  CRC64   data/et_caselaw_train.0.jsonl.xz
    1       1    387.2 KiB  2’310.9 KiB  0.168  CRC64   data/et_caselaw_validation.0.jsonl.xz
    1       1     22.9 MiB    179.6 MiB  0.128  CRC64   data/et_contracts_train.0.jsonl.xz
    1       1  3’164.3 KiB     26.8 MiB  0.115  CRC64   data/et_contracts_validation.0.jsonl.xz
    1       1    255.2 MiB  1’908.2 MiB  0.134  CRC64   data/et_legislation_train.0.jsonl.xz
    1       1  9’239.2 KiB     64.7 MiB  0.140  CRC64   data/et_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/et_other_validation.0.jsonl.xz
    1       1    100.5 MiB    408.8 MiB  0.246  CRC64   data/et_wikipedia_train.0.jsonl.xz
    1       1  1’352.2 KiB  4’921.0 KiB  0.275  CRC64   data/et_wikipedia_validation.0.jsonl.xz
    1       1    194.5 MiB  1’359.0 MiB  0.143  CRC64   data/fi_caselaw_train.0.jsonl.xz
    1       1    604.1 KiB  3’656.1 KiB  0.165  CRC64   data/fi_caselaw_validation.0.jsonl.xz
    1       1     26.0 MiB    219.8 MiB  0.118  CRC64   data/fi_contracts_train.0.jsonl.xz
    1       1  2’971.2 KiB     27.4 MiB  0.106  CRC64   data/fi_contracts_validation.0.jsonl.xz
    1       1    334.7 MiB  2’599.3 MiB  0.129  CRC64   data/fi_legislation_train.0.jsonl.xz
    1       1  7’476.3 KiB     53.9 MiB  0.136  CRC64   data/fi_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/fi_other_validation.0.jsonl.xz
    1       1    255.6 MiB  1’118.0 MiB  0.229  CRC64   data/fi_wikipedia_train.0.jsonl.xz
    1       1  2’464.2 KiB      9.9 MiB  0.242  CRC64   data/fi_wikipedia_validation.0.jsonl.xz
    1       1    476.9 MiB  3’128.1 MiB  0.152  CRC64   data/fr_caselaw_train.0.jsonl.xz
    1       1    476.9 MiB  3’104.4 MiB  0.154  CRC64   data/fr_caselaw_train.1.jsonl.xz
    1       1    350.2 MiB  2’194.9 MiB  0.160  CRC64   data/fr_caselaw_train.2.jsonl.xz
    1       1    603.0 KiB  3’778.7 KiB  0.160  CRC64   data/fr_caselaw_validation.0.jsonl.xz
    1       1     31.9 MiB    278.3 MiB  0.115  CRC64   data/fr_contracts_train.0.jsonl.xz
    1       1  3’034.4 KiB     26.6 MiB  0.111  CRC64   data/fr_contracts_validation.0.jsonl.xz
    1       1    477.0 MiB  3’721.8 MiB  0.128  CRC64   data/fr_legislation_train.0.jsonl.xz
    1       1     89.3 MiB    670.9 MiB  0.133  CRC64   data/fr_legislation_train.1.jsonl.xz
    1       1  3’185.5 KiB     22.6 MiB  0.138  CRC64   data/fr_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/fr_other_validation.0.jsonl.xz
    1       1    476.9 MiB  2’150.5 MiB  0.222  CRC64   data/fr_wikipedia_train.0.jsonl.xz
    1       1    476.9 MiB  2’151.4 MiB  0.222  CRC64   data/fr_wikipedia_train.1.jsonl.xz
    1       1    476.9 MiB  2’151.2 MiB  0.222  CRC64   data/fr_wikipedia_train.2.jsonl.xz
    1       1    384.8 MiB  1’736.1 MiB  0.222  CRC64   data/fr_wikipedia_train.3.jsonl.xz
    1       1    937.8 KiB  3’777.6 KiB  0.248  CRC64   data/fr_wikipedia_validation.0.jsonl.xz
    1       1    721.9 KiB  5’663.9 KiB  0.127  CRC64   data/ga_caselaw_validation.0.jsonl.xz
    1       1  1’246.1 KiB     15.6 MiB  0.078  CRC64   data/ga_contracts_validation.0.jsonl.xz
    1       1     41.2 MiB    419.0 MiB  0.098  CRC64   data/ga_legislation_train.0.jsonl.xz
    1       1     14.9 MiB    123.2 MiB  0.121  CRC64   data/ga_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/ga_other_validation.0.jsonl.xz
    1       1     11.0 MiB     52.9 MiB  0.207  CRC64   data/ga_wikipedia_train.0.jsonl.xz
    1       1    782.4 KiB  3’438.9 KiB  0.228  CRC64   data/ga_wikipedia_validation.0.jsonl.xz
    1       1     72.7 MiB    460.3 MiB  0.158  CRC64   data/hr_caselaw_train.0.jsonl.xz
    1       1    359.9 KiB  2’214.8 KiB  0.162  CRC64   data/hr_caselaw_validation.0.jsonl.xz
    1       1     21.2 MiB    158.3 MiB  0.134  CRC64   data/hr_contracts_train.0.jsonl.xz
    1       1  3’785.9 KiB     26.6 MiB  0.139  CRC64   data/hr_contracts_validation.0.jsonl.xz
    1       1    160.6 MiB  1’258.7 MiB  0.128  CRC64   data/hr_legislation_train.0.jsonl.xz
    1       1     11.2 MiB     86.1 MiB  0.130  CRC64   data/hr_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/hr_other_validation.0.jsonl.xz
    1       1    110.3 MiB    425.5 MiB  0.259  CRC64   data/hr_wikipedia_train.0.jsonl.xz
    1       1  1’743.8 KiB  6’170.1 KiB  0.283  CRC64   data/hr_wikipedia_validation.0.jsonl.xz
    1       1    150.6 MiB  1’320.5 MiB  0.114  CRC64   data/hu_caselaw_train.0.jsonl.xz
    1       1    423.8 KiB  3’496.6 KiB  0.121  CRC64   data/hu_caselaw_validation.0.jsonl.xz
    1       1     26.9 MiB    266.0 MiB  0.101  CRC64   data/hu_contracts_train.0.jsonl.xz
    1       1  3’532.6 KiB     36.1 MiB  0.096  CRC64   data/hu_contracts_validation.0.jsonl.xz
    1       1    337.6 MiB  3’129.4 MiB  0.108  CRC64   data/hu_legislation_train.0.jsonl.xz
    1       1  3’913.7 KiB     94.8 MiB  0.040  CRC64   data/hu_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/hu_other_validation.0.jsonl.xz
    1       1    364.2 MiB  1’835.0 MiB  0.198  CRC64   data/hu_wikipedia_train.0.jsonl.xz
    1       1  1’719.5 KiB  8’000.8 KiB  0.215  CRC64   data/hu_wikipedia_validation.0.jsonl.xz
    1       1    459.8 MiB  2’742.8 MiB  0.168  CRC64   data/it_caselaw_train.0.jsonl.xz
    1       1    577.8 KiB  3’194.2 KiB  0.181  CRC64   data/it_caselaw_validation.0.jsonl.xz
    1       1     31.2 MiB    240.4 MiB  0.130  CRC64   data/it_contracts_train.0.jsonl.xz
    1       1  3’068.9 KiB     24.0 MiB  0.125  CRC64   data/it_contracts_validation.0.jsonl.xz
    1       1    476.9 MiB  3’362.3 MiB  0.142  CRC64   data/it_legislation_train.0.jsonl.xz
    1       1     38.9 MiB    238.7 MiB  0.163  CRC64   data/it_legislation_train.1.jsonl.xz
    1       1  3’211.3 KiB     25.3 MiB  0.124  CRC64   data/it_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/it_other_validation.0.jsonl.xz
    1       1    476.9 MiB  1’864.5 MiB  0.256  CRC64   data/it_wikipedia_train.0.jsonl.xz
    1       1    476.9 MiB  1’864.8 MiB  0.256  CRC64   data/it_wikipedia_train.1.jsonl.xz
    1       1    184.6 MiB    726.2 MiB  0.254  CRC64   data/it_wikipedia_train.2.jsonl.xz
    1       1  1’334.0 KiB  4’843.5 KiB  0.275  CRC64   data/it_wikipedia_validation.0.jsonl.xz
    1       1    136.6 MiB    975.7 MiB  0.140  CRC64   data/lt_caselaw_train.0.jsonl.xz
    1       1    397.0 KiB  2’660.9 KiB  0.149  CRC64   data/lt_caselaw_validation.0.jsonl.xz
    1       1     24.9 MiB    211.8 MiB  0.118  CRC64   data/lt_contracts_train.0.jsonl.xz
    1       1  3’275.5 KiB     26.1 MiB  0.123  CRC64   data/lt_contracts_validation.0.jsonl.xz
    1       1    274.0 MiB  2’174.1 MiB  0.126  CRC64   data/lt_legislation_train.0.jsonl.xz
    1       1  9’780.7 KiB     73.4 MiB  0.130  CRC64   data/lt_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/lt_other_validation.0.jsonl.xz
    1       1     72.6 MiB    349.5 MiB  0.208  CRC64   data/lt_wikipedia_train.0.jsonl.xz
    1       1  1’251.2 KiB  5’369.5 KiB  0.233  CRC64   data/lt_wikipedia_validation.0.jsonl.xz
    1       1    141.0 MiB  1’106.7 MiB  0.127  CRC64   data/lv_caselaw_train.0.jsonl.xz
    1       1    410.3 KiB  3’004.0 KiB  0.137  CRC64   data/lv_caselaw_validation.0.jsonl.xz
    1       1     24.9 MiB    224.5 MiB  0.111  CRC64   data/lv_contracts_train.0.jsonl.xz
    1       1  3’629.0 KiB     33.6 MiB  0.106  CRC64   data/lv_contracts_validation.0.jsonl.xz
    1       1    271.5 MiB  2’377.4 MiB  0.114  CRC64   data/lv_legislation_train.0.jsonl.xz
    1       1     10.5 MiB     87.5 MiB  0.120  CRC64   data/lv_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/lv_other_validation.0.jsonl.xz
    1       1     47.5 MiB    254.7 MiB  0.186  CRC64   data/lv_wikipedia_train.0.jsonl.xz
    1       1    984.1 KiB  4’559.4 KiB  0.216  CRC64   data/lv_wikipedia_validation.0.jsonl.xz
    1       1    132.2 MiB    956.6 MiB  0.138  CRC64   data/mt_caselaw_train.0.jsonl.xz
    1       1    396.1 KiB  2’680.0 KiB  0.148  CRC64   data/mt_caselaw_validation.0.jsonl.xz
    1       1     25.6 MiB    201.0 MiB  0.127  CRC64   data/mt_contracts_train.0.jsonl.xz
    1       1  4’178.4 KiB     34.0 MiB  0.120  CRC64   data/mt_contracts_validation.0.jsonl.xz
    1       1    270.7 MiB  2’121.7 MiB  0.128  CRC64   data/mt_legislation_train.0.jsonl.xz
    1       1     11.4 MiB     84.2 MiB  0.135  CRC64   data/mt_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/mt_other_validation.0.jsonl.xz
    1       1  4’608.3 KiB     19.5 MiB  0.231  CRC64   data/mt_wikipedia_train.0.jsonl.xz
    1       1  1’405.0 KiB  5’754.4 KiB  0.244  CRC64   data/mt_wikipedia_validation.0.jsonl.xz
    1       1    223.1 MiB  1’338.9 MiB  0.167  CRC64   data/nl_caselaw_train.0.jsonl.xz
    1       1    566.0 KiB  3’152.2 KiB  0.180  CRC64   data/nl_caselaw_validation.0.jsonl.xz
    1       1     31.6 MiB    242.3 MiB  0.130  CRC64   data/nl_contracts_train.0.jsonl.xz
    1       1  2’663.9 KiB     22.4 MiB  0.116  CRC64   data/nl_contracts_validation.0.jsonl.xz
    1       1    476.9 MiB  3’311.9 MiB  0.144  CRC64   data/nl_legislation_train.0.jsonl.xz
    1       1     41.1 MiB    268.7 MiB  0.153  CRC64   data/nl_legislation_train.1.jsonl.xz
    1       1  3’678.8 KiB     72.9 MiB  0.049  CRC64   data/nl_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/nl_other_validation.0.jsonl.xz
    1       1    476.9 MiB  1’856.9 MiB  0.257  CRC64   data/nl_wikipedia_train.0.jsonl.xz
    1       1     59.9 MiB    236.4 MiB  0.253  CRC64   data/nl_wikipedia_train.1.jsonl.xz
    1       1    979.4 KiB  3’414.8 KiB  0.287  CRC64   data/nl_wikipedia_validation.0.jsonl.xz
    1       1    147.9 MiB  1’034.1 MiB  0.143  CRC64   data/pl_caselaw_train.0.jsonl.xz
    1       1    416.2 KiB  2’737.2 KiB  0.152  CRC64   data/pl_caselaw_validation.0.jsonl.xz
    1       1     24.8 MiB    208.9 MiB  0.119  CRC64   data/pl_contracts_train.0.jsonl.xz
    1       1  4’241.9 KiB     34.6 MiB  0.120  CRC64   data/pl_contracts_validation.0.jsonl.xz
    1       1    325.0 MiB  2’646.2 MiB  0.123  CRC64   data/pl_legislation_train.0.jsonl.xz
    1       1  3’593.0 KiB     29.0 MiB  0.121  CRC64   data/pl_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/pl_other_validation.0.jsonl.xz
    1       1    476.9 MiB  2’144.7 MiB  0.222  CRC64   data/pl_wikipedia_train.0.jsonl.xz
    1       1    189.5 MiB    864.0 MiB  0.219  CRC64   data/pl_wikipedia_train.1.jsonl.xz
    1       1  1’233.2 KiB  4’965.9 KiB  0.248  CRC64   data/pl_wikipedia_validation.0.jsonl.xz
    1       1    476.9 MiB  3’494.2 MiB  0.136  CRC64   data/pt_caselaw_train.0.jsonl.xz
    1       1    476.9 MiB  3’392.1 MiB  0.141  CRC64   data/pt_caselaw_train.10.jsonl.xz
    1       1    476.9 MiB  3’505.3 MiB  0.136  CRC64   data/pt_caselaw_train.11.jsonl.xz
    1       1    476.9 MiB  3’524.1 MiB  0.135  CRC64   data/pt_caselaw_train.12.jsonl.xz
    1       1    476.9 MiB  3’458.4 MiB  0.138  CRC64   data/pt_caselaw_train.13.jsonl.xz
    1       1    476.9 MiB  3’602.9 MiB  0.132  CRC64   data/pt_caselaw_train.14.jsonl.xz
    1       1    476.9 MiB  4’923.4 MiB  0.097  CRC64   data/pt_caselaw_train.15.jsonl.xz
    1       1    476.9 MiB  6’648.8 MiB  0.072  CRC64   data/pt_caselaw_train.16.jsonl.xz
    1       1    476.9 MiB  7’461.0 MiB  0.064  CRC64   data/pt_caselaw_train.17.jsonl.xz
    1       1    476.9 MiB  6’866.4 MiB  0.069  CRC64   data/pt_caselaw_train.18.jsonl.xz
    1       1    476.9 MiB  3’455.7 MiB  0.138  CRC64   data/pt_caselaw_train.19.jsonl.xz
    1       1    476.9 MiB  3’513.7 MiB  0.136  CRC64   data/pt_caselaw_train.1.jsonl.xz
    1       1    476.9 MiB  3’477.3 MiB  0.137  CRC64   data/pt_caselaw_train.20.jsonl.xz
    1       1    476.9 MiB  3’492.8 MiB  0.137  CRC64   data/pt_caselaw_train.21.jsonl.xz
    1       1    476.9 MiB  3’528.6 MiB  0.135  CRC64   data/pt_caselaw_train.22.jsonl.xz
    1       1     94.1 MiB    694.3 MiB  0.135  CRC64   data/pt_caselaw_train.23.jsonl.xz
    1       1    476.9 MiB  3’436.5 MiB  0.139  CRC64   data/pt_caselaw_train.2.jsonl.xz
    1       1    476.9 MiB  3’527.9 MiB  0.135  CRC64   data/pt_caselaw_train.3.jsonl.xz
    1       1    476.9 MiB  3’492.2 MiB  0.137  CRC64   data/pt_caselaw_train.4.jsonl.xz
    1       1    476.9 MiB  3’554.8 MiB  0.134  CRC64   data/pt_caselaw_train.5.jsonl.xz
    1       1    476.9 MiB  3’494.7 MiB  0.136  CRC64   data/pt_caselaw_train.6.jsonl.xz
    1       1    476.9 MiB  3’439.1 MiB  0.139  CRC64   data/pt_caselaw_train.7.jsonl.xz
    1       1    476.9 MiB  3’625.6 MiB  0.132  CRC64   data/pt_caselaw_train.8.jsonl.xz
    1       1    476.9 MiB  3’726.4 MiB  0.128  CRC64   data/pt_caselaw_train.9.jsonl.xz
    1       1    798.9 KiB  4’820.6 KiB  0.166  CRC64   data/pt_caselaw_validation.0.jsonl.xz
    1       1     28.4 MiB    243.2 MiB  0.117  CRC64   data/pt_contracts_train.0.jsonl.xz
    1       1  3’899.7 KiB     32.6 MiB  0.117  CRC64   data/pt_contracts_validation.0.jsonl.xz
    1       1    406.2 MiB  3’217.5 MiB  0.126  CRC64   data/pt_legislation_train.0.jsonl.xz
    1       1  8’350.4 KiB     58.4 MiB  0.140  CRC64   data/pt_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/pt_other_validation.0.jsonl.xz
    1       1    476.9 MiB  2’050.4 MiB  0.233  CRC64   data/pt_wikipedia_train.0.jsonl.xz
    1       1    140.6 MiB    617.4 MiB  0.228  CRC64   data/pt_wikipedia_train.1.jsonl.xz
    1       1  1’480.0 KiB  6’344.8 KiB  0.233  CRC64   data/pt_wikipedia_validation.0.jsonl.xz
    1       1    124.9 MiB    956.9 MiB  0.131  CRC64   data/ro_caselaw_train.0.jsonl.xz
    1       1    400.4 KiB  2’785.0 KiB  0.144  CRC64   data/ro_caselaw_validation.0.jsonl.xz
    1       1     24.6 MiB    210.5 MiB  0.117  CRC64   data/ro_contracts_train.0.jsonl.xz
    1       1  3’886.3 KiB     34.3 MiB  0.111  CRC64   data/ro_contracts_validation.0.jsonl.xz
    1       1    476.9 MiB  4’496.4 MiB  0.106  CRC64   data/ro_legislation_train.0.jsonl.xz
    1       1     97.6 MiB  1’053.6 MiB  0.093  CRC64   data/ro_legislation_train.1.jsonl.xz
    1       1  3’691.3 KiB     33.4 MiB  0.108  CRC64   data/ro_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/ro_other_validation.0.jsonl.xz
    1       1    179.7 MiB    833.0 MiB  0.216  CRC64   data/ro_wikipedia_train.0.jsonl.xz
    1       1  2’089.4 KiB  9’053.5 KiB  0.231  CRC64   data/ro_wikipedia_validation.0.jsonl.xz
    1       1    143.6 MiB  1’094.2 MiB  0.131  CRC64   data/sk_caselaw_train.0.jsonl.xz
    1       1    415.8 KiB  3’012.4 KiB  0.138  CRC64   data/sk_caselaw_validation.0.jsonl.xz
    1       1     25.9 MiB    226.7 MiB  0.114  CRC64   data/sk_contracts_train.0.jsonl.xz
    1       1  3’933.6 KiB     35.2 MiB  0.109  CRC64   data/sk_contracts_validation.0.jsonl.xz
    1       1    322.4 MiB  2’745.5 MiB  0.117  CRC64   data/sk_legislation_train.0.jsonl.xz
    1       1  3’735.8 KiB     31.7 MiB  0.115  CRC64   data/sk_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/sk_other_validation.0.jsonl.xz
    1       1     91.2 MiB    435.3 MiB  0.210  CRC64   data/sk_wikipedia_train.0.jsonl.xz
    1       1  1’724.4 KiB  7’568.3 KiB  0.228  CRC64   data/sk_wikipedia_validation.0.jsonl.xz
    1       1    131.9 MiB    815.8 MiB  0.162  CRC64   data/sl_caselaw_train.0.jsonl.xz
    1       1    392.8 KiB  2’328.2 KiB  0.169  CRC64   data/sl_caselaw_validation.0.jsonl.xz
    1       1     22.9 MiB    172.4 MiB  0.133  CRC64   data/sl_contracts_train.0.jsonl.xz
    1       1  3’493.7 KiB     27.2 MiB  0.125  CRC64   data/sl_contracts_validation.0.jsonl.xz
    1       1    388.1 MiB  2’732.3 MiB  0.142  CRC64   data/sl_legislation_train.0.jsonl.xz
    1       1  3’429.8 KiB     24.3 MiB  0.138  CRC64   data/sl_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/sl_other_validation.0.jsonl.xz
    1       1    104.6 MiB    425.6 MiB  0.246  CRC64   data/sl_wikipedia_train.0.jsonl.xz
    1       1  1’392.8 KiB  5’004.9 KiB  0.278  CRC64   data/sl_wikipedia_validation.0.jsonl.xz
    1       1    189.5 MiB  1’325.4 MiB  0.143  CRC64   data/sv_caselaw_train.0.jsonl.xz
    1       1    581.2 KiB  3’566.7 KiB  0.163  CRC64   data/sv_caselaw_validation.0.jsonl.xz
    1       1     25.3 MiB    211.7 MiB  0.119  CRC64   data/sv_contracts_train.0.jsonl.xz
    1       1  2’890.6 KiB     26.0 MiB  0.108  CRC64   data/sv_contracts_validation.0.jsonl.xz
    1       1    324.5 MiB  2’570.4 MiB  0.126  CRC64   data/sv_legislation_train.0.jsonl.xz
    1       1  6’984.8 KiB     50.1 MiB  0.136  CRC64   data/sv_legislation_validation.0.jsonl.xz
    1       0         32 B          0 B    ---  CRC64   data/sv_other_validation.0.jsonl.xz
    1       1    333.4 MiB  1’668.1 MiB  0.200  CRC64   data/sv_wikipedia_train.0.jsonl.xz
    1       1  1’088.6 KiB  4’372.9 KiB  0.249  CRC64   data/sv_wikipedia_validation.0.jsonl.xz
-------------------------------------------------------------------------------
  374     351     90.1 GiB    579.9 GiB  0.155  CRC64   374 files

Dataset Creation

This dataset has been created by combining the following datasets: Native Multi Legal Pile, Eurlex Resources, MC4 Legal, Pile of Law, EU Wikipedias. It has been filtered to remove short documents (less than 64 whitespace-separated tokens) and documents with more than 30% punctuation or numbers (see prepare_legal_data.py for more details).

Curation Rationale

[More Information Needed]

Source Data

Initial Data Collection and Normalization

[More Information Needed]

Who are the source language producers?

[More Information Needed]

Annotations

Annotation process

[More Information Needed]

Who are the annotators?

[More Information Needed]

Personal and Sensitive Information

[More Information Needed]

Considerations for Using the Data

Social Impact of Dataset

[More Information Needed]

Discussion of Biases

[More Information Needed]

Other Known Limitations

[More Information Needed]

Additional Information

Dataset Curators

[More Information Needed]

Licensing Information

[More Information Needed]

Citation Information

TODO add citation

Contributions

Thanks to @JoelNiklaus for adding this dataset.