多语言BERT在SQuADv1.1上微调

WandB run link

GPU：Tesla P100-PCIE-16GB

训练参数

max_seq_length              = 512
doc_stride                  = 256
max_answer_length           = 64
bacth_size                  = 16
gradient_accumulation_steps = 2
learning_rate               = 5e-5
weight_decay                = 3e-7
num_train_epochs            = 3
warmup_ratio                = 0.1
fp16                        = True
fp16_opt_level              = "O1"
seed                        = 0

结果

零-shot表现

在ARCD上

在XQuAD上