reStructured Pre-training (RST)

official repository , paper , easter eggs

RST is a new paradigm for language pre-training, which

• unifies 26 different types of signal from 10 data sources (Totten Tomatoes, Dailymail, Wikipedia, Wikidata, Wikihow, Wordnet, arXiv etc ) in the world structurally, being pre-trained with a monolithcal model,
• surpasses strong competitors (e.g., T0) on 52/55 popular datasets from a variety of NLP tasks (classification, IE, retrieval, generation etc)
• achieves superior performance in National College Entrance Examination (Gaokao-English, 高考-英语) achieves 40 points higher than the average scores made by students and 15 points higher than GPT3 with 1/16 parameters. In particular, Qin gets a high score of 138.5 (the full mark is 150) in the 2018 English exam

In such a pre-training paradigm,

• Data-centric Pre-training: the role of data will be re-emphasized, and model pre-training and fine-tuning of downstream tasks are viewed as a process of data storing and accessing
• Pre-training over JSON instead of TEXT: a good storage mechanism should not only have the ability to cache a large amount of data but also consider the ease of access.

Model Description

We release all models introduced in our paper , covering 13 different application scenarios. Each model contains 11 billion parameters.

Model	Description	Recommended Application
rst-all-11b	Trained with all the signals below except signals that are used to train Gaokao models	All applications below （specialized models are recommended first if high performance is preferred）
rst-fact-retrieval-11b	Trained with the following signals: WordNet meaning, WordNet part-of-speech, WordNet synonym, WordNet antonym, wikiHow category hierarchy, Wikidata relation, Wikidata entity typing, Paperswithcode entity typing	Knowledge intensive tasks, information extraction tasks,factual checker
rst-summarization-11b	Trained with the following signals: DailyMail summary, Paperswithcode summary, arXiv summary, wikiHow summary	Summarization or other general generation tasks, meta-evaluation (e.g., BARTScore)
rst-temporal-reasoning-11b	Trained with the following signals: DailyMail temporal information, wikiHow procedure	Temporal reasoning, relation extraction, event-based extraction
rst-information-extraction-11b	Trained with the following signals: Paperswithcode entity, Paperswithcode entity typing, Wikidata entity typing, Wikidata relation, Wikipedia entity	Named entity recognition, relation extraction and other general IE tasks in the news, scientific or other domains
rst-intent-detection-11b	Trained with the following signals: wikiHow goal-step relation	Intent prediction, event prediction
rst-topic-classification-11b	Trained with the following signals: DailyMail category, arXiv category, wikiHow text category, Wikipedia section title	general text classification
rst-word-sense-disambiguation-11b	Trained with the following signals: WordNet meaning, WordNet part-of-speech, WordNet synonym, WordNet antonym	Word sense disambiguation, part-of-speech tagging, general IE tasks, common sense reasoning
rst-natural-language-inference-11b	Trained with the following signals: ConTRoL dataset, DREAM dataset, LogiQA dataset, RACE & RACE-C dataset, ReClor dataset, DailyMail temporal information	Natural language inference, multiple-choice question answering, reasoning
rst-sentiment-classification-11b	Trained with the following signals: Rotten Tomatoes sentiment, Wikipedia sentiment	Sentiment classification, emotion classification
rst-gaokao-rc-11b	Trained with multiple-choice QA datasets that are used to train the T0pp model	General multiple-choice question answering
rst-gaokao-cloze-11b	Trained with manually crafted cloze datasets	General cloze filling
rst-gaokao-writing-11b	Trained with example essays from past Gaokao-English exams and grammar error correction signals	Essay writing, story generation, grammar error correction and other text generation tasks

Have a try?

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("XLab/rst-all-11b")
model = AutoModelForSeq2SeqLM.from_pretrained("XLab/rst-all-11b")

inputs = tokenizer.encode("TEXT: this is the best cast iron skillet you will ever buy. QUERY: Is this review \"positive\" or \"negative\"", return_tensors="pt")
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True))

Data for reStructure Pre-training

This dataset is a precious treasure, containing a variety of naturally occurring signals. Any downstream task you can think of (e.g., the college entrance exam mentioned in the RST paper) can benefit from being pre-trained on some of our provided signals. We spent several months collecting the following 29 signal types, accounting for a total of 46,926,447 data samples. We hope this dataset will be a valuable asset for everyone in natural language processing research.

We provide collected signals through DataLab . For efficiency, we only provide 50,000 samples at most for each signal type. If you want all the samples we collected, please fill this form . More specifically, we collected the following signals.

We will be happy :smiley: to know if the resource is helpful for your work, and please cite our work :blush:

Mine	Signal	#Sample	Use in DataLab	Some Applications
Rotten Tomatoes	(review, rating)	5,311,109	load_dataset("rst", "rotten_tomatoes_sentiment")	Sentiment classification
Daily Mail	(text, category)	899,904	load_dataset("rst", "daily_mail_category")	Topic classification
Daily Mail	(title, text, summary)	1,026,616	load_dataset("rst", "daily_mail_summary")	Summarization; Sentence expansion
Daily Mail	(text, events)	1,006,412	load_dataset("rst", "daily_mail_temporal")	Temporal reasoning
Wikidata	(entity, entity_type, text)	2,214,274	load_dataset("rst", "wikidata_entity")	Entity typing
Wikidata	(subject, object, relation, text)	1,526,674	load_dataset("rst", "wikidata_relation")	Relation extraction; Fact retrieval
wikiHow	(text, category)	112,109	load_dataset("rst", "wikihow_text_category")	Topic classification
wikiHow	(low_category, high_category)	4,868	load_dataset("rst", "wikihow_category_hierarchy")	Relation extraction; Commonsense reasoning
wikiHow	(goal, steps)	47,956	load_dataset("rst", "wikihow_goal_step")	Intent detection
wikiHow	(text, summary)	703,278	load_dataset("rst", "wikihow_summary")	Summarization; Sentence expansion
wikiHow	(goal, first_step, second_step)	47,787	load_dataset("rst", "wikihow_procedure")	Temporal reasoning
wikiHow	(question, description, answer, related_questions)	47,705	load_dataset("rst", "wikihow_question")	Question generation
Wikipedia	(text, entities)	22,231,011	load_dataset("rst", "wikipedia_entities")	Entity recognition
Wikipedia	(texts, titles)	3,296,225	load_dataset("rst", "wikipedia_sections")	Summarization
WordNet	(word, sentence, pos)	27,123	load_dataset("rst", "wordnet_pos")	Part-of-speech tagging
WordNet	(word, sentence, meaning, possible_meanings)	27,123	load_dataset("rst", "wordnet_meaning")	Word sense disambiguation
WordNet	(word, sentence, synonyms)	17,804	load_dataset("rst", "wordnet_synonym")	Paraphrasing
WordNet	(word, sentence, antonyms)	6,408	load_dataset("rst", "wordnet_antonym")	Negation
ConTRoL	(premise, hypothesis, label)	8,323	load_dataset("rst", "qa_control")	Natural language inference
DREAM	(context, question, options, answer)	9,164	load_dataset("rst", "qa_dream")	Reading comprehension
LogiQA	(context, question, options, answer)	7,974	load_dataset("rst", "qa_logiqa")	Reading comprehension
ReClor	(context, question, options, answer)	5,138	load_dataset("rst", "qa_reclor")	Reading comprehension
RACE	(context, question, options, answer)	44,880	load_dataset("rst", "qa_race")	Reading comprehension
RACE-C	(context, question, options, answer)	5,093	load_dataset("rst", "qa_race_c")	Reading comprehension
TriviaQA	(context, question, answer)	46,636	load_dataset("rst", "qa_triviaqa")	Reading comprehension
Arxiv	(text, category)	1,696,348	load_dataset("rst", "arxiv_category")	Topic classification
Arxiv	(text, summary)	1,696,348	load_dataset("rst", "arxiv_summary")	Summarization; Sentence expansion
Paperswithcode	(text, entities, datasets, methods, tasks, metrics)	4,731,233	load_dataset("rst", "paperswithcode_entity")	Entity recognition
Paperswithcode	(text, summary)	120,924	load_dataset("rst", "paperswithcode_summary")	Summarization; Sentence expansion

Bibtext for Citation Info

@article{yuan2022restructured,
  title={reStructured Pre-training},
  author={Yuan, Weizhe and Liu, Pengfei},
  journal={arXiv preprint arXiv:2206.11147},
  year={2022}
}