Lit-6B - 用于虚构故事的大型精调模型

Lit-6B是一个在各种轻小说、色情作品和注释文学的2GB数据集上进行精调的GPT-J 6B模型，旨在生成类似小说的虚构文本。

模型描述

用于精调的模型是 GPT-J ，它是一个6亿参数的自回归语言模型，训练数据来自 The Pile 。

训练数据和注释提示

用于精调的数据来自各种来源，例如 Gutenberg Project 。注释的虚构数据集在生成特定风格时会加上前置标签。下面是一个示例提示，展示了如何使用这些注释。

[ Title: The Dunwich Horror; Author: H. P. Lovecraft; Genre: Horror; Tags: 3rdperson, scary; Style: Dark ]
***
When a traveler in north central Massachusetts takes the wrong fork...

注释可以随意组合，以帮助生成特定风格的文本。

下游用途

这个模型可以用于娱乐目的，也可以作为虚构作家的创作助手。

示例代码

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained('hakurei/lit-6B')
tokenizer = AutoTokenizer.from_pretrained('hakurei/lit-6B')

prompt = '''[ Title: The Dunwich Horror; Author: H. P. Lovecraft; Genre: Horror ]
***
When a traveler'''

input_ids = tokenizer.encode(prompt, return_tensors='pt')
output = model.generate(input_ids, do_sample=True, temperature=1.0, top_p=0.9, repetition_penalty=1.2, max_length=len(input_ids[0])+100, pad_token_id=tokenizer.eos_token_id)

generated_text = tokenizer.decode(output[0])
print(generated_text)

运行该代码的示例输出结果大致如下：

[ Title: The Dunwich Horror; Author: H. P. Lovecraft; Genre: Horror ]
***
When a traveler comes to an unknown region, his thoughts turn inevitably towards the old gods and legends which cluster around its appearance. It is not that he believes in them or suspects their reality—but merely because they are present somewhere else in creation just as truly as himself, and so belong of necessity in any landscape whose features cannot be altogether strange to him. Moreover, man has been prone from ancient times to brood over those things most connected with the places where he dwells. Thus the Olympian deities who ruled Hyper

团队成员和致谢

感谢 TPU Research Cloud 慷慨提供的计算资源，本项目才得以实现。

• Anthony Mercurio
• Imperishable_NEET