Lit-6B - 一个用于虚构故事的大型微调模型

Lit-6B是基于2GB各种轻小说、色情小说和注释文学进行微调的GPT-J 6B模型，用于生成类似小说的虚构文本。

模型描述

用于微调的模型是 GPT-J ，这是一个训练在 The Pile 上的60亿参数的自回归语言模型。

训练数据和注释提示

在微调中使用的数据来自各种来源，如 Gutenberg Project 。注释小说数据集在生成特定风格时有帮助的预先添加了标签。下面是一个示例提示，显示如何使用这些注释。

[ Title: The Dunwich Horror; Author: H. P. Lovecraft; Genre: Horror; Tags: 3rdperson, scary; Style: Dark ]
***
When a traveler in north central Massachusetts takes the wrong fork...

注释可以组合使用，以生成特定风格的文本。

下游应用

此模型可用于娱乐目的，并作为虚构作家的创作助手。

示例代码

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained('hakurei/lit-6B')
tokenizer = AutoTokenizer.from_pretrained('hakurei/lit-6B')

prompt = '''[ Title: The Dunwich Horror; Author: H. P. Lovecraft; Genre: Horror ]
***
When a traveler'''

input_ids = tokenizer.encode(prompt, return_tensors='pt')
output = model.generate(input_ids, do_sample=True, temperature=1.0, top_p=0.9, repetition_penalty=1.2, max_length=len(input_ids[0])+100, pad_token_id=tokenizer.eos_token_id)

generated_text = tokenizer.decode(output[0])
print(generated_text)

此代码的示例输出将类似于：

[ Title: The Dunwich Horror; Author: H. P. Lovecraft; Genre: Horror ]
***
When a traveler comes to an unknown region, his thoughts turn inevitably towards the old gods and legends which cluster around its appearance. It is not that he believes in them or suspects their reality—but merely because they are present somewhere else in creation just as truly as himself, and so belong of necessity in any landscape whose features cannot be altogether strange to him. Moreover, man has been prone from ancient times to brood over those things most connected with the places where he dwells. Thus the Olympian deities who ruled Hyper

团队成员和致谢

如果没有 TPU Research Cloud 提供的计算资源的大力支持，本项目将无法实现。

• Anthony Mercurio
• Imperishable_NEET