Literature-7B-16384 -一个用于长篇小说讲述的大型精调模型

Literature-7B-16384 是一个GPT-NeoX 7B模型，该模型使用2GB的各种轻小说、情色小说和带注释的文学作品进行了精细调整，以生成类似小说的虚构文本。

模型描述

用于精细调整的模型是 RedPajama-7B-16384 ，这是一个70亿参数的自回归语言模型，使用16384个标记的上下文窗口进行调整。

训练数据和注释提示

在精细调整中使用的数据是从各种来源收集而来的，例如 Gutenberg Project 。带注释的小说数据集在生成特定风格时会添加标签。下面是一个示例提示，展示了如何使用这些注释。

[ Title: The Dunwich Horror; Author: H. P. Lovecraft; Genre: Horror; Tags: 3rdperson, scary; Style: Dark ]
***
When a traveler in north central Massachusetts takes the wrong fork...

注释可以混合使用，以帮助生成特定风格的文本。

下游应用

这个模型可以用于娱乐目的，也可以作为小说作家的创作助手。

示例代码

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained('hakurei/Literature-7B-16384')
tokenizer = AutoTokenizer.from_pretrained('hakurei/Literature-7B-16384')

prompt = '''[ Title: The Dunwich Horror; Author: H. P. Lovecraft; Genre: Horror ]
***
When a traveler'''

input_ids = tokenizer.encode(prompt, return_tensors='pt')
output = model.generate(input_ids, do_sample=True, temperature=1.0, top_p=0.9, repetition_penalty=1.2, max_length=len(input_ids[0])+100, pad_token_id=tokenizer.eos_token_id)

generated_text = tokenizer.decode(output[0])
print(generated_text)

使用此代码的示例输出结果将类似于：

[ Title: The Dunwich Horror; Author: H. P. Lovecraft; Genre: Horror ]
***
When a traveler comes to an unknown region, his thoughts turn inevitably towards the old gods and legends which cluster around its appearance. It is not that he believes in them or suspects their reality—but merely because they are present somewhere else in creation just as truly as himself, and so belong of necessity in any landscape whose features cannot be altogether strange to him. Moreover, man has been prone from ancient times to brood over those things most connected with the places where he dwells. Thus the Olympian deities who ruled Hyper

团队成员和致谢

• Anthony Mercurio ：训练
• 11b ：长上下文
• Imperishable_NEET：数据集