《文学-3B-4096》——一种用于生成长篇小说故事的大型精细调优模型

《文学-3B-4096》是一个将2GB各种类型的轻小说、情色小说和注释文学用于精细调优的GPT-NeoX 3B模型，用于生成类似小说的虚构文本。

模型描述

用于精细调优的模型是 RedPajama-INCITE-Base-3B-v1 ，是一个30亿参数的自回归语言模型。

训练数据和注释提示

在精细调优中使用的数据来自于各种来源，例如 Gutenberg Project 。这个带注释的虚构数据集在生成特定风格的文本时会添加标签。以下是一个示例提示，展示如何使用这些注释。

[ Title: The Dunwich Horror; Author: H. P. Lovecraft; Genre: Horror; Tags: 3rdperson, scary; Style: Dark ]
***
When a traveler in north central Massachusetts takes the wrong fork...

注释可以混合搭配，以帮助生成特定的风格。

下游用途

这个模型可用于娱乐目的，作为创作虚构作品的辅助工具。

示例代码

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained('hakurei/Literature-3B-4096')
tokenizer = AutoTokenizer.from_pretrained('hakurei/Literature-3B-4096')

prompt = '''[ Title: The Dunwich Horror; Author: H. P. Lovecraft; Genre: Horror ]
***
When a traveler'''

input_ids = tokenizer.encode(prompt, return_tensors='pt')
output = model.generate(input_ids, do_sample=True, temperature=1.0, top_p=0.9, repetition_penalty=1.2, max_length=len(input_ids[0])+100, pad_token_id=tokenizer.eos_token_id)

generated_text = tokenizer.decode(output[0])
print(generated_text)

使用此代码的示例输出结果将类似于：

[ Title: The Dunwich Horror; Author: H. P. Lovecraft; Genre: Horror ]
***
When a traveler comes to an unknown region, his thoughts turn inevitably towards the old gods and legends which cluster around its appearance. It is not that he believes in them or suspects their reality—but merely because they are present somewhere else in creation just as truly as himself, and so belong of necessity in any landscape whose features cannot be altogether strange to him. Moreover, man has been prone from ancient times to brood over those things most connected with the places where he dwells. Thus the Olympian deities who ruled Hyper

团队成员和致谢

• Anthony Mercurio ：训练
• 11b ：长上下文
• Imperishable_NEET：数据集