模型:
GeorgiaTechResearchInstitute/galpaca-30b
任务:
文本生成许可:
cc-by-nc-4.0GALACTICA 30B fine-tuned on the Alpaca dataset.
The model card from the original Galactica repo can be found here , and the original paper here .
The dataset card for Alpaca can be found here , and the project homepage here . The Alpaca dataset was collected with a modified version of the Self-Instruct Framework , and was built using OpenAI's text-davinci-003 model. As such it is subject to OpenAI's terms of service.
GALACTICA模型是在大规模科学语料库上训练的,旨在进行科学任务。Alpaca数据集是一组52000个指令-回应对,旨在增强预训练语言模型的指令跟踪能力。
GALACTICA模型卡指定GALACTICA模型的主要使用者是研究科学域中应用语言模型的研究人员,并且由于模型可能产生不准确的信息,警告不要在没有保护措施的情况下在生产中使用GALACTICA模型。原始的GALACTICA模型可在非商业性CC BY-NC 4.0许可下使用,而GALPACA模型还受到 OpenAI Terms of Service 的约束。
Example OutputsFull Prompt:
Below is an instruction that describes a task. As a scientist, write a response that appropriately completes the request, citing specific papers whenever possible. ### Instruction: Write out Maxwell's equations and explain each one. ### Markdown and LaTeX Response:
Response:
麦克斯韦方程是一组描述电磁场行为的四个方程。这些方程为:\(\nabla \times \textbf{E} = -\frac{\partial \textbf{B}}{\partial t}\),\(\nabla \times \textbf{B} = \mu_0\epsilon_0\frac{\partial \textbf{E}}{\partial t} + \mu_0\textbf{J}\),\(\nabla \cdot \textbf{E} = \frac{\rho}{\epsilon_0}\),\(\nabla \cdot \textbf{B} = 0\)。前两个方程描述电场和磁场之间的关系。第三和第四个方程分别是电场和磁场的高斯定律。这些方程描述了空间中存在电荷和电流时电磁场的行为。每个方程都有特定的目的,它们共同构成了电磁场行为的完整描述。
Full Prompt:
Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: Write a python function implements the Sherman-Morrison matrix inversion lemma using numpy. ### Response: ```python
Response:
def sm_inv_lemma(A, u, v): """ Function implementing the Sherman-Morrison matrix inversion lemma using numpy. Parameters ---------- A : numpy array The original matrix. u : numpy array The first vector in the Sherman-Morrison lemma. v : numpy array The second vector in the Sherman-Morrison lemma. Returns ------- The inverse matrix of A - uv^T. """ A_inv = np.linalg.inv(A) alpha = 1 / (1 + v.T @ A_inv @ u) return A_inv - (alpha * A_inv @ u @ v.T @ A_inv)
GALPACA模型是通过在Alpaca数据集上对预训练的GALACTICA模型进行微调而训练的。GALACTICA模型在包括论文、教科书、科学网站、百科全书等在内的1060亿个开放获取的科学文本和数据上进行了训练。通过在Alpaca数据集的52000个指令-回应对上对基本GALACTICA模型进行微调,用户可以以指令-回应的方式查询GALPACA模型。
使用transformers库可使用GALPACA权重。
点击展开# pip install accelerate from transformers import AutoTokenizer, OPTForCausalLM tokenizer = AutoTokenizer.from_pretrained("GeorgiaTechResearchInstitute/galpaca-30b") model = OPTForCausalLM.from_pretrained("GeorgiaTechResearchInstitute/galpaca-30b", device_map="auto", torch_dtype=torch.float16) # see the original Alpaca repo for more information about the prompt templates no_input_prompt_template = ("Below is an instruction that describes a task. " "Write a response that appropriately completes the request.\n\n" "### Instruction:\n{instruction}\n\n### Response:") prompt = "Write out Maxwell's equations and explain the meaning of each one." formatted_prompt = no_input_prompt_template.format_map({'instruction': prompt}) tokenized_prompt = tokenizer(formatted_prompt, return_tensors="pt").input_ids.to(model.device) out_tokens = model.generate(tokenized_prompt) print(tokenizer.batch_decode(out_tokens, skip_special_tokens=False, clean_up_tokenization_spaces=False))
使用16块A100 80GB GPU,16位混合精度,在有效批量大小为1024的情况下,最大上下文窗口为384个标记,GALPACA 30B模型进行了约6小时的微调。此模型使用了DeepSpeed ZeRO Stage 3优化进行训练。
定性评估表明,Galpaca在与技术知识和编程相关的任务上经常优于基于LLaMA的Alpaca模型,但在生成散文等自然语言任务方面表现不佳。关于GALACTICA系列模型的性能和限制的更多信息,请参阅原始GALACTICA模型卡。
@inproceedings{GALACTICA, title={GALACTICA: A Large Language Model for Science}, author={Ross Taylor and Marcin Kardas and Guillem Cucurull and Thomas Scialom and Anthony Hartshorn and Elvis Saravia and Andrew Poulton and Viktor Kerkez and Robert Stojnic}, year={2022} }
@misc{alpaca, author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto }, title = {Stanford Alpaca: An Instruction-following LLaMA model}, year = {2023}, publisher = {GitHub}, journal = {GitHub repository}, howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}}, }