GALPACA 30B (large)

GALACTICA 30B fine-tuned on the Alpaca dataset.

The model card from the original Galactica repo can be found here , and the original paper here .

The dataset card for Alpaca can be found here , and the project homepage here . The Alpaca dataset was collected with a modified version of the Self-Instruct Framework , and was built using OpenAI's text-davinci-003 model. As such it is subject to OpenAI's terms of service.

Model Details

GALACTICA模型是在大规模科学语料库上训练的，旨在进行科学任务。Alpaca数据集是一组52000个指令-回应对，旨在增强预训练语言模型的指令跟踪能力。

Model Use

GALACTICA模型卡指定GALACTICA模型的主要使用者是研究科学域中应用语言模型的研究人员，并且由于模型可能产生不准确的信息，警告不要在没有保护措施的情况下在生产中使用GALACTICA模型。原始的GALACTICA模型可在非商业性CC BY-NC 4.0许可下使用，而GALPACA模型还受到 OpenAI Terms of Service 的约束。

Full Prompt:

Below is an instruction that describes a task. As a scientist, write a response that appropriately completes the request, citing specific papers whenever possible.

### Instruction:
Write out Maxwell's equations and explain each one.

### Markdown and LaTeX Response:

Response:

麦克斯韦方程是一组描述电磁场行为的四个方程。这些方程为：\(\nabla \times \textbf{E} = -\frac{\partial \textbf{B}}{\partial t}\)，\(\nabla \times \textbf{B} = \mu_0\epsilon_0\frac{\partial \textbf{E}}{\partial t} + \mu_0\textbf{J}\)，\(\nabla \cdot \textbf{E} = \frac{\rho}{\epsilon_0}\)，\(\nabla \cdot \textbf{B} = 0\)。前两个方程描述电场和磁场之间的关系。第三和第四个方程分别是电场和磁场的高斯定律。这些方程描述了空间中存在电荷和电流时电磁场的行为。每个方程都有特定的目的，它们共同构成了电磁场行为的完整描述。

Full Prompt:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Write a python function implements the Sherman-Morrison matrix inversion lemma using numpy.

### Response:
```python

Response:

def sm_inv_lemma(A, u, v):
  """ Function implementing the Sherman-Morrison matrix inversion lemma using numpy.
  
  Parameters
  ----------
  A : numpy array
      The original matrix.
  
  u : numpy array
      The first vector in the Sherman-Morrison lemma.
  
  v : numpy array
      The second vector in the Sherman-Morrison lemma.
  
  Returns
  -------
  The inverse matrix of A - uv^T.
  """
  A_inv = np.linalg.inv(A)
  alpha = 1 / (1 + v.T @ A_inv @ u)
  return A_inv - (alpha * A_inv @ u @ v.T @ A_inv)

Training Data

GALPACA模型是通过在Alpaca数据集上对预训练的GALACTICA模型进行微调而训练的。GALACTICA模型在包括论文、教科书、科学网站、百科全书等在内的1060亿个开放获取的科学文本和数据上进行了训练。通过在Alpaca数据集的52000个指令-回应对上对基本GALACTICA模型进行微调，用户可以以指令-回应的方式查询GALPACA模型。

How to Use

使用transformers库可使用GALPACA权重。

# pip install accelerate
from transformers import AutoTokenizer, OPTForCausalLM

tokenizer = AutoTokenizer.from_pretrained("GeorgiaTechResearchInstitute/galpaca-30b")
model = OPTForCausalLM.from_pretrained("GeorgiaTechResearchInstitute/galpaca-30b", device_map="auto", torch_dtype=torch.float16)

# see the original Alpaca repo for more information about the prompt templates
no_input_prompt_template = ("Below is an instruction that describes a task. "
                            "Write a response that appropriately completes the request.\n\n"
                            "### Instruction:\n{instruction}\n\n### Response:")
prompt = "Write out Maxwell's equations and explain the meaning of each one."
formatted_prompt = no_input_prompt_template.format_map({'instruction': prompt})

tokenized_prompt = tokenizer(formatted_prompt, return_tensors="pt").input_ids.to(model.device)
out_tokens = model.generate(tokenized_prompt)

print(tokenizer.batch_decode(out_tokens, skip_special_tokens=False, clean_up_tokenization_spaces=False))

Training Resources

使用16块A100 80GB GPU，16位混合精度，在有效批量大小为1024的情况下，最大上下文窗口为384个标记，GALPACA 30B模型进行了约6小时的微调。此模型使用了DeepSpeed ZeRO Stage 3优化进行训练。

Performance and Limitations

定性评估表明，Galpaca在与技术知识和编程相关的任务上经常优于基于LLaMA的Alpaca模型，但在生成散文等自然语言任务方面表现不佳。关于GALACTICA系列模型的性能和限制的更多信息，请参阅原始GALACTICA模型卡。

Works Cited

@inproceedings{GALACTICA,
    title={GALACTICA: A Large Language Model for Science},
    author={Ross Taylor and Marcin Kardas and Guillem Cucurull and Thomas Scialom and Anthony Hartshorn and Elvis Saravia and Andrew Poulton and Viktor Kerkez and Robert Stojnic},
    year={2022}
}

@misc{alpaca,
  author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
  title = {Stanford Alpaca: An Instruction-following LLaMA model},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
}