Chat & support: my new Discord server

Want to contribute? TheBloke's Patreon page

Falcon-40B-Instruct 4bit GPTQ

This repo contains an experimantal GPTQ 4bit model for Falcon-40B-Instruct .

It is the result of quantising to 4bit using AutoGPTQ .

Repositories available

• 4-bit GPTQ model for GPU inference
• 3-bit GPTQ model for GPU inference
• Unquantised bf16 model in pytorch format, for GPU inference and for further conversions

EXPERIMENTAL

Please note this is an experimental GPTQ model. Support for it is currently quite limited.

It is also expected to be VERY SLOW . This is currently unavoidable, but is being looked at.

This 4bit model requires at least 35GB VRAM to load. It can be used on 40GB or 48GB cards, but not less.

Please be aware that you should currently expect around 0.7 tokens/s on 40B Falcon GPTQ.

AutoGPTQ

AutoGPTQ is required: pip install auto-gptq

AutoGPTQ provides pre-compiled wheels for Windows and Linux, with CUDA toolkit 11.7 or 11.8.

If you are running CUDA toolkit 12.x, you will need to compile your own by following these instructions:

git clone https://github.com/PanQiWei/AutoGPTQ
cd AutoGPTQ
pip install .

These manual steps will require that you have the Nvidia CUDA toolkit installed.

text-generation-webui

There is provisional AutoGPTQ support in text-generation-webui.

This requires text-generation-webui as of commit 204731952ae59d79ea3805a425c73dd171d943c3.

So please first update text-genration-webui to the latest version.

How to download and use this model in text-generation-webui

About trust_remote_code

Please be aware that this command line argument causes Python code provided by Falcon to be executed on your machine.

This code is required at the moment because Falcon is too new to be supported by Hugging Face transformers. At some point in the future transformers will support the model natively, and then trust_remote_code will no longer be needed.

In this repo you can see two .py files - these are the files that get executed. They are copied from the base repo at Falcon-7B-Instruct .

Simple Python example code

To run this code you need to install AutoGPTQ and einops:

pip install auto-gptq
pip install einops

You can then run this example code:

import torch
from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM

# Download the model from HF and store it locally, then reference its location here:
quantized_model_dir = "/path/to/falcon40b-instruct-GPTQ"

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, use_fast=False)

model = AutoGPTQForCausalLM.from_quantized(quantized_model_dir, device="cuda:0", use_triton=False, use_safetensors=True, torch_dtype=torch.float32, trust_remote_code=True)

prompt = "Write a story about llamas"
prompt_template = f"### Instruction: {prompt}\n### Response:"

tokens = tokenizer(prompt_template, return_tensors="pt").to("cuda:0").input_ids
output = model.generate(input_ids=tokens, max_new_tokens=100, do_sample=True, temperature=0.8)
print(tokenizer.decode(output[0]))

Provided files

gptq_model-4bit--1g.safetensors

This will work with AutoGPTQ as of commit 3cb1bf5 ( 3cb1bf5a6d43a06dc34c6442287965d1838303d3 )

It was created without groupsize to reduce VRAM requirements, and with desc_act (act-order) to improve inference quality.

• gptq_model-4bit--1g.safetensors
  • Works only with latest AutoGPTQ CUDA, compiled from source as of commit 3cb1bf5
    • At this time it does not work with AutoGPTQ Triton, but support will hopefully be added in time.
  • Works with text-generation-webui using --autogptq --trust_remote_code
    • At this time it does NOT work with one-click-installers
  • Does not work with any version of GPTQ-for-LLaMa
  • Parameters: Groupsize = None. Act order (desc_act)

Discord

For further support, and discussions on these models and AI in general, join us at: TheBloke AI's Discord server

Thanks, and how to contribute.

Thanks to the chirper.ai team!

I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.

If you're able and willing to contribute, it'd be most gratefully received and will help me to keep providing models, and work on new AI projects.

Donaters will get priority support on any and all AI/LLM/model questions, plus other benefits.

• Patreon: https://patreon.com/TheBlokeAI
• Ko-Fi: https://ko-fi.com/TheBlokeAI

Patreon special mentions : Aemon Algiz; Johann-Peter Hartmann; Talal Aujan; Jonathan Leane; Illia Dulskyi; Khalefa Al-Ahmad; senxiiz; Sebastain Graf; Eugene Pentland; Nikolai Manek; Luke Pendergrass.

Thank you to all my generous patrons and donaters.

✨ 原始模型卡片：Falcon-40B-Instruct

✨ Falcon-40B-Instruct

Falcon-40B-Instruct 是由 TII 基于 Falcon-40B 构建的一个有40B参数的仅解码的模型，并在混合了 Baize 的数据集上进行了微调。它遵循 TII Falcon LLM License 进行发布。

论文即将发布 ?。

为什么选择使用Falcon-40B-Instruct?

• 如果您正在寻找基于 Falcon-40B 的即用型聊天/指导模型。
• Falcon-40B是目前可用的最佳开源模型。它优于 LLaMA ， StableLM ， RedPajama ， MPT 等其他模型。请参阅 OpenLLM Leaderboard 。
• 它具有针对推理进行优化的架构，使用了FlashAttention（ Dao et al., 2022 ）和multiquery（ Shazeer et al., 2019 ）。

? 这是一个指导模型，可能不适合进一步微调。如果您有兴趣构建自己的指导/聊天模型，建议从 Falcon-40B 开始。

? 是否在寻找一个更小、更便宜的模型？ Falcon-7B-Instruct 是Falcon-40B-Instruct的小兄弟！

from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch

model = "tiiuae/falcon-40b-instruct"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)
sequences = pipeline(
   "Girafatron is obsessed with giraffes, the most glorious animal on the face of this Earth. Giraftron believes all other animals are irrelevant when compared to the glorious majesty of the giraffe.\nDaniel: Hello, Girafatron!\nGirafatron:",
    max_length=200,
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")

Falcon-40B-Instruct模型卡片

模型详细信息

模型描述

• 由开发： https://www.tii.ae ;
• 模型类型： 仅解码的有针对性模型;
• 语言（NLP）： 英语和法语;
• 许可证： TII Falcon LLM License ;
• 在模型上进行微调： Falcon-7B .

模型来源

• 论文： 即将发布 .

使用方法

直接使用

Falcon-40B-Instruct 在聊天数据集上进行了微调。

超出范围的使用

生产中的使用需要充分评估风险并采取适当的预防措施；任何被视为不负责任或有害的用途。

偏见、风险和限制

Falcon-40B-Instruct 主要在英语数据上进行训练，不适合正确推广到其他语言。此外，由于其训练数据来自代表网络的大规模语料库，它会携带在线上常见的刻板印象和偏见。

建议

我们建议Falcon-40B-Instruct的用户制定适当的防范措施，并为任何生产使用采取适当的预防措施。

如何开始使用该模型

from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch

model = "tiiuae/falcon-40b-instruct"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)
sequences = pipeline(
   "Girafatron is obsessed with giraffes, the most glorious animal on the face of this Earth. Giraftron believes all other animals are irrelevant when compared to the glorious majesty of the giraffe.\nDaniel: Hello, Girafatron!\nGirafatron:",
    max_length=200,
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")

培训细节

培训数据

Falcon-40B-Instruct 在 Bai ze 上微调，使用了 RefinedWeb 数据的5%。

数据使用Falcon- 7B / 40B 分词器进行分词。

评估

论文即将发布。

请参阅 OpenLLM Leaderboard 以获取早期结果。

技术规格

有关预训练的更多信息，请参阅 Falcon-40B .

模型架构和目标

Falcon-40B是一个仅针对因果关系的解码模型，训练任务是预测下一个令牌。

该架构在很大程度上借鉴了GPT-3论文（ Brown et al., 2020 ），但有以下差异：

• 位置嵌入：旋转嵌入（ Su et al., 2021 ）；
• 注意力：multiquery（ Shazeer et al., 2019 ）和FlashAttention（ Dao et al., 2022 ）；
• 解码器块：平行注意力/MLP和单层规范。

对于multiquery，我们使用的是一种内部变体，每个张量并行度都使用独立的键和值。

计算基础设施

Falcon-40B-Instruct在AWS SageMaker上训练，使用P4d实例的64个A100 40GB GPU。

Falcon-40B-Instruct使用自定义的分布式训练代码库Gigatron进行训练。它使用三维并行性方法结合了ZeRO和高性能Triton内核（FlashAttention等）。

引用

论文即将发布 ?.

许可证

Falcon-40B-Instruct在 TII Falcon LLM License 下发布。一般来说，

• 您可以自由地用于研究和/或个人目的；
• 您可以共享和构建这些模型的衍生作品，但您需要提供归属和相同许可证的共享；
• 对于商业用途，如果可归因的收入低于每年100万美元，您无需支付版权费用；否则，您应该与TII签订商业协议。

联系方式

falconllm@tii.ae