模型:

ethzanalytics/stablelm-tuned-alpha-7b-sharded

英文

StableLM-Tuned-Alpha 7b: sharded checkpoint

这是一个分片检查点(带有~4GB的分片)的模型。有关所有详细信息,请参见 original model

  • 这使得低RAM加载成为可能,例如Colab :)

基本用法

安装transformers,accelerate和bitsandbytes。

pip install -U -q transformers bitsandbytes accelerate

在8位中加载模型,然后 run inference

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "ethzanalytics/stablelm-tuned-alpha-7b-sharded"
tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
    model_name, load_in_8bit=True, device_map="auto"
)