模型:
ethzanalytics/stablelm-tuned-alpha-7b-sharded
这是一个分片检查点(带有~4GB的分片)的模型。有关所有详细信息,请参见 original model 。
安装transformers,accelerate和bitsandbytes。
pip install -U -q transformers bitsandbytes accelerate
在8位中加载模型,然后 run inference :
from transformers import AutoTokenizer, AutoModelForCausalLM model_name = "ethzanalytics/stablelm-tuned-alpha-7b-sharded" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained( model_name, load_in_8bit=True, device_map="auto" )