Levanter 背包

示例:

import torch
import transformers
from transformers import AutoModelForCausalLM


model_id = "ivanzhouyq/levanter-backpacks"
config = transformers.AutoConfig.from_pretrained(model_id, trust_remote_code=True)
torch_model = AutoModelForCausalLM.from_pretrained(model_id, config=config, trust_remote_code=True)
torch_model.eval()

input = torch.randint(0, 50264, (1, 512), dtype=torch.long)
torch_out = torch_model(
    input,
    position_ids=None,
)
torch_out = torch.nn.functional.softmax(torch_out.logits, dim=-1)

请注意，您需要一起安装 safetensors、transformers 和 torch 才能使其正常工作。