roberta-base-reward-model-falcon-dolly: 一个使用Dolly v2数据集和Falcon-7b-instruct生成的对比数据进行训练的实验性奖励模型

这是一个使用TRL进行训练的实验性奖励模型，使用了Dolly v2数据集的对比数据和Falcon生成的回答。

出于测试目的，我们假设人工编写的回答（由Databricks员工编写）优先于Falcon生成的回答。实际情况可能并非总是如此，但您可以使用 Argilla 设置一个对比数据收集，以获取关于优选回答的真实反馈。

要使用此模型进行评分：

from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("argilla/roberta-base-reward-model-falcon-dolly")

model = AutoModelForSequenceClassification.from_pretrained("argilla/roberta-base-reward-model-falcon-dolly")

def get_score(model, tokenizer, prompt, response):
    # Tokenize the input sequences
    inputs = tokenizer.encode_plus(prompt, response, truncation=True, padding="max_length", max_length=512, return_tensors="pt")

    # Perform forward pass
    with torch.no_grad():
        outputs = model(**inputs)

    # Extract the logits
    logits = outputs.logits

    return logits.item()

# Example usage
prompt = "What is Depreciation"
example_less_pref_response = "What is Depreciation – 10 Important Facts to Know? When a business buys a new asset, the purchase price of that asset is depreciated over time to reflect its usage and eventual obsolescence. Depreciation expense can be a tax deductible expense and is usually a non-cash expense reported on a company’s income statement and balance sheet. The amount of depreciation expense a company reports each year is the difference between the original purchase price of the asset and what the current value of that asset might be. Here are 10 important facts to know about depreciation: 1. Depreciation is a non-cash expense. It is an expense that is reported in a business’s income statement and balance sheet and not a cash flow expense. 2. Depreciation is an accounting standard and it is required to be disclosed in a business’s financial statements. 3. The amount of depreciation is usually a tax expense and not a cash expense reported on a company’s income statement"
example_preferred_response = "Depreciation is the drop in value of an asset due to wear and tear, age and obsolescence (going out of date) as recorded in an organization's financial records."

score = get_score(model, tokenizer, prompt, example_less_pref_response)
print(score)
# >> -3.915163993835449

score = get_score(model, tokenizer, prompt, example_preferred_response)
print(score)
# >> 7.460323333740234

评分示例

以下是一些示例的评分（请记住，此奖励模型已经经过训练以优选原始回答）：