GPT-J for preference modeling

• Dataset: https://huggingface.co/datasets/reciprocate/oasst_hh_shp_hellaswag_webgpt_rm_dataset
• Logs: https://wandb.ai/sorry/autocrit/runs/y9des5kz?workspace=user-sorry

Usage:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("reciprocate/gpt-j_rm_format-oa", revision="501f895")
model = AutoModelForSequenceClassification.from_pretrained("reciprocate/gpt-j_rm_format-oa", revision="501f895")

input = "<|prompter|>Are you lying right now?</s><|assistant|>I am :)</s><|endoftext|>"
model(**tokenizer(input, return_tensors="pt"))[0].item()

Output:

-4.877448558807373