As a base model used https://huggingface.co/eachadea/vicuna-13b-1.1

Finetuned on Teknium's GPTeacher dataset, unreleased Roleplay v2 dataset, GPT-4-LLM dataset Uncensored, WizardLM Uncensored and Nous Research Instruct Dataset

Approx 180k instructions, all from GPT-4, all cleaned of any OpenAI censorship/"As an AI Language Model" etc.

Base model still has OpenAI censorship. Soon, a new version will be released with cleaned vicuna from https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltere

Trained on 8 A100-80GB GPUs for 5 epochs following Alpaca deepspeed training code.

Nous Research Instruct Dataset will be released soon.

Prompt format is Alpaca:

### Instruction:

### Response:

or

### Instruction:

### Input:

### Response:

GPTeacher, Roleplay v2 by https://huggingface.co/teknium

Wizard LM by https://github.com/nlpxucan

Nous Research Instruct Dataset by https://huggingface.co/karan4d and https://huggingface.co/huemin

Benchmark results:

    "arc_challenge": {
      "acc": 0.4189419795221843,
      "acc_stderr": 0.01441810695363901,
      "acc_norm": 0.439419795221843,
      "acc_norm_stderr": 0.014503747823580123
    },
    "arc_easy": {
      "acc": 0.7159090909090909,
      "acc_stderr": 0.009253921261885768,
      "acc_norm": 0.5867003367003367,
      "acc_norm_stderr": 0.010104361780747527
    },
    "boolq": {
      "acc": 0.8137614678899082,
      "acc_stderr": 0.006808882985424063
    },
    "hellaswag": {
      "acc": 0.5790679147580163,
      "acc_stderr": 0.004926996830194234,
      "acc_norm": 0.7518422624975104,
      "acc_norm_stderr": 0.004310610616845708
    },
    "openbookqa": {
      "acc": 0.288,
      "acc_stderr": 0.02027150383507522,
      "acc_norm": 0.436,
      "acc_norm_stderr": 0.0221989546414768
    },
    "piqa": {
      "acc": 0.7529923830250272,
      "acc_stderr": 0.010062268140772622,
      "acc_norm": 0.749727965179543,
      "acc_norm_stderr": 0.01010656188008979
    },
    "winogrande": {
      "acc": 0.6495659037095501,
      "acc_stderr": 0.01340904767667019
    }

Compute provided by our project sponsor https://redmond.ai/