japanese-gpt-neox-small

This repository provides a small-sized Japanese GPT-NeoX model. The model was trained using code based on EleutherAI/gpt-neox .

Update log

• 2023/03/20 Update the model weight and config files such that it can be loaded via Huggingface's official GPT-NeoX implementation.

How to use the model

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("rinna/japanese-gpt-neox-small", use_fast=False)
model = GPTNeoXForCausalLM.from_pretrained("rinna/japanese-gpt-neox-small")

Model architecture

A 12-layer, 768-hidden-size transformer-based language model.

Training

The model was trained on Japanese CC-100 , Japanese C4 , and Japanese Wikipedia to optimize a traditional language modelling objective.

Tokenization

The model uses a sentencepiece -based tokenizer.

A toy prefix-tuning weight file

Along with pretrained model, we also release a prefix-tuning weight file named smileface_suffix.task0.weight for demonstration. The toy prefix-tuning weights here is trained to encourage the model to end every generated sentence with a smiling face emoji ?. Find the training/inference code for prefix-tuning at our Github repo prefix-tuning-gpt .

Here are a few samples generated with and without the toy prefix weights, respectively.

3 samples without the prefix weights

3 samples with the prefix weights:

Inference with FasterTransformer

After version 5.1, NVIDIA FasterTransformer now supports both inference for GPT-NeoX and a variety of soft prompts (including prefix-tuning). The released pretrained model and prefix weights in this repo have been verified to work with FasterTransformer 5.1.

Licenese

The MIT license