grammar-synthesis-small (beta)

This model is a fine-tuned version of google/t5-small-lm-adapt for grammar correction on an expanded version of the JFLEG dataset.

usage in Python (after pip install transformers ):

from transformers import pipeline
corrector = pipeline(
              'text2text-generation',
              'pszemraj/grammar-synthesis-small',
              )
raw_text = 'i can has cheezburger'
results = corrector(raw_text)
print(results)

Check out a simple demo in Google Colab here .

Model description

The intent is to create a text2text language model that successfully completes "single-shot grammar correction" on a potentially grammatically incorrect text that could have a lot of mistakes with the important qualifier of it does not semantically change text/information that IS grammatically correct.

Compare some of the heavier-error examples on other grammar correction models to see the difference :)

Limitations

• dataset: cc-by-nc-sa-4.0
• model: apache-2.0
• this is still a work-in-progress and while probably useful for "single-shot grammar correction" in a lot of cases, give the outputs a glance for correctness ok?

Use Cases

Obviously, this section is quite general as there are many things one can use "general single-shot grammar correction" for. Some ideas or use cases:

original response:
                ive heard it attributed to a bunch of different philosophical schools, including stoicism, pragmatism, existentialism and even some forms of post-structuralism. i think one of the most interesting (and most difficult) philosophical problems is trying to let dogs (or other animals) out of cages. the reason why this is a difficult problem is because it seems to go against our grain (so to
synthesizing took 306.12 seconds
Final response in 1294.857 s:
        I've heard it attributed to a bunch of different philosophical schools, including solipsism, pragmatism, existentialism and even some forms of post-structuralism. i think one of the most interesting (and most difficult) philosophical problems is trying to let dogs (or other animals) out of cages. the reason why this is a difficult problem is because it seems to go against our grain (so to speak)

Note: that I have some other logic that removes any periods at the end of the final sentence in this chatbot setting to avoid coming off as passive aggressive

Training and evaluation data

More information needed ?

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0004
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 32
• total_train_batch_size: 512
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 4

Training results

Framework versions

• Transformers 4.20.1
• Pytorch 1.11.0+cu113
• Datasets 2.3.2
• Tokenizers 0.12.1