Asking gpt-2 to finish sentence with huggingface transformers

Question

I am currently generating text from left context using the example script run_generation.py of the huggingface transformers library with gpt-2:

$ python transformers/examples/run_generation.py \
  --model_type gpt2 \
  --model_name_or_path gpt2 \
  --prompt "Hi, " --length 5

=== GENERATED SEQUENCE 1 ===
Hi,  could anyone please inform me

I would like to generate short complete sentences. Is there any way to tell the model to finish a sentence before length words?

Note: I don't mind changing model, but would prefer an auto-regressive one.

I don't mind changing model, but would prefer an auto-regressive one. — r0levrai, Apr 09 '20 at 13:13
Avoid posting additional content as a comment: update the question instead. — Berriel, Apr 09 '20 at 13:23
I am trying to use `StoppingCriteria` to stop earlier. Anyone with luck on that? — Lei Zhang, Oct 01 '21 at 09:20

score 3 · Accepted Answer · answered Oct 26 '20 at 18:25

3

Unfortunately there is no way to do so. You can set the length parameter to a greater value and then just discard the incomplete part at the end.

Even GPT3 doesn't support completing a sentence before a specific length. GPT3 support "sequences" though. Sequences force the model to stop when certain condition is fulfilled. You can find more information about in thi article

answered Oct 26 '20 at 18:25

Sergei Krivosheenko

1,136
7
17

What about StoppingCriteria? This option exists according to the source code but I can't get it to work and there's not a single example of it on the entire internet. – pete Nov 21 '22 at 00:10

score -1 · Answer 2 · answered Apr 30 '22 at 02:20

This parameter helps you to get the results:

    '''
model_name is the model name, such as "124M" or "345M" and relies on 
models_dir.
• models_dir defines the directory containing the models.
• seed sets a random integer for random generators. The seed can be set to 
reproduce results.
• nsamples is the number of samples to return. If it is set to 0, it will continue 
to generate samples until you double-click on the run button of the cell or 
press Ctrl + M.
• batch_size determines the size of a batch and has an impact on memory 
and speed.
• length is the number of tokens of generated text. If set to none, it relies on 
the hyperparameters of the model.
• temperature determines the level of Boltzmann distributions. If the 
temperature is high, the completions will be more random. If the temperature 
is low, the results will become more deterministic.
• top_k controls the number of tokens taken into consideration by Top-k at 
each step. 0 means no restrictions. 40 is the recommended value.
• top_p controls Top-p
'''

Hope it will help you!

Asking gpt-2 to finish sentence with huggingface transformers

2 Answers2