Stack Overflow
Stack Overflow

Questions tagged [gpt-2]

Use this tag with Generative Pre-trained Transformer 2 (GPT-2). Do not use with GPT-3 or the ad tagging library (GPT).

References

See the GPT-2 definition on Wikipedia.

Related Tags

199 questions
12
votes
2 answers

OpenAI GPT-2 model use with TensorFlow JS

Is that possible to generate texts from OpenAI GPT-2 using TensorFlowJS? If not what is the limitation, like model format or ...?
jay
  • 477
  • 5
  • 13
6
votes
2 answers

How can I create and fit vocab.bpe file (GPT and GPT2 OpenAI models) with my own corpus text?

This question is for those who are familiar with GPT or GPT2 OpenAI models. In particular, with the encoding task (Byte-Pair Encoding). This is my problem: I would like to know how I could create my own vocab.bpe file. I have a spanish corpus text…
rafaelmg07
  • 63
  • 1
  • 3
5
votes
2 answers

Generate the probabilities of all the next possible word for a given text

i have the following code import transformers from transformers import pipeline # Load the language model pipeline model = pipeline("text-generation", model="gpt2") # Input sentence for generating next word predictions input_sentence = "I enjoy…
datadigger
  • 101
  • 7
5
votes
1 answer

Colab: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory warn(f"Failed to load image Python extension: {e}")

I'm trying to use the python package aitextgen in google Colab so I can fine-tune GPT. First, when I installed the last version of this package I had this error when importing it. Unable to import name '_TPU_AVAILABLE' from…
Peyman
  • 3,097
  • 5
  • 33
  • 56
5
votes
2 answers

Can i clear up gpu vram in colab

I'm trying to use aitextgen to finetune 774M gpt 2 on a dataset. unfortunately, no matter what i do, training fails because there are only 80 mb of vram available. how can i clear the vram without restarting the runtime and maybe prevent the vram…
Blazeolmo 343
  • 51
  • 1
  • 2
5
votes
4 answers

HuggingFace | ValueError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet con

Not always, but occasionally when running my code this error appears. At first, I doubted it was a connectivity issue but to do with cashing issue, as discussed on an older Git Issue. Clearing cache didn't help runtime: $ rm…
DanielBell99
  • 896
  • 5
  • 25
  • 57
5
votes
1 answer

On-the-fly tokenization with datasets, tokenizers, and torch Datasets and Dataloaders

I have a question regarding "on-the-fly" tokenization. This question was elicited by reading the "How to train a new language model from scratch using Transformers and Tokenizers" here. Towards the end there is this sentence: "If your dataset is…
Pietro
  • 415
  • 6
  • 16
4
votes
0 answers

can't change embedding dimension to pass it through gpt2

I'm practicing image captioning and have some problems with different dimensions of tensors. So I have image embedding aka size [1, 512], but GPT2, which I use for caption generation, needs size [n, 768], where n is number of tokens of the caption's…
kat0ewww
  • 41
  • 1
4
votes
2 answers

Fine-Tuning GPT2 - attention mask and pad token id errors

I have been trying to fine-tune GPT2 on the wikitext-2 dataset (just to help myself learn the process) and I am running into a warning message that I have not seen before: "The attention mask and the pad token id were not set. As a consequence, you…
Toakley
  • 182
  • 3
  • 13
4
votes
2 answers

Fine tuning GPT2 for generative question anwering

I am trying to finetune gpt2 for a generative question answering task. Basically I have my data in a format similar to: Context : Matt wrecked his car today. Question: How was Matt's day? Answer: Bad I was looking on the huggingface documentation to…
Matei Neagu
  • 71
  • 1
  • 2
4
votes
1 answer

Why some weights of GPT2Model are not initialized?

I am using the GPT2 pre-trained model for a research project and when I load the pre-trained model with the following code, from transformers.models.gpt2.modeling_gpt2 import GPT2Model gpt2 = GPT2Model.from_pretrained('gpt2') I get the following…
K.N
  • 871
  • 2
  • 10
  • 30
4
votes
2 answers

How do I make a paraphrase generation using BERT/ GPT-2

I am trying hard to understand how to make a paraphrase generation using BERT/GPT-2. I cannot understand how do I make it. Could you please provide me with any resources where I will be able to make a paraphrase generation model? "The input would be…
Tazin Islam
  • 41
  • 1
  • 2
4
votes
2 answers

Asking gpt-2 to finish sentence with huggingface transformers

I am currently generating text from left context using the example script run_generation.py of the huggingface transformers library with gpt-2: $ python transformers/examples/run_generation.py \ --model_type gpt2 \ --model_name_or_path gpt2 \ …
r0levrai
  • 43
  • 5
4
votes
1 answer

Can we use GPT-2 sentence embedding for classification tasks?

I am experimenting on the use of transformer embeddings in sentence classification tasks without finetuning them. I have used BERT embeddings and those experiments gave me very good results. Now I want to use GPT-2 embeddings (without fine-tuning).…
Shamane Siriwardhana
  • 3,951
  • 6
  • 33
  • 73
4
votes
3 answers

Python gpt-2-simple, load multiple models at once

I'm working on a discord bot and one of the functions I want to implement responds with text generated by the gpt-2-simple library. I want to have more then one model loaded to have multiple models available to respond to messages from my…
Jade Harley
  • 41
  • 1
1
2 3
…
13 14