Stack Overflow
Stack Overflow

Questions tagged [distilbert]

37 questions
38
votes
5 answers

ValueError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]] - Tokenizing BERT / Distilbert Error

def split_data(path): df = pd.read_csv(path) return train_test_split(df , test_size=0.1, random_state=100) train, test = split_data(DATA_DIR) train_texts, train_labels = train['text'].to_list(), train['sentiment'].to_list() test_texts,…
Raoof Naushad
  • 526
  • 1
  • 5
  • 7
7
votes
1 answer

Text generation using huggingface's distilbert models

I've been struggling with huggingface's DistilBERT model for some time now, since the documentation seems very unclear and their examples (e.g.…
George
  • 3,521
  • 4
  • 30
  • 75
6
votes
2 answers

what 's the meaning of "Using bos_token, but it is not set yet."

When I run the demo.py from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("distilbert-base-multilingual-cased") model = AutoModel.from_pretrained("distilbert-base-multilingual-cased", return_dict=True) #…
young
  • 61
  • 1
  • 4
4
votes
0 answers

huggingface distillbert classification using multiprocessing

I am trying to use torch multiprocessing to parallelize the predictions from two separate huggingface distillbert classification models. It seems to be deadlocked at the prediction step. I am using python 3.6.5, torch 1.5.0 and huggingface…
user3245722
  • 323
  • 2
  • 3
  • 9
2
votes
0 answers

How can I train distilBERT more efficiently on my large text classification task?

I've been thrown into the deep end a bit with a task at work. I need to use DistilBERT for a multi-class text classification problem, but here's the kicker the dataset is gigantic - we're talking millions of samples! I've been messing around with…
user22280248
2
votes
1 answer

ValueError: The model did not return a loss from the inputs, only the following keys: logits

I want to make a metaphor detector model. As a pretrained model, I use a DistilBert model that I have, previously, fine-tuned with masking (this is the model I use to make the new metaphor detection model). The new model is the one that gives me an…
Cristi Fernandez
  • 21
  • 1
2
votes
0 answers

HuggingFace Trainer() does nothing - only on Vertex AI workbench, works on colab

I am having issues getting the Trainer() function in huggingface to actually do anything on Vertex AI workbench notebooks. I'm totally stumped and have no idea how to even begin to try debug this. I made this small notebook:…
andrewm4894
  • 1,451
  • 4
  • 17
  • 37
2
votes
1 answer

How to get output_attentions of a pretrained Distilbert Model?

I am using a pretrained DistilBert model: from transformers import TFDistilBertModel,DistilBertConfig dbert = 'distilbert-base-uncased' config = DistilBertConfig(max_position_embeddings=256 , dropout=0.2, …
user8888
  • 31
  • 4
2
votes
3 answers

How to use DistilBERT Huggingface NLP model to perform sentiment analysis on new data?

I am using DistilBERT to do sentiment analysis on my dataset. The dataset contains text and a label for each row which identifies whether the text is a positive or negative movie review (eg: 1 = positive and 0 = negative). Here is the code from the…
brownie_coder
  • 17
  • 1
  • 4
2
votes
1 answer

Input to reshape doesn't match requested shape

I know others have posted similar questions already, but I couldn't find a solution that was appropriate here. I've written a custom keras layer to average outputs from DistilBert based on a mask. That is, I have dim=[batch_size, n_tokens_out, 768]…
bmt
  • 21
  • 2
2
votes
1 answer

cannot import name 'DISTILBERT_PRETRAINED_MODEL_ARCHIVE_MAP' from 'transformers.modeling_distilbert'

I am trying to train the distil BERT model for Question Answering purpose. I have installed simple transformers and everything but when I try to run the following command: model = QuestionAnsweringModel('distilbert', …
swapnil agashe
  • 69
  • 1
  • 12
1
vote
0 answers

DistilBert for self-supervision - switch heads for pre-training: MaskedLM and SequenceClassification

Say I want to train a model for sequence classification. And so I define my model to be: model = DistilBertForSequenceClassification.from_pretrained("bert-base-uncased") My question is - what would be the optimal way if I want to pre-train this…
R90
  • 21
  • 3
1
vote
1 answer

Difference between from_config and from_pretrained in HuggingFace

num_labels = 3 if task.startswith("mnli") else 1 if task=="stsb" else 2 preconfig = DistilBertConfig(n_layers=6) model1 = AutoModelForSequenceClassification.from_config(preconfig) model2 =…
strnlz
  • 13
  • 5
1
vote
1 answer

PyTorch - FineTuning bert - Oscillating loss - Very bad accuracy

I have been trying to train a model on vulnerability detection through source code. And, after a little bit of searching, I thought a very good starting point could be using a pre-trained transformer model from HuggingFace with PyTorch and…
Desperados
  • 434
  • 5
  • 13
1
vote
1 answer

How to solve this problem with distilbert tokenizer?

from transformers import DistilBertTokenizerFast tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased') tokenized_input = tokenizer( sentences, truncation=True, is_split_into_words=True, padding='max_length',…
Gozdi
  • 41
  • 1
  • 1
  • 6
1
2 3