Stack Overflow
Stack Overflow

Questions tagged [roberta-language-model]

64 questions
4
votes
1 answer

Dutch sentiment analysis RobBERT

I have a question about Dutch sentiment analysis in Python. For a project at school I want to analyse the sentiment of a Dutch interview. I have worked with Vader but that doesn't work in Dutch. So I found RobBERT: a Dutch RoBERTa-based language…
Niels
  • 41
  • 3
3
votes
1 answer

Finetuning a LM vs prompt-engineering an LLM

Is it possible to finetune a much smaller language model like Roberta on say, a customer service dataset and get results as good as one might get with prompting GPT-4 with parts of the dataset? Can a fine-tuned Roberta model learn to follow…
Tolu
  • 1,081
  • 1
  • 8
  • 23
3
votes
2 answers

Adding new tokens to BERT/RoBERTa while retaining tokenization of adjacent tokens

I'm trying to add some new tokens to BERT and RoBERTa tokenizers so that I can fine-tune the models on a new word. The idea is to fine-tune the models on a limited set of sentences with the new word, and then see what it predicts about the word in…
Jigsaw
  • 405
  • 5
  • 11
2
votes
1 answer

How to apply random forests to the output produced by Bert?

I'm trying to get the output embeddings of a RoBERTa model, so I can train a random forests classifier on it for text classification (sentiment analysis). The original dataset this is based on is 500 news articles that each have a left/center/right…
pyroshark
  • 49
  • 6
2
votes
1 answer

How to change AllenNLP BERT based Semantic Role Labeling to RoBERTa in AllenNLP

Currently i'm able to train a Semantic Role Labeling model using the config file below. This config file is based on the one provided by AllenNLP and works for the default bert-base-uncased model and also GroNLP/bert-base-dutch-cased. { …
Thijs
  • 45
  • 5
2
votes
1 answer

Issue with loading the Roberta-base model

I am trying to use the Roberta-base model using AutoTokenizer.from_pretrained('roberta-base') but I get the following error: RuntimeError: Failed to import transformers.modeling_tf_utils because of the following error (look up to see its…
JoB
  • 23
  • 3
2
votes
2 answers

Is it necessary to re-train BERT models, specifically RoBERTa model?

I am looking for a sentiment analysis code with atleast 80%+ accuracy. I tried Vader and it I found it easy and usable, however it was giving accuracy of 64% only. Now, I was looking at some BERT models and I noticed it needs to be re-trained? Is…
Shrumo
  • 47
  • 7
2
votes
1 answer

Input/output format for Fine Tuning Huggingface RobertaForQuestionAnswering

I'm trying to fine-tune "RobertaForQuestionAnswering" on my custom dataset and I'm confused about the input params it takes. Here's the sample code. >>> from transformers import RobertaTokenizer, RobertaForQuestionAnswering >>> import torch >>>…
tarang ranpara
  • 129
  • 1
  • 1
  • 7
2
votes
0 answers

How to fix strict error when loading RoBerta using PyTorch

Any tips on how to fix this? Trying to follow the basic torch guide here: https://pytorch.org/hub/pytorch_fairseq_roberta/ but running into this error: in 1 import torch ----> 2 roberta =…
user16484288
  • 21
  • 1
2
votes
0 answers

How to perform Batch inferencing with RoBERTa ONNX quantized model?

I have converted RoBERTa PyTorch model to ONNX model and quantized it. I am able to get the scores from ONNX model for single input data point (each sentence). I want to understand how to get batch predictions using ONNX Runtime inference session by…
Yamini Preethi K
  • 21
  • 2
2
votes
1 answer

XLM-RoBERTa token - id relationship

I used the XLM-RoBERTa tokenizer in order to get the IDs for a bunch of sentences such as: ["loving is great", "This is another example"] I see that the IDs returned are not always as many as the number of whitespace-separated tokens in my…
user4180911
2
votes
1 answer

RPC failed with status = "Unavailable: Socket closed" Error when training FairSeq RoBERTa on Cloud TPU using PyTorch

I followed the tutorials "Pre-training FairSeq RoBERTa on Cloud TPU using Pytorch" to setup a Preemptible (v2-8) TPU env and train my RoBERTa model. The PyTorch env is based on torch-xla-1.6 as instructed by the document. However, it does not output…
user3786340
  • 190
  • 1
  • 8
2
votes
1 answer

Train RoBERTa from scratch where dataset is larger than the capacity of RAM?

I have a corpus that is 16 GB large and my ram IS around 16 GB ish. If I load the entire dataset to train the language model RoBERTa from scratch, I am going to have a memory issue. I intend to train my RoBERTa using the script provided from…
Realdeo
  • 449
  • 6
  • 19
1
vote
1 answer

Using RoBERTa-base for QA model outputs the context not an answer

I'm trying to use this model from deepset/roberta-base-squad2 to essentially go through a column of work related activities and have it answer the question what are the necessary skills for this job ? However the model is simply handing me back my…
Moe_blg
  • 71
  • 4
1
vote
0 answers

Roberta with GRU is not training

I'm trying to fine-tune RoBERTa and integrate external knowledge via a BiGRU block. But the model is not learning (the train loss is around 0.8 and is not decreasing). There is no problem with the data, I tried some other RoBERTa-based models on the…
atlas
  • 11
  • 1
1
2 3 4 5