Questions tagged [huggingface-evaluate]
5 questions
3
votes
1 answer
How to compute sentence level perplexity from hugging face language models?
I have a large collection of documents each consisting of ~ 10 sentences. For each document, I wish to find the sentence that maximises perplexity, or equivalently the loss from a fine-tuned causal LM. I have decided to use Hugging Face and the…
pilu
- 720
- 5
- 16
1
vote
1 answer
Load accuracy metric with evaluate ,sometime mistakes happen: TypeError: 'NoneType' object is not callable
I'm using Bert and other encoder models for text classification tasks,but when I try to load accuracy metric with evaluate in huggingface,sometime mistakes happen: TypeError: 'NoneType' object is not callable.
I am searching for a long time on net.…
bill yao
- 21
- 5
0
votes
0 answers
Compute BLEU score of a Pandas DataFrame with valid rows filtered
I have a Pandas DataFrame from an Excel file, which contains text data which need to calculate the BLEU score row-by-row.
import evaluate
import pandas as pd
sacrebleu = evaluate.load("sacrebleu")
testset = pd.read_excel(xlsx_filename)
# find out…
Raptor
- 53,206
- 45
- 230
- 366
0
votes
1 answer
How to load a .py script directly with evaluate.load?
If I have a script like this https://huggingface.co/spaces/evaluate-metric/frugalscore/blob/main/frugalscore.py and save it as fgscore.py with a directory locally like:
./
my_script.py
fgscore/
fgscore.py
And in my_script.py, I can do…
alvas
- 115,346
- 109
- 446
- 738
0
votes
2 answers
How to resolve Error in seqeval in NER bert finetuning?
I'm trying to finetune a NER model, (BERT/BioBERT) and after first epoch of training, in Evaluation part, I got the following error, Any idea what is wrong?
ValueError: Predictions and/or references don't match the expected format.
Expected format:…
Raha
- 17
- 5