3

I’m trying to do a finetuning without an evaluation dataset. For that, I’m using the following code:

training_args = TrainingArguments(
    output_dir=resume_from_checkpoint,
    evaluation_strategy="epoch",
    per_device_train_batch_size=1,
)
def compute_metrics(pred: EvalPrediction):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    f1 = f1_score(labels, preds, average="weighted")
    acc = accuracy_score(labels, preds, average="weighted")
    return {"accuracy": acc, "f1": f1}
trainer = Trainer(
    model=self.nli_model,
    args=training_args,
    train_dataset=tokenized_datasets,
    compute_metrics=compute_metrics,
)

However, I get

ValueError: Trainer: evaluation requires an eval_dataset

I thought that by default, Trainer does no evaluation… at least in the docs, I got this idea…

alvas
  • 115,346
  • 109
  • 446
  • 738
An old man in the sea.
  • 1,169
  • 1
  • 13
  • 30

3 Answers3

2

First do this Splitting dataset into Train, Test and Validation using HuggingFace Datasets functions

from datasets import load_dataset
from datasets import DatasetDict

...

# Split dataset into 80-20%
ds_train, ds_valid = tokenized_datasets.train_test_split(test_size=0.2, seed=42)

Then

trainer = Trainer(
    model=self.nli_model,
    args=training_args,
    train_dataset=ds_train,
    eval_dataset=ds_valid,
    compute_metrics=compute_metrics,
)

It is important to understand why when "training" you will always need a "validation" set.

The train_dataset changes the gradient during optimization and parameters of the model. Reports training loss.

The eval_dataset is sort of like a validation (aka development set) to validate the results of the model updates independent of the data it is trained on. Reports validation loss.

alvas
  • 115,346
  • 109
  • 446
  • 738
2

I set evaluation_strategy="no" and do_eval=False when setting the TrainingArguments and then I was able to call the trainer.train() without passing any eval dataset. Note, I tested this on transformers version 4.31.0

Sarah Masud
  • 79
  • 1
  • 6
  • This way the evaluation will be bypassed, which is not a good practice. alvas' answer must be the accepted one. – subhojit777 Aug 22 '23 at 09:53
0

Looking through source code here: Source Huggingface Trainer

We see the following error in get_eval_dataloader() function

if eval_dataset is None and self.eval_dataset is None:
   raise ValueError("Trainer: evaluation requires an eval_dataset.")

so it seems you need an eval_dataset passed in the __init__ (where you call trainer=Trainer(...) or eventually because predict() calls evaluate() which calls get_eval_dataloader()

smoot
  • 21
  • 5