How can i solve ImportError: Using the `Trainer` with `PyTorch` requires `accelerate>=0.20.1` when using Huggingface's TrainArguments?

Question

I'm using the transformers library in Google colab, and When i am using TrainingArguments from transformers library i'm getting Import error with this code:

from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir = "/content/our-model",
    learning_rate=2e-5,
    per_device_train_batch_size= 64,
    per_device_eval_batch_size = 16,
    num_train_epochs = 2,
    weight_decay = 0.01,
    evaluation_strategy = "epoch",
    save_strategy = "epoch",
    load_best_model_at_end = True,
    push_to_hub = False
)

This is the error i'm getting:

<ipython-input-28-0518ea5ff407> in <cell line: 2>()
      1 from transformers import TrainingArguments
----> 2 training_args = TrainingArguments(
      3     output_dir = "/content/our-model",
      4     learning_rate=2e-5,
      5     per_device_train_batch_size= 64,

4 frames
/usr/local/lib/python3.10/dist-packages/transformers/training_args.py in _setup_devices(self)
   1670         if not is_sagemaker_mp_enabled():
   1671             if not is_accelerate_available(min_version="0.20.1"):
-> 1672                 raise ImportError(
   1673                     "Using the `Trainer` with `PyTorch` requires `accelerate>=0.20.1`: Please run `pip install transformers[torch]` or `pip install accelerate -U`"
   1674                 )

ImportError: Using the `Trainer` with `PyTorch` requires `accelerate>=0.20.1`: Please run `pip install transformers[torch]` or `pip install accelerate -U 

I already tried pip install for 0.20.1 version of accelerate and pip install transformers[torch] and both didn't worked.

Answer 1

If you're not particular about which transformers and accelerate version to tie to, then do this to use the most up-to-date version in Google Colab:

! pip install -U accelerate
! pip install -U transformers

Then the issue you are having with accelerate should auto-resolve itself.

Note:

• Underspecifying pip install -U transformers instead of pip install transformers[pytorch] might be easier since that's what most of the users do and the developers of the library will make sure that the basic pip works with the common functions and class like TrainingArguments

• Instead of specifying accelerate to the pip install accelerate>=0.20.1, if you have no particular need to fixed the version, automatically upgrading to the latest version might get you more stability when using the library, esp. with "hot"/"trending" libraries that are constantly changing (almost) daily.

---

If further debugging is necessary, i.e. if the above didn't work. To check your transformers and accelerate version, do this:

import accelerate

accelerate.__version__

Most probably you might have an ImportError at the first line if accelerate is not already installed when you installed transformers.

And then if the first line works and the 2nd line is not outputting a version >=0.20.1, then that is the cause of your issue.

The current versions to-date (July 2023) are:

import accelerate
import transformers

transformers.__version__, accelerate.__version__

[out]:

('4.30.1', '0.21.0')

Here's an example notebook with the model that you wish to use as per the comments in your question, https://colab.research.google.com/drive/1D79AjHMeE6HAZC-g2S83baTgsHtDUu5i?usp=sharing

---

If the error persist after the pip install ..., try restarting the runtime.

If you can't find the buttons to press to restart, try this in the cell Restart kernel in Google Colab then re-run the cells for import ...

import os
os._exit(00)

Answer 2

I faced the same problem on Google COLAB and I followed the instructions above (see the answer rated with 4) to resolve it, namely, both libraries were installed like:

!pip install -U accelerate
!pip install -U transformers