No module named 'transformers.models' while trying to import BertTokenizer

Question

I am trying to import BertTokenizer from the transformers library as follows:

import transformers
from transformers import BertTokenizer
from transformers.modeling_bert import BertModel, BertForMaskedLM

However, I get the following error:

I am using transformers version 3.5.1 because I had a problem with the updated version which can be found here.

Does anyone know how to fix this? Apart from updating the transformers library to its latest version (that will unfortunately cause more errors).

Any help is appreciated!

I can not reproduce this with transformers 3.5.1. Have you maybe created a package called `transformers` by yourself? — cronoik, Mar 27 '21 at 14:56
@Gleichmut The answer that lvjiujin gave below solved the issue at that time, however I haven't used any upgraded versions of transformers lately : / — Icaru5, Aug 20 '23 at 16:39

score 6 · Accepted Answer · answered May 08 '21 at 13:27

6

you can change your code from

transformers.modeling_bert import BertModel, BertForMaskedLM

to

from transformers.models.bert.modeling_bert import BertModel,BertForMaskedLM

answered May 08 '21 at 13:27

lvjiujin

501
4
5

score 0 · Answer 2 · answered Dec 17 '21 at 04:00

How you call the package depends on how you installed the transformers package.

If you've installed it from pip you need to find the directory where the package is installed, for example, if installing with pip with conda as a virtual environment this is where the BertModel package will be stored:

/home/{USER_NAME}/anaconda3/envs/{CONDA_ENV_NAME}/lib/{PYTHON_VERSION}/site-packages/transformers/modeling_bert.py

while the BertTokenizer is automatically called in the init.py, hence can be directly called.

Therefore you should be able to call

from transformers.modeling_bert import BertModel, BertForMaskedLM from transformers import BertTokenizer

otherwise, if you've installed it from source, you need to map to the correct file. For Example, with the following file directory structure:

Code_folder
     transformers #package
         models
             bert
                 modeling_bert.py
     main.py # your file that wants to call the transformer package

Then you can call the following packages in the following way:

from transformers.models.bert.modeling_bert import BertModel,BertForMaskedLM
from transformers import BertTokenizer

No module named 'transformers.models' while trying to import BertTokenizer

2 Answers2

Linked