R Reticulate transformers library cannot find torch

Question

Using R and the reticulate package I am trying to use a pre-trained model from Huggingface. This partcular model requires PyTorch and transformers. Both are available in R via reticulate, however even though I can install and load both, the transformers package can't find the PyTorch installation.

use_virtualenv("r-reticulate")
reticulate::py_install('transformers', pip = TRUE)
reticulate::py_install("PyTorch")

transformer = reticulate::import('transformers')
torch = reticulate::import('torch')
tokenizer = transformer$AutoTokenizer$from_pretrained("gagan3012/keytotext-small")
model = transformer$AutoModel$from_pretrained("gagan3012/keytotext-small")

and the error:

Error in py_call_impl(callable, dots$args, dots$keywords): ImportError: 
AutoModel requires the PyTorch library but it was not found in your environment. Checkout the instructions on the
installation page: https://pytorch.org/get-started/locally/ and follow the ones that match your environment.


Detailed traceback:
  File "/miniconda/envs/r-reticulate/lib/python3.7/site-packages/transformers/utils/dummy_pt_objects.py", line 364, in from_pretrained
    requires_backends(cls, ["torch"])
  File "/miniconda/envs/r-reticulate/lib/python3.7/site-packages/transformers/file_utils.py", line 683, in requires_backends
    raise ImportError("".join([BACKENDS_MAPPING[backend][1].format(name) for backend in backends]))

Traceback:

1. transformer$AutoModel$from_pretrained("gagan3012/keytotext-small")
2. py_call_impl(callable, dots$args, dots$keywords)

but PyTorch is definitely installed - I can call methods i.e torch$cudnn_convolution_add_relu so how can I tell the transformers package where torch is?

score 0 · Answer 1 · answered Mar 23 '22 at 18:42

It looks like reticulate is pointing to miniconda, and you have likely installed PyTorch, etc. in a different environment.

Check out which python environment your Pytorch is installed. Then check where your reticulate environment is using Sys.getenv() and look for RETICULATE_PYTHON variable.

The best way to handle a mis-match is to create a .Renviron file in the home directory listed in your Sys.getenv()

I install all of my packages into a Python environment called reticulate and then point to it in my .Renviron file.

In your .Renviron file you need the following code:

RETICULATE_PYTHON="path to environment with Pytorch"

This post talks about it also.

.Renviron File for Reticulate

R Reticulate transformers library cannot find torch

1 Answers1