Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

Question

I just installed the latest version of Tensorflow via pip install tensorflow and whenever I run a program, I get the log message:

W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found

Is this bad? How do I fix the error?

score 144 · Accepted Answer · edited Jun 20 '20 at 09:12

Tensorflow 2.1+

What's going on?

With the new Tensorflow 2.1 release, the default tensorflow pip package contains both CPU and GPU versions of TF. In previous TF versions, not finding the CUDA libraries would emit an error and raise an exception, while now the library dynamically searches for the correct CUDA version and, if it doesn't find it, emits the warning (The W in the beginning stands for warnings, errors have an E (or F for fatal errors) and falls back to CPU-only mode. In fact, this is also written in the log as an info message right after the warning (do note that if you have a higher minimum log level that the default, you might not see info messages). The full log is (emphasis mine):

2020-01-20 12:27:44.554767: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found

2020-01-20 12:27:44.554964: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

Should I worry? How do I fix it?

If you don't have a CUDA-enabled GPU on your machine, or if you don't care about not having GPU acceleration, no need to worry. If, on the other hand, you installed tensorflow and wanted GPU acceleration, check your CUDA installation (TF 2.1 requires CUDA 10.1, not 10.2 or 10.0).

If you just want to get rid of the warning, you can adapt TF's logging level to suppress warnings, but that might be overkill, as it will silence all warnings.

Tensorflow 1.X or 2.0:

Your CUDA setup is broken, ensure you have the correct version installed.

Thanks. But I add another point: If you installed python from **Microsoft Store**, you may still see this issue, even if you've done the required steps! See [my answer here](https://stackoverflow.com/a/63547977/5318303). — Mir-Ismaili, Aug 23 '20 at 17:34
I uninstalled every component of CUDA and still got the same error as if CUDA still existed — pete, Jan 14 '22 at 02:14
@pete the error appears if it can't find the right CUDA version. Since you uninstalled it, of course it won't find it and print the error... That's the normal behaviour. — GPhilo, Jan 14 '22 at 07:39
@brainslugs83 CUDA minor versions are not interchangeable, that's a limitation coming from CUDA itself — GPhilo, Jun 23 '22 at 07:20
Is there a better way to avoid those warnings on import time? without affecting TF_CPP_MIN_LOG_LEVE. Or modify those warnings from stderr to stdout? — Mussi, Mar 20 '23 at 09:17

score 76 · Answer 2 · edited Apr 06 '20 at 17:27

76

To install the prerequisites for GPU support in TensorFlow 2.1:

Install your latest GPU drivers.
Install CUDA 10.1.
- If the CUDA installer reports "you are installing an older driver version", you may wish to choose a custom installation and deselect some components. Indeed, note that software bundled with CUDA including GeForce Experience, PhysX, a Display Driver, and Visual Studio integration are not required by TensorFlow.
- Also note that TensorFlow requires a specific version of the CUDA Toolkit unless you build from source; for TensorFlow 2.1 and 2.2, this is currently version 10.1.
Install cuDNN.
1. Download cuDNN v7.6.4 for CUDA 10.1. This will require you to sign up to the NVIDIA Developer Program.
2. Unzip to a suitable location and add the bin directory to your PATH.
Install tensorflow by pip install tensorflow.
You may need to restart your PC.

edited Apr 06 '20 at 17:27

Ninjakannon

3,751
7
53
76

answered Jan 28 '20 at 16:11

Chayan Bansal

1,857
1
13
23

3

Thanks. But I add another point: If you installed python from **Microsoft Store**, you may still see this issue, even if you've done the required steps! See [my answer here](https://stackoverflow.com/a/63547977/5318303). – Mir-Ismaili Aug 23 '20 at 17:34
7

6. Restart your IDE after modifying the PATH variable. – t3chb0t Sep 27 '20 at 17:09
2

Note: I was a bit confused that after installing the latest GPU drivers, `nvidia-smi` showed me `CUDA Version: 11.2` (CUDA 11 is needed for TF 2.4) but the real installed and used CUDA version by my TF was 10.1. So never skip the CUDA installation step and always ensure your PATH is correctly set up. – Nerxis Feb 05 '21 at 15:18
2

Wow, restarting my pc could have saved me 2h. Thanks :) – Beolap Jun 08 '21 at 16:29

score 32 · Answer 3 · answered Sep 11 '20 at 15:22

32

TensorFlow 2.3.0 works fine with CUDA 11. But you have to install tf-nightly-gpu (after you installed tensorflow and CUDA 11): https://pypi.org/project/tf-nightly-gpu/

Try:

pip install tf-nightly-gpu

Afterwards you'll get the message in your console:

I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_110.dll

answered Sep 11 '20 at 15:22

Jenya Pu

733
1
7
13

3

Thank you! I thought I was going to have to roll back to 10.1. – Mark Sep 20 '20 at 03:11
3

Thank you! "pip install tf-nightly" did the trick for me. – Hagbard Dec 02 '20 at 09:57
1

Thank you! This was my only solution after trying for more than 15 hours. – Upulie Han Jan 21 '21 at 12:57
1

Had to do four steps: 1) Install `tf-nightly-gpu`, 2) install latest nvidia cuda toolkit (11.2+) using custom install but don't overwrite newer drivers (unselect), 3) Restart terminal. 4) Reinstall Nvidia driver if above steps borked it. e.g. nvidia-smi results in `Failed to initialize NVML: GPU access blocked by the operating system`. – BSalita Mar 07 '21 at 20:12
In March 2021, tensorflow new version is stated to be compatible with cuda 11, and I don't feel fine at using a nightly tensorflow-gpu version, it sounds like a hack. Nonetheless, this was the only solution working – Christian Vincenzo Traina Mar 20 '21 at 22:52

score 20 · Answer 4 · edited May 08 '22 at 00:32

20

I solved this another way. First of all I installed cuda 10.1 toolkit from this link.

Where I selected installer type: exe(local) (for windows) and installed 10.1 in custom mode (without visual studio integration, NVIDIA PhysX because previously I installed CUDA 10.2 so required dependencies were installed automatically)

After installation, From the Following Path (C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin) , in my case, I copied cudart64_101.dll file and pasted in (C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin).

Then importing Tensorflow worked smoothly.

edited May 08 '22 at 00:32

darclander

1,526
1
13
35

answered Feb 02 '20 at 07:54

Shantanu Nath

363
3
13

I've had the same problem when using Conda env. The solution was the same: copied the Dlls mentioned in the warning from "...\Anaconda\Envs\\Library\bin" to the "...\Anaconda3\Library\bin" The required Dlls should be in a path that os knows about it. – SaeedM May 21 '20 at 18:03
3

Just a note: Adding the folder `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin` to `PATH` would've worked as well (and is the recommended way to go, to avoid messing up with CUDA installations) – GPhilo May 25 '20 at 08:13
https://developer.nvidia.com/cuda-10.1-download-archive-base?target_os=Windows&target_arch=x86_64&target_version=10&target_type=exelocal Better use the NVIDIA site for downloading the .dll file. – Upulie Han Jan 21 '21 at 09:20

score 17 · Answer 5 · answered Nov 26 '20 at 00:27

In my case the tensorflow install was looking for cudart64_101.dll

The 101 part of cudart64_101 is the Cuda version - here 101 = 10.1

I had downloaded 11.x, so the version of cudart64 on my system was cudart64_110.dll

This is the wrong file!! cudart64_101.dll ≠ cudart64_110.dll

Solution

Download Cuda 10.1 from https://developer.nvidia.com/

Install (mine crashes with NSight Visual Studio Integration, so I switched that off)

When the install has finished you should have a Cuda 10.1 folder, and in the bin the dll the system was complaining about being missing

Check that the path to the 10.1 bin folder is registered as a system environmental variable, so it will be checked when loading the library

You may need a reboot if the path is not picked up by the system straight away

Confirmed that a reboot was needed in my case after installing CUDA 11.0 to get tensorflow to find it (cudart64_xyz.DLL requires CUDA version xy.z, in my case, xyz=110, not 101) — JoseOrtiz3, Mar 17 '21 at 22:40

Nicolas Gervais · Answer 6 · 2020-07-02T19:31:27.983

12

In a conda environment, this is what solved my problem (I was missing cudart64-100.dll:

Downloaded it from dll-files.com/CUDART64_100.DLL
Put it in my conda environment at C:\Users\<user>\Anaconda3\envs\<env name>\Library\bin

That's all it took! You can double check if it's working:

import tensorflow as tf
tf.config.experimental.list_physical_devices('GPU')

edited Jul 02 '20 at 19:31

answered May 18 '20 at 12:23

Nicolas Gervais

33,817
13
115
143

7

If you installed CUDA 10.1 for example you probably already have the DLL in "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin\cudart64_101.dll". I'm personally averse to downloading DLLs from secondary sources. – Dustin Andrews Jul 02 '20 at 19:29
For me, the directory location is C:\Users\\.conda\envs\\Library\bin – quarkz Jan 28 '21 at 08:43

Scott · Answer 7 · 2020-10-16T02:15:17.927

6

This answer might be helpful if you see above error but actually you have CUDA 10 installed:

pip install tensorflow-gpu==2.0.0

output:

I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll

which was the solution for me.

edited Oct 16 '20 at 02:15

answered Oct 16 '20 at 02:10

Scott

4,974
6
35
62

score 5 · Answer 8 · edited Feb 05 '21 at 23:43

5

I installed cudatoolkit 11 and copy dll C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin to C:\Windows\System32. It fixed for PyCharm but not for Anaconda jupyter:

[name: "/device:CPU:0" device_type: "CPU" memory_limit: 268435456 locality { } incarnation: 6812190123916921346 , name: "/device:GPU:0" device_type: "GPU" memory_limit: 13429637120 locality { bus_id: 1
links { } } incarnation: 18025633343883307728 physical_device_desc: "device: 0, name: Quadro P5000, pci bus id: 0000:02:00.0, compute capability: 6.1" ]

edited Feb 05 '21 at 23:43

Nerxis

3,452
2
23
39

answered Dec 07 '20 at 14:17

user3452134

362
4
11

Thank you, copying the dll file there worked great: tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll – Mitch Dec 23 '20 at 22:48

score 4 · Answer 9 · edited Jul 30 '20 at 12:53

This solution worked for me :

I preinstalled the environnement with anaconda (here is the code)

conda create -n YOURENVNAME python=3.6 // 3.6> incompatible with keras
conda activate YOURENVNAME
conda install tensorflow-gpu
conda install -c anaconda keras
conda install -c anaconda scikit-learn
conda install matplotlib

but after I had still these warnings

2020-02-23 13:31:44.910213: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found

2020-02-23 13:31:44.925815: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll

2020-02-23 13:31:44.941384: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll

2020-02-23 13:31:44.947427: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll

2020-02-23 13:31:44.965893: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll

2020-02-23 13:31:44.982990: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll

2020-02-23 13:31:44.990036: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudnn64_7.dll'; dlerror: cudnn64_7.dll not found

How I solved the first warning : I just download a zip file wich contained all the cudnn files (dll, etc) here : https://developer.nvidia.com/cudnn

How I solved the second warning : I looked the last missing file (cudart64_101.dll) in my virtual env created by conda and I just copy/pasted it in the same lib folder than for the .dll cudnn

score 4 · Answer 10 · edited May 18 '20 at 12:06

4

Tensorflow 2.1 works with Cuda 10.1.

If you want a quick hack:

Just download cudart64_101.dll from here. Extract the zip file and copy the cudart64_101.dll to your CUDA bin directory

Else:

Install Cuda 10.1

edited May 18 '20 at 12:06

Nicolas Gervais

33,817
13
115
143

answered Apr 30 '20 at 04:35

S Chakraborty

144
2
8

Where is located the cuda `bin` directory? – Maf Nov 18 '20 at 23:41
11

Recommending that people install an executable from an unofficial source is not a good solution. – Michael Hoffmann Nov 25 '20 at 17:14

score 3 · Answer 11 · edited Jun 20 '20 at 09:12

3

Tensorflow gpu 2.2 and 2.3 nightly

(along CUDA Toolkit 11.0 RC)

To solve the same issue as OP, I just had to find cudart64_101.dll on my disk (in my case C:\Program Files\NVIDIA Corporation\NvStreamSrv) and add it as variable environment (that is add value C:\Program Files\NVIDIA\Corporation\NvStreamSrv)cudart64_101.dll to user's environment variable Path).

edited Jun 20 '20 at 09:12

Community

1
1

answered Jun 13 '20 at 19:00

tagoma

3,896
4
38
57

score 2 · Answer 12 · answered Jul 08 '21 at 07:37

download CUDA Toolkit 11.0 RC

To solve the issue, I just find cudart64_101.dll on my disk ( C:\Program Files\NVIDIA Corporation\NvStreamSrv) and add it as variable environment that is add value (C:\Program Files\NVIDIA\Corporation\NvStreamSrv)cudart64_101.dll to user's environment variable Path).

score 2 · Answer 13 · answered May 03 '22 at 18:13

2

This could be caused by the version of python you are running as well, I was using the python 3.7 from the microsoft store and I run into this error, switching to python 3.10 fixed it.

answered May 03 '22 at 18:13

Cash Lo

1,052
1
8
20

I suspect the issue was the fact that you used the Microsoft store python. I vaguely remember something about it having some issues – GPhilo May 03 '22 at 18:17
Yes, I this answer just now https://stackoverflow.com/questions/60636785/tensorflow-not-recognising-cudart64-101-dll/63547977#63547977 but I remember having issue with another version of python that was from the offical site, caused by the change of how dll are loaded in 3.8 – Cash Lo May 03 '22 at 18:23

score 1 · Answer 14 · answered Jun 23 '20 at 05:13

1

Was able to fix the issue by updating NVIDIA device drivers to the latest (v446.14). NVIDIA drivers download link here.

answered Jun 23 '20 at 05:13

razmik

199
1
7

score 1 · Answer 15 · answered Aug 03 '21 at 15:53

I ran into this problem when mixing pip & conda to get tensorflow 2.3 installed. (I used pip to install tensorflow 2.3 b/c at the time conda's install of tensorflow 2.3 was broken.)

I ended up with the incorrect versions of cudatoolkit and cudnn installed.

To solve the problem, I simply did conda install with specific versions of cudatoolkit and cuda specified.

Look at https://www.tensorflow.org/install/source_windows?force_isolation=true#tested_build_configurations for info on tensorflow, cudatoolkit, and cuda versions that should work together.

Tran Quoc Bao · Answer 16 · 2022-09-13T07:46:12.480

This is just a Warning and Information message that CUDA libraries cannot be found.

If you are using NVIDIA GPU, you can refer to how to install the missing files.

If you don't use NVIDIA GPU, or simply want to ignore the I and W messages, you can add the 2 lines below at the beginning of your code:

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'

You can see more about TF_CPP_MIN_LOG_LEVEL at TensorFlow logging.

Mohamed Fathallah · Answer 17 · 2022-09-14T03:33:23.007

I had this error here with tensorboard , it happen after I update the GPU driver the thing is i was running tensorboard from the cmd where I didn't install any CUDA since I was using anaconda when you install TensorFlow using anaconda all the required cuda and cudnn are downloaded for you you will miss the files if you didn't use anaconda env where you install TensorFlow in it

Solution

* just open tensoboard from anaconda

1-or just download the last Cuda toolkit and add it to paths

https://developer.nvidia.com/cuda-toolkit-archive

and use this 2-conda install -c anaconda cudatoolkit

3-then restart your pc

score 0 · Answer 18 · answered Nov 09 '22 at 05:59

I too have faced similar issues and realized that the issue was with CUDA and CUDNN version mismatch.

Can refer here for the proper versions. From the reference below for TensorFlow 2.4.0 it is recommended to use CUDA 11.0 and cuDNN 8.0.

Or you can refer here to download cuDNN for suitable CUDA.

score -4 · Answer 19 · edited Feb 05 '21 at 15:20

-4

A simpler way would be to create a link called cudart64_101.dll to point to cudart64_102.dll. This is not very orthodox but since TensorFlow is looking for cudart64_101.dll exported symbols and the nvidia folks are not amateurs, they would most likely not remove symbols from 101 to 102. It works, based on this assumption (mileage may vary).

edited Feb 05 '21 at 15:20

Nerxis

3,452
2
23
39

answered Feb 26 '20 at 23:26

Dan Marinescu

111
5

1

This is a terrible idea, CUDA minor versions (i.e., 10.1, 10.2, etc) are not compatible with each other. Linking to 10.2 "as if" it was 10.1 will cause random crashes (that's assuming the DLL loads at all). Besides, this implies you installed cuda 10.2 on your machine, which for a CPU-only installation is not necessary – GPhilo Feb 27 '20 at 01:45
of course it is not necessary for cpu only. as far as 10.2 being completely different than 10.1 and random crashes, i would recommend you learn about dynamic linking and exported symbols – Dan Marinescu Feb 28 '20 at 02:02
the dll loads and works just fine (for your information) – Dan Marinescu Feb 28 '20 at 02:03
but end users should install prescribed dynamic libraries, especially if they are not sure what this is all about – Dan Marinescu Feb 28 '20 at 02:04
1

"of course it's not necessary for CPU-only": read again the question title. – GPhilo Feb 28 '20 at 02:19

Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

19 Answers19

Tensorflow 2.1+

What's going on?

Should I worry? How do I fix it?

Tensorflow 1.X or 2.0:

Solution

Tensorflow gpu 2.2 and 2.3 nightly

Solution

* just open tensoboard from anaconda

1-or just download the last Cuda toolkit and add it to paths

Linked

Related