Loaded runtime CuDNN library: 7.1.2 but source was compiled with: 7.6.0; Ubuntu 18.04

Question

I am trying to address the issue in the title:

Loaded runtime CuDNN library: 7.1.2 but source was compiled with: 7.6.0.  CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version

I have read several other posts (example: Loaded runtime CuDNN library: 5005 (compatibility version 5000) but source was compiled with 5103 (compatibility version 5100))

that basically tells me that my machine has CuDNN 7.1.2 but I need 7.6.0. The answer is then to download and install 7.6.*

the only issue is that I thought I did that by following the instructions on nvidia archive (https://developer.nvidia.com/rdp/cudnn-archive)

and if I go to /usr/local/cuda/include and read cudnn.h it shows

#if !defined(CUDNN_H_)
#define CUDNN_H_

#define CUDNN_MAJOR 7
#define CUDNN_MINOR 6
#define CUDNN_PATCHLEVEL 4

Currently I have CUDA-10.0, 10.1, and 10.2 installed with my .bashrc set to 10.0 (although nvcc --version states I have cuda 9.1 --another issue I cant seem to fix).

Any suggestions? I have been trying to tackle this for days but no luck.

UPDATE:

Here are the paths I have

export PATH=$PATH:/usr/local/cuda-10.0/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.0/lib64
export CUDA_HOME=/usr/local/cuda

Before this is closed could you help with either suggesting a proper path to set or to find old cudnn please?

You almost certainly have multiple CuDNN versions installed, and the solution will be to find and remove them, or carefully set paths so that the correct version is found. It's your system, only you know what you did and where things are to be found. — talonmies, Apr 05 '20 at 10:05
You may have a mismatch between what version of CUDA toolkit is in your `PATH` and what in your `LD_LIBRARY_PATH`. — Hristo Iliev, Apr 05 '20 at 15:37
@HristoIliev any suggestions for the proper path? I have added my current paths above. — MikeDoho, Apr 05 '20 at 16:33
@talonmies ok well I was hoping you would give a suggestion on the proper way to identify it and or all the files associated with it but I will go back online and look for how to do that — MikeDoho, Apr 05 '20 at 16:50
Use `ldd your_executable` to see the paths to all shared libraries. No one knows where you’ve installed things on your system - only you, hence no one can tell you what the correct paths are. — Hristo Iliev, Apr 05 '20 at 19:39
Also, `CUDA_HOME` is obviously different from what you add to `PATH` and `LD_LIBRARY_PATH`. Unless `/usr/local/cuda` is a symlink to `/usr/local/cuda-10.0`. Set `CUDA_HOME` first, then use it when setting `PATH` and `LD_LIBRARY_PATH`. — Hristo Iliev, Apr 05 '20 at 19:40

score 6 · Answer 1 · answered Oct 14 '20 at 06:45

I hit a very similar error:

Loaded runtime CuDNN library: 7.1.4 but source was compiled with: 7.6.5.  CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.

and tracked it down to accidentally having an older CuDNN in my ldconfig:

$ sudo ldconfig -p | grep libcudnn
    libcudnn.so.7 (libc6,x86-64) => /usr/local/cuda-9.0/lib64/libcudnn.so.7
    libcudnn.so.7 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libcudnn.so.7
    libcudnn.so (libc6,x86-64) => /usr/local/cuda-9.0/lib64/libcudnn.so
    libcudnn.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libcudnn.so

The libcudnn.so.7 file in the cuda-9.0 directory was pointing to the older version:

ls -alh /usr/local/cuda-9.0/lib64/libcudnn.so.7
lrwxrwxrwx 1 root root 17 Dec 16  2018 /usr/local/cuda-9.0/lib64/libcudnn.so.7 -> libcudnn.so.7.1.4

But I had compiled tensorflow against the newer version:

ls -alh /usr/lib/x86_64-linux-gnu/libcudnn.so.7
lrwxrwxrwx 1 root root 17 Oct 27  2019 /usr/lib/x86_64-linux-gnu/libcudnn.so.7 -> libcudnn.so.7.6.5

Since ldconfig uses /etc/ld.so.conf to determine where to look for libraries (I guess in conjunction with LD_LIBRARY_PATH), I checked it and it showed:

include /etc/ld.so.conf.d/*.conf

When I listed the files in that directory, I spotted the problem file and removed it:

$ cat /etc/ld.so.conf.d/cuda9.conf
/usr/local/cuda-9.0/lib64
$ sudo rm /etc/ld.so.conf.d/cuda9.conf

After that I re-ran ldconfig to reload the config, and then everything worked as expected and the error disappeared.

Thank you so so much man. You saved me from a bad day because of this. Really appreciate this. — PanDe, Oct 29 '21 at 07:41

Loaded runtime CuDNN library: 7.1.2 but source was compiled with: 7.6.0; Ubuntu 18.04

1 Answers1