CuDNN library compatibility error after loading model weights

Question

I am trying to load NSynth weights and I am using tf version 1.7.0

from magenta.models.nsynth import utils
from magenta.models.nsynth.wavenet import fastgen

def wavenet_encode(file_path):

 # Load the model weights.
 checkpoint_path = './wavenet-ckpt/model.ckpt-200000'

 # Load and downsample the audio.
 neural_sample_rate = 16000
 audio = utils.load_audio(file_path, 
                          sample_length=400000, 
                          sr=neural_sample_rate)

 encoding = fastgen.encode(audio, checkpoint_path, len(audio))

 # Reshape to a single sound.
 return encoding.reshape((-1, 16))

# An array of n * 16 frames. 
wavenet_z_data = wavenet_encode(file_path)

I get the following error:

tensorflow/stream_executor/cuda/cuda_dnn.cc:396] Loaded runtime CuDNN library: 7103 (compatibility version 7100) but source was compiled with 7005 (compatibility version 7000). If using a binary install, upgrade your CuDNN library to match. If building from sources, make sure the library loaded at runtime matches a compatible version specified during compile configuration.

What should I do and, which version of tf should I install, and exactly which CUDA version do I need?

score 16 · Answer 1 · answered Apr 21 '18 at 21:10

16

As the error says, the Tensorflow version you are using is compiled for CuDNN 7.0.5 while your system has CuDNN 7.1.3 installed.

As the error also suggests, you can solve this problem:

Either by installing CuDNN 7.0.5 (follow instructions here: https://developer.nvidia.com/cudnn);
Or by compiling Tensorflow yourself for your system (follow instructions here: https://www.tensorflow.org/install/install_sources).

answered Apr 21 '18 at 21:10

benjaminplanche

14,689
5
57
69

1

Instead of recompiling TensorFlow, even downgrading it works. I did this `pip install -U tensorflow==1.9.0` and the problem was solved. – paradocslover Oct 26 '19 at 20:44

score 6 · Answer 2 · edited Oct 19 '18 at 17:07

At the env:

ubuntu16.04   
cuda9.0   
cudnn7.0  
tensorflow 1.11.0  
python 3.5

I try to train the object detection with tensorflow, I meet this problem:

2018-10-18 21:31:36.796017: E tensorflow/stream_executor/cuda/cuda_dnn.cc:343] Loaded runtime CuDNN library: 7.0.5 but source was compiled with: 7.2.1.  CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
Segmentation fault (core dumped)

it's because the tensorflow version is higher;

I use pip3 install --upgrade --force-reinstall tensorflow-gpu==1.9.0 --user to fix the problem.

Thank you, it works for me. I have (1) the same error (with a different ersion), (2) upgrading cuDNN to 7.4.x does not work, (3) 7.2.1 is only for CUDA 9.2, (4) I do not want to mess with rebuild the package myself and (5) nightly version does not help. Your solution is the one that I overlooked twice before trying it but it actually works ! — BachT, Nov 27 '18 at 08:39

Falguni Das Shuvo · Answer 3 · 2020-06-27T13:12:42.433

1

benjaminplanche's answer clearly answers the question. I would like to add a bit and I hope you would find it helpful.

For installing cuDNN, cudnnenv is very convenient.

Cudnnenv is like a manager that can manage various versions of cuDNN. Installing cudnnenv is easy using pip:

pip install cudnnenv

edited Jun 27 '20 at 13:12

answered Jun 27 '20 at 13:05

Falguni Das Shuvo

66
5

score 0 · Answer 4 · answered Apr 26 '18 at 08:42

0

I suggest installing the version of cudnn that tensorflow is compiled with:

sudo apt install libcudnn7-dev=7.0.5.15-1+cuda<x> libcudnn7=7.0.5.15-1+cuda<x>

<x> symbol must be replaced with cuda version you have e.g. for cuda version 9.0 replace it with 9.0.

Later freeze the version in apt that would not automatically update:

sudo apt-mark hold libcudnn7 libcudnn7-dev

answered Apr 26 '18 at 08:42

Primoz

1,324
2
16
34

Mine was the other way around; TF was somehow compiled with the newer version. Since the default from Nvidia seems to be downloadable .deb files, no alternate versions were known to apt. So, I needed to download the newer .deb files from Nvidia. – tsbertalan Oct 08 '18 at 15:46

score 0 · Answer 5 · answered Dec 18 '19 at 22:08

0

Installing Tensorflow Nightly solved the problem for me:

pip3 install tf-nightly-gpu

answered Dec 18 '19 at 22:08

Brody Higby

137
1
8

score 0 · Answer 6 · edited Sep 18 '21 at 14:04

I removed:

  cudnn_adv_infer64_8.dll
  cudnn_adv_train64_8.dll
  cudnn_cnn_infer64_8.dll
  cudnn_cnn_train64_8.dll
  cudnn_ops_infer64_8.dll
  cudnn_ops_train64_8.dll
  cudnn64_8.dll

from

C:\Users\<username>\AppData\Local\Programs\Python\Python38\Lib\site-packages\~orch\lib

and

C:\Users\<username>\AppData\Local\Programs\Python\Python38\Lib\site-packages\torch\lib

I think tensorflow automatically loads these dlls if they exit.

I'm not sure if this causes other issues!

CuDNN library compatibility error after loading model weights

6 Answers6

Linked