System setup:
Ubuntu 16.04, Tesla V100 on AWS p3-2xlarge, Nvidia driver 396.54, Cuda 9.0.176_384.81, CuDNN 9.0
Tensorflow GPU 1.9.0, Python 3.6 using pyenv
I was curious about the Google Quickdraw game and was doing some research on how they trained the model.
I followed the file at
https://github.com/tensorflow/models/blob/master/tutorials/rnn/quickdraw/train_model.py
to run the following command
python train_model.py \
--training_data train_data \
--eval_data eval_data \
--model_dir /tmp/quickdraw_model/ \
--cell_type cudnn_lstm
The training and eval data were generated using
https://github.com/tensorflow/models/blob/master/tutorials/rnn/quickdraw/create_dataset.py
and using the files here: https://console.cloud.google.com/storage/browser/quickdraw_dataset/full/simplified
Then the program stops after giving the following errors:
2019-02-01 06:41:15.770071: E tensorflow/stream_executor/cuda/cuda_dnn.cc:943] CUDNN_STATUS_EXECUTION_FAILED Failed to set dropout descriptor with state memory size: 3932160 bytes.
2019-02-01 06:41:15.770123: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at cudnn_rnn_ops.cc:1214 : Unknown: CUDNN_STATUS_EXECUTION_FAILED Failed to set dropout descriptor with state memory size: 3932160 bytes.
Traceback (most recent call last):
File "/home/ubuntu/.pyenv/versions/abc/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/ubuntu/.pyenv/versions/abc/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/ubuntu/.pyenv/versions/abc/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: CUDNN_STATUS_EXECUTION_FAILED Failed to set dropout descriptor with state memory size: 3932160 bytes.
[[Node: cudnn_lstm/CudnnRNN = CudnnRNN[T=DT_FLOAT, direction="bidirectional", dropout=0.3, input_mode="linear_input", is_training=true, rnn_mode="lstm", seed=0, seed2=0, _device="/job:localhost/replica:0/task:0/device:GPU:0"](transpose, cudnn_lstm/zeros, cudnn_lstm/zeros, cudnn_lstm/opaque_kernel/read)]]
[[Node: OptimizeLoss/clip_by_global_norm/mul_1/_239 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_354_OptimizeLoss/clip_by_global_norm/mul_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
After doing some research, it seems the error was given by calling function cudnnSetDropoutDescriptor.
https://github.com/tensorflow/tensorflow/blob/r1.9/tensorflow/stream_executor/cuda/cuda_dnn.cc#L932
After checking the API docs, it seems CUDNN_STATUS_EXECUTION_FAILED is probably caused by library bugs or wrong installation.
I checked the installation by running the mnist test, and it passed.
Btw, I also tried running the above command without the cell_type param, which means it will run on CPU. It was able to run without any problem. Also, I tried running the same thing using the following setup, and it gave the same errors.
Ubuntu 18.04, Tesla V100 on AWS p3.2xlarge, NVidia Driver 410.79, Cuda 10.0.130_410.48, CuDNN 10.0,
Tensorflow GPU 12.0/10.0, Python 3.6 using pyenv
Has anyone ever tried doing this and encountered similar problems?
