1

Im currently running my LSTM training script on a CPU-only setup (thats right no GPU) and i get a large amount of lines between every training step. What to do about these 'dlerror's and how to solve it?

Does this affect the performance? If not, how to hide it?

Epoch 1/20
2020-09-02 20:49:06.592450: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-09-02 20:49:06.599065: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2020-09-02 20:49:12.746036: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
2020-09-02 20:49:12.751769: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-09-02 20:49:12.761444: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: DESKTOP-6NCF44I
2020-09-02 20:49:12.763713: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: DESKTOP-6NCF44I
2020-09-02 20:49:12.800139: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x229365115b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-09-02 20:49:12.802774: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-09-02 20:49:16.012887: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started.
  1/369 [..............................] - ETA: 0s - loss: 115175354838024192.0000 - mse: 116338741219426304.0000 - mae: 174372896.0000 - mape: 65089616.0000 - cosine_similarity: -0.0534
2020-09-02 20:49:06.592450: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-09-02 20:49:06.599065: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2020-09-02 20:49:12.746036: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
2020-09-02 20:49:12.751769: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-09-02 20:49:12.761444: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: DESKTOP-6NCF44I
2020-09-02 20:49:12.763713: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: DESKTOP-6NCF44I
2020-09-02 20:49:12.800139: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x229365115b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-09-02 20:49:12.802774: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-09-02 20:49:16.012887: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started.
2020-09-02 20:50:39.226962: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started.

2020-09-02 20:49:06.592450: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-09-02 20:49:06.599065: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2020-09-02 20:49:12.746036: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
2020-09-02 20:49:12.751769: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-09-02 20:49:12.761444: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: DESKTOP-6NCF44I
2020-09-02 20:49:12.763713: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: DESKTOP-6NCF44I
2020-09-02 20:49:12.800139: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x229365115b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-09-02 20:49:12.802774: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-09-02 20:49:16.012887: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started.
2020-09-02 20:50:39.226962: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started.
2020-09-02 20:51:55.732652: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: C:\Users\Ben\Desktop\colab_comparison\logs.txt\train\plugins\profile\2020_09_02_18_51_53

2020-09-02 20:49:06.592450: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-09-02 20:49:06.599065: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2020-09-02 20:49:12.746036: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
2020-09-02 20:49:12.751769: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-09-02 20:49:12.761444: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: DESKTOP-6NCF44I
2020-09-02 20:49:12.763713: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: DESKTOP-6NCF44I
2020-09-02 20:49:12.800139: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x229365115b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-09-02 20:49:12.802774: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-09-02 20:49:16.012887: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started.
2020-09-02 20:50:39.226962: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started.
2020-09-02 20:51:55.732652: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: C:\Users\Ben\Desktop\colab_comparison\logs.txt\train\plugins\profile\2020_09_02_18_51_53
2020-09-02 20:51:56.905231: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to C:\Users\Ben\Desktop\colab_comparison\logs.txt\train\plugins\profile\2020_09_02_18_51_53\DESKTOP-6NCF44I.trace.json.gz
  2/369 [..............................] - ETA: 4:00:34 - loss: 108831026716868608.0000 - mse: 109930332186214400.0000 - mae: 169439408.0000 - mape: 116326512.0000 - cosine_similarity: -0.2260
2020-09-02 20:49:06.592450: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-09-02 20:49:06.599065: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2020-09-02 20:49:12.746036: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
2020-09-02 20:49:12.751769: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-09-02 20:49:12.761444: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: DESKTOP-6NCF44I
2020-09-02 20:49:12.763713: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: DESKTOP-6NCF44I
2020-09-02 20:49:12.800139: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x229365115b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-09-02 20:49:12.802774: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-09-02 20:49:16.012887: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started.
2020-09-02 20:50:39.226962: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session started.
2020-09-02 20:51:55.732652: I tensorflow/core/profiler/rpc/client/save_profile.cc:168] Creating directory: C:\Users\Ben\Desktop\colab_comparison\logs.txt\train\plugins\profile\2020_09_02_18_51_53
2020-09-02 20:51:56.905231: I tensorflow/core/profiler/rpc/client/save_profile.cc:174] Dumped gzipped tool data for trace.json.gz to C:\Users\Ben\Desktop\colab_comparison\logs.txt\train\plugins\profile\2020_09_02_18_51_53\DESKTOP-6NCF44I.trace.json.gz
2020-09-02 20:51:57.700621: I tensorflow/core/profiler/utils/event_span.cc:288] Generation of step-events took 54.008 ms

2020-09-02 20:51:57.763778: I tensorflow/python/profiler/internal/profiler_wrapper.cc:87] Creating directory: C:\Users\Ben\Desktop\colab_comparison\logs.txt\train\plugins\profile\2020_09_02_18_51_53Dumped tool data for overview_page.pb to C:\Users\Ben\Desktop\colab_comparison\logs.txt\train\plugins\profile\2020_09_02_18_51_53\DESKTOP-6NCF44I.overview_page.pb
Dumped tool data for input_pipeline.pb to C:\Users\Ben\Desktop\colab_comparison\logs.txt\train\plugins\profile\2020_09_02_18_51_53\DESKTOP-6NCF44I.input_pipeline.pb
Dumped tool data for tensorflow_stats.pb to C:\Users\Ben\Desktop\colab_comparison\logs.txt\train\plugins\profile\2020_09_02_18_51_53\DESKTOP-6NCF44I.tensorflow_stats.pb
Dumped tool data for kernel_stats.pb to C:\Users\Ben\Desktop\colab_comparison\logs.txt\train\plugins\profile\2020_09_02_18_51_53\DESKTOP-6NCF44I.kernel_stats.pb

  4/369 [..............................] - ETA: 5:43:42 - loss: 99759720780267520.0000 - mse: 100767406007255040.0000 - mae: 139372400.0000 - mape: 186370256.0000 - cosine_similarity: -0.2431

I already looked into the other questions but couldnt find a practical solution.

talonmies
  • 70,661
  • 34
  • 192
  • 269
La-Li-Lu-Le-Lo
  • 91
  • 1
  • 7

2 Answers2

2

Technically, it's not an error, it's a warning. It's mainly just verbose, so you can safely ignore it. To silence it, do this in the correct order. You will also need to restart your kernel for now.

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'

import tensorflow as tf
Nicolas Gervais
  • 33,817
  • 13
  • 115
  • 143
  • the main problem is that it creates a new loading bar at every step ( by loading bar i mean 1/369 [..............................] 2/369 [..............................] 3/369 [..............................] instead of finishing the 369. Is there a way to only stop this? – La-Li-Lu-Le-Lo Sep 02 '20 at 21:03
  • I was feeling annoyed for a year or so on this warning ! At last found the simple three-line solution ! Kudos !! – Partha D. Apr 04 '23 at 17:04
1

The line below, tells you that it could not load the library "cudart64_101.dll". This library is the runtime for CUDA, an API for parallel computing on Nvidia GPUs. Other errors mention "nvcuda.dll" which is related to CUDA too.

2020-09-02 20:49:06.592450: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found

If you look at the log you've pasted, the next line tells you not to worry about it if you aren't using GPU. Since you are not using GPU, you should not worry.

2020-09-02 20:49:06.599065: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

If you still want to hide them, I think that changing the logging level could help. Although, it could hide other important information too... I, personally, would just ignore them as they are telling me.

frangaren
  • 498
  • 3
  • 9
  • the main problem is that it creates a new loading bar at every step ( by loading bar i mean 1/369 [..............................] 2/369 [..............................] 3/369 [..............................] instead of finishing the 369. Is there a way to only stop this? – La-Li-Lu-Le-Lo Sep 02 '20 at 21:03
  • It's certainly annoying. But, I can't think of anything other than changing the logging level and hiding the warnings. – frangaren Sep 03 '20 at 11:17