Why does onnxruntime fail to create CUDAExecutionProvider in Linux(Ubuntu 20)?

Question

import onnxruntime as rt
ort_session = rt.InferenceSession(
            "my_model.onnx",
            providers=["CUDAExecutionProvider"],
)

onnxruntime (onnxruntime-gpu 1.13.1) works (in Jupyter VsCode env - Python 3.8.15) well when providers is ["CPUExecutionProvider"]. But for ["CUDAExecutionProvider"] it sometimes(not always) throws an error as:

[W:onnxruntime:Default, onnxruntime_pybind_state.cc:578 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.

I tried following the provided link in the error, and tried different setups in the conda environment to test the code with various version combinations.

score 9 · Answer 1 · answered Jan 28 '23 at 12:02

9

Replacing:

import onnxruntime as rt

with

import torch
import onnxruntime as rt

somehow perfectly solved my problem.

answered Jan 28 '23 at 12:02

Oguz Hanoglu

161
1
6

1

The reason might be related to the fact that requirements include *CUDA* and *cuDNN* and these are installed within *pytorch* in *conda*. Importing torch would be enabling CUDA/cuDNN in the kernel. – Oguz Hanoglu Jan 28 '23 at 12:07
Because pytorch contains all cuda libraries. When you import pytorch, it also imports cuda libraries in memory. – RedEyed Apr 26 '23 at 10:36

peng yao · Answer 2 · 2023-05-09T02:51:01.227

You should check your onnxruntime-gpu version ,cuda version and cudnn version in this link 'https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html'
Use 'nvcc -V' and 'cat /usr/local/cuda/include/cudnn_version.h' make sure your env is useful.
'import torch' is necessary,the code as is follows:

import torch
import onnxruntime as rt
providers = ['CUDAExecutionProvider', 'CPUExecutionProvider']
session = rt.InferenceSession("yolo.onnx", providers=providers)

score 0 · Answer 3 · answered Jun 13 '23 at 09:45

0

In my case the following helped:

uninstall onnxruntime
uninstall onnxruntime-gpu
install optimum[onnxruntime-gpu]

more here

answered Jun 13 '23 at 09:45

thawro

31
2

score 0 · Answer 4 · answered Aug 20 '23 at 20:37

I had the same issue but with TensorRT TensorrtExecutionProvider:

[W:onnxruntime:Default, onnxruntime_pybind_state.cc:614 CreateExecutionProviderInstance] Failed to create TensorrtExecutionProvider. Please reference https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html#requirements to ensure all dependencies are met.

Look like in this case one have to import tensorrt and have to do so before importing onnxruntime (GPU):

import tensorrt
import onnxruntime as ort

# Inference with ONNX TensorRT

Why does onnxruntime fail to create CUDAExecutionProvider in Linux(Ubuntu 20)?

4 Answers4

Linked