Stack Overflow
Stack Overflow

Questions tagged [onnxruntime]

ONNX Runtime is a cross-platform inference and training machine-learning accelerator.

See onnxruntime github project.

292 questions
12
votes
3 answers

How do you run a ONNX model on a GPU?

I'm trying to run an ONNX model import onnxruntime as ort import onnxruntime.backend model_path = "model.onnx" #https://microsoft.github.io/onnxruntime/ ort_sess = ort.InferenceSession(model_path) print( ort.get_device() ) This prints…
djacobs7
  • 11,357
  • 3
  • 25
  • 33
7
votes
1 answer

How to multi-thread in ONNX Runtime?

Using ONNX Runtime to run inference on deep learning models. Lets say I have 4 different models, each with its own input image, can I run them in parallel in 4 threads? Would there be one "environment" and then 4 sessions (using same…
Tullhead
  • 565
  • 2
  • 7
  • 17
6
votes
1 answer

Efficient Bitmap to OnnxRuntime Tensor in C#

I am using Microsoft OnnxRuntime to detect and classify objects in images and I want to apply it to real-time video. To do that, I have to convert each frame into an OnnxRuntime Tensor. Right now I have implemented a method that takes around…
Ignacio
  • 806
  • 1
  • 10
  • 29
5
votes
1 answer

i am unable to install onnxruntime with pip3 .please resolve it

ERROR: Could not find a version that satisfies the requirement onnxruntime (from versions: none) ERROR: No matching distribution found for onnxruntime
Varsha Ladkani
  • 116
  • 1
  • 4
5
votes
1 answer

Getting a prediction from an ONNX model in python

I can't find anyone who explains to a layman how to load an onnx model into a python script, then use that model to make a prediction when fed an image. All I could find were these lines of code: sess =…
Grant Allan
  • 189
  • 1
  • 4
  • 10
5
votes
2 answers

onnxruntime inference is way slower than pytorch on GPU

I was comparing the inference times for an input using pytorch and onnxruntime and I find that onnxruntime is actually slower on GPU while being significantly faster on CPU I was tryng this on Windows 10. ONNX Runtime installed from source - ONNX…
sn710
  • 581
  • 5
  • 20
5
votes
1 answer

Running Multiple ONNX Model for Inferencing in Parallel in Python

Is there a way to run multiple ONNX models in parallel and use multiple cores available? Currently, I have trained two ONNX models and want to infer using them. I have used threading from Python but that doesn’t really use multiple cores. After that…
Abhishek Gangwar
  • 1,697
  • 3
  • 17
  • 29
5
votes
1 answer

How to know input/output layer names and sizes for Pytorch model?

I have Pytorch model.pth using Detectron2's COCO Object Detection Baselines pretrained model R50-FPN. I am trying to convert the .pth model to onnx. My code is as follows. import io import numpy as np from torch import nn import…
batuman
  • 7,066
  • 26
  • 107
  • 229
4
votes
0 answers

Onnxruntime: inference with CUDNN on GPU only working if pytorch imported first

I am trying to perform inference with the onnxruntime-gpu. Therefore, I installed CUDA, CUDNN and onnxruntime-gpu on my system, and checked that my GPU was compatible (versions listed below). When I attempt to start an inference session, I receive…
mutableVoid
  • 1,284
  • 2
  • 11
  • 29
4
votes
4 answers

Why does onnxruntime fail to create CUDAExecutionProvider in Linux(Ubuntu 20)?

import onnxruntime as rt ort_session = rt.InferenceSession( "my_model.onnx", providers=["CUDAExecutionProvider"], ) onnxruntime (onnxruntime-gpu 1.13.1) works (in Jupyter VsCode env - Python 3.8.15) well when providers is…
Oguz Hanoglu
  • 161
  • 1
  • 6
4
votes
1 answer

ONNX with custom ops from TensorFlow in Java

in order to make use of Machine Learning in Java, I'm trying to train a model in TensorFlow, save it as ONNX file and then use the file for inference in Java. While this works fine with simple models, it's getting more complicated using…
nf3lix
  • 58
  • 4
4
votes
1 answer

onnxruntime not using CUDA

Environment: CentOS 7 python 3.9.5 CUDA: 11.4 cudnn: 8.2.4 onnxruntime-gpu: 1.9.0 nvidia driver: 470.82.01 1 tesla v100 gpu while onnxruntime seems to be recognizing the gpu, when inferencesession is created, no longer does it seem to recognize…
kwagjj
  • 807
  • 1
  • 13
  • 23
4
votes
1 answer

How to use onnxruntime with .ort model in Android Studio

I'm trying to create an Android App that incorporates a Machine Learning Model. I had an onnx model, along with a Python script file, two json files with the label names, and some numpy data for mel spectrograms computation. I tried to go with…
ZookKep
  • 481
  • 5
  • 13
3
votes
0 answers

tf_rep.export_graph() gives me KeyError: 'input.1'

I've written pytorch model,than converted it to .onnx format, buit a tf representation, tried to convert it to tflite and while exporting graph error occured this is how I saved the model and converted it to .onnx: (LeNet5 is the name of the model…
Yaroslav
  • 56
  • 3
3
votes
2 answers

OnnxRuntime vs OnnxRuntime+OpenVinoEP inference time difference

I'm trying to accelerate my model's performance by converting it to OnnxRuntime. However, I'm getting weird results, when trying to measure inference time. While running only 1 iteration OnnxRuntime's CPUExecutionProvider greatly outperforms…
TBM - VOICE
  • 123
  • 1
  • 6
1
2 3
…
19 20