Deploy model from Roboflow Universe platform

Community Help
1

I found a model on the Roboflow Universe platform that aligns well with my project. I wish to deploy it on my laptop and utilize the GPU to accelerate video inference. However, my laptop is an office machine, so I plan to implement this process on a cloud server like Kaggle. Yet, when I tried to obtain the model, it provided me with the following warning.

UserWarning: Specified provider ‘OpenVINOExecutionProvider’ is not in available provider names
[W:onnxruntime:Default, onnxruntime_pybind_state.cc:640 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider.

Ignoring this warning, when I attempted to infer my video, I discovered that the GPU on the cloud server was not utilized; my video was still running on the CPU. I searched through the Roboflow documentation, but found no mention of this issue.

1. I want to know if it’s officially allowed to run Roboflow’s publicly available Universe models on cloud servers or on one’s own GPU device.
2. If it is permitted, how can this issue be resolved?

Thank you sincerely for taking the time to read through my question. Regardless of whether you can provide assistance, I wish you a fulfilling life and smooth sailing ahead.

2

Hey @ChianWin

Yes, you can run Universe models locally on-device using our open-source Inference package.

To accelerate inference using a NVIDIA GPU (which are the only types of accelerators supported at the time of writing), you need to install inference-gpu with pip: pip install inference-gpu

See the ONNX GPU runtime section of the Inference docs for more info on the installation process

Hope this helps

1 Like
3

Thank you very much for your response. In fact, I followed the documentation’s suggestion to install inference-gpu using pip install inference-gpu . However, the warning mentioned above and the issue of the GPU not being utilized still persist.

4

Just to provide further context, I referred to the following link Track Objectsto run model inference on videos. The example provided in that link demonstrates using both Universe models and the YOLO official models for video inference. Interestingly, when I ran the latter model on the cloud platform, my GPU was properly utilized.

5

Hey @ChianWin

I did some searching online regarding your error message and came across these two posts: https://stackoverflow.com/questions/75267445/why-does-onnxruntime-fail-to-create-cudaexecutionprovider-in-linuxubuntu-20/75267493#75267493
FAQ/GUIDE: I've figured out why CUDA isn't available in Insightface! · Issue #238 · cubiq/ComfyUI_IPAdapter_plus · GitHub

Could you try importing PyTorch import torch to see if that resolves your issue? If not, try uninstalling onnxruntime-gpu and reinstalling it.

6

As I mentioned in my previous response, when I ran the YOLO official model for video inference, my GPU was indeed utilized for acceleration. Additionally, when I used a model I trained myself, the GPU was also effectively invoked. It seems the issue might not lie with the configuration of my GPU. Of course, I’ve also attempted to configure the GPU as per the article you provided, but it seems none of these attempts have been successful so far. Here are the steps I’ve tried:

  1. First, I imported torch and then onnxruntime.
  2. I uninstalled and reinstalled onnxruntime-gpu to ensure its compatibility with the CUDA version.
7

Hey @ChianWin

Could you please try the following steps so we can determine if CUDA is installed properly on your machine:

  • Try running nvidia-smi in your Command Line.
    • If it doesn’t run properly, it likely means that CUDA isn’t installed properly. Go to this NVIDIA page and follow the prompts.
  • If it does run properly, please sent a screenshot as well as some info about the GPU and OS on your machine.