RSTP Local inference running slow/jerky

[peter_roboflow]

maybe try uninstalling and then doing the correct install commands

[peter_roboflow]

also can you post what error you run into?

[Automatez]

Okay, here’s where I ended up tonight. (If we figure this out I’ll be so impressed!)

After following the onnx install for CUDA 11.x, I got the below error, seeming to indicate my onnx is too new for inference-gpu. So I uninstalled onnx
again and this time tried putting in an old onnx.

(I did notice during my uninstalls of everything, that items like numpy were located in anaconda. VS Code was opening to a conda shell by default
and I did deactivate initially but I must have restarted along the way and not deactivated again. So now I’ve got everything installed outside
the conda env. But still getting similar errors so not sure that was my issue.)

PS C:\Users\zacra\Documents\python\training\roboflow\supervision> pip install onnxruntime-gpu --index-url xxxxx://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-11/pypi/simple/
Looking in indexes: xxxxx://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-11/pypi/simple/
Collecting onnxruntime-gpu

• Downloading xxxxx://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/9c975526-0258-4aac-9e34-f8b3551decdd/pypi/download/onnxruntime-gpu/1.20.1/onnxruntime_gpu-1.20.1-cp311-cp311-win_amd64.whl (179.7 MB)*

• ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 179.7/179.7 MB 3.0 MB/s eta 0:00:00*
  

Requirement already satisfied: coloredlogs in c:\users\zacra\appdata\local\programs\python\python311\lib\site-packages (from onnxruntime-gpu) (15.0.1)
Requirement already satisfied: flatbuffers in c:\users\zacra\appdata\local\programs\python\python311\lib\site-packages (from onnxruntime-gpu) (24.3.25)
Requirement already satisfied: numpy>=1.21.6 in c:\users\zacra\appdata\local\programs\python\python311\lib\site-packages (from onnxruntime-gpu) (1.26.4)
Requirement already satisfied: packaging in c:\users\zacra\appdata\local\programs\python\python311\lib\site-packages (from onnxruntime-gpu) (24.2)
Requirement already satisfied: protobuf in c:\users\zacra\appdata\local\programs\python\python311\lib\site-packages (from onnxruntime-gpu) (5.29.1)
Requirement already satisfied: sympy in c:\users\zacra\appdata\local\programs\python\python311\lib\site-packages (from onnxruntime-gpu) (1.13.3)
*iendly>=9.1->coloredlogs->onnxruntime-gpu) (3.5.4) sympy->onnxruntime-gpu) (1.3.0) *
Installing collected packages: onnxruntime-gpu iendly>=9.1->coloredlogs->onnxruntime-
ERROR: pip’s dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
inference-gpu 0.29.2 requires onnxruntime-gpu<1.20.0,>=1.15.1, but you have onnxruntime-gpu 1.20.1 which is incompatible. rce of the following dependency confli
Successfully installed onnxruntime-gpu-1.20.1
PS C:\Users\zacra\Documents\python\training\roboflow\supervision> python sandbox2.py
*2024-12-09 22:22:20.4955084 [E:onnxruntime:Default, provider_bridge_ort.cc:1862 onnxruntime::TryGetProviderInfo_CUDA] D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1539 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary *
failed with error 126 “” when trying to load “C:\Users\zacra\AppData\Local\Programs\Python\Python311\Lib\site-packages\onnxruntime\c\onnxruntime\core\session\provider_briapi\onnxruntime_providers_cuda.dll” o load "C:\Users\zacra\AppData\Local\P

2024-12-09 22:22:20.5060190 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:993 onnxruntime::python::CreateExecutionProviderInstance] load. Failed to create CUDAExecutionProvider. Require cuDNN 8.* and CUDA 11., and the latest MSVC runtime. Please install all dependencies as mentioned in the GPU requirements page (xxxxx://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they’re in the PATH, and that your GPU is supported.

So now I tried: pip install onnxruntime-gpu==1.18.0
And that got me this whole new batch of errors:

Successfully installed onnxruntime-gpu-1.18.0
PS C:\Users\zacra\Documents\python\training\roboflow\supervision> python sandbox2.py
*2024-12-09 22:31:11.5651465 [E:onnxruntime:Default, provider_bridge_ort.cc:1744 onnxruntime::TryGetProviderInfo_CUDA] C:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1426 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary *
failed with error 126 “” when trying to load “C:\Users\zacra\AppData\Local\Programs\Python\Python311\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll”

**************** EP Error ****************
EP Error C:\a_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:866 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded. Please install the correct version of CUDA andcuDNN as mentioned in the GPU requirements page (xxxxx://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they’re in the PATH, and that your GPU is supported.

• when using [‘CUDAExecutionProvider’]*
  Falling back to [‘CUDAExecutionProvider’, ‘CPUExecutionProvider’] and retrying.

---

*2024-12-09 22:31:11.6185059 [E:onnxruntime:Default, provider_bridge_ort.cc:1744 onnxruntime::TryGetProviderInfo_CUDA] C:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1426 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary *
failed with error 126 “” when trying to load “C:\Users\zacra\AppData\Local\Programs\Python\Python311\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll”

Traceback (most recent call last):

• File “C:\Users\zacra\AppData\Local\Programs\Python\Python311\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py”, line 419, in init*
• self._create_inference_session(providers, provider_options, disabled_optimizers)*
• File “C:\Users\zacra\AppData\Local\Programs\Python\Python311\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py”, line 483, in _create_inference_session*
• sess.initialize_session(providers, provider_options, disabled_optimizers)*
  RuntimeError: C:\a_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:866 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded. Please install the correct version of CUDA andcuDNN as mentioned in the GPU requirements page (xxxxx://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they’re in the PATH, and that your GPU is supported.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

• File “C:\Users\zacra\AppData\Local\Programs\Python\Python311\Lib\site-packages\inference\core\models\roboflow.py”, line 712, in initialize_model*
• self.onnx_session = onnxruntime.InferenceSession(*

•                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^*
  

• File “C:\Users\zacra\AppData\Local\Programs\Python\Python311\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py”, line 432, in init*
• raise fallback_error from e*
• File “C:\Users\zacra\AppData\Local\Programs\Python\Python311\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py”, line 427, in init*
• self._create_inference_session(self._fallback_providers, None)*
• File “C:\Users\zacra\AppData\Local\Programs\Python\Python311\Lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py”, line 483, in _create_inference_session*
• sess.initialize_session(providers, provider_options, disabled_optimizers)*
  RuntimeError: C:\a_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:866 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded. Please install the correct version of CUDA andcuDNN as mentioned in the GPU requirements page (xxxxx://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they’re in the PATH, and that your GPU is supported.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

• File “C:\Users\zacra\Documents\python\training\roboflow\supervision\sandbox2.py”, line 19, in *
• model = get_model(model_id=“yolov8n-640”)*

•        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^*
  

• File “C:\Users\zacra\AppData\Local\Programs\Python\Python311\Lib\site-packages\inference\models\utils.py”, line 362, in get_model *
• return ROBOFLOW_MODEL_TYPES[(task, model)](model_id, api_key=api_key, *kwargs)

•       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^*
  

• File “C:\Users\zacra\AppData\Local\Programs\Python\Python311\Lib\site-packages\inference\core\models\roboflow.py”, line 611, in init*
• self.initialize_model()*
• File “C:\Users\zacra\AppData\Local\Programs\Python\Python311\Lib\site-packages\inference\core\models\roboflow.py”, line 719, in initialize_model*
• raise ModelArtefactError(*
  inference.core.exceptions.ModelArtefactError: Unable to load ONNX session. Cause: C:\a_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:866 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded. Please install the correct version of CUDA andcuDNN as mentioned in the GPU requirements page (xxxxx://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they’re in the PATH, and that your GPU is supported.

[Automatez]

Let’s go!!! It works. Thanks for all the help! Below is where I ended up, in case it can help anyone else out.

1. my GPU is NVIDIA GeForce GTX 1660 Ti (run ‘nvidia-smi’ from command line to see all this info) - this mattered for the CUDA capabilities

2. installed PyTorch version 2.3.0 - to be compatible with the CUDA 11.8 below (per xxxxx://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements)

3. installed CUDA version 11.8 - NVIDIA advised this older version for my GPU

4. installed cuDNN version 8.9.7 - so that it works for CUDA 11.x (had to unzip an archived set and manually move the files into the corresponding CUDA folders)

5. installed these dependencies for onnxruntime-gpu - flatbuffers numpy packaging protobuf sympy

6. installed onnxruntime-gpu version 1.18.0 - to match the CUDA/cuDNN combo

7. Added these lines to my script - to hide the UserWarning for missing providers
   import os
   os.environ[“ONNXRUNTIME_EXECUTION_PROVIDERS”] = ‘[CUDAExecutionProvider]’

Just about shed a tear when that image popped up on the screen after running the script. Thanks again @peter_roboflow and @Grzegorz!

[peter_roboflow]

Let’s gooooo!!!