RSTP Local inference running slow/jerky

Pete_Olesen · November 26, 2024, 4:26am

I am a newbie, so doing my first object detection project (traffic counting). Running inference on a Windows 10 machine with an Nvidia RTX 3060 GPU. I am running Python 3.10.15 and I use Conda to manage the environment. I have built a model and connected to an RSTP feed from a Reolink NVR. I got it to work but it is definitely herky jerky.

I have attempted to change the camera settings to a lower bit rate and resolution. When I view the camera via the Reolink NVR, it runs smoothly. I have a POE camera that is connected via a 1GB wireless network. The video streaming through the NVR is smooth.

I believe I have the 3060 RTX working correctly. It took me a while, but I got CUDA 12 and cuDnn 9 working.

I know there will be some overhead with inference, but wondering if there is something I can do in the Python code to smooth things out. The herky jerky is OK, as long as I get an accurate traffic count, but if I can run smoother, that would be better.

Thanks very much. This is my first programming project, and first with a webcam.

brad · November 27, 2024, 2:26am

If you run nvidia-smi while it’s going is your GPU being utilized?

Pete_Olesen · December 1, 2024, 8:15pm

Brad, Thanks for your help. When I run this, it is showing no memory usage on the GPU. As I read the documentation, it looks like I need to install Docker. I am not too familiar with that.

I am not sure how I need to modify my code to make that happen, but will read some more.

Grzegorz · December 2, 2024, 1:18pm

Hi @Pete_Olesen!

I’d love to help you solving this issue, many thanks for providing all details you already shared.

You mentioned you created conda environment, so I assume you are running directly on host (not in docker container)

Can you confirm if you have installed inference-gpu?

I created very simple script which is loading on my GPU. I have CUDA 12 in my local setup (however I’m running on Linux, not on Windows).
I created my venv like below:

python3.10 -m venv venv
source venv/bin/activate
pip install inference-gpu

Can you share if you see any warnings when running this script?

from typing import Any, Dict

import cv2 as cv
import supervision as sv

from inference.core.interfaces.stream.inference_pipeline import
InferencePipeline


box_annotator = sv.BoundingBoxAnnotator()


def custom_sink(prediction: Dict[str, Any], video_frame) -> None:
     detections = sv.Detections.from_inference(prediction)
     annotated_frame = box_annotator.annotate(
         video_frame.image.copy(), detections=detections
     )
     cv.imshow("", annotated_frame)
     cv.waitKey(1)


inference_pipeline = InferencePipeline.init(
     video_reference=0,
     model_id="yolov8n-640",
     on_prediction=custom_sink,
)

inference_pipeline.start()
inference_pipeline.join()

Please run above script like below:

ONNXRUNTIME_EXECUTION_PROVIDERS='[CUDAExecutionProvider]' python /path/to/this_script.py

Many thanks!
Grzegorz

Pete_Olesen · December 3, 2024, 2:10pm

Grzegorz,

Thanks very much. I will take a look. It may be a few days before I can try this. I work a day job and getting back from Thanksgiving and playing catch up. Thanks again for your help.

Pete_Olesen · December 5, 2024, 4:25am

Grzegorz,

I did some more work in this and I think I made some changes. I did not do the venv approach, but instead I am using my Conda environment. However, I did update inference with pip install --upgrade inference. This did install some addtional things and made one of the error messages go away.

I did run the second block of code and these are the warnings I am getting:

PS C:\Users\Owner> & D:/AnacondaReinstall/envs/roboflow_env3/python.exe c:/Users/Owner/RoboflowHelp1.py
SupervisionWarnings: BoundingBoxAnnotator is deprecated: BoundingBoxAnnotator is deprecated and has been renamed to BoxAnnotator. BoundingBoxAnnotator will be removed in supervision-0.26.0.
UserWarning: Specified provider ‘CUDAExecutionProvider’ is not in available provider names.Available providers: ‘AzureExecutionProvider, CPUExecutionProvider’
UserWarning: Specified provider ‘OpenVINOExecutionProvider’ is not in available provider names.Available providers: ‘AzureExecutionProvider, CPUExecutionProvider’
UserWarning: Specified provider ‘CoreMLExecutionProvider’ is not in available provider names.Available providers: ‘AzureExecutionProvider, CPUExecutionProvider’

It looks like the CUDA execution provider is not available. Not sure how I can fix this.

I asked Chat GPT and it suggested I run nvidia-smi. It looks like the GPU is recognized.

Anway, It seems you are on the right track. I was unsure about the ONYXRUNTIME_EXECUTION_PROVIDERS… script. Can include this line in the other PY files instead of running separately?

Thanks for all of your assistance. I think we are close, just haven’t found the right answer yet.

Grzegorz · December 5, 2024, 12:04pm

Hi @Pete_Olesen ,

Can you share result of pip freeze and uname -a?
Can you also install nvidia-ml-py in your environment, and share results of below:

python -c "from pynvml import nvmlInit, nvmlDeviceGetHandleByIndex, nvmlDeviceGetName;handle = nvmlDeviceGetHandleByIndex(0);print(nvmlDeviceGetName(handle))"

This will help us confirm if we are running inference at the right version, and hopefully that CUDA can be loaded from your environment.

Thanks,
Grzegorz

Pete_Olesen · December 6, 2024, 1:42am

Thanks. I ran the pip install from the conda prompt. Here are the results. Thanks for your diligence.

(base) C:\Users\Owner>conda activate roboflow_env3

(roboflow_env3) C:\Users\Owner>pip install nvidea-ml-py
ERROR: Could not find a version that satisfies the requirement nvidea-ml-py (from versions: none)
ERROR: No matching distribution found for nvidea-ml-py

(roboflow_env3) C:\Users\Owner>pip install nvidia-ml-py
Collecting nvidia-ml-py
Downloading nvidia_ml_py-12.560.30-py3-none-any.whl.metadata (8.6 kB)
Downloading nvidia_ml_py-12.560.30-py3-none-any.whl (40 kB)
Installing collected packages: nvidia-ml-py
Successfully installed nvidia-ml-py-12.560.30

(roboflow_env3) C:\Users\Owner>python -c “from pynvml import nvmlInit, nvmlDeviceGetHandleByIndex, nvmlDeviceGetName;handle = nvmlDeviceGetHandleByIndex(0);print(nvmlDeviceGetName(handle))”
Traceback (most recent call last):
File “”, line 1, in
File “D:\AnacondaReinstall\envs\roboflow_env3\lib\site-packages\pynvml.py”, line 2435, in nvmlDeviceGetHandleByIndex
fn = _nvmlGetFunctionPointer(“nvmlDeviceGetHandleByIndex_v2”)
File “D:\AnacondaReinstall\envs\roboflow_env3\lib\site-packages\pynvml.py”, line 994, in _nvmlGetFunctionPointer
raise NVMLError(NVML_ERROR_UNINITIALIZED)
pynvml.NVMLError_Uninitialized: Uninitialized

(roboflow_env3) C:\Users\Owner>

Grzegorz · December 6, 2024, 1:31pm

Hi @Pete_Olesen ,

Thanks for running the script. Please accept my apologies, I forgot to include call to nvmlInit(), can you run below corrected script?

python -c "from pynvml import nvmlInit, nvmlDeviceGetHandleByIndex, nvmlDeviceGetName;nvmlInit();handle = nvmlDeviceGetHandleByIndex(0);print(nvmlDeviceGetName(handle))"

Thanks,
Grzegorz

Pete_Olesen · December 7, 2024, 12:39am

Gregorz,

No apologies needed! Here is what it returned:

(base) C:\Users\Owner>conda activate roboflow_env3

(roboflow_env3) C:\Users\Owner>python -c “from pynvml import nvmlInit, nvmlDeviceGetHandleByIndex, nvmlDeviceGetName;nvmlInit();handle = nvmlDeviceGetHandleByIndex(0);print(nvmlDeviceGetName(handle))”
NVIDIA GeForce RTX 3060

(roboflow_env3) C:\Users\Owner>

I am thinking there may be some conflict with some of the versions of CUDA and cdnn ? that I have installed. I did some more Q&A with Chat GPT and it suggested I install Pytorch. I ran a script after I installed it to see if Pytorch could see the GPU and it now could, but that was only after I installed something related to CUDA 12.4. I also have CUDA 12.6 installed on this machine.

Below is what I can see from conda list for this environment. It is confusing to me the relationship between what you install at the machine level in Windows via downloads from Nvidia and what is installed in each environment.

Thanks for all of your help.

(roboflow_env3) C:\Users\Owner>conda list

packages in environment at D:\AnacondaReinstall\envs\roboflow_env3:

(roboflow_env3) C:\Users\Owner>

Grzegorz · December 7, 2024, 8:03pm

Hi Pete,

It seems you have both inference and inference-gpu installed, can you try uninstalling inference (and maybe updating inference-gpu). I also see you have onnxruntime-gpu as well as onnxruntime - I only have onnxruntime-gpu in my venv.

For your reference here are my dependencies that I have after building fresh venv with inference-gpu:

aiofiles==24.1.0
aiohappyeyeballs==2.4.4
aiohttp==3.10.11
aiohttp-retry==2.8.3
aioice==0.9.0
aiortc==1.9.0
aiosignal==1.3.1
aiosqlite==0.20.0
annotated-types==0.7.0
anthropic==0.34.2
anyio==4.7.0
APScheduler==3.11.0
asyncua==1.1.5
attrs==24.2.0
av==12.3.0
backoff==2.2.1
boto3==1.35.60
botocore==1.35.76
certifi==2024.8.30
cffi==1.17.1
charset-normalizer==3.4.0
click==8.1.7
coloredlogs==15.0.1
commonmark==0.9.1
contourpy==1.3.1
cryptography==44.0.0
cycler==0.12.1
Cython==3.0.11
dataclasses-json==0.6.7
defusedxml==0.7.1
distro==1.9.0
dnspython==2.7.0
docker==7.1.0
fastapi==0.110.3
filelock==3.16.1
flatbuffers==24.3.25
fonttools==4.55.2
frozenlist==1.5.0
fsspec==2024.10.0
google-crc32c==1.6.0
h11==0.14.0
httpcore==1.0.7
httpx==0.25.2
huggingface-hub==0.26.5
humanfriendly==10.0
idna==3.10
ifaddr==0.2.0
imageio==2.36.1
inference-gpu==0.29.2
iniconfig==2.0.0
jiter==0.8.0
jmespath==1.0.1
kiwisolver==1.4.7
lazy_loader==0.4
marshmallow==3.23.1
matplotlib==3.9.3
mpmath==1.3.0
multidict==6.1.0
mypy-extensions==1.0.0
networkx==3.4.2
numpy==1.26.4
nvidia-ml-py==12.560.30
onnxruntime-gpu==1.19.2
openai==1.57.0
opencv-python==4.10.0.84
opencv-python-headless==4.10.0.84
packaging==24.2
pandas==2.2.3
pillow==10.4.0
pluggy==1.5.0
prometheus-fastapi-instrumentator==6.0.0
prometheus_client==0.21.1
propcache==0.2.1
protobuf==5.29.1
py-cpuinfo==9.0.0
pybase64==1.0.2
pycparser==2.22
pydantic==2.10.3
pydantic-settings==2.6.1
pydantic_core==2.27.1
pydot==2.0.0
pyee==12.1.1
Pygments==2.18.0
PyJWT==2.10.1
pylibsrtp==0.10.0
pyOpenSSL==24.3.0
pyparsing==3.2.0
pytest==8.3.4
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
pytz==2024.2
PyYAML==6.0.2
redis==5.0.8
requests==2.32.3
requests-file==2.1.0
requests-toolbelt==1.0.0
rich==13.0.1
s3transfer==0.10.4
scikit-image==0.24.0
scipy==1.14.1
setuptools==75.6.0
shapely==2.0.6
shellingham==1.5.4
six==1.17.0
slack_sdk==3.33.5
sniffio==1.3.1
sortedcontainers==2.4.0
starlette==0.37.2
structlog==24.4.0
supervision==0.22.0
sympy==1.13.3
tifffile==2024.9.20
tldextract==5.1.3
tokenizers==0.20.3
tqdm==4.67.1
twilio==9.3.8
typer==0.12.5
typing-inspect==0.9.0
typing_extensions==4.12.2
tzdata==2024.2
tzlocal==5.2
urllib3==2.2.3
wheel==0.45.0
yarl==1.18.3
zxing-cpp==2.2.0

Maybe it would be easier to build fresh conda environment and test there?

Hope this helps,
Grzegorz

Pete_Olesen · December 8, 2024, 4:15am

Grzegorz,

I took your advice and created a new python 3.10 environment using conda. I am getting a similar package list as your. Please see below:

(roboflow) C:\Users\Owner>conda list

packages in environment at D:\AnacondaReinstall\envs\roboflow:

(roboflow) C:\Users\Owner>

However, rerunning the VS code getting some errors still. It seems that it detects CUDA as a provider, but still having difficulties. Here are the errors I am seeing. I am running CUDA 12.6 and cuDnn 9.6. Based on the Onyx requirements, this should work. Very perplexing.

PS C:\Users\Owner> & D:/AnacondaReinstall/envs/roboflow/python.exe “c:/Users/Owner/Roboflow Test 20241123 RSTP.py”
UserWarning: Specified provider ‘OpenVINOExecutionProvider’ is not in available provider names.Available providers: ‘TensorrtExecutionProvider, CUDAExecutionProvider, CPUExecutionProvider’
UserWarning: Specified provider ‘CoreMLExecutionProvider’ is not in available provider names.Available providers: ‘TensorrtExecutionProvider, CUDAExecutionProvider, CPUExecutionProvider’
2024-12-07 21:11:52.7607413 [E:onnxruntime:Default, provider_bridge_ort.cc:1992 onnxruntime::TryGetProviderInfo_CUDA] D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1637 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 “” when trying to load “D:\AnacondaReinstall\envs\roboflow\lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll”

2024-12-07 21:11:52.7752091 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:965 onnxruntime::python::CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Require cuDNN 9.* and CUDA 12.*, and
the latest MSVC runtime. Please install all dependencies as mentioned in the GPU requirements page (NVIDIA - CUDA | onnxruntime), make sure they’re in the PATH, and that your GPU is supported.

peter_roboflow · December 8, 2024, 4:37am

@Pete_Olesen do you have the latest Visual C++ runtime installed? Latest supported Visual C++ Redistributable downloads | Microsoft Learn

See this thread for more info: Cant get GPU to work with ONNX Runtime 1.19 Cuda 12.6 CuDNN 9 RTXA4000 · Issue #21825 · microsoft/onnxruntime · GitHub

Maybe try pip install msvc-runtime==14.40.33807 in your environment if that doesn’t work?

Grzegorz · December 8, 2024, 9:18pm

Hi @Pete_Olesen ,

In addition to what @peter_roboflow suggested, I guess you already have your PATH updated with paths to CUDA_INSTALL_DIR/bin, CUDA_INSTALL_DIR/libnvvp and cuDNN_INSTALL_DIR/bin?

We can also try to capture more verbose logs from onnx by setting ORT_LOG_LEVEL=VERBOSE and running your script

Thanks,
Grzegorz

Automatez · December 9, 2024, 7:21pm

I am having a similar issue. I’ve worked through most of the steps on this page (and about 100 other sites ). The only difference right now is I’m not in a venv for this. I was, but then certain global things had to be updated anyways so I just stayed global as I worked through this. My current error is below. I have had other errors come and go but this one will not go away. (GPU is NVIDIA GeForce GTX 1660 Ti). I’m not wanting to derail this current solve for @Pete_Olesen. Just wanted to chime in that I’m following the discussion with a similar issue.

UserWarning: Specified provider ‘OpenVINOExecutionProvider’ is not in available provider names.Available providers: ‘TensorrtExecutionProvider, CUDAExecutionProvider, CPUExecutionProvider’
UserWarning: Specified provider ‘CoreMLExecutionProvider’ is not in available provider names.Available providers: ‘TensorrtExecutionProvider, CUDAExecutionProvider, CPUExecutionProvider’

peter_roboflow · December 9, 2024, 7:39pm

These warnings are a bit misleading, but they are not actually a problem for you

OpenVINO is only available if you install it (it’s an optimized Intel runtime) and same with CoreML (an optimized Apple silicon runtime)

So both of those providers not being available is ok – it’ll just select CUDAExecutionProvider which is the one using your GPU

Grzegorz · December 9, 2024, 8:50pm

Hi @Automatez ,

To add on top of @peter_roboflow explanation, you can narrow down list of available ONNX execution providers by setting ONNXRUNTIME_EXECUTION_PROVIDERS='[CUDAExecutionProvider]' as mentioned in one of replies above. Setting this env variable should result in those warnings to go away.

Can you share whole log?

Hope this helps,
Grzegorz

Automatez · December 9, 2024, 10:29pm

Update on my end - following some guidance in one of those links, I looked at the NVIDIA matrix for CUDA version use. It said on my “older” GPU generation, I should be using CUDA 11.8. So I uninstalled 12.6 and loaded that up and now I have the below errors.

These errors seem to indicate two things. First, that maybe I need to be on a 12.x CUDA. Now, the matrix does say “for best performance” go to 11.8 for my GPU, so maybe I don’t have to be on 11.8 to make it work, it just might not run as fast?

And the onnx piece complicates things because their site (in the error log) only gives options for CUDA 11.8 with cuDNN <= 8.x. But the NVIDIA site said I can go 9.x on cuDNN for 11.8 CUDA. So do I just need to back cuDNN down to 8.x to get onnx to work? (so much going on!)

https://docs.nvidia.com/deeplearning/cudnn/latest/reference/support-matrix.html

UserWarning: Specified provider ‘OpenVINOExecutionProvider’ is not in available provider names.Available providers: ‘TensorrtExecutionProvider, CUDAExecutionProvider, CPUExecutionProvider’
UserWarning: Specified provider ‘CoreMLExecutionProvider’ is not in available provider names.Available providers: ‘TensorrtExecutionProvider, CUDAExecutionProvider, CPUExecutionProvider’
2024-12-09 16:17:11.2470127 [E:onnxruntime:Default, provider_bridge_ort.cc:1992 onnxruntime::TryGetProviderInfo_CUDA] D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1637 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 “” when trying to load “C:\Users\zacra\anaconda3\lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll”

2024-12-09 16:17:11.2573155 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:965 onnxruntime::python::CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Require cuDNN 9.* and CUDA 12.*, and the latest MSVC runtime. Please install all dependencies as mentioned in the GPU requirements page (NVIDIA - CUDA | onnxruntime), make sure they’re in the PATH, and that your GPU is supported.

THEN
When I set the env variable as you said, the errors do reduce to the below.

I set it via 2 lines in my python file:
import os
os.environ[“ONNXRUNTIME_EXECUTION_PROVIDERS”] = ‘[“CUDAExecutionProvider”]’

UserWarning: Specified provider ‘“CUDAExecutionProvider”’ is not in available provider names.Available providers: ‘TensorrtExecutionProvider, CUDAExecutionProvider, CPUExecutionProvider’
*************** EP Error ***************
EP Error Unknown Provider Type: “CUDAExecutionProvider” when using [‘“CUDAExecutionProvider”’]
Falling back to [‘CPUExecutionProvider’] and retrying.

AND THEN

When I changed the provider line to below, I got those new errors (further below)

os.environ[“ONNXRUNTIME_EXECUTION_PROVIDERS”] = ‘[CUDAExecutionProvider]’

Result:
2024-12-09 16:44:39.7958310 [E:onnxruntime:Default, provider_bridge_ort.cc:1992 onnxruntime::TryGetProviderInfo_CUDA] D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1637 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 “” when trying to load “C:\Users\zacra\anaconda3\lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll”

2024-12-09 16:44:39.8053160 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:965 onnxruntime::python::CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Require cuDNN 9.* and CUDA 12.*, and the latest MSVC runtime. Please install all dependencies as mentioned in the GPU requirements page (onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they’re in the PATH, and that your GPU is supported.

peter_roboflow · December 9, 2024, 11:25pm

@Automatez please follow the steps here to reinstall onnxruntime-gpu for CUDA 11 Install ONNX Runtime | onnxruntime

Automatez · December 10, 2024, 3:54am

Tried these two lines and got Requirement already satisfied for everything. Rats.

# For Cuda 11.x, please use the following instructions to install from ORT Azure Devops Feed for 1.19.2 or later

pip install flatbuffers numpy packaging protobuf sympy
pip install onnxruntime-gpu --index-url xxxxx://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-11/pypi/simple/

And I was hoping there’d be a common fix with @Pete_Olesen but maybe what I’m trying to run is part of the problem so I dropped my short script below in case that sheds any light on things. (Really appreciate the help here, btw!)

# xxxxx://inference.roboflow.com/quickstart/run_a_model/#install-inference

import os

# Set the ONNX Runtime execution providers

os.environ[“ONNXRUNTIME_EXECUTION_PROVIDERS”] = ‘[CUDAExecutionProvider]’

os.environ[“ORT_LOG_LEVEL”] = “VERBOSE”

# import a utility function for loading Roboflow models

from inference import get_model

# define the image url to use for inference

image = “https://media.roboflow.com/inference/people-walking.jpg”

# load a pre-trained yolov8n model

model = get_model(model_id=“yolov8n-640”)

# run inference on our chosen image, image can be a url, a numpy array, a PIL image, etc.

results = model.infer(image)

Topic		Replies	Views
Error deploying custom trained model on NVIDIA Jetson Xavier 🤝 Community Help	25	1948	April 17, 2024
Need Support 🤝 Community Help bugs	8	87	March 11, 2025
Deploy model from Roboflow Universe platform 🤝 Community Help bugs	6	654	May 7, 2024
Inference on Jetson (Jetpack 4.x, 5.x) 🤝 Community Help	3	141	January 28, 2025
Tensorrt converted weights not working with supervision 🤝 Community Help bugs	27	370	July 3, 2024

RSTP Local inference running slow/jerky

packages in environment at D:\AnacondaReinstall\envs\roboflow_env3:

Name Version Build Channel

packages in environment at D:\AnacondaReinstall\envs\roboflow:

Name Version Build Channel

RSTP Local inference running slow/jerky

packages in environment at D:\AnacondaReinstall\envs\roboflow_env3:

Name Version Build Channel

packages in environment at D:\AnacondaReinstall\envs\roboflow:

Name Version Build Channel

Related topics