How to use my GPU with inference library

🤝 Community Help
1

how can i use the inference library with my NVIDIA GPU, is there a way to explicitly initialize the model to use the CUDA device (similar to PyTorch’s device argument)

Note: i have already installed inference and inference-gpu, and i can use my GPU with PyTorch
For example how can i change this code to use my NVIDIA GPU.

from inference import get_model

image = "https://media.roboflow.com/inference/people-walking.jpg"
model = get_model(model_id="rfdetr-medium")

while True:
    results = model.infer(image)

The documentation is not clear for how i use the gpu Architecture - Roboflow Inference

Edit: infrence appears to have access to GPU, but it does not work when using the serverless API

$ inference server start 
GPU detected. Using a GPU image.
...

Edit: the execution providers appear to include CUDA

(Pdb) p model.onnxruntime_execution_providers
['CUDAExecutionProvider', 'OpenVINOExecutionProvider', 'CoreMLExecutionProvider', 'CPUExecutionProvider']
2

I figured out you need to:

model = get_model(model_id="rfdetr-medium", onnxruntime_execution_providers=["CUDAExecutionProvider"] )
1 Like
3

Hi @Kallinteris-Andreas!
Thats fantastic! Glad you were able to resolve the issue!

4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.