Been loving the interface, but having issues with the inference module required on Edge devices. I am working on accessing the NPU of two of my devices a RPi5 with the Hailo 8L, and an Orange Pi 5 Pro (RK3588S). For example, the Orange Pi 5 Pro has a 6TOPS NPU that can accelerate my work quite a bit vs the CPU, Yolov8 is supported, but it needs the .pt file (or ONNX) to convert into it’s own optimized format. The Hailo for RPi 5 has a similar (albeit simpler) process and from the documentation it seems that there is built in support already for Hailo via inference. For the life of me I cannot figure out how to properly get a model export going so I can get things converted for the RK3588s’ npu. There have been past forum posts asking about this but its been a while. I really want to avoid being hardware/ecosystem locked if I can avoid it.
Hi @ahillelt, to export models for NPUs like Rockchip RK3588S and Hailo 8L, convert your .pt or ONNX models into the format required by each NPU… For the RK3588S, you might need to use Rockchip’s NPU SDK or tools for model conversion.
For Hailo, use the Hailo toolbox to optimize and deploy your models.
Check their respective documentation for specific conversion steps.
Hi @ahillelt, many thanks for sharing your feedback!
If you want to use specialized model format with inference, you can achieve this by developing your custom workflows block and then use inference with workflows. In that block you would handle model loading and running image through it. Have a look how core blocks are implemented, and example of how to run workflow.
Thanks for this @Grzegorz Grzegorz, looks like if I have a ready-to-go specialized model I can find a way to introduce it into the workflow of my project on Roboflow. However, if I trained on roboflow is there any way to get that trained model out of the roboflow environment, for example as onnx or even .pt? @mitchellmill is right, Rockchip has it’s own SDK for conversion and it starts with either .pt or onnx, and it takes that to convert to their own format. I already am able to work with converting my pytorch models without issue, but I can’t figure out how to get the newly trained model out of Roboflow. Once I have it converted and optimized for the RK3588s then I will be able to introduce the model as part of the workflow. However, I need to first get the model free standing. I am trying to avoid exporting the Dataset as Yolov8 and then run the train locally, again…
In order to stay compliant with licensing requirements, fine-tuned model weights can only be used in an Inference deployment. For free and Starter Plan customers, your Roboflow license permits use of your models on one device. Enterprise customers can deploy models on multiple devices.
You need enterprise plan if you are looking to deploy models within your infrastructure:
Offline Mode: Deploy models offline, in your VPC, on-premise, and entirely within your cloud.
Got it. I just want to make sure it’s clear to me and any others that read this. As of right now unless we use the inference feature from Roboflows own platform it is not possible to take a trained model from a roboflow project and export the weights to use for whichever model we selected. The only way this is possible is if we purchase an enterprise plan, this is even true if we are not an enterprise.
I assume this means Roboflow cannot use the NPUs of various hardware out there, except for CUDA, otherwise it all runs on the CPU of whatever our edge device is?
Hi, please accept my apologies for missing your reply!
As of right now unless we use the inference feature from Roboflows own platform it is not possible to take a trained model from a roboflow project and export the weights to use for whichever model we selected. The only way this is possible is if we purchase an enterprise plan, this is even true if we are not an enterprise.
This is current state of our licensing.
I assume this means Roboflow cannot use the NPUs of various hardware out there, except for CUDA, otherwise it all runs on the CPU of whatever our edge device is?
