-
Project Type: RF-DETR Object detection.
-
Operating System & Browser: Win 10
-
Project Universe Link or Workspace/Project ID: N/A
-
**Do you grant Roboflow Support permission to access your Workspace for troubleshooting? (Yes/No): No
Hey guys, I have just started messing with RF-DETR!
I have some questions.So currently, I am using Ultralytics and YOLO.
Whenever training my models, I capture at multiple image sizes, ranging from 160 to 640, Image size is randomly selected at values divisible by 32 in that range.I then train dynamically on this same range, I do this because at run-time with my model, the dimensions are overridden to the values within this range.
Is it possible to do this with RF-DETR?
Furthermore, am I even taking the right approach? If my run-time capture size ranges from 160px to 640px, Should I be dynamically training in that range aswell as capturing my datasets images within that range or is there a better answer?
Thanks for any and all input! If you have any questions or considerations, shoot!**
@Tyler_Armistead are your original images of fixed or variable resolution? I read ācapture at multiple image sizesā as you resizing them yourself, is that correct?
The original images are at varying resolutions(160,320,480,640). I try to avoid resizing when capturing, inferring, and training.
Doing so voids all loss of data with resizing the images, and also matches what it will see at inference most closely.
I see, thanks for clarification. What do you think about āFill (with center crop) inā preprocessing on Roboflow platform? If you choose the input size for RF-DETR model to be 640x640, the preprocessing will put your original image into the middle of 640x640 frame and pad the rest with black pixels, so model will see original pixels during training and inference. The ācropā part in the name of preprocessing wonāt apply here, since all your images are smaller than 640x640.
But for this to work best at inference, I have to capture at 640, the point of having these different sizes baked into the model, is so that at run-time the dims can be overridden and save tons of resources on lower end computers.
I donāt think I fully understand.. and Iāll explain why below, but I think NAS does what you want automatically (you should feed full resolution images in, it handles maintaining resolution invariance during training automatically).
The reason this doesnāt make sense though is that running RF-DETR at higher resolution is orders of magnitude more compute intensive than resizing images is. Its runtime is ~quadratic with respect to the resolution.
For other model types besides RF-DETR, the input resolution is baked at training time. If you pass images at a different resolution it resides them to the input size behind the scenes anyway.
When would an Image ever be resized?
Here let me try to re-explain here.
I have a product, in that product there is an option to change the net-size from 128-640, Currently I use YOLO, So it is as simple as called override dims from onnx, and it overwrites the net-size in the graph!
So for maximum accuracy, I have captured all images at 640, and then whenever training I need to figure out a way to pass the images in without resizing them. If they are resized, you lose a ton of data!
Ideally they are cropped, all images collected are centered, so this means, capturing at 640 and cropping to values within my range will create data that would be identical to what actually capturing at those lower imgsz is!
But it seems roboflow has nothing for this, even though it would be the most accurate approach! Additionally it seems roboflow also has no way to have dynamic dims.
if itās required, I suppose a model can be created for every single size, but that is just terribly inefficient AND less accurate.
I understand it resizes them whenever training, this is why I am wanting to fix that!
For example with YOLO/Ultralytics you can do a custom data loader that will choose a specific size each batch. Then you get all your image sizes without resizing.
@Tyler_Armistead super interesting considerations here around multiple image sizes! Iām not sure Iāll help at all, but hereās my brain dump and you can decide if it adds/clarifies anything.
- Data capture - Interesting that you are needing to bring in various image sizes. Many long-term CV solutions intentionally create a single camera set up to avoid that variable, but without more details, Iām just assuming you have no choice in this case so weāll move on.
- One model for all image sizes - seems like this is what you are looking for. You train a model and then at runtime it accepts that image and just runs it through the layers. I agree that Iāve not seen that as an option at Roboflow so I think the question is - why? And my guess is the answer is that in MOST projects the target object is large enough that scaling down only speeds up compute as opposed to losing accuracy. Although there are cases like drone footage where scaling down would be detrimental. In those cases, it seems the most common solutions are higher resolution and/or tiling so the model does NOT lose the information normally lost when resizing down.
- Could models for each size work? You mentioned itās inefficient - seems like that would depend on the scale of the project. Could make sense to create all those if used at large scale. You mentioned itās less accurate - I think this would require testing. A single model needing to tune weights to understand all objects (banana) at all different pixel sizes (40 pixels long, then 200 pixels long, etc) means less room for storing other knowledge in the weights. I would definitely expect model A for 640x640, model B for 480x480, and model C for 320x320 would each outperform a generalized model X for all those image sizes. BUTā¦
- Edge use. To your point about low-end computers being used - this could be a deciding factor. If you can only store one model at a time, a single generalized model would be the choice. If you have enough memory to hold three separate models then that would be my preference. And then thereās always the cloud option if you have that capability - you could have fifty models and just pick the needed one at runtime. (But then we are back to - is it worth it to build all those out or will they only be used a couple times.)
Anyways, itās a great conversation starter and if you wanted to share more about the use-case, itād be interesting just from my own learning perspective to hear the constraints that created the need for multiple image sizes and edge compute! (Though I know sometimes projects are on ālockdownā and I totally get that.) Best of luck on the build!
Hey man, I would love to talk more about it!
Do you have discord or somewhere else we could talk?