RF-DETR Training Glitch: 10.5 Hours, 20+ Credits, and No Progress Past Epoch 12

[Algorithm_Artist]

Hi Roboflow team,

I recently trained a custom object detection model using the new RF-DETR architecture on a dataset of approximately 3,000 images. While I understand that RF-DETR base is a larger model compared to YOLO small/nano variants (which typically train in 1-2 hours), I was surprised to see that training took over 10.5 hours and consumed more than 20 credits (especially because projected training time was 3 hours).

Even more concerning, the training process stalled at epoch 12, yet credits continued to be consumed after the model had seemingly stopped progressing. I had to manually early stop the training in the morning, or it would have kept running indefinitely. This led to an overcharge of 10 additional credits beyond my available balance. At $3 per credit, this single training session ended up costing me around $60, which feels disproportionate for a 12-epoch run on a moderate-sized dataset.

I’d appreciate clarification on:

• Why the training continued to consume credits after stalling at epoch 12.
• Whether there’s a safeguard to stop credit usage when no progress is being made.
• If there’s a possibility for reimbursement of the excess credits due to this apparent malfunction.

I’ve been a supporter of Roboflow for a while, and I really want to continue using the platform. However, incidents like this are discouraging, especially when experimenting with new models. I hope this can be addressed fairly.

Thanks in advance for your help!

[Jacob_Witt]

Hi @Algorithm_Artist - I’m sorry about this, particularly the difference in estimated train times.

I’ve added 25 credits to your account and will work with the team to update our train time estimation + look for bugs related with your specific training run.

RF-DETR is brand new and our ML team is full-time on this; improvements coming soon!

[Algorithm_Artist]

Hi @Jacob_Witt,

Thank you very much for the quick response and for adding the 25 credits - I really appreciate it.

I noticed that while the credits are now showing on my account, the “Flex Usage” section still shows an overdraw of 10.95 credits. It doesn’t seem to have been offset by the 25 reimbursed credits. Is this just a visual bug, or does it still count as a balance that needs to be covered?

Thanks again!

[Matvei_Popov]

Hello @Algorithm_Artist I’m very sorry you are experiencing this issue! I’m currently looking into this problem. I wanted to ask for your permission to add myself to your workspace as it would enable me to troubleshoot it better. Please let me know if it is ok with you.

[MYAKAM_ANIRUDH_IIITK]

Hi!
I have faced a similar issue with training RF-DETR. I have been searching for relevant help and finally found this chat. I have attached the image proof of the issue. I tried to train it around last week on small plant dataset. The credits for the image augmentations, version generation and Model training have been consumed but the error message says model stalled and no result. I am currently on Student Research plan that I use for my Honors Project. My April month credits are all used up and I’m not able to make further progress in my project. It is affecting my work. Please Please Kindly respond.

[Matvei_Popov]

Hello @MYAKAM_ANIRUDH_IIITK I’m very sorry you are experiencing this issue. We will look into your problem asap and reach back out when we find an issue. Sorry once again you are experiencing this inconvenience.