During the past few days while trying to run a training with Yolov5 I encountered a “CUDA error: device-side assert”. While doing some debugging I see to have narrowed the problem down to the Intersection Over Union (IoU) part of the training script. Upon careful reviewing of the dataset and the annotations I noticed that in some cases there where different objects in front of other larger objects resulting in a Bounding Box within another Bounding box.
I wanted to ask if someone has had the “CUDA error: device-side assert” before and what happens in the training script when a bounding box is present within another bounding box.
Thanks a lot for the support