Does Intersection over Union filter out good detections when using instance segmentation?

[Reinier_Zeldenrust]

We have a trained instance segmentation that we’re using via hosted inference, using workflows. In workflows, you can set IoU, but it’s not clear how it works. The documentation points to this: How to Code Non-Maximum Suppression (NMS) in Plain NumPy seems to refer to object detection specifically.

However I’m seeing a few cases where my model is missing a really obvious roof face, while the adjacent roof face is well detected. I wonder if the IoU setting uses bounding boxes, or the pixel masks themselves.

Especially when the roof faces are diagonal on the image, the bounding boxes of the detection may overlap quite a lot even when the detections themselves do not overlap at all, and this setting may filter out too many real, valid detections!

I have experimented with setting the IoU value to be high (e.g. 0.95) but then we do get quite a few more genuinely overlapping detections. I’m hoping the model can be smart enough to actually compare the masks rather than the bounding boxes.

[Reinier_Zeldenrust]

This is the screenshot in the workflow mode:

and the link goes here again - and the workflows documentation also takes you there.

[Thomas_Hansen]

Hi Reiner,

Yes it looks like the NMS is only using axis aligned bounding boxes as far as I can tell. I think this is the code that does NMS post processing on instance segmentation in inference here: inference/inference/core/models/instance_segmentation_base.py at main · roboflow/inference · GitHub

I think it’s be possible to implement NMS using actual polygon overlapp calculations. I don’t know if it should be default in this code given potential speed implications. But I it being optional and have an flag on the Workflow block e.g (inference/inference/core/workflows/core_steps/models/roboflow/instance_segmentation/v2.py at main · roboflow/inference · GitHub)

The overlap calculation of the NMS function itself is here: inference/inference/core/nms.py at main · roboflow/inference · GitHub

I understand that doesn’t solve your problem, but hopefully confirming at least is somewhat helpful?

We don’t have this on the roadmap right now at roboflow, but I’ll note it as a feature request in the backlog. Since inference is open source, we’d be happy to have you make a PR if it’s something you want to implement yourself.

[Reinier_Zeldenrust]

Hi Thomas, thanks for the response, definitely helpful. I guess we can set the value to 0.99 (1 seems to error for some reason) and then do the filtering on our end afterwards. We’ll take a look at doing a PR too.