[Help Wanted] Comparing predictions from multiple models

Community Help
1

Inference on multiple models concurrently seems to not work

  • Project type: Object detection
  • Operating System & Browser: Ubuntu

Hi everyone!

So for my project I’m training four RF-DETR versions on the same images but with different annotations to compare them.

When visually comparing I’d like to use inference on each model and overlay the predictions together with the ground truths on the same image plot with supervision, color coded by model. When run on a single image it works perfectly.

image
image610×427 51.9 KB

When doing this for multiple images in a row however the inference only seems to work on the first image, since all following plots only has the ground truths plotted.
I know that it is not a prediction problem since I have verified that one of the later images in my list does in fact produce predictions upon inference when done by itself.

Relevant code as follows:

for image in images:
    image = cv2.imread(os.path.join(data_dir, image["file_name"]))

    # annotate the image with our ground truth results
    bounding_box_annotator = sv.BoxAnnotator(thickness=1, color=sv.Color.WHITE)
    image_annotations = annotations_by_image[image["id"]]
    gt_boxes = []
    for annotation in image_annotations:
        x, y, w, h = annotation["bbox"]
        gt_boxes.append([x, y, x+w, y+h])
    gt_boxes = np.array(gt_boxes)
    gt = sv.Detections(
        xyxy=gt_boxes,
        class_id=np.full(len(gt_boxes), 2),
        data={'class_name': np.full(len(gt_boxes), 'GT', dtype='<U6')}
    )
    annotated_image = bounding_box_annotator.annotate(scene=image, detections=gt)

    # run inference on the models (it's a zip of (range(num_models), str{custom name}, model{from get_model})
    for i, model_name, model in models:
        bounding_box_annotator = sv.BoxAnnotator(thickness=1, color=colors[i])
        results = model.infer(image, confidence=0.3)[0]
        detections = sv.Detections.from_inference(results)
        # annotate the image with our inference results
        annotated_image = bounding_box_annotator.annotate(scene=image, detections=detections)

    sv.plot_image(annotated_image)

Is there some limit on running inference on multiple models simultaneously on roboflow with the inference package? Or is it possibly a bug (either in my code or the package). Any help to understand the api better and also suggestions on how to fix this would be helpful.

2

There’s no Roboflow “one-image” or “one-model” limit—the issue is in the drawing loop.
supervision.BoxAnnotator.annotate() edits the array you give it in-place.

Because you:
1. draw the ground-truth on image,
2. immediately pass that mutated array to every model’s .infer(), and
3. then keep drawing on the same array again,

the first pass works, but for the next images the detector is looking at an image that already has big white boxes (and then coloured boxes) painted over the objects, so it often returns zero detections.

Run inference on a clean copy (or before you draw anything), and keep a separate canvas for visualisation.

⸻

How to fix it

for img_dict in images:
    # --- load once ---------------------------------------------------------
    img_path = os.path.join(data_dir, img_dict["file_name"])
    raw      = cv2.imread(img_path)           # pristine copy for the models
    canvas   = raw.copy()                     # we’ll draw on this only

    # --- ground-truth ------------------------------------------------------
    gt_detections = sv.Detections(
        xyxy=xyxy_from_coco(annotations_by_image[img_dict["id"]]),
        class_id=np.full(len(annotations_by_image[img_dict["id"]]), 2),
        data={'class_name': np.full(len(annotations_by_image[img_dict["id"]]), 'GT')}
    )
    canvas = sv.BoxAnnotator(thickness=1, color=sv.Color.WHITE)\
              .annotate(canvas, gt_detections)

    # --- every model -------------------------------------------------------
    for idx, (name, model) in enumerate(models):   # models = [(“RF-DET-A”, modelA), ...]
        preds  = model.infer(raw, confidence=0.3)[0]          # run on *raw*
        dets   = sv.Detections.from_inference(preds)

        canvas = sv.BoxAnnotator(thickness=1, color=colors[idx])\
                  .annotate(canvas, dets)

    sv.plot_image(canvas)

Key changes
• raw never gets drawn on → detectors always see the true pixels.
• All annotation happens on canvas, so every model’s boxes stack up.
• No variables are reused in a way that hides earlier drawings.

A Better Solution

This code is really inefficient. Running a model across a folder of images and doing something with the output is exactly what we built Batch Processing for. It will run the models in a Workflow in parallel on the GPU and the rest of the Blocks across multiple CPU to use the underlying system’s maximum processing potential (and by building with a Workflow each individual Block has extensive tests to avoid bugs like the one above).

3

Thank you for the bug fixes, I hadn’t noticed that the canvas messed with my predictions.

The actual issue however was the models only predicting once, which turns out was because I used a zip for the models instead of a standard tuple list. The code now works wonderfully for the purpose I have (visualizing a small subset of images)

I attempted batch processing, however it proved quite difficult to visualize the ground truths through the Roboflow interface, either GUI or serverless API. There seemed to not be an easy way to directly use a dataset version from a project as input, and I could only upload images or videos, not annotations. The restrictions for using the project annotations when using a local image as input were not very user-friendly.