Inaccurate Annotations upon export to YOLOv5-Oriented Bounding Boxes

Hello All,

I’m trying to create an object detection model that can detect a custom made robot from an aerial image. For this project, it is crucial to know the orientation of the vehicle, therefore the model will be based off of the YOLOv5-OBB repository.

To build the model, I’ve followed this guide from Roboflow.

I was able to get the model up and running, but wasn’t happy with my results. After adjusting parameters like batch size and epochs I got slightly better results, but the bounding box wasn’t as tight to the robot as I would like.

I started to look into why this may be the case and I noticed that the Roboflow annotations when exported aren’t as accurate when I drew them.

The technical term for the YOLOv5-OBB annotation format is called “DOTA”. DOTA works by storing pixel coordinates of the vertices of the bounding box in a text file starting from the top left corner heading clockwise.

I’ve created a small Python script that verifies the Roboflow annotations by drawing a bounding box over the object by using the pixel coordinates found in the exported DOTA annotation text file.

To use it, just change the PATH variable to the path of your dataset. It will then store the results in a folder called “Annotations Drawn” In the root of the PATH. Please note that this script will only work if your model has only 1 class or object its trying to detect.

import cv2 as cv
import os

PATH = r"C:\Users\XXX\Desktop\Test script\Exported Annotation"
os.chdir(PATH)

Ipath = PATH + r"\train\images"
Apath = PATH + r"\train\labelTxt"
Spath = PATH + r"\Annotations Drawn"

if os.path.exists(Spath):
    print("Save Directory already exists")
else:
    os.mkdir(Spath)
    print("Save Directory Created")

IMG_Path = os.listdir(Ipath)
Annot_Path = os.listdir(Apath)

for img in IMG_Path:
    img = img[:-4] #remove file extension 
    for annot in Annot_Path:
        annot = annot[:-4] #remove extension again

        if img == annot:   
            #reassign extensions
            img = img + ".jpg"
            annot = annot + ".txt"
            
            os.chdir(Apath)

            #S/O to Geeks4Geeks
            with open(annot, 'r') as file:
                for line in file:
                    x1, y1, x2, y2, x3, y3, x4, y4, obj, difficulty = line.split()
                    file.close
                    #this algorithm only works for single class annotations
            
            x1 = int(float(x1))
            y1 = int(float(y1))
            x2 = int(float(x2))
            y2 = int(float(y2))
            x3 = int(float(x3))
            y3 = int(float(y3))
            x4 = int(float(x4))
            y4 = int(float(y4))
            
            os.chdir(Ipath)
            print(img)
            
            image = cv.imread(img)
            # cv.imshow('Car', image)
            # cv.waitKey(0)

            cv.line(image, (x1, y1), (x2,y2), (0,0,255), thickness=3)
            cv.line(image, (x2, y2), (x3,y3), (0,0,255), thickness=3)
            cv.line(image, (x3, y3), (x4,y4), (0,0,255), thickness=3)
            cv.line(image, (x4, y4), (x1,y1), (0,0,255), thickness=3)

            os.chdir(Spath)
            cv.imwrite(img, image)

            break

print("Done")

Here’s an example of the problem:

Annotation done in Roboflow:

Annotation verified by script:

As you can see, the bounding box coordinates when outputted isn’t as tight as the annotation I drew in Roboflow.

Initially I thought this was because the DOTA format uses only 4 points and when I used the polygon tool to draw my annotation I used more than 4 points (in an effort to get the tightest bounding box possible).

Therefore I generated a new version where I only used 4 points from the polygon tool to represent my annotation like so:

After confirming my annotation with script above I received:

As you can see the exported annotation is still wildly skewed and inaccurate.

I’ve tested multiple permutations changing the augmentation steps and enabling auto rotate (which made the problem way worse as the annotation would be completely shifted like so: )

No matter what changes I made the annotations would still show skewed although resizing the image to a lower resolution reduced the skewing effect.

One thing I did notice is if the annotation is relatively axis aligned i.e. the bounding box is roughly 90 degrees, the exported annotation will remain tight.

Please do not confuse this with the rotation augmentation step, but rather if the object itself is rotated.
Notice in the above images, no augmentations or preprocessing was applied but because the object is rotated, the corresponding annotation is skewed.

When the robot is relatively axis aligned like so:

The annotations are as follows:
Roboflow:

Verified:

As you can see when the annotations are axis aligned the exported annotation remains accurate as well.

Would one of the devs be able to look into this and provide a fix possibly? Thank you!

I think one solution would be to allow users to draw a bounding box that is perfectly a rectangle and then rotate it. Then just export the vertices of that rectangle.

TLDR: When an object is rotated, the exported annotation is skewed and looks like a rhombus.

EDIT: The forum won’t let me attach multiple images so I’ve removed them.

What was the original image size (pixel width and pixel height) prior to exporting?

And preprocessing steps were included on the version you exported?

I’m wondering if Resize was added to the generated version for export, and you’re comparing annotations for a resized image to those of the original image size.

Hello,

The original image is 1944 x 2592.

Interestingly enough, with no preprocessing steps the image is scaled down to 1536 x 2048.

Also, I’ve tried many permutations with different preprocessing and augmentation (and without) steps, the results are unchanged.

In my last experiment I did confirm that if the robot is naturally upright but the rotation augmentation step is applied, then the skewing effect will also occur like so:

I’m wondering if Resize was added to the generated version for export, and you’re comparing annotations for a resized image to those of the original image size.

And yes, when I do purposely resize the image in the preprocessing steps the resized image is generated as well & the script verifies the annotations based on that as well like so:

The above photo is resized to 1280 x 1280.