Roboflow HELP! I have an ai model that is being weird after having done object detection for the preview model. This means I tested images not seen in the model, but of course relating to the object and it showed amazing high accuracies and tight boxes, however, when I left the model for a bit for class, I came back and continued testing….This is where the problem starts…The model started decreasing scores, and then eventually missing the objects entirely? AS IF it was an entirely different model! I have the confusion matrix with the testing images labeled and I have the noted down percentages I recorded. EXCEPT, I was curious to try those images, and once I did IT DECREASED PERCENTAGES and MISSED images that were previously accurate and boxed in. What does this mean? It’s the same model version and labeled model??? Why? It even says images cannot be altered, like you can go back to the original dataset, which might be my case because the images I had in that model were only like 1638 with preprocessing and augmentations, and then the current dataset we have now has almost 1000 annotated images, so when I was getting images not tested in the model I just moved annotations around, but those annotations I put back and were not in the model versions previous dataset. MAYBE I went too far back to those images originally in the model, but I need help SOOON URGENTLY PLEASE!!! Model : Marine Microplastic AI Detection & natalia.alvarado100@student.tsc.edu is my email for the website.
Project Type: Object Detection
Operating System & Browser: Windows 10/11 & Google Chrome
Project Universe Link or Workspace/Project ID: marine-microplastic-ai-detector/29
Do you grant Roboflow Support permission to access your Workspace for troubleshooting? (Yes/No): Yes
I believe I tested some images on the preview of the model, and I did not modify sliders on the Confidence Threshold or Overlap Threshold…I know modifying those sliders can cause percentages not to show if it hits a certain threshold, HOWEVER, this should not be the case for percentages that are already supposed to be certain. The first time I put the first image into the model, it was 98% and was tightly boxed. The second time after I realized percentages were going down and images were being missed entirely, I was curious to check a former image I had tested in the preview.
It was missed entirely.
I have no idea how this happened. Could it be the model is getting dumber or?
I understand the questions you are asking. I have seen these pop up for mistakes I might have made, but this model and the steps I took are not the case.
To restate, I put images that were not in the model to test if the model could detect the microplastics, and kept the same confidence/overlap thresholds THE ENTIRE TIME.
Model trained on diverse dataset should respond well to images outside of train dataset. Model not detecting object in some new images suggest there might be something on those images that was not captured in train set - i.e. specimens distribution in the train set might not be uniform enough (all images present specimens in certain area of the scene missing other areas), angle at which photo was taken might not be represented well in the train set, lightning conditions might be under-represented in train set, specimens with some features unseen in train set might be presented in new images, and so on - there can be many reasons why your perceived model performance might look like it’s degraded but in reality it’s under-representation of some features that is resulting in suboptimal results. It is important that all of the features are equally represented in your train dataset.
Once trained, model is not updated, model artifacts are static and never change.
You shared:
the images I had in that model were only like 1638 with preprocessing and augmentations, and then the current dataset we have now has almost 1000 annotated images
Please confirm - you trained your model on dataset containing 1000 images annotated by yourself and your team; when configuring new version you added augumentations so total number of images included in training dataset was 1638; what is your judgement regarding quality of annotations - did you have any images with objects visible in the scene but not annotated? If you ask yourself a question regarding new images compared to train dataset - is there anything that stands out on those images that is missing from train dataset? If so - include those images and retrain the model.
You shared:
so when I was getting images not tested in the model I just moved annotations around, but those annotations I put back and were not in the model versions previous dataset
Do I understand correctly that you was using previously trained model to help with labelling of new images? This is one of great features made available when labelling with our platform, with new versions of your model the assistance in labelling of new photos should be better.
MAYBE I went too far back to those images originally in the model
Please help me understand what do you mean?
and kept the same confidence/overlap thresholds THE ENTIRE TIME
The sliders are there to help you build some intuition about model performance. If by decreasing confidence threshold you notice more objects are appearing that could be a sign there is still room for improvement by adding more samples into the train set.
