Trackers fail compared to frame-by-frame Keypoint Detection

šŸ¤ Community Help
1

How do I tune trackers for video inference?

My Keypoint Detection model reached the following metrics:

  • mAP@50: 96.6%
  • Precision: 100%
  • Recall: 90.9%
  • F1: 95.2%

And it really shows good quality keypoint detection on a bunch of frames Iā€™ve provided.

But when it comes to the video inference, model seems to get mad. Standard code from documentation doesnā€™t contain any tracking at all. So Iā€™ve tried to add trackers:

  • OC-SORT doesnā€™t outperform any other tracker
  • SORTTracker also gives insufficient results
  • DetectionSmoother changed almost nothing in the way bbox coordinates are calculated
  • ByteTrack often loses track of an object and produces hundreds of IDs (only ~2 needed)

All of the them often ā€˜loseā€™ backward objects (probably because of incorrect IoU threshold).

Iā€™ve added hard poses, frames with vertices missing, like GPT on Roboflow Documentation advised, but it made almost no improvement, comparing to frame-by-frame inference.

Iā€™ve also tried to tune ByteTrack:

tracker = sv.ByteTrack(

       lost_track_buffer=75,

       track_activation_threshold=0.18,

       minimum_matching_threshold=0.7,

       minimum_consecutive_frames=1,

)

But the suggested param values seems to be bad.

You can compare the inference quality on the image:

trackers
trackers1920Ɨ460 261 KB

How should I tune this trackers or change video inference module to get better results?

2

Hi @fitwist ,

Could you also share a video of just detection (visualization) on a video? Could be that inference speed is too slow for tracker to consistently maintain ID over frames.

Thanks, Erik

3

Of course, here they are

OC-SORT: https://youtu.be/AZ3smKoeL2Y

Frame-by-frame: https://youtu.be/CPvP3gULlEo

4

Hi @fitwist , could you also share the full code with detection, tracking, and visualization? The first video seems to have an issue with visualization (scaling?), but would like to verify it locally.

Thanks, Erik