Handling keypoint classes

Feedback
1

Currently the keypoint classes allow you to create n-classes-to-n-skeleton datasets. However, most keypoint models probably require n-classes-to-one-skeleton datasets. For example, if you wanted to classify exercise poses for a human you’d have to create a separate duplicate skeleton for each pose. And even then, I’m not sure you’d easily be able to change the class inside the annotator without reseting the points.

2

Thanks @Hart_Woolery!

Would love to explore this a bit more. Do you mind sharing what model you’re training?

While we don’t offer n-classes-to-one-skeleton datasets at present, what I’ve seen most customers do is calculate angles (or run a model on skeleton points) to classify poses.

3

I’m training a yolov8-pose model, to detect both the racing team and keypoints for F1 cars. What I have to do currently is train a separate detection model, run my pose data through it, clean up the classification, then re-upload the dataset somewhere, which is a somewhat lengthy workaround : )

I think the classes could be abstracted completely from the keypoints in order to solve this issue. Then skeletons could be selected from a separate control during annotation.

4

How many classes do you have? Would this be solved with some ability to duplicate the same skeleton structure when configuring new classes?

5

I have one class per F1 team (10) and could possibly extend to one per driver (20). While duplicating the skeletons might help in the interim, it presents a few possible issues:

  1. Modifying or deleting points from one skeleton would not affect the rest
  2. If somebody mislabeled the class while annotating and then changed it, would it reset the keypoints?
  3. Would the output layer have num_keypoints * num_skeletons or just num_keypoints? Ideally the the latter to allow the model to dedicate more weights toward each keypoint.

I think a way that would encapsulate all possible scenarios is to allow any skeleton to be any class. I think you could migrate to this without changing any existing data, though I’m sure there’s challenges I haven’t thought of.

6
  1. Yeah that’d be a limitation unfortunately – how often would you need to change the skeleton once it’s setup?
  2. No if the skeletons match they should propagate; let me know if this isn’t working properly
  3. I’m not sure I follow here; I think YOLOv8-pose requires the same number of keypoints on each skeleton (if you have multiple you have to pad the ones with less points). They’d all have num_keypoints points.

I think a way that would encapsulate all possible scenarios is to allow any skeleton to be any class.

How would you communicate that to the model?

7

Ok thanks for clarifying…

  1. Probably not often
  2. Great!
  3. Yes, was just curious if creating more duplicate skeletons would result in padding the keypoints for the other skeletons, but sounds like no.

It sounds like based on 3 that it should work as needed, so ignore my last thought there

8

Awesome lmk if you encounter any bugs. I put creating a “duplicate skeleton” option on my list to make that much easier (currently would have to be done manually)