I have tried the same thing that you have mentioned but the resulting mask is in binary format, I need a polygon points in coco-format.
I split off your topic since it was somewhat of a different topic. If you are trying to export SAM annotations in COCO format, you can use Smart Polygon.
Then, you can export in COCO format.
Hope this helps!