Hi, we just released YOLOv8 instance segmentation weights upload after 8pm central time (United States), yesterday.
That is why your upload didn’t work.
If you try it now, by retraining your model, it will work.
You’ll need to use the python package version >=1.0.1, pip install roboflow>=1.0.1
https://discuss.roboflow.com/t/model-weights-upload-support-for-yolov5-yolov7-and-yolov8/1682