Hi support, we’ve been unable to train a new model for a few days now as it always fails with this response: This training job did not complete successfully. This can happen for a few reasons but often means that the chosen model dimension (which corresponds to image size) was too large to fit into GPU memory.
We’ve tried adjusting settings as mentioned, as well as disabling all augmentations but have had the same issue every time. We have not changed the original dataset and tried using the same settings as a version that worked fine 2 weeks ago but it also ended up failing. At this point we believe it might be an issue specific to Semantic segmentation.
Our project is stalled due to this, so any help would be really appreciated!
Project Type: Semantic Segmentation
Operating System & Browser: Windows 10 - Chrome
Project Universe Link or Workspace/Project ID: park-path