We present a simple, fully-convolutional model for real-time instance segmentation that achieves 29.8 mAP on MS COCO at 33.5 fps evaluated on a single Titan Xp, which is significantly faster than any previous competitive approach. Moreover, we obtain this result after training on only one GPU. We accomplish this by breaking instance segmentation into two parallel subtasks: (1) generating a set of prototype masks and (2) predicting per-instance mask coefficients. Then we produce instance masks by linearly combining the prototypes with the mask coefficients. We find that because this process doesn't depend on repooling, this approach produces very high-quality masks and exhibits temporal stability for free. Furthermore, we analyze the emergent behavior of our prototypes and show they learn to localize instances on their own in a translation variant manner, despite being fully-convolutional. Finally, we also propose Fast NMS, a drop-in 12 ms faster replacement for standard NMS that only has a marginal performance penalty.
A simple, fully convolutional model for real-time instance segmentation. This is the code for our paper:
For a real-time demo, check out our ICCV video:
Here are our YOLACT models along with their FPS on a Titan Xp and mAP on COCO's val:
| Image Size | GPU x BS | Backbone | *FPS | mAP | Weights | Configs | Download |
|---|---|---|---|---|---|---|---|
| 550 | 1x8 | Resnet50-FPN | 42.5 | 29.0 | config | model | |
| 550 | 8x8 | Resnet50-FPN | 42.5 | 28.4 | config | model | |
| 550 | 1x8 | Resnet101-FPN | 33.5 | 30.4 | config | model |
*Note: The FPS is evaluated by the original implementation. When calculating FPS, only the model inference time is taken into account. Data loading and post-processing operations such as converting masks to RLE code, generating COCO JSON results, image rendering are not included.
All the aforementioned models are trained with a single GPU. It typically takes ~12GB VRAM when using resnet-101 as the backbone. If you want to try multiple GPUs training, you may have to modify the configuration files accordingly, such as adjusting the training schedule and freezing batch norm.
# Trains using the resnet-101 backbone with a batch size of 8 on a single GPU.
./tools/dist_train.sh configs/yolact/yolact_r101.py 1Please refer to mmdetection/docs/getting_started.md.
If you use YOLACT or this code base in your work, please cite
@inproceedings{yolact-iccv2019,
author = {Daniel Bolya and Chong Zhou and Fanyi Xiao and Yong Jae Lee},
title = {YOLACT: {Real-time} Instance Segmentation},
booktitle = {ICCV},
year = {2019},
}