5

Now I'm reproducing the Mask R-CNN(Instance segmentation task.)

I can't figure out how to use the MS COCO test dataset.

There exists 'instances_train2014', 'instances_val2014' which have specific annotations.

But in testing data, there are only image_id, image_url, image height and width. I couldn't figure out how to use it at all.

Could anyone figure this out? :( Sorry for terrible English

Kim
  • 61
  • 1
  • 3

2 Answers2

9

The test data is just that, test data. They did not release annotations for it. Instead, you train on the train/val datasets and submit results for the test data to the evaluation server. This ensures a fair comparison between different methods.

pjreddie
  • 390
  • 2
  • 10
1

So could you tell me how Researcher evaluate mAP of a model, like ssd_mobilenet_v1_coco have COCO mAP[^1] = 21(according to This)? Did they calculate on COCO validation set instead of COCO Test set and record the mAP number?

mrSmith91
  • 338
  • 1
  • 6
  • 18
  • See here: https://github.com/AlexeyAB/darknet/issues/2145 - it can't be done locally, but check out CodaLab. – Tomasz Gandor Mar 06 '19 at 17:22
  • This didn't answer my question. Did Researcher calculate mAP[^1] on COCO validation set instead of COCO Test set, or they calculate mAP[^1] on COCO Test set using CodaLab? – mrSmith91 Mar 07 '19 at 01:12
  • 1
    They calculated mAP on COCO validation set. COCO2014 minival but different split. Can be referred to here: [^1]: See MSCOCO evaluation protocol. The COCO mAP numbers here are evaluated on COCO 14 minival set (note that our split is different from COCO 17 Val). A full list of image ids used in our split could be fould here. – Anchal Gupta Jan 16 '20 at 09:24