How to run eval.py job for tensorflow object detection models

Question

I have trained an object detector using tensorflow's object detection API on Google Colab. After researching on the internet for most of the day, I haven't been able to find a tutorial about how to run an evaluation for my model, so I can get metrics like mAP.

I figured out that I have to use the eval.py from the models/research/object_detection folder, but I'm not sure which parameters should I pass to the script.

Shortly, what I've done so far is, generated the labels for the test and train images and stored them under the object_detection/images folder. I have also generated the train.record and test.record files, and I have written the labelmap.pbtxt file. I am using the faster_rcnn_inception_v2_coco model from the tensorflow model zoo, so I have configured the faster_rcnn_inception_v2_coco.config file, and stored it in the object_detection/training folder. The training process ran just fine and all the checkpoints are stored also in the object_detection/training folder.

Now that I have to evaluate the model, I ran the eval.py script like this:

!python eval.py --logtostderr --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config --checkpoint_dir=training/ --eval_dir=eval/

Is this okay? Because this started running fine, but when I opened tensorboard there were only two tabs, namely images and graph, but no scalars. Also, I ran tensorboard with logdir=eval.

I am new to tensorflow, so any kind of help is welcome. Thank you.

score 9 · Answer 1 · answered Sep 18 '18 at 01:36

The setup looks good. I had to wait a long time for the Scalars tab to load/show up alongside the other two - like 10 minutes after the evaluation job finished.

But at the end of the evaluation job, it prints in the console all the scalar metrics that will be displayed in the Scalars tab:

Accumulating evaluation results...
DONE (t=1.57s).
Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.434
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.693
Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.470
Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000

etc.

If you want to use the new model_main.py script instead of legacy/eval.py, you can call it like

python model_main.py --alsologtostderr --run_once --checkpoint_dir=/dir/with/checkpoint/at/one/timestamp --model_dir=eval/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config

Note that this new API would require the optimizer field in train_config, which is probably already in your pipeline.config since you're using the same for both training and evaluation.

where do i get detection rectangles? – Stepan Yakovenko Jul 13 '19 at 12:45 — Stepan Yakovenko, Jul 13 '19 at 12:45

score 5 · Answer 2 · edited Oct 01 '19 at 14:13

5

For those who are looking to run the new model_main.py in evaluation mode only. There is a flag in the parameter that you can set that does just that. That flag is checkpoint_dir, if you set it equal to a folder containing past training checkpoints, the model will run in evaluation only.

Hope I can help a few that missed it like myself! Cheers,

edited Oct 01 '19 at 14:13

Prelude

151
1
10

answered Aug 01 '19 at 19:42

ChowderII

157
1
8

score 5 · Answer 3 · answered May 07 '20 at 00:30

I'll try to expand and complement the previous answers.

If you want to evaluate your model on validation data you should use:

python models/research/object_detection/model_main.py --pipeline_config_path=/path/to/pipeline_file --model_dir=/path/to/output_results --checkpoint_dir=/path/to/directory_holding_checkpoint --run_once=True

If you want to evaluate your model on training data, you should set 'eval_training_data' as True, that is:

python models/research/object_detection/model_main.py --pipeline_config_path=/path/to/pipeline_file --model_dir=/path/to/output_results --eval_training_data=True --checkpoint_dir=/path/to/directory_holding_checkpoint --run_once=True

I also add comments to clarify some of previous options:

--pipeline_config_path: path to "pipeline.config" file used to train detection model. This file should include paths to the TFRecords files (train and test files) that you want to evaluate, i.e. :

    ...
    train_input_reader: {
        tf_record_input_reader {
                #path to the training TFRecord
                input_path: "/path/to/train.record"
        }
        #path to the label map 
        label_map_path: "/path/to/label_map.pbtxt"
    }
    ...
    eval_input_reader: {
        tf_record_input_reader {
            #path to the testing TFRecord
            input_path: "/path/to/test.record"
        }
        #path to the label map 
        label_map_path: "/path/to/label_map.pbtxt"
    }
    ...

--model_dir: Output directory where resulting metrics will be written, particularly "events.*" files that can be read by tensorboard.

--checkpoint_dir: Directory holding a checkpoint. That is the model directory where checkpoint files ("model.ckpt.*") has been written, either during training process, or after export it by using "export_inference_graph.py".

--run_once: True to run just one round of evaluation.

How to run eval.py job for tensorflow object detection models

3 Answers3

Linked