1

Tensorflow have few benchmark tools:

For .pb model and for .tflite model

I have few questions regarding parameters of .pb benchmark tool:

  1. Is num_threads related to number of parallel runs of single threaded experiments or to internal threads used by tensorflow?
  2. Is it possible to use GPU when tool build for desktop, i.e. not for mobile? if it so, how to ensure that GPU is not used?

Also few questions regarding result interpretation:

  1. What is count in result output? How Timings (microseconds): count= related to --max_num_runs parameter?

Example:

Run --num_threads=-1 --max_num_runs=1000:
    2019-03-20 14:30:33.253584: I tensorflow/core/util/stat_summarizer.cc:85] Timings (microseconds): count=1000 first=3608 curr=3873 min=3566 max=8009 avg=3766.49 std=202
    2019-03-20 14:30:33.253584: I tensorflow/core/util/stat_summarizer.cc:85] Memory (bytes): count=1000 curr=3301344(all same)
    2019-03-20 14:30:33.253591: I tensorflow/core/util/stat_summarizer.cc:85] 207 nodes observed
    2019-03-20 14:30:33.253597: I tensorflow/core/util/stat_summarizer.cc:85]
    2019-03-20 14:30:33.378352: I tensorflow/tools/benchmark/benchmark_model.cc:636] FLOPs estimate: 116.65M
    2019-03-20 14:30:33.378390: I tensorflow/tools/benchmark/benchmark_model.cc:638] FLOPs/second: 46.30B

Run --num_threads=1 --max_num_runs=1000:
    2019-03-20 14:32:25.591915: I tensorflow/core/util/stat_summarizer.cc:85] Timings (microseconds): count=1000 first=7502 curr=7543 min=7495 max=7716 avg=7607.22 std=34
    2019-03-20 14:32:25.591934: I tensorflow/core/util/stat_summarizer.cc:85] Memory (bytes): count=1000 curr=3301344(all same)
    2019-03-20 14:32:25.591952: I tensorflow/core/util/stat_summarizer.cc:85] 207 nodes observed
    2019-03-20 14:32:25.591970: I tensorflow/core/util/stat_summarizer.cc:85]
    2019-03-20 14:32:25.805970: I tensorflow/tools/benchmark/benchmark_model.cc:636] FLOPs estimate: 116.65M
    2019-03-20 14:32:25.806007: I tensorflow/tools/benchmark/benchmark_model.cc:638] FLOPs/second: 15.46B

Run --num_threads=-1 --max_num_runs=10000:
    2019-03-20 14:38:48.045824: I tensorflow/core/util/stat_summarizer.cc:85] Timings (microseconds): count=3570 first=3961 curr=3899 min=3558 max=6997 avg=3841.2 std=175
    2019-03-20 14:38:48.045829: I tensorflow/core/util/stat_summarizer.cc:85] Memory (bytes): count=3570 curr=3301344(all same)
    2019-03-20 14:38:48.045833: I tensorflow/core/util/stat_summarizer.cc:85] 207 nodes observed
    2019-03-20 14:38:48.045837: I tensorflow/core/util/stat_summarizer.cc:85]
    2019-03-20 14:38:48.169368: I tensorflow/tools/benchmark/benchmark_model.cc:636] FLOPs estimate: 116.65M
    2019-03-20 14:38:48.169412: I tensorflow/tools/benchmark/benchmark_model.cc:638] FLOPs/second: 48.66B

Run --num_threads=1 --max_num_runs=10000:
    2019-03-20 14:35:50.826722: I tensorflow/core/util/stat_summarizer.cc:85] Timings (microseconds): count=1254 first=7496 curr=7518 min=7475 max=7838 avg=7577.23 std=50
    2019-03-20 14:35:50.826735: I tensorflow/core/util/stat_summarizer.cc:85] Memory (bytes): count=1254 curr=3301344(all same)
    2019-03-20 14:35:50.826746: I tensorflow/core/util/stat_summarizer.cc:85] 207 nodes observed
    2019-03-20 14:35:50.826757: I tensorflow/core/util/stat_summarizer.cc:85]
    2019-03-20 14:35:51.053143: I tensorflow/tools/benchmark/benchmark_model.cc:636] FLOPs estimate: 116.65M
    2019-03-20 14:35:51.053180: I tensorflow/tools/benchmark/benchmark_model.cc:638] FLOPs/second: 15.55B

i.e. when --max_num_runs=10000 is used count is count=3570 and count=1254 what does it mean?

For .tflite benchmark tool:

--num_threads=1 --num_runs=10000
    Initialized session in 0.682ms
    Running benchmark for at least 1 iterations and at least 0.5 seconds
    count=54 first=23463 curr=8019 min=7911 max=23463 avg=9268.5 std=2995
    Running benchmark for at least 1000 iterations and at least 1 seconds
    count=1000 first=8022 curr=6703 min=6613 max=10333 avg=6766.23 std=337
    Average inference timings in us: Warmup: 9268.5, Init: 682, no stats: 6766.23

What does no stats: 6766.23 mean?

mrgloom
  • 20,061
  • 36
  • 171
  • 301

1 Answers1

7

After digging in the code a bit I've found the following (All times are in microseconds):

  • count: number of actual runs
  • first: time the first iteration took
  • curr: time the last iteration took
  • min: minimum time an iteration took
  • max: maximum time an iteration took
  • avg: mean time an iteration took
  • std: standard deviation of timing over all runs
  • Warmup: warm up run average
  • Init: start up time (should always be same as the Initialized session in)
  • no stats: Is the very poorly named average run time (matches the avg= in the previous line)
  • num_threads: This is used to set intra_op_parallelism_threads and inter_op_parallelism_threads (more info here)

The relevant files (linked to the proper lines) are:

I'm not so sure about the using GPU vs not using GPU. If you are using the freeze_graph to export the .pb file then it will store the device of each node in the graph. You can use device placement to do this before exporting. If you need to change it after you could try setting the environment variable CUDA_VISIBLE_DEVICES="" to make sure the GPU is not used.

McAngus
  • 1,826
  • 18
  • 34