31

Is there a way to calculate the time cost for each node in a TensorFlow network?
I find it hard to locate the performance bottlenecks.

EDIT: The Timeline profiler is really awesome (https://stackoverflow.com/a/37774470/3632556).

Community
  • 1
  • 1
bgshi
  • 1,174
  • 2
  • 12
  • 14
  • 4
    Possible duplicate: http://stackoverflow.com/questions/34293714/tensorflow-can-i-measure-the-execution-time-of-individual-operations – user3559888 Jan 29 '16 at 19:27

3 Answers3

7

If you want to find how much time was spent on each operation at TF, you can do this in tensorboard using runtime statistics. You will need to do something like this (check the full example in the above-mentioned link):

run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
run_metadata = tf.RunMetadata()
sess.run(<values_you_want_to_execute>, options=run_options, run_metadata=run_metadata)
your_writer.add_run_metadata(run_metadata, 'step%d' % i)

Better than just printing it you can see it in tensorboard:

Additionally, clicking on a node will display the exact total memory, compute time, and tensor output sizes.

Also now tensorflow has a debugger. Here is a tutorial of how to use it.

[Example from link

benjaminplanche
  • 14,689
  • 5
  • 57
  • 69
Salvador Dali
  • 214,103
  • 147
  • 703
  • 753
  • The current (late 2017, TensorFlow 1.4) way of getting the Timeline is using a [ProfilerHook](https://www.tensorflow.org/api_docs/python/tf/train/ProfilerHook). This works with the MonitoredSessions in tf.Estimator where tf.RunOptions are not available. – Urs Dec 16 '17 at 00:53
2

Right now the best external way is to compile with a CPU/GPU profiler, but this has to be done manually by changing options in the BUILD and tensorflow.bzl files (not sure where).

You can then get profile information about which functions are taking the most time, etc. It's up to you, you can run by using gperftools and visualizing by pprof

Since TensorFlow's code is linked via swig, i'm not 100% sure how easy it will be to get the symbols when reading the profile via pprof. You may be able to point it to the swig .so file that was built.

Give it a try and let us know if it works for you!

WY Hsu
  • 1,837
  • 2
  • 22
  • 33
Eugene Brevdo
  • 899
  • 7
  • 8
-1

https://github.com/tensorflow/tensorflow/tree/master/tensorflow/core/profiler

Major Features:

Measure model parameters, float operations, tensor shapes.

Profile op execution times, requested memory size and device placement.

Inspect checkpoint tensors' shapes and their values.

Selectively group, filter, account and order ops.

Peter
  • 108
  • 2
  • The question says *"Is there a way to calculate the time cost for each node in a TensorFlow network?"* Can you improve your answer by showing how to use the tool to "calculate the time cost"? – Pang Jun 06 '17 at 01:01
  • Too bad, link not found – WY Hsu Jul 31 '17 at 08:25