1

There is a relevant question here already TensorFlow: Is there a way to measure FLOPS for a model?

However, the answer given by @Tobias Scheck is the forward pass stats.

Is there a way to measure/estimate the backward pass as well?

NOBOND_GFX
  • 11
  • 1
  • 3

1 Answers1

1

If you just want to get a quick number, you can simply add 

grads = tf.gradients(C, [A, B])

to @Tobias Scheck's code to construct the gradient computation nodes. Then, subtract the new number (with gradient ops) from the original one (without gradient ops) to get the estimated flops.

A word of caution about using this method in larger projects. This method uses static analysis of the whole graph. This has a few problems including:

  • The flops from ops in a while loop will be added only once.
  • Ops that are never normally run (some TF functionalities can leave garbage ops in the graph) will be added.
  • This analysis heavily depends on shape inference. It might not be available for all ops.
  • This analysis depends on registering functions that can estimate the flops of a given op. There can be ops without such functions and such functions don't precisely model the flops done by the actual kernel your TF will pick to execute the op.

For more info see: https://github.com/tensorflow/tensorflow/blob/r1.8/tensorflow/core/profiler/g3doc/profile_model_architecture.md

It is better to use this in conjunction with an actual run record (RunMetadata) or use a purely runtime based approach, e.g. Can I measure the execution time of individual operations with TensorFlow?, and do some filtering/aggregation on the results.

iga
  • 3,571
  • 1
  • 12
  • 22
  • Hi, what about the high level case which through Keras loading an established model already (2nd example by @Tobias). Do I need to hack into the Keras model to add the explicit the gradient OP? – NOBOND_GFX May 02 '18 at 00:46
  • If you use `cmd="scope"` instead of `cmd="op"`, it should group the ops by name scope instead of op type. All gradient ops should have "gradient" somewhere in their name. You should be able to just sum their flop estimates. – iga May 02 '18 at 03:16
  • when using keras, it is loading a native model (mobilenet), where I can find the OP? – NOBOND_GFX May 02 '18 at 16:24
  • What op do you want to find and why? You just invoke `tf.profiler.profile` with a different argument. If you don't see any names with `gradient` in them, you might need to execute the model once for the backward pass to be built. – iga May 02 '18 at 20:14
  • All the OPS in whole backward pass. I just want to estimate the computation cost of the backward pass. But since I am using the keras on top, so only thing I can see is the model (wrapped already) and the fit_generator. Where should I put the profile object? – NOBOND_GFX May 03 '18 at 16:04
  • There is an example in the link in your question. Do the same, except use `cmd='scope'`. If you don't see gradient ops, run one training step first. – iga May 03 '18 at 18:40
  • The "scope view" is working as iga said. It outputs the following info: _TFProfRoot (--/1.21b flops) training/Adam/gradients/model_1/mobilenet_1.00_224/conv_pw_7/convolution_grad/Conv2DBackpropFilter (102.76m/102.76m flops). @iga - what the numerator and denominator split number as above means. I thought 102.76m is the ops associated with Conv2DBackpropFiter. What info it carry for splitting it into 2 parts? – NOBOND_GFX May 14 '18 at 13:31