0

Does tf.train.get_global_step() return the value of current training step? If not then how is currentglobal_step different from current training step?

This is with reference to the following code:

optimizer=lambda: tf.train.FtrlOptimizer(
      learning_rate=tf.train.exponential_decay(
          learning_rate=0.1,
          global_step=tf.train.get_global_step(),
          decay_steps=10000000,
          decay_rate=0.96)))

Will tf.train.get_global_step() increment global_step at every training step and affect the learning_rate accordingly? Or more specifically, will the global_step value be the same as the current training step value and influence the learning_rate accordingly?

Pulah
  • 123
  • 12

1 Answers1

1

Edited after question's edit

Will tf.train.get_global_step() increment global_step at every training step?

No. The optimizer will take care of the increment, tf.train.get_global_step() only gets you the current variable defined to store the global step (if any was already defined).

and affect the learning_rate accordingly?

Yes, the learning rate schedule will internally get the value of the current global step and adjust the LR accordingly.

Update: some clarification

In TF there's a key difference between "variables" as are commonly intended in python (not tf.Variable()) and tensors (tf.Variable is a tensor).

When you call

global_step = tf.train.get_global_step()

(assuming a global step was prevously defined somewhere) you get a Tensor object back, not an integer.

The underlying idea is to separate the construction phase of the computation, where you describe the operations applied to the data, from the actual execution, where you feed the data and get results. This often causes confusion at first, but it's a key point of TF's programming model (at least until TF 2.0).

If you want to get the current value of global_step, you need to evaluate the graph. Assuming you already have a tf.Session() defined, you either:

step_value = sess.run(global_step)

or alternatively:

step_value = global_step.eval(session=sess)

This is done internally from the LR schedule. At each step, it will get the current value of the global step and calculate the LR from it with the given parameters. Similarly, the optimizer internally will take care of updating the current global step value at each step, so unless you need the value for logging/debugging, you normally wouldn't explicitly evaluate global_step yourself.

Community
  • 1
  • 1
GPhilo
  • 18,519
  • 9
  • 63
  • 89
  • So if the current training step is 50, will the value of global_step also be 50 which will be returned to me by tf.train.get_global_step() ? – Pulah Sep 17 '18 at 08:58
  • No. In ("classic", as in "non-eager") TF there's a key difference between graph creation and execution. When you call `tf.train.get_global_step` you get the *tensor* used to store the global step (or `None` if it wasn't yet defined). A tensor doesn't have an associated value until it is evaluated in a session. – GPhilo Sep 17 '18 at 09:05
  • @PulahD.Dhandekar see my edit and the update beneath it – GPhilo Sep 17 '18 at 09:25
  • Thanks a ton for your answer. I haven't defined `global_step` anywhere in my code. Is it necessary to do so? If I haven't defined it, does `tensorflow` assign a default value to `global_step`? How should `global_step` be defined ideally if it is necessary for me to define it initially? – Pulah Sep 17 '18 at 09:39
  • Just use [` tf.train.get_or_create_global_step`](https://www.tensorflow.org/api_docs/python/tf/train/get_or_create_global_step) instead and it will take care of defining one for you in case none is already there – GPhilo Sep 17 '18 at 09:41
  • Could this be related to my issue here: https://stackoverflow.com/questions/57919530/getting-interrupted-by-signal-11-sigsegv ? – Stefan Falk Sep 13 '19 at 09:23