14

In Caffe, the SGD solver has a momentum parameter (link). In TensorFlow, I see that tf.train.GradientDescentOptimizer does not have an explicit momentum parameter. However, I can see that there is tf.train.MomentumOptimizer optimizer. Is it the equivalent of Caffe SGD with momentum optimizer?

Alp Aribal
  • 370
  • 1
  • 11
A Das
  • 817
  • 2
  • 10
  • 21

2 Answers2

19

Yes it is. tf.train.MomentumOptimizer = SGD + momentum

nessuno
  • 26,493
  • 5
  • 83
  • 74
  • Thanks @nessuno. The tensorflow documentation is a little inadequate, I'd say. – A Das Oct 28 '16 at 17:26
  • if you look at formula of SGD+momentum in Caffe (http://caffe.berkeleyvision.org/tutorial/solver.html) and in TF(https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/train/MomentumOptimizer?hl=en) you can see these are little different. – Vladimir Bystricky Jun 22 '20 at 11:57
1

tf.keras.optimizers.SGD has a momentum parameter. Applying a Nesterov momentum is also possible by using nesterov=True.

Alp Aribal
  • 370
  • 1
  • 11
  • 1
    Just to clarify tf.keras.optimizers.SGD has no minimize method for tensorflow 1.6. It is suitable for passing as an argument to keras sequential model.fit. However, tf.train.MomentumOptimizer can be used. – rajesh Mar 10 '20 at 15:17