6

I added the built-in regularizer tf.contrib.layers.l2_regularizer as such:

regularizer = tf.contrib.layers.l2_regularizer(scale=0.1)
layer1 = tf.layers.dense(tf_x, 50, tf.nn.relu, kernel_regularizer=regularizer)
layer2 = tf.layers.dense(layer1, 50, tf.nn.relu, kernel_regularizer=regularizer) 
output = tf.layers.dense(layer2, 5, tf.nn.relu)

I tried different values for scale (0.1-1), but it didn't seem to do much. I was wondering whether I have to do add the regularizer at some other position (i.e. optimizer, train etc.) or whether this might simply be due to my data.

I didn't adjust any other position in my code for the regularizer then in the tf.layers like above.

sandboxj
  • 1,234
  • 3
  • 21
  • 47
  • 1
    Check: https://stackoverflow.com/questions/37107223/how-to-add-regularizations-in-tensorflow – Vijay Mariappan Aug 16 '17 at 19:58
  • 2
    Possible duplicate of [How to add regularizations in TensorFlow?](https://stackoverflow.com/questions/37107223/how-to-add-regularizations-in-tensorflow) – GeertH Aug 16 '17 at 20:05
  • 1
    None of the answers in the referred question apply the regularization directly into the `tf.layers`. They either explicitly add it to the loss, or specify it in the weights. Also, some comments there suggest that adding it to the loss actually applies it twice. If applied like I specified above, i.e. without specifying weights explicitly, is it correct to specify it under `kernel_regularizer` as the only step? – sandboxj Aug 16 '17 at 20:12
  • 1
    If you specify the `kernel_regularizer` argument, a tensor will be added to the `tf.GraphKeys.REGULARIZATION_LOSSES` collection. You need to incorporate these tensors into your loss as described in the accepted answer in the referenced question. – GeertH Aug 16 '17 at 20:32

0 Answers0