Loss functions in GANs

Question

I'm trying to build a simple mnist GAN and need less to say, it didn't work. I've searched a lot and fixed most of my code. Though I can't really understand how loss functions are working.

This is what I did:

loss_d = -tf.reduce_mean(tf.log(discriminator(real_data))) # maximise
loss_g = -tf.reduce_mean(tf.log(discriminator(generator(noise_input), trainable = False))) # maxmize cuz d(g) instead of 1 - d(g)
loss = loss_d + loss_g

train_d = tf.train.AdamOptimizer(learning_rate).minimize(loss_d)
train_g = tf.train.AdamOptimizer(learning_rate).minimize(loss_g)

I get -0.0 as my loss value. Can you explain how to deal with loss functions in GANs?

without looking too in-depth, you are probably not setting the correct learning rate causing your weights to explode, giving you NaN's. — OneRaynyDay, Apr 23 '18 at 19:24
I haven't coded this myself yet so I won't answer, but I believe you need to set the `var_list` property on the `minimize` function. What you're doing is defining two optimizers over all variables. If your discriminator and generator are in the same graph you're doing two opposite updates. The discriminator optimizer should only update the weights of the discriminator, and likewise the optimizer for the generator. You should use `tf.variable_scope` to help organize your variables into two groups. — David Parks, Apr 23 '18 at 20:06
An alternative to `tf.variable_scope` is to organize things by inheriting from `tf.keras.Model`, as in the [eager GAN example](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/eager/python/examples/gan). (That example is executing eagerly, but switching it to graph building should be relatively easy with some tweaks to the training loop). Then you get a `.variables` property for each component. — Allen Lavoie, Apr 25 '18 at 21:37
I have mostly been relying on this : http://blog.aylien.com/introduction-generative-adversarial-networks-code-tensorflow/ it uses var_list. — ParmuTownley, Apr 26 '18 at 03:09

score 2 · Accepted Answer · edited Aug 30 '18 at 00:13

It seems you try to sum the generator and discriminator losses together which is completely wrong! Since the Discriminator train with both real and generated data you have to create two distinct losses, one for real data and other one for noise data(generated) that you pass into the discriminator network.

Try to change your code as follows:

1)

loss_d_real = -tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=discriminator(real_data),labels= tf.ones_like(discriminator(real_data))))

2)

loss_d_fake=-tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=discriminator(noise_input),labels= tf.zeros_like(discriminator(real_data))))

then the discriminator loss will be equal to = loss_d_real+loss_d_fake. Now create loss for your generator:

3)

loss_g= tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=discriminator(genereted_samples), labels=tf.ones_like(genereted_samples)))

Hi, could you improve the code formatting to clarify your answer — see https://stackoverflow.com/help/formatting — , Aug 29 '18 at 20:40

score 1 · Answer 2 · answered Aug 29 '18 at 19:45

Maryam seems to have identified the cause of your spurious loss values(i.e. summing the generator and discriminator losses). Just wanted to add that you should probably opt for the Stochastic Gradient Descent optimizer for the discriminator in lieu of Adam - doing so provides stronger theoretical guarantees of the network's convergence when playing the minimax game(source: https://github.com/soumith/ganhacks).

Loss functions in GANs

2 Answers2