0

I am trying to run a simple logistic regression over dummy data. Apparently, the loss and the weights do not change over different epochs. I have tried to change the learning rate to different values but it did not work. Also, I have tried different random initialization of the weights and increased the number of epochs, but it did not work as well. Here is an example of what I have run:

train_set_x_dummy = np.random.rand(3,100) #100 examples with 3 features
train_set_y_dummy = np.random.randint(high=2,low=0,size=(1,10))
n_features = train_set_x_dummy.shape[0]

X = tf.placeholder(name="X", dtype=tf.float32,shape=(n_features,None))
Y = tf.placeholder(name="Y", dtype=tf.float32,shape=(1,None))


W = tf.Variable(np.random.rand(n_features,1), name="weights",dtype=tf.float32)
b = tf.Variable(0 ,name="bias",dtype=tf.float32)

y_preds = tf.nn.sigmoid( tf.matmul(tf.transpose(W),X) + b)
loss = tf.nn.softmax(Y - y_preds)
mse = tf.reduce_sum(loss)

optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01).minimize(loss)

tf.set_random_seed(1103)
with tf.Session() as sess:    
    sess.run(tf.global_variables_initializer())
    loss_summary = tf.summary.scalar("loss_scaler",mse)
    weights_summary = tf.summary.histogram("weights_summary",W)
    bias_summary = tf.summary.histogram("bias_summary",b)
    writer = tf.summary.FileWriter('logs', tf.get_default_graph())

    for epoch in range(1000):

        (_,l,y_preds_log,weights,mse_values) = sess.run([optimizer,mse,y_preds,W,mse],feed_dict={X:train_set_x_dummy , Y:train_set_y_dummy})
        mse_str = loss_summary.eval(feed_dict={X:train_set_x_dummy , Y:train_set_y_dummy})
        writer.add_summary(mse_str,epoch)

        writer.add_summary(weights_summary.eval(),epoch)
        writer.add_summary(bias_summary.eval(),epoch)

    writer.flush()

So, any idea why this is happening?

Yahia
  • 1,209
  • 1
  • 15
  • 18
  • 2
    Softmax for a single prediction always returns `1`. (exp(x)/sum(exp(x) = 1, when there is one term to sum). I have answered a similar question here: https://stackoverflow.com/questions/45652597/cost-function-always-returning-zero-for-a-binary-classification-in-tensorflow/45654404#45654404. So use sigmoid instead. – Vijay Mariappan Jun 01 '18 at 22:45
  • 1
    Related discussion: [Tensorflow sigmoid and cross entropy vs sigmoid_cross_entropy_with_logits](https://stackoverflow.com/a/47238223/712995) – Maxim Jun 02 '18 at 13:00

0 Answers0