0

I'm new to Tensorflow and I'm trying to understand how it processes data. Currently, this is what I want to have as my input. My full code is up on github should you want to download it.

print (y_train[0])
>>> [0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 
1.0, 1.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 
1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 
1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 
1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 
0.0, 1.0, 0.0, 1.0, 0.0, 0.0]
# list of 80 elements

print (np.array(y_train))
>>> [[0. 0. 1. ... 1. 0. 0.]
 [0. 1. 0. ... 0. 0. 0.]
 [1. 0. 1. ... 0. 0. 0.]
 ...
 [0. 0. 1. ... 1. 1. 0.]
 [1. 0. 0. ... 0. 0. 1.]
 [0. 0. 0. ... 1. 0. 1.]]

print (np.array(y_train).shape)
>>> (11645, 80)

print (x_train[0])
>>> [1.0, 4.0, 5.0, 2.0, 5.0, 3.0, 5.0, 3.0, 4.0, 5.0, 3.0, 5.0, 4.0, 3.0, 
3.0, 4.0, 5.0, 4.0, 4.0, 5.0, 4.0, 3.0, 3.0, 4.0, 4.0, 5.0]

print (np.array(x_train)/5)
>>> [[0.2 0.8 1.  ... 0.8 0.8 1. ]
[0.6 0.8 1.  ... 1.  1.  0.8]
[0.8 0.4 1.  ... 1.  0.6 1. ]
...
[1.  0.6 0.8 ... 0.4 0.8 0.6]
[1.  0.8 0.8 ... 0.4 0.6 1. ]
[0.6 0.8 0.8 ... 1.  0.8 0.6]]

print (np.array(x_train).shape)
>>> (11645, 26)

So basically I have 11645 pieces of data in my dataset. For the input, I wish to have 26 inputs normalized from 0 to 1. For the output, I wish to have 80 binary outputs. I don't think TF can give binary outputs, so I probably will use a sigmoid activation function.

How do I get Tensorflow to understand that I have 11645 pieces of data I want to process and that the input shape should be 26x1 and the output 80x1? There are some pieces of Tensorflow and Keras that I don't understand how they fit together. For instance, if I want Tensorflow to understand that my input should be 1x26 and not some other input shape, should I use x_train = tf.reshape(x_train, [-1,1*26]) and y_train = tf.reshape(y_train, [-1,1*80])? From the documentations it seems like it will shape x_train into a tensor of only 1 row and 26 columns, and I will have 11645 of those. But does that specify to Tensorflow that the input should only be 1x26 and it won't go off grabbing some other number (eg. 26x2). Or do I have to do something more explicit like this where I specify the input shape into the model? model.add(tf.keras.layers.Dense(26, activation=keras.activations.relu, input_shape=(26,)))?

Again, for my output, I want to have a 1x80 tensor that I can reshape and stuff. Do I have to specify to tensorflow explicitly? Or will something like model.add(tf.keras.layers.Dense(80, activation=keras.activations.sigmoid)) be enough to tell Tensorflow that I want a 1x80 matrix, and (for eg, using the sigmoid function) that it should compare every piece of data in that predicted 1x80 with the 1x80 matrix I have in y_train to calculate the loss function?

Basically, I am confused as to how Tensorflow 'knows' what data to accept as an individual input and output. Is there a way to specify it or is it a step one can omit?

EDIT: Based on the answers, I have used the code:

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(26, input_dim=26,activation='relu'))
model.add(tf.keras.layers.Dense(80, activation='sigmoid'))
model.compile(optimizer='rmsprop',
      loss='binary_crossentropy',
      metrics=['accuracy'])
model.fit(x_train, y_train, epochs=10, batch_size=32)

I'm getting the following matrix:

[0.38176608 0.34900635 0.36545524 0.36806932 0.36692804 0.37398493
  0.36821148 0.35577637 0.38441166 0.3676901  0.41162464 0.40428266
  0.41464344 0.4040607  0.39316037 0.428753   0.3547327  0.35693064
  0.3422352  0.36919317 0.36431065 0.3515264  0.3889933  0.33974153
  0.37329385 0.35898593 0.3891792  0.42334762 0.40694237 0.41910493
  0.39983115 0.47813386 0.37625512 0.35567597 0.36811477 0.38242644
  0.36549032 0.35696995 0.37058106 0.3556903  0.37096408 0.34965912
  0.4247738  0.41512045 0.41622216 0.38645518 0.40850884 0.43454456
  0.3655926  0.34644917 0.36782715 0.34224963 0.35035127 0.3502
  0.3607877  0.38218996 0.37265536 0.3653391  0.41620222 0.41124558
  0.3916335  0.41291553 0.39959764 0.4649614  0.34603494 0.36731967
  0.34146535 0.34573284 0.33941117 0.35885242 0.3493014  0.35866526
  0.37188208 0.34971312 0.38165745 0.3962399  0.38913697 0.4078925
  0.38799426 0.4709055 ]

This is a far cry from the 0 and 1 matrix I want. What should I do to get closer to that? I've tried Googling my problem, but to no avail. Should I simply apply a threshold to this (eg. 0.4?) and convert it to a binary matrix that way?

koifish
  • 424
  • 1
  • 6
  • 16

1 Answers1

0

Usually in tensorflow we specify placeholders for when we create the graph. These specify the datatypes, shape, and sometimes name of the input data. A basic example that matches your code:

x = tf.placeholder(tf.float32,[None,26])
y = tf.placeholder(tf.float32,[None,80])
W = tf.get_variable('W',shape=[26,80],initializer=tf.truncated_normal_initializer(mean=0.0, stddev=0.01))
output = tf.matmul(x,W)
cost = tf.losses.sigmoid_cross_entropy(y,outputs,reduction=tf.losses.Reduction.MEAN)
with tf.Session() as sess:
    loss = sess.run(cost,feed_dict={x:your_input_here,y:your_output_here})

So, tensorflow knows how big your input is because you specified it to be so, and uses this to calculate the output shape of each of the subsequent layers. The batch dimension (the first dimension) doesn't matter because this can be a variable, as if your input is size [50x26], your output would be size [50,80]. The number of data samples is irrelevant because you can feed them into the model as you please.

But in keras, it's a bit simpler:

model = Sequential()
model.add(Dense(32, input_dim=26,activation='relu'))
model.add(Dense(80,activation='sigmoid'))
model.compile(optimizer='rmsprop',
          loss='binary_crossentropy',
          metrics=['accuracy'])
model.fit(data, labels, epochs=10, batch_size=32)

You can see that we have to specify the input dimensions in the first layer, and again, batch size does not need to be specified. The output layer can then be specified to be the same shape as your expected number of outputs.

Also, as a side note I would recommend that you split your data into batches (anywhere between 10 and 200 samples, depending on memory/performance), and not put in the entire 11k samples at once!

jhso
  • 3,103
  • 1
  • 5
  • 13
  • Hi I used your code and I'm getting a matrix that has values closer to 0.5 rather than 1. How should I go about getting the matrix closer to 1? I've put some edits into the question if you are not sure what kind of matrix I am getting. – koifish Jan 21 '19 at 06:30
  • You should definitely train for more than 3 epochs, and your first layer should read: model.add(tf.keras.layers.Dense(180, activation=keras.activations.relu, input_dim=26)) which means you've set your input layer to size 26 in both INPUT and OUTPUT! Also, don't use steps_per_epoch instead of batch size! This code you've attached means that it's going through 32*10*3 samples and then stopping! (32 is the batch size default). Use the model fit parameters in my answer and you should find your results closer to what you expect. – jhso Jan 21 '19 at 07:06
  • I have used your code and I am still getting the about the same matrix. It seems for some reason that TF is levelling out the matrix to minimise the loss rather than having individual terms of it be closer to 1 and 0. – koifish Jan 21 '19 at 07:35
  • Yes, but currently you have one layer of 26 nodes which is very tiny. Change your first layer to have something between 100 and 500 nodes and you should see your loss changing. What is your loss if you use the History function from: https://stackoverflow.com/questions/36952763/how-to-return-history-of-validation-loss-in-keras – jhso Jan 21 '19 at 08:05
  • Also, after reading your question, a sigmoid function will never return only 0/1 values. You should be using a threshold so separate them, but this should be done through analysing the performance of a validation set. – jhso Jan 21 '19 at 08:11
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/187035/discussion-between-yip-jung-hon-and-jhso). – koifish Jan 21 '19 at 08:18