Setting up a CNN network with multi-label classification

Question

I have a set of 100x100 images, and an output array corresponding to the size of the input (i.e. length of 10000), where each element can be an 1 or 0.

I am trying to write a python program using TensorFlow/Keras to train a CNN on this data, however, I am not sure how to setup the layers to handle it, or the type of network to use.

Currently, I am doing the following (based off the TensorFlow tutorials):

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(100, 100)),
    keras.layers.Dense(128, activation=tf.nn.relu),
    keras.layers.Dense(10000, activation=tf.nn.softmax)
])
model.compile(optimizer=tf.train.AdamOptimizer(), 
          loss='sparse_categorical_crossentropy',
          metrics=['accuracy'])

However, I can't seem to find what type of activation I should be using for the output layer to enable me to have multiple output values? How would I set that up?

This [answer](https://stackoverflow.com/a/51892084/2099607) might help you for the activation and the loss function you need to use. If you want a convolutional architecture, then obviously you need to use convolution layers like `Conv2D`. Usually a set of `Conv2D` and `MaxPooling2D` layers are used successively and at the end you might have one or a few dense layers. — today, Aug 24 '18 at 18:41

Amir · Accepted Answer · 2018-08-24T19:44:36.483

I am not sure how to setup the layers to handle it.

Your code is one way to handle that but as you might read in literature, is not the best one. State-of-the-art models usually use 2D Convolution Neural Networks. E.g:

    img_input = keras.layers.Input(shape=img_shape)
    conv1 = keras.layers.Conv2D(16, 3, activation='relu', padding='same')(img_input)
    pol1 = keras.layers.MaxPooling2D(2)(conv1)
    conv2 = keras.layers.Conv2D(32, 3, activation='relu', padding='same')(pol1)
    pol2 = keras.layers.MaxPooling2D(2)(conv2)
    conv3 = keras.layers.Conv2D(64, 3, activation='relu', padding='same')(pol2)
    pol3 = keras.layers.MaxPooling2D(2)(conv3)
    flatten = keras.layers.Flatten()(pol3)
    dens1 = keras.layers.Dense(512, activation='relu')(flatten)
    dens2 = keras.layers.Dense(512, activation='relu')(dens1)
    drop1 = keras.layers.Dropout(0.2)(dens2)
    output = keras.layers.Dense(10000, activation='softmax')(drop1)

I can't seem to find what type of activation I should be using for the output layer to enable me to have multiple output values

Softmax is a good choice. It squashes a K-dimensional vector of arbitrary real values to a K-dimensional vector of real values, where each entry is in the range (0, 1].

You can pas output of your Softmax to top_k function to extract top k prediction:

softmax_out = tf.nn.softmax(logit)
tf.nn.top_k(softmax_out, k=5, sorted=True)

If you need multi-label classification you should change the above network. Last Activation function will change to sigmoid:

output = keras.layers.Dense(10000, activation='sigmoid')(drop1)

Then use tf.round and tf.where to extract labels:

indices = tf.where(tf.round(output) > 0.5)
final_output = tf.gather(x, indices)

The OP wants to do multi-label classification and not single-label multi-class classification. Therefore, `sigmoid` must be used as the activation function of last layer and not the `softmax`. — today, Aug 24 '18 at 19:28
@today Thanks for notice. I'll changed my answer to cover both situation — Amir, Aug 24 '18 at 19:33
@AmirHadifar , 1) Can you expand on the reason why all those layers are needed? Are multiple Conv2D/MaxPooling layers common, or does the number depend on network size? 2) When I try to create a model based on that, using keras.Model(inputs=img_input,outputs=output), I get a failure: 'Node' object has no attribute 'output_masks'. Am I missing something? — Hawkeye001, Aug 28 '18 at 03:43
@Hawkeye001 take a look at [here](https://cloud.google.com/blog/products/gcp/learn-tensorflow-and-deep-learning-without-a-phd) to understand why we need those convolutions and poolings. For the error you can take look at [here](https://stackoverflow.com/questions/51821537/attributeerror-node-object-has-no-attribute-output-masks/51831434) and [here](https://github.com/keras-team/keras/issues/10907) — Amir, Aug 28 '18 at 05:34
@AmirHadifar thanks for those links; regarding my error, I actually had seen those links, but re-running my samples, I may have previously loaded keras from tensorflow, instead of the native one, which was what was causing my error. When i started clean, and verified which version of keras I was using, it worked correctly. — Hawkeye001, Aug 29 '18 at 04:34

Setting up a CNN network with multi-label classification

1 Answers1