Using softmax in Neural Networks to decide label of input

Question

I am using the keras model with the following layers to predict a label of input (out of 4 labels)

embedding_layer = keras.layers.Embedding(MAX_NB_WORDS, 
                                         EMBEDDING_DIM, 
                                         weights=[embedding_matrix], 
                                         input_length=MAX_SEQUENCE_LENGTH,
                                         trainable=False)

sequence_input = keras.layers.Input(shape = (MAX_SEQUENCE_LENGTH,), 
                                    dtype = 'int32')

embedded_sequences = embedding_layer(sequence_input)

hidden_layer = keras.layers.Dense(50, activation='relu')(embedded_sequences)
flat = keras.layers.Flatten()(hidden_layer)
preds = keras.layers.Dense(4, activation='softmax')(flat)
model = keras.models.Model(sequence_input, preds)
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['acc'])
model.fit(X_train, Y_train, batch_size=32, epochs=100)

However, the softmax function returns a number of outputs of 4 (because I have 4 labels)

When I'm using the predict function to get the predicted Y using the same model, I am getting an array of 4 for each X rather than one single label deciding the label for the input.

model.predict(X_test, batch_size = None, verbose = 0, steps = None)

How do I make the output layer of the keras model, or the model.predict function, decide on one single label, rather than output weights for each label?

score 0 · Answer 1 · answered Dec 02 '17 at 16:26

The following is a common function to sample from a probability vector

def sample(preds, temperature=1.0):
    # helper function to sample an index from a probability array
    preds = np.asarray(preds).astype('float64')
    preds = np.log(preds) / temperature
    exp_preds = np.exp(preds)
    preds = exp_preds / np.sum(exp_preds)
    probas = np.random.multinomial(1, preds, 1)
    return np.argmax(probas)

Taken from here.

The temperature parameter decides how much the differences between the probability weights are weightd. A temperature of 1 is considering each weight "as it is", a temperature larger than 1 reduces the differences between the weights, a temperature smaller than 1 increases them.

Here an example using a probability vector on 3 labels:

p = np.array([0.1, 0.7, 0.2]) # The first label has a probability of 10% of being chosen, the second 70%, the third 20%

print(sample(p, 1)) # sample using the input probabilities, unchanged
print(sample(p, 0.1)) # the new vector of probabilities from which to sample is [  3.54012033e-09,   9.99996371e-01,   3.62508322e-06]
print(sample(p, 10)) # the new vector of probabilities from which to sample is [ 0.30426696,  0.36962778,  0.32610526]

To see the new vector make sample return preds.

isn't there a way to make the softmax layer itself normalize the values or add any layer than can normalize the value rather than working with the output that it produces (meaning have the output value of the predict function belong to one of the 4 possible layers) ? — spark problems, Dec 02 '17 at 17:14
Not that I know of. Consider that a softmax layer is just an activation layer, and the softmax is a given function that cannot be changed. If you need it, you can create your [own activation function](https://stackoverflow.com/questions/43915482/how-do-you-create-a-custom-activation-function-with-keras) — gionni, Dec 02 '17 at 17:23

Using softmax in Neural Networks to decide label of input

1 Answers1