I want my Neural network output to be either 0 or 1, and not probabilities between 0 and 1, with customised step function at output layer

Question

I want my Neural network output to be either 0 or 1, and not probabilities between 0 and 1.

for the same I have designed step function for output layer, I want my output layer to just roundoff the output of previous(softmax) layer i.e. converting probabilities into 0 and 1.

My customised function is not giving the expected results . Kindly help.

My code is :

from keras.layers.core import Activation
from keras.models import Sequential
from keras import backend as K
# Custom activation function
from keras.layers import Activation
from keras import backend as K
from keras.utils.generic_utils import get_custom_objects

@tf.custom_gradient
def custom_activation(x):
    print("tensor ",x)
    ones = tf.ones(tf.shape(x), dtype=x.dtype.base_dtype)
    zeros = tf.zeros(tf.shape(x), dtype=x.dtype.base_dtype)
    def grad(dy):
        return dy
    print(" INSIDE ACTOVATION FUNCTION ")
    return keras.backend.switch(x > .5, ones, zeros), grad



model = keras.models.Sequential()

model.add(keras.layers.Dense(32,input_dim=a,activation='relu'))
model.add(keras.layers.Dropout(0.2))
model.add(keras.layers.Dense(64,activation="relu"))   
model.add(keras.layers.Dropout(0.2))
model.add(keras.layers.Dense(2,activation='softmax'))  
model.add(Activation(custom_activation, name='custom_activation'))#output layer
### Compile the model
model.compile(loss="binary_crossentropy",optimizer='adam',metrics=["accuracy"])

I think you can use `model.predict_classes(..)` instead of `model.predict(..)`. This will output classes instead of probabilities. — sagi, Jun 07 '21 at 12:58
"My customised function is not giving the expected results", this is too vague, you need to be very precise on what is expected and what is actually happening. — Dr. Snoopy, Jun 07 '21 at 13:01
@Dr.Snoopy I want my customised function to just convert the probabilities into binary values (0 or 1). Output layer is expected to just convert the output(probabilities) of softmax layer into 1 or 0, while with my code I am getting different confusion matrix in case 1 and case 2. Case 1: softmax layer is output layer Case 2: softmax layer followed by customised activation function layer (customised layer is output layer ) I want in the form 1 and 0 i.e. not the probabilities customised layer should simply round off values. i.e. 0.7 to 1 , 0.3 to 0, .87 to 1. etc. — Himani Deshpande, Jun 07 '21 at 17:51
You sure knowing what you are doing? You sure the binary cross-entropy loss will actually work in the (highly awkward) case you are trying to build? Or you just decided to ignore all the ML theory of the last 50 years and try to [optimize "directly" the accuracy instead of the loss](https://stackoverflow.com/questions/47891197/cost-function-training-target-versus-accuracy-desired-goal), so you need hard classes in the output? — desertnaut, Jun 07 '21 at 18:06

score 1 · Answer 1 · answered Jun 07 '21 at 15:20

First of all, note that obtaining the class probabilities is always yielding more information than a pure 0-1 classification, and thus your model will almost always train better and faster.

That being said, and considering that you do have an underlying reason to limit your NN, a hard decision like the one you want to implement as activaction function is know as step function or heaviside function. The main problem with this function is that, by default, the function is non-differentiable (there is an infinite slope in the threshold, 0.5 in your case). To address this you have two options:

Create a custom "approximative" gradient that is differentiable. This SO answer covers it well.
Use tf.cond(), which, relying on TF's AutoGrad, will only execute one branch of the graph at runtime, and omit the unused branch.

class MyHeavisideActivation(tf.keras.layers.Layer):
  def __init__(self, num_outputs, threshold=.5, **kwargs):
    super(MyHeavisideActivation, self).__init__(**kwargs)
    self.num_outputs = num_outputs
    self.threshold = threshold

  def build(self, input_shape):
    pass

  def call(self, inputs):
    return tf.cond(inputs > self.threshold, 
                   lambda: tf.add(tf.multiply(inputs,0), 1), # set to 1
                   lambda: tf.multiply(inputs, 0))           # set to 0


#> ...same as above
model.add(keras.layers.Dense(2,activation='softmax'))  
model.add(MyHeavisideActivation(2, name='custom_activation'))#output layer

I used your code followed by my customised activation function( with def grad(Div): return Div*1; as grad function,) but getting error tensorflow.python.framework.errors_impl.InvalidArgumentError: The second input must be a scalar, but it has shape [32,2] [[{{node gradient_tape/sequential/module_wrapper/custom_activation/cond/StatelessIf/switch_pred/_11}}]] [Op:__inference_train_function_884] Function call stack: train_function Kindly suggest — Himani Deshpande, Jun 07 '21 at 17:46

I want my Neural network output to be either 0 or 1, and not probabilities between 0 and 1, with customised step function at output layer

1 Answers1