0

When i run my neural network the cost always stays around its initial value. sometime it increases and sometimes it decreases but never by much.

I tried some different seeds but this didn't work. I also tried with different input but this also didn't work.

I looked over the math for the training but I couldn't find anything wrong with it.

class NeuralNet():
    def __init__(self):
        np.random.seed(1)

        self.w0 = 2*np.random.random((3, 4))-1
        self.w1 = 2*np.random.random((4, 1))-1

    def sigmoid(self, x):
        return 1/(1+np.exp(-x))

    def sigmoid_d(self, x):
        x = self.sigmoid(x)
        return x * (1-x)

    def train(self, training_length, x, y, print_cost_cycle=1000):
        for i in range(training_length):
            l0 = x
            l1 = self.sigmoid(np.dot(l0, self.w0))
            l2 = self.sigmoid(np.dot(l1, self.w1))

            l2_error = (l2 - y)**2
            l2_d = 2*(y - l2) * self.sigmoid_d(l2)
            l1_cost = np.dot(l2_d, self.w1.T)
            l1_d = l1_cost * self.sigmoid_d(l1)

            self.w0 += np.dot(l0.T, l1_d)
            self.w1 += np.dot(l1.T, l2_d)

            if i % print_cost_cycle == 0:
                print(f"cost = {np.mean(l2_error)}")


x_train = np.array([
    [0, 0, 1],
    [0, 1, 1],
    [1, 0, 1],
    [1, 1, 1]
])
y_train = np.array([
    [0],
    [1],
    [1],
    [0]
])

neural_net = NeuralNet()
neural_net.train(10000, x_train, y_train)

the output:

cost = 0.2475932677010128
cost = 0.24844153430705612
cost = 0.21300873346681945
cost = 0.21497568086911634
cost = 0.22868625594402198
cost = 0.23076981080756687
cost = 0.23242035817817885
cost = 0.23560061594143974
cost = 0.23578477135747
cost = 0.2356102896524483
TBchip
  • 1
  • 2
  • 1
    Please update your post to also include the actual code you use to *run* your NN. – desertnaut Apr 15 '20 at 15:21
  • If you are allowed to: 1. try a different activation function (e.g. ReLU instead of sigmoid) 2. try adding more examples to your training dataset. – asymmetryFan Apr 15 '20 at 15:23
  • This looks like the exact same data, and a very similar implementation, to this question: https://stackoverflow.com/questions/61239698/simple-neural-network-gives-wrong-output-after-training. As pointed out there, this network model also does not have a bias parameter, which would add a lot of power to the model. See this SO post for a very good discussion: https://stackoverflow.com/questions/2480650/what-is-the-role-of-the-bias-in-neural-networks – mcskinner Apr 15 '20 at 22:52

0 Answers0