4

We have weights and optimizer in the neural network.
Why cant we just W * input then apply activation, estimate loss and minimize it?
Why do we need to do W * input + b?

Thanks for your answer!

Vladimir Tsyshnatiy
  • 989
  • 1
  • 10
  • 20
  • 1
    Refer this : http://stackoverflow.com/questions/2480650/role-of-bias-in-neural-networks – Rishi Feb 06 '17 at 09:10

1 Answers1

18

There are two ways to think about why biases are useful in neural nets. The first is conceptual, and the second is mathematical.

Neural nets are loosely inspired by biological neurons. The basic idea is that human neurons take a bunch of inputs and "add" them together. If the sum of the inputs is greater than some threshold, then the neuron will "fire" (produce an output that goes to other neurons). This threshold is essentially the same thing as a bias. So, in this way, the bias in artificial neural nets helps to replicate the behavior of real, human neurons.

Another way to think about biases is simply by considering any linear function, y = mx + b. Let's say you are using y to approximate some linear function z. If z has a non-zero z-intercept, and you have no bias in the equation for y (i.e. y = mx), then y can never perfectly fit z. Similarly, if the neurons in your network have no bias terms, then it can be harder for your network to approximate some functions.

All that said, you don't "need" biases in neural nets--and, indeed, recent developments (like batch normalization) have made biases less frequent in convolutional neural nets.

Bill
  • 436
  • 3
  • 6