2

I have read some other questions (and related answers) about this, but I still have doubts: will adding a bias to a threshold activation function change the threshold? As far as I know, adding a bias should move the activation function along the x-axis, so it should also change the threshold.

Let's say that we have only one input node and one output node, and the input node has a threshold activation function with the threshold set to 0. Now if we give 1 as input, the neuron will activate and return 1 * weight to the output node, but if we add a bias node a_0 = -1 with a weight of 2 ,connected to the input node, and give the previous input 1, the neuron won't activate anymore, because now we have to reach at least 2 to activate it. Can this be considered as "changing" the threshold or not?

seaotternerd
  • 6,298
  • 2
  • 47
  • 58
Andrea Tucci
  • 187
  • 2
  • 8
  • 2
    Adding a bias allows you to learn a whole new class of functions. Consider a linear decision line in 2D: without a bias it must pass the origin, with a bias it is not constrained to that. – alfa May 17 '13 at 15:04

1 Answers1

4

Have you read these very good explanations about bias: bias explanation and bias explanation 2?

As is said in the first link the bias will shift the curve so the result of the calculation will be more varied. I think if you already use the bias, you don't need to use threshold (set the threshold to 0) because both bias and threshold do the same thing to shift activation function along the x-axis.

But I think bias is much more efficient than threshold. This is because the bias values are just weights and can be calculated exactly like any other weight in the neural network. Threshold values require separate calculation apart from the weights. There are some interesting bias and threshold comparison in encog forum.

this is a neural network calculated with bias :

calculated with bias

and this is with threshold

calculated with threshold

both will give same result. If you are interested in full calculation you can read the encog wiki above.

So I think the answer to your question "does a bias change the threshold of an activation function" is yes. In my thesis about hybrid GA and NN I've tried both and end up only using bias and set the threshold to 0.

I hope my answer can help you, but if you have other question about my answer feel free to ask in the comment :)

Community
  • 1
  • 1
Niko Adrianus Yuwono
  • 11,012
  • 8
  • 42
  • 64