Proper way to implement biases in Neural Networks

Question

I can make a neural network, I just need a clarification on bias implementation. Which way is better: Implement the Bias matrices B1, B2, .. Bn for each layer in their own, seperate matrix from the weight matrix, or, include the biases in the weight matrix by adding a 1 to the previous layer output (input for this layer). In images, I am asking whether this implementation:

Or this implementation:

Is the best. Thank you

The gradients for the bias are often simpler to calculate than the gradients for the normal nodes. In Andrew Ng's original ML course (2012), I remember including the biases in the matrix with the column of `1`s but in his 2016 course he has them separate. I assuming that it is more performent to keep them separate as matrix-multiplication is worse than quadratic time. I'm not sure if that changes when you have GPUs. You could try look into the source code for popular libraries and see how they're doing it. — Dan, Nov 13 '18 at 11:20
In terms of computational cost, in any sane implementation doing it either way won't make a difference: if the lower layer has N neurons and next, upper layer has M neurons, then the computational cost of adding M biases will be dwarfed by the M*MxN operation of passing the activity in the lower layer via the weights to the upper. — Paul Brodersen, Nov 13 '18 at 22:48
IMO, the only substantial distinction is in terms of code readability and maintainability. Having implemented several backprop and RBM variants from scratch, I would argue having the biases separate from the weights results in much cleaner code, especially if you want to play around with learning rules or different initializations. However, as prefixed, that is just an opinion. — Paul Brodersen, Nov 13 '18 at 22:50
Correction: `M * M x N` should read `N * N x M` in the comment above. — Paul Brodersen, Nov 13 '18 at 22:56

WristMan · Answer 1 · 2018-11-27T09:18:48.700

I think the best way is to have two separate matrices, one for the weitghts and one for the bias. Why? :

I don't believe there is an increase on the computational load since W*x and W*x + b should be equivalent running on GPU. Mathematically and computationally they are equivalent.
Greater modularity. Let's say you want to initialize the weights and the bias using different initializers (ones, zeros, glorot...). By having two separate matrices this is straightforward.
Easier to read and maintain.

score 0 · Answer 2 · answered Nov 13 '18 at 11:15

include the biases in the weight matrix by adding a 1 to the previous layer output (input for this layer)

This seems to be what is implemented here: Machine Learning with Python: Training and Testing the Neural Network with MNIST data set in the paragraph "Networks with multiple hidden layers".

I don't know if it's the best way to do it though. (Maybe not related but still: in the mentioned example code, it worked with sigmoid, but failed when I replaced it with ReLU).

score 0 · Answer 3 · answered Nov 15 '18 at 12:03

In my opinion I think implementing the bias matrices separately for each layer is the way to go. This will create a lot of hyper-parameters that your model will have to learn but it will give your model more freedom to converge.

For more information read this.

Proper way to implement biases in Neural Networks

3 Answers3