Vectorize Forward Propagation

Question

I have made a neural network from scratch and want to make it run a little bit faster. I'm wondering if vectorizing my forward prop would make it faster. My current forward prop code is:

def forwardProp(self, inputs):
    for i in range (self.dimensions[1]):
        self.secondLayerNeurons[i] = self.relu(np.dot(self.firstLayerWeights[i], inputs)+self.firstLayerBiases[i])
    for i in range (self.dimensions[2]):
        self.outputNeurons[i] = self.sigmoid(np.dot(self.secondLayerWeights[i], self.secondLayerNeurons)+self.secondLayerBiases[i])

If vectorization will make this faster, how would I vectorize this? Thanks in advance!

Pranav Hosangadi · Accepted Answer · 2020-08-10T18:10:49.573

0

I'm wondering if vectorizing my forward prop would make it faster

Yes!

How would I vectorize this?

You want to eliminate loops and figure out vector algebra that will do the same thing.

Let's say self.firstLayerWeights.shape is (N, D). You want to calculate the row-wise dot product of this matrix with inputs. Let's say you implement this logic in a function called rowwise_dot

def rowwise_dot(inpA, inpB):
    # Calculate and return rowwise dot

Now you have the rowwise_dot function, you can add the entire vector of self.firstLayerBiases without having to loop,

rowwise_dot(self.firstLayerWeights, inputs) + self.firstLayerBiases

Next, make sure self.relu and self.sigmoid can take vectors and return whatever you need them to for each element of the vector. This might involve similar shenanigans as vectorizing the rowwise dot product.

So in the end you have:

def forwardProp(self, inputs):
    self.secondLayerNeurons = self.relu(rowwise_dot(self.firstLayerWeights, inputs) + self.firstLayerBiases)
    self.outputNeurons = self.sigmoid(rowwise_dot(self.secondLayerWeights, self.secondLayerNeurons) + self.secondLayerBiases)

edited Aug 10 '20 at 18:10

answered Aug 10 '20 at 18:02

Pranav Hosangadi

23,755
7
44
70

Current Sigmoid: return 1/(1+np.exp(-x)) where x is a double, how would I vectorize that? – Joey Aug 10 '20 at 18:07
It's already vectorized. `1/(1 + np.exp(-x))` already gives you the element-wise sigmoid function of the vector `x` – Pranav Hosangadi Aug 10 '20 at 18:11
Would there be any numpy functions that function like the rowwise function? – Joey Aug 10 '20 at 18:14
Many numpy functions have an `axis` argument that perform the operation along the specified axis. – Pranav Hosangadi Aug 10 '20 at 18:15
Thanks, how would I make relu so that it can take in an array without using a loop? The code is currently return max(0.0, x) – Joey Aug 10 '20 at 18:18
For `relu`, `max(0, x)` will not work because you want `x_return[i] = max(0, x[i])`. – Pranav Hosangadi Aug 10 '20 at 18:18
`x_return = x[:]; x_return[x < 0] = 0; return x_return;` should work. – Pranav Hosangadi Aug 10 '20 at 18:19
Thanks a lot, i'll try it. – Joey Aug 10 '20 at 18:19

score -1 · Answer 2 · answered Oct 12 '20 at 19:37

Vectorizing forward prop makes the MLP run a lot faster. I used the @ operator.

    def forwardProp(self, inputs):
        self.secondLayerNeurons = self.sigmoid(self.w1 @ inputs + self.b1)
        self.outputNeurons = self.sigmoid(self.w2 @ self.secondLayerNeurons + self.b2)

Vectorize Forward Propagation

2 Answers2