When we say "non-linearity of deep neural networks", what do we actually mean by the term "non-linearity" in this context ?
Also, the purpose of the activation function is to introduce non-linearity into the network. What does this non-linearity means ? (I am new to Deep learning.)
non-linear means that the output cannot be reproduced from a linear combination of the inputs (which is not the same as output that renders to a straight line--the word for this is affine).
another way to think of it: without a non-linear activation function in the network, a NN, no matter how many layers it had, would behave just like a single-layer perceptron, because summing these layers would give you just another linear function (see definition just above).
