Connecting perceptrons with output of previous ones?

Question

Because of the help I received and researched here I was able to create a simple perceptron in C#, code of which goes like:

       int Input1 = A;
       int Input2 = B;
        //weighted sum
        double WSum = A * W1 + B * W2 + Bias;

        //get the sign: -1 for negative, +1 for positive
        int Sign=Math.Sign(WSum);

        double error = Desired - Sign;
        //updating weights
        W1 += error * Input1 * 0.1; //0.1 being a learning rate
        W2 += error * Input2 * 0.1;
        return Sign;

I do not use Sigmoid here and just get -1 or 1.
I would have two questions:
1) Is that correct that my weights get values like -5 etc? When input is e.g. 100,50 it goes like: W1+=error*100*0.1
2) I want to proceed deeper and create more connected neurons - I guess I would need at least two to provide inputs to the third. Is that correct that the third will be fed with values just -1..1? I am aiming to a simple pattern recognition but so far do not understand how it should work.

score 0 · Answer 1 · answered Mar 28 '13 at 08:15

0

Try to set your weights as double.

Also i think it's much better to work with arrays, especially in neural networks and perceptron is the only way.

And you will need some for or while loops to succeed what you want.

answered Mar 28 '13 at 08:15

Mhche

143
14

Yes this is a just a raw code, I do have weights as a double but there are like 5,1, I mean it is not just -1 and 1. The articles I read all say weights are in this interval. – John V Mar 28 '13 at 08:25
Are you sure it's not any overflow problem?Have you tried long double?Also sure you need arrays i think there is the problem. – Mhche Mar 28 '13 at 08:28
Not sure what you mean. There is no overflow, just I train my perceptron and weights become around 4-5 (well I cannot see how they can be just -1 1 in this case). – John V Mar 28 '13 at 08:31
double Sign=Math.Sign(WSum); I am not sure either what you want, i don't want to confuse you more. – Mhche Mar 28 '13 at 08:38

score 0 · Accepted Answer · edited May 23 '17 at 10:32

It is perfectly valid that the values of your weights range from -Infinity to +Infinity. You should always use real numbers instead of integers (so as mentioned above, double will work. 32 bit floats precision is perfectly sufficient for neural networks).

Moreover, you should decay your learning rate with every learning step, e.g. reduce it by a factor of 0.99 after each update. Else, your algorithm will oscillate when approaching an optimum.

If you want to go "deeper", you will need to implement a Multilayer Perceptron (MLP). There exists a proof that a neural network with simple threshold neurons and multiple layers alsways has an equivalent with only 1 layer. This is why several decades ago the research community temporarily abandoned the idea of artificial neural networks. 1986, Geoffrey Hinton made the Backpropagation algorithm popular. With it you can train MLPs with multiple hidden layers.

To solve non-linear problems like XOR or other complex problems like pattern recognition, you need to apply a non-linear activation function. Have a look at the logistic sigmoid activation function for a start. f(x) = 1. / (1. + exp(-x)). When doing this you should normalize your input as well as your output values to the range [0.0; 1.0]. This is especially important for the output neurons since the output of the logistic sigmoid activation function is defined in exactly this range.

A simple Python implementation of feed-forward MLPs using arrays can be found in this answer.

Edit: You also need at least 1 hidden layer to solve e.g. XOR.

Thanks. Im not that strong in math, though. If you dont mind: why the input should be normalized to 0..1 range? So basically, each neuron will produce an output of 0..1 and it is the output for the others? — John V, Mar 28 '13 at 15:36
This is well explained in [this answer](http://stackoverflow.com/questions/4674623/why-do-we-have-to-normalize-the-input-for-an-artificial-neural-network) and [this link](http://www.faqs.org/faqs/ai-faq/neural-nets/part2/). In a multilayer perceptron, the output of one neuron is one of the inputs for each neuron in the following layer (except for the output layer, naturally). — schreon, Mar 28 '13 at 16:42

Connecting perceptrons with output of previous ones?

2 Answers2