3

I have been trying to implement back-propagation neural networks for a while now and i am facing issues time after time. The progress so far is that my neural network works fine for XOR, AND and OR.

Following image shows the training of my neural network for XOR over 100000 iterations and it seems to converge well. For this i had 2 input neurons and a single output neuron with a hidden layer with 2 neurons it [ though 1 would have been sufficient ] enter image description here

Now progressing forward i trained the same network two distinguish coordinates in XY plane into two classes with the same structure of 2 input neuron and 1 output neuron and single hidden layer with two neurons: Dataset enter image description here

For the next i trained it for two classes only but with 2 output neurons and keeping rest of the structure same and it did take long this time to converge but it did. enter image description here But now i increased to three classes; class A will be 100 and class B will be 010 and class C will be 001 but now when i train it then it never converges and gives me the following result for the data shown below: enter image description here enter image description here

It never seems to converge. And i have observed this pattern of that if i increase the number of neurons in my output layer the error rate increases like anything? Can i anyone direct me to where i am going wrong?

Community
  • 1
  • 1
KungFu_Panda
  • 111
  • 11
  • What are output neurons levels of logic? 0-0.3 for green, 0.3-0.7 for blue, 0.7-1.0 for red? – huseyin tugrul buyukisik Feb 17 '15 at 23:58
  • No! actually my output layer has 3 neurons so if its classified as of class A then the output layer's first neuron should be triggered and complete output from the layer will be 100 and similarly if class b then complete output will be 010 and so on – KungFu_Panda Feb 18 '15 at 00:09
  • What is sum of those outputs? – huseyin tugrul buyukisik Feb 18 '15 at 00:11
  • sum of outputs??.. i am taking the one as described above as my final output only.. like my output will be either [1.0,0.0,0.0] or [0.0,1.0,0.0] or [0.0,0.0,1.0] in ideal cases.. i hope this makes my problem more clear – KungFu_Panda Feb 18 '15 at 00:22
  • 1
    Just out of interest, I noticed the Java tag.. What NN library are you using for this? – Richard Feb 18 '15 at 13:20
  • @Richard i have implemented neural network with back-propagation myself.. so no library has been used. – KungFu_Panda Feb 18 '15 at 14:49

1 Answers1

5

If you move from binary classification to multiclass classification, you'll have to generalize your backpropagation algorithm to properly handle more than two classes.

The main differences to binary classification are that the update changes to:

update

with:

y

being the new score where the argument y (output) is chosen that yields the highest score for the features times the weight vector w. This strategy is called one-vs.-rest. Written as pseudocode (from here):

pseudocode

Keep in mind that depending on your own specific code you might have to perform additional changes (e.g. map the real valued output to binary output per output unit) to your current code.

Your architecture of having 1 binary output node per class is perfectly fine if you have multiple independent attributes by which you classify. Otherwise, you should consider using a softmax output layer (see here for an implementation example). Using softmax activation in the output layer will convert your raw values to posterior probabilities (as opposed to binary outputs per class). Since this gives you a measure of certainty it also gives you more insight.

Community
  • 1
  • 1
runDOSrun
  • 10,359
  • 7
  • 47
  • 57
  • 3
    I don't know anything about neural networks, but this is a cool answer. – Luminous Feb 18 '15 at 13:16
  • thanks for the answer! but i will have study a lot to understand the points you have made here and then get to implement those changes you suggested.. but really appreciate for all the resources you have included :) – KungFu_Panda Feb 18 '15 at 15:00