Caffe, joining outputs from 2 models

Question

I have 2 different models, let's say NM1 and NM2.

So, what I'm looking is something that works like in the example below.

Let's say that we have a picture of a dog.

NM1 predicts that it's a cat on the picture with a probability 0.52 and that it's a dog with a probability 0.48. NM2 predicts that it's a dog with a probability 0.6 and that it's a cat with a probability 0.4.

NM1 - will predict wrong NM2 - will predict correctly

NM1 + NM2 - connection will predict correctly (because 0.48 + 0.6 > 0.52 + 0.4)

So, each model ends with InnerProducts (after Softmax) which give me 2 vectors of probabilities.

Next step, I have those 2 vectors and I want to add them. Here I use Eltwise layer.

layer {
  name: "eltwise-sum"
  type: "Eltwise"
  bottom: "fc8"
  bottom: "fc8N"
  top: "out"
  eltwise_param { operation: SUM }
}

Before joining NM1 had accuracy ~70% and NM2 ~10%.

After joining accuracy can't reach even 1%.

Thus, my conclusion is that I understand something wrong and I'd be grateful if someone could explain to me where I'm wrong.

PS. I did turn off shuffle when creating lmdb.

UPDATE

layer {
  name: "eltwise-sum"
  type: "Eltwise"
  bottom: "fc8L"
  bottom: "fc8NL"
  top: "out"
  eltwise_param { 
  operation: SUM 
  coeff: 0.5
  coeff: 0.5
  }

}


#accur for PI alone
layer {
  name: "accuracyPINorm"
  type: "Accuracy"
  bottom: "fc8L"
  bottom: "label"
  top: "accuracyPiNorm"
  include {
    phase: TEST
  }
}

#accur for norm images alone
layer {
  name: "accuracyIMGNorm"
  type: "Accuracy"
  bottom: "fc8NL"
  bottom: "labelN"
  top: "accuracyIMGNorm"
  include {
    phase: TEST
  }
}

#accur for them together
layer {
  name: "accuracy"
  type: "Accuracy"
  bottom: "out"
  bottom: "label"
  top: "accuracy"
  include {
    phase: TEST
  }
}

[**DO NOT TURN SHUFFLING OFF**](http://stackoverflow.com/a/37659171/1714410). — Shai, Mar 05 '17 at 10:42
@Shai I understand that it's a problem but turning shuffling off was only one way to be sure that I get same pictures from both inputs. With shuffling it could return different images on each input. — Aleksander Monk, Mar 05 '17 at 13:25
shuffle the lines of the text file used for generating the lmdbs — Shai, Mar 05 '17 at 13:27

score 1 · Accepted Answer · answered Mar 04 '17 at 17:56

1

If you want to add (element-wise) the probabilities, you need to add after the "Softmax" layer, and not after the "InnerProduct" layer. You should have something like

layer {
  type: "InnerProduct"
  name: "fc8"
  top: "fc8"
  # ... 
}
layer {
  type: "Softmax"
  name: "prob_nm1"
  top: "prob_nm1"
  bottom: "fc8"
}
layer {
  type: "InnerProduct"
  name: "fc8N"
  top: "fc8N"
  # ... 
}
layer {
  type: "Softmax"
  name: "prob_nm2"
  top: "prob_nm2"
  bottom: "fc8N"
}
# Joining the probabilites
layer {
  type: "Eltwise"
  name: "prob_sum"
  bottom: "prob_nm1"
  bottom: "prob_nm2"
  top: "prob_sum"
  eltwise_param {
    operation: SUM
    coeff: 0.5
    coeff: 0.5
  }
}

answered Mar 04 '17 at 17:56

Shai

111,146
38
238
371

Thank you for your answer. I wrote wrong before, I used Softmax after InnerProducts. The thing that I didn't know was argument "coeff" in Eltwise layer. Maybe that was the problem! I'll try it out now and write results here. Thanks again! – Aleksander Monk Mar 04 '17 at 20:36
1

@AleksanderMonk I'm not sure this will do the trick. Are you training/finetuning the combined model? how are you measuring accuracy? – Shai Mar 04 '17 at 20:53
Yes, I'm trying to train combined model with normal images on one of the inputs and 'sharpened' pictures on the second input. I updated my question with accuracy layers. In a few words, I check accuracy for output from the first model, check accuracy for output from the second model and check accuracy for Eltwise output. – Aleksander Monk Mar 04 '17 at 21:07
I tried to run blvc_reference_caffenet only on the normal images and after 15000 iterations accuracy is 0.001. So, my guess is that without shuffling caffe can't learn anything and I would have to think about some way of creating 4D images(RGB + my other input) in LMDB. – Aleksander Monk Mar 05 '17 at 10:36

Caffe, joining outputs from 2 models

1 Answers1