3

I am trying to build up caffe's infogain loss layer to work. I have seen the posts, solutions but for me it's still doesn't work

My data lmdb dimensions are Nx1xHxW (grayscale images) and my target image lmdb dimensions are Nx3xH/8xW/8 (rgb images). My last convolutional layer's dimension is 1x3x20x80. The output_size is 3, so I have 3 classes as my label numbers are (0,1,2) in the target lmdb image dataset.

I want to try out infogain loss layer, because I think I have class imbalance problem. Most of my images contains too much background.

After My last convolutional layer (conv3) I have these:

 layer {
    name: "loss"
    type: "SoftmaxWithLoss"
    bottom: "conv3"
    top: "loss"
 } 


 layer {
    bottom: "loss"
    bottom: "label"
    top: "infoGainLoss"
    name: "infoGainLoss"
    type: "InfogainLoss"
    infogain_loss_param {
      source: "infogainH.binaryproto"
    }
 }

My infogain matrix was generated by InfogainLoss layer post (as Shai suggested) so my H matrix is 1x1x3x3 dimension (an identity matrix). So my L is 3 as I have 3 classes. When I run the prototxt file everything is fine (dimensions are ok), but after my last convolution layer (conv3 layer) I get the following error:

 I0320 14:42:16.722874  5591 net.cpp:157] Top shape: 1 3 20 80 (4800)
 I0320 14:42:16.722882  5591 net.cpp:165] Memory required for data:   2892800
 I0320 14:42:16.722892  5591 layer_factory.hpp:77] Creating layer loss
 I0320 14:42:16.722900  5591 net.cpp:106] Creating Layer loss
 I0320 14:42:16.722906  5591 net.cpp:454] loss <- conv3
 I0320 14:42:16.722913  5591 net.cpp:411] loss -> loss
 F0320 14:42:16.722928  5591 layer.hpp:374] Check failed: ExactNumBottomBlobs() == bottom.size() (2 vs. 1) SoftmaxWithLoss Layer takes 2 bottom blob(s) as input.

I double checked, every lmdb dataset filename has been set correctly. I don't know what can be the problem. Any idea?

Dear @Shai

Thank you for your answer. I did the following as you mentioned:

 layer {
    name: "prob"
    type: "Softmax"
    bottom: "conv3"
    top: "prob"
    softmax_param { axis: 1 }
 }

 layer {
    bottom: "prob"
    bottom: "label"
    top: "infoGainLoss"
    name: "infoGainLoss"
    type: "InfogainLoss"
    infogain_loss_param {
        source: "infogainH.binaryproto"
    }
 }

But I still have error:

Top shape: 1 3 20 80 (4800)
I0320 16:30:25.110862  6689 net.cpp:165] Memory required for data: 2912000
I0320 16:30:25.110867  6689 layer_factory.hpp:77] Creating layer infoGainLoss
I0320 16:30:25.110877  6689 net.cpp:106] Creating Layer infoGainLoss
I0320 16:30:25.110884  6689 net.cpp:454] infoGainLoss <- prob
I0320 16:30:25.110889  6689 net.cpp:454] infoGainLoss <- label
I0320 16:30:25.110896  6689 net.cpp:411] infoGainLoss -> infoGainLoss
F0320 16:30:25.110965  6689 infogain_loss_layer.cpp:35] Check failed: bottom[1]->height() == 1 (20 vs. 1)
Community
  • 1
  • 1
experimenter
  • 63
  • 5

1 Answers1

4

What went wrong?

Your error comes from "loss" layer, not the "InfogainLoss" layer: You confused "Softmax" layer that outputs class probabilities, with "SoftmaxWithLoss" layer that outputs a (scalar) loss value.

What should you do?

  1. Replace the "loss" layer with a "prob" layer of type "Softmax" layer:

    layer {
  name: "prob"
  type: "Softmax" # NOT SoftmaxWithLoss
  bottom: "conv3"
  top: "prob"
  softmax_param { axis: 1 } # compute prob along 2nd axis
}

  2. You need to compute the loss across the second dimension, currently it seems like "InfogainLoss" layer does not support this feature. You might need to tweak "InfogainLoss" layer to have functionality like "SoftmaxWithLoss" that allows loss to be computed along arbitrary axis.
    Update: I created a pull request on BVLC/caffe that "upgrades" infogain loss layer. This upgraded version supports "loss along axis" like you are after. Moreover, it makes the "Softmax" layer redundant as it computes the probabilities internally (see this thread).
    The upgraded layer can be used like this:

    layer {
  bottom: "conv3" # prob is computed internally
  bottom: "label"
  top: "infoGainLoss"
  name: "infoGainLoss"
  type: "InfogainLoss"
  infogain_loss_param {
    source: "infogainH.binaryproto"
    axis: 1  # compute loss and probability along axis
  }
}
Community
  • 1
  • 1
Shai
  • 111,146
  • 38
  • 238
  • 371
  • I have another error. I edited my question. I tried out without infogain layer, just with the Softmax layer and caffe is writing me out the labels, and the probabilities. The latter is always 0.33333 so it doesn't learn anything. As for the labels I barely have 1, just 0 and 2 those are the background on my images, so that's why I thought this could be a class imbalance problem – experimenter Mar 20 '16 at 15:38
  • @experimenter 1. the error you get from the infogain layer now is due to the fact that you are defining a per-pixel loss, rather than a per-image loss. SoftmaxWithLoss supports this, but infogain does not. 2. it seems like you are having a class imbalance issues, you might want to tweak the infogain layer to handle per-pixel loss – Shai Mar 20 '16 at 15:59
  • Thank you very much. Can I try this modified version? I think I need to paste the code from github to my infogain_loss_layer.cpp and hpp as well. And recompile caffe with this modified version of files. Do I need to change anything else in caffe files or I just have to modify these two files? – experimenter Mar 21 '16 at 09:07
  • you can [clone from the PR](http://stackoverflow.com/a/14947921/1714410): `git clone https://github.com/shaibagon/caffe.git -b upgrade_infogain` should do the trick. You might want to consider "cherry pick" this commit to your own repository... I strongly suggest you do this change using git tools, rather than hack it yorself and copy files. It would be easier to track changes and keep updating your code if you don't break the git flow. – Shai Mar 21 '16 at 09:18