2

I'm working on a RBM (for a DBN) for image classification. I' working with two RBM layers. The first has Gaussian visible units and binary hidden units and the second has binary visible units and softmax hidden units. It works quite well. I now want to try to use Noise Rectified Linear Unit as the hidden layer, but I fail to understand how to implement them. Everything I've tried has just led to terrible results.

Now, if I understand correctly: The activation probability of a ReLU is simply p = max(0, x + N(0, 1)), but then how do I sample the values to activate the visible units ? Should the noise only be used in sampling and not in activation probabilities ?

Another thing: in some papers I saw that the noise is to be N(0,1) and some others use N(0,sigmoid(x)).

So, what should be the activation function and how the values should be sampled ?

Baptiste Wicht
  • 7,472
  • 7
  • 45
  • 110

1 Answers1

3

Apparently:

Using max(0, x) as activation probability function and using max(0, x + N(0, Sigmoid(x)) for sampling seems to work for the RBM.

Baptiste Wicht
  • 7,472
  • 7
  • 45
  • 110
  • Hi, I would like to ask you how you managed connection to second RBM. prolem I am having: I have perfectly working gaussian-rectified RBM, where input for gaussian units comes from N(0, 1) for each element in vector separately. Rectified units are calculated by ln(1 + e^x). This machine works well and minimizes error better than gaussian-binary. Now what if I would like to create DBN? In sense adding extra RBM behind rectified hidden units? Do I need some sort of normalization or something? Gaussian vs rectified isn't compatible, is it? Does it correspond with fine-tuning later? thanks – Snurka Bill Jul 12 '15 at 13:31