10

I want my Restricted Boltzmann Machine to learn a new representation of real-valued data (see: Hinton - 2010 - A Practical Guide to Training RBMs). I'm struggling with an implementation of Gaussian linear units.

With Gaussian linear units in the visible layer the energy changes to E(v,h)= ∑ (v-a)²/2σ - ∑ bh - ∑v/σ h w. Now I don't know how to change the Contrastive Divergence Learning Algorithm. The visible units won't be sampled any more as they are linear. I use the expectation (mean-fied activation) p(v_i=1|h)= a +∑hw + N(0,1) as their state. The associations are left unchangend ( pos: data*p(h=1|v)' neg: p(v=1|h)*p(h=1|v)' ). But this only leads to random noise when I want to reconstruct the data. The error rate will stop improving around 50%.

Finally I want to use Gaussian linear units in both layers. How will I get the states of the hidden units then? I suggest by using the mean-field activation p(h_i=1|v)= b +∑vw + N(0,1) but I'm not sure.

tessi
  • 13,313
  • 3
  • 38
  • 50
theo
  • 111
  • 2
  • 5

2 Answers2

3

You could take a look at the gaussian RBM that Hinton himself has provided Please find it here. http://www.cs.toronto.edu/~hinton/code/rbmhidlinear.m

kunjans1
  • 39
  • 2
  • 1
    Note that [link-only answers](http://meta.stackoverflow.com/tags/link-only-answers/info) are discouraged, SO answers should be the end-point of a search for a solution (vs. yet another stopover of references, which tend to get stale over time). Please consider adding a stand-alone synopsis here, keeping the link as a reference. – kleopatra Oct 18 '13 at 09:48
  • Useful example. It's slightly different than what OP wanted - the hidden layer is linear-gaussian and the visible is bernoulli and sigma is not learned. – Peter Jun 01 '14 at 16:22
1

I have been working on a similar project implementing an RBM with c++ and matlab mexfunction. i have found from the implementation of Professor Hinton (in matlab) that the the binary activation for visible units uses the simoid function. ie : vs = sigmoid(bsxfun(@plus, hs*obj.W2', obj.b)); But when implementing the gaussian visible units RBM you simply have to sample the visible units without using the simoid. ie: vs = bsxfun(@plus, h0*obj.W2', obj.b); or better look at the Professor Hinton implementation (see: http://www.cs.toronto.edu/~hinton/code/rbmhidlinear.m)

JNW
  • 79
  • 1
  • 2
  • 8