Unsupervised pre-training for convolutional neural network in theano

Question

I would like to design a deep net with one (or more) convolutional layers (CNN) and one or more fully connected hidden layers on top.
For deep network with fully connected layers there are methods in theano for unsupervised pre-training, e.g., using denoising auto-encoders or RBMs.

My question is: How can I implement (in theano) an unsupervised pre-training stage for convolutional layers?

I do not expect a full implementation as an answer, but I would appreciate a link to a good tutorial or a reliable reference.

Note that unsupervised pre-training is hardly ever used any more (as far as I know). — pir, Apr 08 '16 at 08:27
@pir: Why is it hardly used anymore? Is there something new that's better? — Øystein Schønning-Johansen, Nov 16 '16 at 11:38
Better initialization (He et. al 2016) and optimizers (Adam) has rendered it mostly unnecessary. — pir, Nov 16 '16 at 17:00

score 29 · Accepted Answer · answered Sep 25 '14 at 20:29

This paper describes an approach for building a stacked convolutional autoencoder. Based on that paper and some Google searches I was able to implement the described network. Basically, everything you need is described in the Theano convolutional network and denoising autoencoder tutorials with one crucial exception: how to reverse the max-pooling step in the convolutional network. I was able to work that out using a method from this discussion - the trickiest part is figuring out the right dimensions for W_prime as these will depend on the feed forward filter sizes and the pooling ratio. Here is my inverting function:

    def get_reconstructed_input(self, hidden):
        """ Computes the reconstructed input given the values of the hidden layer """
        repeated_conv = conv.conv2d(input = hidden, filters = self.W_prime, border_mode='full')

        multiple_conv_out = [repeated_conv.flatten()] * np.prod(self.poolsize)

        stacked_conv_neibs = T.stack(*multiple_conv_out).T

        stretch_unpooling_out = neibs2images(stacked_conv_neibs, self.pl, self.x.shape)

        rectified_linear_activation = lambda x: T.maximum(0.0, x)
        return rectified_linear_activation(stretch_unpooling_out + self.b_prime.dimshuffle('x', 0, 'x', 'x'))

Hi senecaur - I read the paper but I could not understand the architecture of the net. Can you give more details about the architecture of the net you used to pretrain? Layer by Layer. And then - did you use the same net to finetune ? i.e just changed the output layer from images to supervised labels ? Thanks a lot — Run2, Feb 11 '15 at 09:38
I have raised a relevant question in CrossValidated http://stats.stackexchange.com/questions/137537/what-is-the-architecture-of-a-stacked-convolutional-autoencoder. I will be grateful if senecaur - you can drop in an answer there. Thanks — Run2, Feb 13 '15 at 09:47
@senecaur If you're using tied weights, W_prime should be flipped version of the original W (or am I wrong?). How do you manage to create this matrix in Theano notation? — mmohaveri, Feb 18 '15 at 16:43
@senecaur As there seems to be interest in using stacked convolutional autoencoders and afaik no implementation can be found on the internet, would you mind sharing/releasing your implementation? — shapecatcher, Apr 29 '15 at 12:29

Unsupervised pre-training for convolutional neural network in theano

1 Answers1

Linked