Nolearn/Lasagne neural network not starting training

Question

I am using Theano 0.7, nolearn 0.6adev and lasagne 0.2.dev1 to train a neural net on GPU (in an IPython 3.2.1 notebook). However, because of the first layer ('reduc'), the following network does not start training, aven after waiting a few hours:

import theano
from lasagne import layers
from lasagne.updates import nesterov_momentum
from nolearn.lasagne import NeuralNet
from nolearn.lasagne import BatchIterator
from lasagne import nonlinearities
from lasagne import init
import numpy as np

testNet = NeuralNet(
    layers=[(layers.InputLayer, {"name": 'input', 'shape': (None, 12, 1000, )}),

            (layers.Conv1DLayer, {"name": 'reduc', 'filter_size': 1, 'num_filters': 4, 
                                  "nonlinearity":nonlinearities.linear,}),

            (layers.Conv1DLayer, {"name": 'conv1', 'filter_size': 25, 'num_filters': 100,
                                 'pad': 'same', }),
            (layers.MaxPool1DLayer, {'name': 'pool1', 'pool_size': 5, 'stride': 3}),



            (layers.Conv1DLayer, {"name": 'conv2', 'filter_size': 15, 'num_filters': 100,
                                 'pad': 'same', 
                                  'nonlinearity': nonlinearities.LeakyRectify(0.2)}),

            (layers.MaxPool1DLayer, {'name': 'pool2', 'pool_size': 5, 'stride': 2}),

            (layers.Conv1DLayer, {"name": 'conv3', 'filter_size': 9, 'num_filters': 100,
                                 'pad': 'same', 
                                  'nonlinearity': nonlinearities.LeakyRectify(0.2)}),

            (layers.MaxPool1DLayer, {'name': 'pool3', 'pool_size': 2}),

            (layers.Conv1DLayer, {"name": 'conv4', 'filter_size': 5, 'num_filters': 20,
                                 'pad': 'same', }),

            (layers.Conv1DLayer, {"name": 'conv5', 'filter_size': 3, 'num_filters': 20,
                                 'pad': 'same',}),


            (layers.DenseLayer, {"name": 'hidden1', 'num_units': 10, 
                                 'nonlinearity': nonlinearities.rectify}),

            (layers.DenseLayer, {"name": 'output', 'nonlinearity': nonlinearities.sigmoid, 
                                 'num_units': 5})
           ],

    # optimization method:
    update=nesterov_momentum,
    update_learning_rate=5*10**(-3),
    update_momentum=0.9,

    regression=True,  
    max_epochs=1000,
    verbose=1,
    )

testNet.fit(np.random.random([3000, 12, 1000]).astype(np.float32), 
            np.random.random([3000, 5]).astype(np.float32))

If I comment out the first layer, the training starts in a few seconds. Training more complicated networks is not a problem either. Any idea of what's causing the issue?

Edit: oddly enough, if I remove conv4 and conv5, the training also starts within a reasonable time.

Edit2: what's even stranger is, if I change the size of the filters to 10 in the layer reduc, then the training starts in a reasonable amount of time. If after that I stop the cell's execution, change this value to 1, and reexecute the cell, the training goes fine...

Finally I've started using another framework, but if someone's interested, here's the link to the thread I started on the lasagne user group.

Nolearn/Lasagne neural network not starting training

0 Answers0