21

TL.DR. Is there a 3-dimensional friendly implementation of theano.tensor.nnet.neighbours.images2neibs?

I would like to perform voxel-wise classification of a volume (NxNxN) using a neural network that takes in a nxnxn image, where N>n. To classify each voxel in the volume, I have to iterate through each voxel. For each iterration, I obtain and pass the neighborhood voxels as the input to the neural network. This is simply a sliding window operation, which the operation is the neural network.

While my neural network is implemented in Theano, the sliding window implementation is in python/numpy. Since this is not a pure Theano operation, the classification takes forever (> 3 hours) to classify all voxels in one volume. For 2d sliding window operation, Theano has a helper method, theano.tensor.nnet.neighbours.images2neibs, is there a similar implementation for 3-dimensional images?

Edit: There are existing numpy solutions (1 and 2) for n-d sliding window, both use np.lib.stride_tricks.as_strided to provide "views of the sliding window", thus preventing memory issues. In my implementation, the sliding window arrays are being passed from numpy (Cython) to Python and then to Theano. To boost performance, it's likely I have to bypass Python.

galfisher
  • 1,122
  • 1
  • 13
  • 24
pangyuteng
  • 1,749
  • 14
  • 29
  • related discussion. https://github.com/Theano/Theano/issues/2166 – pangyuteng Feb 29 '16 at 04:27
  • 1
    Alternatively, maybe you want to check out `sklearn.feature_extraction.image.extract_patches`. This can give you a view onto the desired `nxnxn` cubes without making a copy of the data. Combine it with an `np.einsum` which also doesn't copy and you may get something that runs in acceptable time (no guarantee, never tried) – eickenberg Feb 29 '16 at 12:03
  • Thanks Eickenberg. I'll need to take a look at `np.einsum`! – pangyuteng Feb 29 '16 at 17:36
  • 1
    fyi `sklearn.feature_extraction.image.extract_patches` also uses stride tricks to do its work. It is just a few lines of code and calculation to get the right shape of the views. – eickenberg Mar 01 '16 at 08:31
  • I think OverfeatTransformer from sklearn_theano.feature_extraction.overfeat is what I'm looking for. Guess who the author is. :) – pangyuteng Mar 02 '16 at 07:00
  • 1
    Hmm, that works, but only for color images. How are you thinking of extending it to 3D volumes? – eickenberg Mar 02 '16 at 10:11
  • OverfeatTransformer would be a good template for me to work with. Thanks for sharing sklearn-theano! – pangyuteng Mar 03 '16 at 05:35
  • For those that ended up here, I recommend using fully convolutional neural network structure as opposed to voxel-wise classification for classifying pixels/voxels. see J Long, Fully Convolutional Networks for Semantic Segmentation; B Hariharan, Hypercolumns for object segmentation and fine-grained localization. – pangyuteng Jan 06 '17 at 00:38
  • 1
    hmm are you sure? Please also see [this comment](https://plus.google.com/+PierreSermanet/posts/VngsFR3tug9) about the naming of these things as though they didn't exist before. (Just adding this for completeness - I haven't looked at the paper you refer to and in the end it is the results that count) – eickenberg Jan 06 '17 at 14:14

1 Answers1

1

Eickenberg and Kastner's OverfeatTransformer utility in sklearn_theano.feature_extraction.overfeat would be a good match for this operation, as mentioned by OP.

J_H
  • 17,926
  • 4
  • 24
  • 44