How to convert the CIFAR10 tutorial to NCHW

Question

I'm trying to convert the Tensorflow CIFAR10 tutorial from NHWC to NCHW, but can't figure out how to do so. I have only found answers such as this, which is a couple of lines of code without an explanation of how it works and where to use it. Here are a couple of unsuccessful attempts I have made using this approach:

def inference(images):

    with tf.variable_scope('conv1') as scope:
    kernel = _variable_with_weight_decay('weights',
                                     shape=[5, 5, 3, 64],
                                     stddev=5e-2,
                                     wd=0.0)

    # ****************************************************************** #

    ### Original
    conv = tf.nn.conv2d(images, kernel, [1, 1, 1, 1], padding='SAME')

    ### Attempt 1
    imgs = tf.transpose(images, [0, 3, 1, 2]) # NHWC -> NCHW
    conv = tf.nn.conv2d(imgs, kernel, [1, 1, 1, 1], padding='SAME')
    conv = tf.transpose(conv, [0, 2, 3, 1]) # NCHW -> NHWC

    ### Attempt 2
    kern = tf.transpose(kernel, [0, 3, 1, 2]) # NHWC -> NCHW
    conv = tf.nn.conv2d(images, kern, [1, 1, 1, 1], padding='SAME')
    conv = tf.transpose(conv, [0, 2, 3, 1]) # NCHW -> NHWC

    # ****************************************************************** #

    biases = _variable_on_cpu('biases', [64], tf.constant_initializer(0.0))
    pre_activation = tf.nn.bias_add(conv, biases)
    conv1 = tf.nn.relu(pre_activation, name=scope.name)
    _activation_summary(conv1)
    
    ...

Which get the errors (respectively):

ValueError: Dimensions must be equal, but are 24 and 3 for 'conv1/Conv2D' (op: 'Conv2D') with input shapes: [64,3,24,24], [5,5,3,64].

ValueError: Dimensions must be equal, but are 3 and 5 for 'conv1/Conv2D' (op: 'Conv2D') with input shapes: [64,24,24,3], [5,64,5,3].

Can someone please provide a set of steps I can follow to convert this example to NCHW successfully.

Andre Holzner · Accepted Answer · 2018-01-08T20:30:13.320

5

In your attempt #1 , try the following:

conv = tf.nn.conv2d(imgs, kernel, [1, 1, 1, 1], padding='SAME', data_format = 'NCHW')

(i.e. add data_format = 'NCHW' to the parameters)

e.g. as follows:

import tensorflow as tf

config = tf.ConfigProto()
config.gpu_options.allow_growth = True

with tf.Session(config=config) as session:

    kernel = tf.ones(shape=[5, 5, 3, 64])
    images = tf.ones(shape=[64,24,24,3])

    imgs = tf.transpose(images, [0, 3, 1, 2]) # NHWC -> NCHW
    conv = tf.nn.conv2d(imgs, kernel, [1, 1, 1, 1], padding='SAME', data_format = 'NCHW')
    conv = tf.transpose(conv, [0, 2, 3, 1]) # NCHW -> NHWC

    print("conv=",conv.eval())

edited Jan 08 '18 at 20:30

answered Jan 08 '18 at 16:08

Andre Holzner

18,333
6
54
63

Whoops, I left that off. I tried that with the original but not with the attempts above. With that now, I get "CUDA_ERROR_OUT_OF_MEMORY; total memory reported 11713708032". My GPU is a 1080 Ti by the way. Any ideas? – Mark Sonn Jan 08 '18 at 16:25
1

do you (or somebody) else have some tensorflow session running on the same GPU(s) ? `nvidia-smi` will show you how much memory is already used on the GPU(s). By default tensorflow grabs all available memory on all GPUs it finds according to my experience (see also https://stackoverflow.com/questions/34199233 ). – Andre Holzner Jan 08 '18 at 16:36
No, just me running this one session – Mark Sonn Jan 08 '18 at 17:26
I added an example (with dummy data) to the answer which does not run out of memory for me on a Titan X (Pascal) – Andre Holzner Jan 08 '18 at 20:32
If you understand how the transpose works, could you please add a small explanation to your answer :) – Mark Sonn Jan 09 '18 at 08:35

How to convert the CIFAR10 tutorial to NCHW

1 Answers1

Linked