Train Caffe CNN to output multidimensional features

Question

I'd like to build a feature extractor using Caffe's CNNs and I already have a large sample of input features and desired output features.

Now I need to train some convolutional layers to learn how to transform the input features into the output.

My question is: How can I achieve this on Caffe?

As a minimal example, suppose I wanted to train a CNN that inverts the values of a 2D array.

For example, if my input is

[[0,1,0],
 [1,1,1],
 [0,1,0]]

the CNN should output

[[1,0,1],
 [0,0,0],
 [1,0,1]].

For

[[0,0,0],
 [0,1,0],
 [0,0,0]]

the output should be

[[1,1,1],
 [1,0,1],
 [1,1,1]]

and so on.

Of course this is just a minimal example to share, the actual problem is nearly impossible to tackle without the use of multiple convolutions.

I was able to create this code for this problem. I used the Euclidean Loss at the end, but unfortunately the CNN is not learning anything.

ROOT_DIR = '/home'

from os.path import join
import numpy as np
import h5py
from itertools import product
import caffe
from caffe import layers
from caffe.proto import caffe_pb2

#%% GENERATE DATA
data_in = np.array([np.array(seq).reshape(1,3,3) for seq in product([0,1], repeat=9)])
data_out = np.array([-1*array+1 for array in data_in])

with open(join(ROOT_DIR, 'data.txt'), 'w') as ftxt:
    with h5py.File(join(ROOT_DIR, 'data.hdf5'), 'w') as fhdf5:
        fhdf5['data'] = data_in.astype(np.float32)
        fhdf5['label'] = data_out.astype(np.float32)
        ftxt.write(join(ROOT_DIR, 'data.hdf5'))

#%%DEFINE NET
net = caffe.NetSpec()
net.data, net.label = layers.HDF5Data(batch_size=64, source=join(ROOT_DIR, 'data.txt'), ntop=2)
net.conv1 = layers.Convolution(net.data, kernel_size=1, num_output=128)
net.relu1 = layers.ReLU(net.conv1, in_place=True)
net.conv2 = layers.Convolution(net.relu1, kernel_size=1, num_output=1)
net.relu2 = layers.ReLU(net.conv2, in_place=True)
net.loss = layers.EuclideanLoss(net.relu2, net.label)
net.to_proto()

with open(join(ROOT_DIR, 'invert_net.prototxt'), 'w') as f:
    f.write(str(net.to_proto()))

#%% DEFINE SOLVER
solver = caffe_pb2.SolverParameter()
solver.train_net = join(ROOT_DIR, 'invert_net.prototxt')
solver.max_iter = 10000
solver.base_lr = 0.01
solver.lr_policy = 'fixed'

with open(join(ROOT_DIR, 'solver.prototxt'), 'w') as f:
    f.write(str(solver))

#%% TRAIN NET
caffe.set_mode_cpu()
solver = caffe.SGDSolver(join(ROOT_DIR, 'solver.prototxt'))
solver.solve()