self-defined conneted network using tesnsorflow take too much storage and failed to train

Question

For my project i write a not-fully connected feedforword network using tensorflow.

I only use scalar variables to generate weights, rather than matrix variables, because of not-fully connectivity.

For this reason i get the flexibility to connet neurons from two neighouring layers respectively.

Unfortunately the training of this network take too much storages. enter image description here Also the training failed,if the network has more layers (i have an iteration to define the number of layers)

I have released my code on github: https://github.com/hezhouyuanren/BP-like-Decoder.git . I'm hopping for your help

Answer 1

Instead of using a single scalar value for each weight, which will probably result in a huge graph structure, you can make regular fully-connected layers and then remove the connections that you don't need. For example, for an input X with N features and a layer of size M, you would have a typical weight matrix W of size N x M. You can then have another matrix C, also N x M such that C_ij is 1 if there is a connection between input feture i and layer neuron j and 0 if there is not. The output of the layer (before activation) would then be X * (W · C), where * is the usual matrix product and · is the element-wise product. This should also play well with the training (assuming C is not trainable), since values where C is 0 will just not take affect the parameter updates. The only issue is that you are using a bigger matrix than you need, but unless you have very large number of very big layers, it shouldn't be much of a problem (surely less problematic than having thousands of scalar graph nodes).