pruning tensorflow connections and weights (using cifar10 cnn)

Question

I'm using tensorflow to run a cnn for image classification. I use tensorflow cifar10 cnn implementation.(tensorflow cifar10) I want to decrease the number of connections, meaning I want to prune the low-weight connections. How can I create a new graph(subgraph) without some of the nuerones?

score 1 · Answer 1 · edited May 23 '17 at 11:53

Tensorflow does not allow you lock/freeze a particular kernel of a particular layer, that I have found. The only I've found to do this is to use the tf.assign() function as shown in

How to freeze/lock weights of one Tensorflow variable (e.g., one CNN kernel of one layer

It's fairly cave-man but I've seen no other solution that works. Essentially, you have to .assign() the values every so often as you iterate through the data. Since this approach is so inelegant and brute-force, it's very slow. I do the .assign() every 100 batches.

Someone please post a better solution and soon!

score 0 · Answer 2 · answered Feb 15 '16 at 00:35

0

The cifar10 model you point to, and for that matter, most models written in TensorFlow, do not model the weights (and hence, connections) of individual neurons directly in the computation graph. For instance, for fully connected layers, all the connections between the two layers, say, with M neurons in the layer below, and 'N' neurons in the layer above, are modeled by one MxN weight matrix. If you wanted to completely remove a neuron and all of its outgoing connections from the layer below, you can simply slice out a (M-1)xN matrix by removing the relevant row, and multiply it with the corresponding M-1 activations of the neurons.

answered Feb 15 '16 at 00:35

keveman

8,427
1
38
46

You could zero-out the corresponding weight in the weight matrix to "prune" an individual connection. Of course, subsequent training may cause the weight to be changed back to a non-zero value. – Nimrand Apr 24 '16 at 02:29
Does TF skip the convolution on masked matrices on runtime or it performs the convolution but with zero weight values ? – Konstantinos Monachopoulos Oct 08 '18 at 14:04

NokiaDDT · Answer 3 · 2018-06-01T02:09:30.170

0

Another way is add an addition mask to control the connections.

The first step involves adding mask and threshold variables to the layers that need to undergo pruning. The variable mask is the same shape as the layer's weight tensor and determines which of the weights participate in the forward execution of the graph.

There is a pruning implementation under tensorflow/contrib/model_pruning to prune the model. Hope this can help you to prune model quickly.

https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/model_pruning

edited Jun 01 '18 at 02:09

answered May 24 '18 at 07:58

NokiaDDT

1
2

So, by adding the masks and re-training, we can identify which weights are not contributing. But how can you actually remove these weight tensors afterwards and save a smaller model (ideally in a .pb format) ? – Konstantinos Monachopoulos Aug 03 '18 at 23:58

score 0 · Answer 4 · answered Oct 08 '18 at 14:09

I think google has an updated answer here : https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/model_pruning

Removing pruning nodes from the trained graph:

$ bazel build -c opt contrib/model_pruning:strip_pruning_vars
$ bazel-bin/contrib/model_pruning/strip_pruning_vars --checkpoint_path=/tmp/cifar10_train --output_node_names=softmax_linear/softmax_linear_2 --filename=cifar_pruned.pb

I suppose that cifar_pruned.pb will be smaller, since the pruned "or zero masked" variables are removed.

Does it actually remove the connections between the layers or just mask and threshold variables? — Illuminati0x5B, May 22 '19 at 21:37

pruning tensorflow connections and weights (using cifar10 cnn)

4 Answers4