0

Let our model as several fully-connected layers:

enter image description here

I want to share middle layers and use two models with same weights like this:

enter image description here

Can I make this with Tensorflow?

Barbora
  • 921
  • 1
  • 6
  • 11
Depth
  • 1
  • 1
  • Short answer is, that it is possible. If you are using the same `tf.Graph` for both models, you could achieve this behavior by reusing the variable scopes. Could you provide a code snippet of what have you tried? – Jindra Helcl Dec 15 '18 at 14:58

1 Answers1

2

Yes, you can share layers! The official TensorFlow tutorial is here.

In your case, layer sharing could be implemented using variable scopes

#Create network starts, which are not shared
start_of_net1 = create_start_of_net1(my_inputs_for_net1)
start_of_net2 = create_start_of_net2(my_inputs_for_net2)

#Create shared middle layers
#Start by creating a middle layer for one of the networks in a variable scope
with tf.variable_scope("shared_middle", reuse=False) as scope:
    middle_of_net1 = create_middle(start_of_net1)
#Share the variables by using the same scope name and reuse=True
#when creating those layers for your other network
with tf.variable_scope("shared_middle", reuse=True) as scope:
    middle_of_net2 = create_middle(start_of_net2)

#Create end layers, which are not shared
end_of_net1 = create_end_of_net1(middle_of_net1)
end_of_net2 = create_end_of_net2(middle_of_net2)

Once a layer has been created in a variable scope, you can reuse the variables in that layer as many times as you like. Here, we just reuse them once.

Jeffrey Ede
  • 601
  • 5
  • 11
  • I think this answer no longer works in Tensorflow 2. Can you update it? Thanks! – a06e Jan 08 '20 at 12:34
  • @becko I'm still using tensorflow 1.7, however, this [accepted answer](https://stackoverflow.com/questions/56889038/tensorflow-2-0-how-make-share-parameters-among-convolutional-layers) may help. In tensorflow 2, you can declare parts of your model like functions to be reused. If you are using tensorflow 1.x, [Sonnet](https://github.com/deepmind/sonnet) provides similar variable sharing. – Jeffrey Ede Jan 13 '20 at 17:21