1

I want to use the keras in multi-gpus with the applications (such VGG16). But there are some error.

I try to use the single-gpus it is correct. But the multi-gpus is wrong. The code like this:

import keras
    with tf.device('/cpu:0'):
        input1 = keras.layers.Input(config.input_shape)
        input2 = keras.layers.Input(config.input_shape)
        sub_model = keras.applications.VGG16(include_top=False, weights=config.VGG_MODEL_PATH,
                                             input_shape=config.input_shape)
        output1 = sub_model(input1)
        output2 = sub_model(input1)
        model = keras.Model(inputs=[input1, input2], outputs=[output1, output2])
    parallel_model = keras.utils.multi_gpu_model(model, gpus=3)
    parallel_model.compile('sgd', loss=['mse', 'mse'])
    parallel_model.fit((np.random.random([10, 128, 128, 3]), np.random.random([10, 128, 128, 3])),
                       (np.random.random([10, 4, 4, 512]), np.random.random([10, 4, 4, 512])))

The error message is

Traceback (most recent call last):
  File "/data00/home/liangdong.tony/PycharmProject/RetrievalCCWebVideo/AE/demo.py", line 145, in <module>
    parallel_model = keras.utils.multi_gpu_model(model, gpus=3)
  File "/data00/home/liangdong.tony/.local/lib/python2.7/site-packages/keras/utils/training_utils.py", line 177, in multi_gpu_model
    return Model(model.inputs, merged)
  File "/data00/home/liangdong.tony/.local/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/data00/home/liangdong.tony/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 1811, in __init__
    'Layer names: ', all_names)
RuntimeError: ('The name "vgg16" is used 2 times in the model. All layer names should be unique. Layer names: ', ['input_1', 'input_2', 'lambda_1', 'lambda_2', 'lambda_3', 'lambda_4', 'lambda_5', 'lambda_6', 'model_1', 'vgg16', 'vgg16'])
nwpuxhld
  • 85
  • 1
  • 1
  • 10

2 Answers2

0

I'm just guessing but it says in your error log "The name "vgg16" is used 2 times in the model".

I guess if you create output1 and output2 with 

    output1 = sub_model(input1)

    output2 = sub_model(input1)

and add it to your model you create a duplicated layer name of the VGG16 model. Maybe you could use another input (input2)?

You can also try to rename your model:

output1 = sub_model(input1)
sub_model.name="VGG16_2"

output2 = sub_model(input1)

I might test your code and try to solve the issue if you could provide a little more code :)

This also seems like a similar problem.

Hope this helps.

Fabian
  • 756
  • 5
  • 12
  • I replace to output2 = sub_model(input2), same error. The code just is the demo of the case, which use multi-input. each input is preprocessed by VGG 16 (shared weights). – nwpuxhld Jun 06 '19 at 13:56
  • Another guess might be that u need a second sub_model? But I'm really just guessing right now :/ Will try it wenn i get home :) – Fabian Jun 06 '19 at 13:58
  • Ok, Thank you very much! I want to used the model with shared weights. If I define the second model, I think maybe they do not shared wight with each other. – nwpuxhld Jun 06 '19 at 14:07
  • You could just rename the model between output1 and output2. I will update the code in my answer. Last guessing try I promise ;) – Fabian Jun 06 '19 at 14:13
  • sad. It rewrites as: sub_model.name = 'vgg16_1' output1 = sub_model(input1) sub_model.name = 'vgg16_2' output2 = sub_model(input2) model = keras.Model(inputs=[input1, input2], outputs=[output1, output2]) same error. – nwpuxhld Jun 06 '19 at 16:04
  • Hi, I found one solution, but in this solution, the usage of gpu is very low. – nwpuxhld Jun 09 '19 at 14:08
0

I find there is a non-wise solution. There is the solution code:

import tensorflow as tf
from tensorflow.keras import backend as K


def slice_batch(x, n_gpus, part):
    sh = K.shape(x)
    L = sh[0] // n_gpus
    if part == n_gpus - 1:
        return x[part * L:]
    return x[part * L:(part + 1) * L]


def multi_gpu_wrapper(single_model, num_gpu):
    inputs = single_model.inputs
    towers = []
    splited_layer = tf.keras.layers.Lambda(lambda x: slice_batch(x, num_gpu, gpu_id))
    concate_layer = tf.keras.layers.Concatenate(axis=0)
    with tf.device('/cpu:0'):
        for gpu_id in range(num_gpu):
            cur_inputs = []
            for input in inputs:
                cur_inputs.append(
                    splited_layer(input)
                )
            towers.append(single_model(cur_inputs))
            print towers[-1]
    outputs = []
    num_output = len(towers[-1])
    with tf.device('/cpu:0'):
        for i in range(num_output):
            tmp_outputs = []
            for j in range(num_gpu):
                tmp_outputs.append(towers[j][i])
            outputs.append(concate_layer(tmp_outputs))
    multi_gpu_model = tf.keras.models.Model(inputs=inputs, outputs=outputs)
    return multi_gpu_model


if __name__ == '__main__':
    import config
    import os
    import numpy as np
    gpu_ids = "0,1,3"
    os.environ["CUDA_VISIBLE_DEVICES"] = gpu_ids
    with tf.device('/cpu:0'):
        input1 = tf.keras.layers.Input(config.input_shape)
        input2 = tf.keras.layers.Input(config.input_shape)
        sub_model = tf.keras.applications.VGG16(include_top=False, weights=config.VGG_MODEL_PATH,
                                                input_shape=config.input_shape)
        output1 = sub_model(input1)
        output2 = sub_model(input2)
        model = tf.keras.Model(inputs=[input1, input2], outputs=[output1, output2])
    multi_gpu_model = multi_gpu_wrapper(model, 3)
    multi_gpu_model.compile('sgd', loss=['mse', 'mse'])
    multi_gpu_model.fit([np.random.random([1000, 128, 128, 3]), np.random.random([1000, 128, 128, 3])],
                        [np.random.random([1000, 4, 4, 512]), np.random.random([1000, 4, 4, 512])], batch_size=128)

But, I found the usage of GPU in this solution is very low.

nwpuxhld
  • 85
  • 1
  • 1
  • 10