Passing tensorflow model to Process of multiprocessing python library

Question

I get an error when I try to pass a copy of a keras model as parameter to a multiprocessing.Process object in Python. The code is the following:

def dumb_fun(x, model):
    print(x)
    print(model)


def clone(model):
    new_model = clone_model(model)
    new_model.set_weights(model.get_weights())
    return new_model


if __name__ == '__main__':

    input = layers.Input(shape=(120,160,2))
    x = layers.Conv2D(32, (3, 3), activation='relu')(input)
    x = layers.MaxPool2D((2, 2))(x)
    x = layers.Conv2D(64, (3, 3), activation='relu')(x)
    x = layers.MaxPool2D((2, 2))(x)
    x = layers.Conv2D(128, (3, 3), activation='relu')(x)
    x = layers.MaxPool2D((2, 2))(x)
    x = layers.Flatten()(x)
    x = layers.Dense(64, activation='relu')(x)
    x = layers.Dense(32, activation='relu')(x)
    actions = layers.Dense(5, activation='softmax')(x)
    # actor network
    actor = keras.Model(input, actions, name="actor")

    numbers = [1,1,2,3,5,8,13]
    num_procs = len(numbers)

    actors = []
    for i in range(num_procs):
        actors.append(clone(actor))

    processes = []

    for i in range(num_procs):
        processes.append(Process(target=dumb_fun, args=(numbers[i],actors[i])))

    for proc in processes:
        proc.start()

    for proc in processes:
        proc.join()

I got the following error:

Traceback (most recent call last): File "", line 1, in File "C:\Users\boezi\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "C:\Users\boezi\AppData\Local\Programs\Python\Python310\lib\multiprocessing\spawn.py", line 126, in _main self = reduction.pickle.load(from_parent) File "C:\Users\boezi\PycharmProjects\AWSDeepRacerChallenge\env\lib\site-packages\keras\saving\pickle_utils.py", line 48, in deserialize_model_from_bytecode model = save_module.load_model(temp_dir) File "C:\Users\boezi\PycharmProjects\AWSDeepRacerChallenge\env\lib\site-packages\keras\utils\traceback_utils.py", line 67, in error_handler raise e.with_traceback(filtered_tb) from None File "C:\Users\boezi\PycharmProjects\AWSDeepRacerChallenge\env\lib\site-packages\tensorflow\python\saved_model\load.py", line 915, in load_partial raise FileNotFoundError( FileNotFoundError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for ram://c7e98835-fe78-46a1-b2b1-fb9241d8ef90/variables/variables You may be trying to load on a different device from the computational device. Consider setting the experimental_io_device option in tf.saved_model.LoadOptions to the io_device such as '/job:localhost'.

I tried the multiprocessing part without passing the model and it works, what is the problem?

score 1 · Answer 1 · answered Jun 29 '22 at 18:42

I was facing a similar issue recently. The thing is that you can't pass unpickable object to the Process method. I would recommand you to try the same approach but with the Thread method from threading library.
https://stackoverflow.com/a/3044626/12763693

import threading
threading.Thread(target=function, args=(,))

Passing tensorflow model to Process of multiprocessing python library

1 Answers1