Get error "The Session graph is empty." when use multiprocessing for Keras networks

Question

I want to evaluate and get the loss of each sample. So I want to apply multiprocessing to accelerate it. But it shows error "The Session graph is empty. Add operations to the graph before calling run()."

model.fit(x=X_measured, y=y_train, batch_size=batch_size, epochs=epochs, verbose=0, 
           validation_data=(X_measured_test,y_test), shuffle=True)
    
def get_loss(i, model, X_measured, y_train):
    samples_loss=model.evaluate(x=X_measured[i:i+1,:],y=y_train[i:i+1,:],batch_size=None,verbose=0,steps=1)
    return samples_loss


pool = mp.Pool(mp.cpu_count())
samples_loss=pool.starmap(get_loss, [(j, model, X_measured, y_train) for j in range(X_measured.shape[0])])
pool.close()

Answer 1

According to this excellent answer about multiprocessing in Keras, the best rule of thumb is to "run work connected with each model in a separate process".

Thus, the way you're constructing things-to train the model on the main environment and then calculate the loss on a separate process, can't be done since Keras/Tensorflow loads/configures a bunch of things into the main process that isn't propagated down to the spawned process.

If you do have separate models to train, it appears the best approach would be to spawn a new process for each model training and evaluation.