How do sessions and parallelism work in TF2.0?

Question

I am trying to run two tensorflow models in parallel in the same process.

In Tensorflow 1.x we could do e.g. Keras Tensorflow - Exception while predicting from multiple threads

graph = tf.Graph()
with graph.as_default():
    session = tf.Session()
    with session.as_default():
        # Create and use the model ...

In Tensorflow 2.0, the session has been removed.

When I try to run two models in parallel within the same process, Tensorflow crashes. Is there a way around this problem is TF 2.0?

Error message:

2019-09-18 16:26:56.978352: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.981418: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.983202: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.985757: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.987541: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.990140: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.992499: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.994886: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.997778: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:56.997802: W tensorflow/stream_executor/stream.cc:1919] attempting to perform BLAS operation using StreamExecutor without BLAS support
2019-09-18 16:26:56.997829: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: Blas GEMM launch failed : a.shape=(1, 2520), b.shape=(2520, 2520), m=1, n=2520, k=2520
     [[{{node dense/MatMul}}]]
2019-09-18 16:26:57.000709: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-09-18 16:26:57.000744: W tensorflow/stream_executor/stream.cc:1919] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
  File 
    result = model.predict([numpy.array([inV,]), numpy.array([inS,])])
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 915, in predict
    use_multiprocessing=use_multiprocessing)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_arrays.py", line 722, in predict
    callbacks=callbacks)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_arrays.py", line 393, in model_iteration
    batch_outs = f(ins_batch)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/keras/backend.py", line 3625, in __call__
    outputs = self._graph_fn(*converted_inputs)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1081, in __call__
    return self._call_impl(args, kwargs)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1121, in _call_impl
    return self._call_flat(args, self.captured_inputs, cancellation_manager)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1224, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 511, in call
    ctx=ctx)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError:  Blas GEMM launch failed : a.shape=(1, 2520), b.shape=(2520, 2520), m=1, n=2520, k=2520
     [[node dense/MatMul (defined at usr/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]] [Op:__inference_keras_scratch_graph_378]

Function call stack:
keras_scratch_graph

I would think about raising this on the tensorflow page. Have you tried using tf.compat.v1.Session? The guys at tensorflow preserved some of the TF1.0 stuff... Worse case, is there a need for you to upgrade? If upgrading to 2.0 breaks your program. Keeping tensorflow to 1.0 might still be viable. — Jason Chia, May 04 '20 at 07:23

score 0 · Answer 1 · answered May 05 '20 at 08:13

Try to allow fraction of GPU for each model.

import tensorflow as tf
config = tf.compat.v1.ConfigProto(gpu_options = 
                     tf.compat.v1.GPUOptions(per_process_gpu_memory_fraction=0.8)
# device_count = {'GPU': 1}
)
config.gpu_options.allow_growth = True
session = tf.compat.v1.Session(config=config)
tf.compat.v1.keras.backend.set_session(session)

How do sessions and parallelism work in TF2.0?

1 Answers1