0

UPDATE:

I found the source code of GPUDevice, it hard-coded max streams to 1, may I know the know reason?

GPUDevice(const SessionOptions& options, const string& name, Bytes memory_limit, const DeviceLocality& locality, TfGpuId tf_gpu_id, const string& physical_device_desc, Allocator* gpu_allocator, Allocator* cpu_allocator) : BaseGPUDevice(options, name, memory_limit, locality, tf_gpu_id, physical_device_desc, gpu_allocator, cpu_allocator, false /* sync every op */, 1 / max_streams /) { if (options.config.has_gpu_options()) { force_gpu_compatible_ = options.config.gpu_options().force_gpu_compatible(); }

======================================

I am wondering whether TensorFlow(1.x version) supports multi-thread or multi-stream on a single GPU. If not, I am curious the underlying reasons, TF did this on some purposes or some libs like CUDA prevents TF from providing or some other reasons?

Like some previous posts[1,2], I tried to run multiple training ops in TF, i.e. sees.run([train_op1, train_op2],feed_dict={...}), I used the TF timeline to profile each iteration. However, TF timeline always showed that two train ops run sequentially (although timeline is not accurate[3], the wall time of each op suggests sequential running). I also looked at some source code of TF, it looks like the each op are computed by in device->ComputeAsync() or device->Compute(), and the GPU is blocked when computing an op. If I am correct, one GPU can only run a single op each time, which may lower GPU utilization.

1.Running multiple tensorflow sessions concurrently

2.Run parallel op with different inputs and same placeholder

3.https://github.com/tensorflow/tensorflow/issues/1824#issuecomment-244251867

  • This question could attract down-votes, follows as SO is a QA site where questions should be specific to a problem, and here you have more than one question also some of the questions could be interpreted as opinion based ect.. more help here https://stackoverflow.com/help/how-to-ask – Nigel Savage Feb 21 '20 at 20:13
  • @NigelSavage I updated my question, thanks. –  Feb 26 '20 at 17:44

1 Answers1

0

I have similar experience with you. I have two GPU, each GPU run three threads, each thread running a session, each session running time fluct a lot. if run only one thread on each GPU, session running time is quite stable.

from these appearence, we can conclude that ,thread in tensorflow not cowork well, the mechanism of tensorflow has problem.

Haoyang Ma
  • 1