1

In general, when implementing multi-processing for Tensorflow, is it necessary to have a single process for each GPU? A CPU can have multiple processes (for each of its cores) right? I read somewhere that they suggested having a single process per GPU, but it just seems to me that a GPU should be able to handle more than that.

cmed123
  • 675
  • 6
  • 18
  • To the best of my knowledge a CPU can execute only one instruction per core at any given time. There can be many of them in the pipeline, but only one is executed at any time. Are you sure that when talking of 'single process per GPU' they weren't talking about a 'single process per GPU core'? – NotAName Dec 28 '19 at 06:53
  • Hm I guess there just seems to be differing opinions in some places such as this: https://stackoverflow.com/questions/46145100/is-it-unsafe-to-run-multiple-tensorflow-processes-on-the-same-gpu. Also, don't GPUs have like hundreds of cores though? (For higher-end NVIDIA ones). So I can run hundreds of processes per GPU? I thought processes were heavier than threads. So GPUs can run even more threads then? – cmed123 Dec 28 '19 at 07:10
  • I don't think that in that post it was meant in the context you understood it as. What comments there say is that you can run one TF process per GPU, which is actually no longer true any more. 'One TF process' means that one TF model can be trained at a time (once again, not true any more), but the process then splits into multiple separate calculations that run on separate cores, such as CUDA cores on Nvidia cards. – NotAName Dec 28 '19 at 07:55
  • Oh I see, thank you for the explanation! So, theoretically it's now possible to run as many TF processes as CUDA cores on Nvidia cards? So, at least 100? So basically as long as I have enough VRAM on the Nvidia card, I should theoretically be able to train 100 TF models at once, right? – cmed123 Dec 28 '19 at 14:42

0 Answers0