1

Due to this great question with the remarkable answers here how-to-use-threading-in-python, make me doubt, is the number of ThreadPool has some significant on speed or memory usage?

For example, should the number of ThreadPool is mutiplied number of number of my CPU cores? (I have two CPU cores, so I should make 4,8,12,.. ThreadPool?).

Thank you in advance.

Community
  • 1
  • 1
chrsow
  • 13
  • 4

1 Answers1

1

In CPython, a ThreadPool (and threading in general) is mainly useful for I/O-bound tasks. As a test, you should keep increasing the pool size until your speed/throughput stops increasing.

For CPU-bound tasks in Python (but not numpy or other extensions that release the GIL while doing their work) it would be better to use multiprocessing.Pool. In that case, I tend to use as many processes as the CPU has cores. This is the default when creating a Pool with processes=None. If you use more processes than you have cores, processes from the pool will be fighting over cores.

Roland Smith
  • 42,427
  • 3
  • 64
  • 94
  • I see, does this mean either I/O. bound or CPU bound, we just use multiprocessing module and forget about "threading" module for any task we do concurrentcy or parallelis? – chrsow Jan 23 '17 at 01:30