0

I am doing a little bit of python multithreading programming and found the result of my code vary strange, not parallel at all (8 cores, 8 threads, 13% cpu utilization). Then I have found a python GIL term and these slides (http://www.dabeaz.com/python/GIL.pdf). Is it real that python is not parallelizable? Will the multiprocessing module help to utilize computational resources or there is another performance issue with that?

itun
  • 3,439
  • 12
  • 51
  • 75
  • Which multithreading library did you use? – Nebril Dec 24 '14 at 10:26
  • http://stackoverflow.com/questions/6821477/python-code-performance-decreases-with-threading – NPE Dec 24 '14 at 10:28
  • @Nebril I use standard threading module can you recommend something else. Because spawning new processes does not really fit me. – itun Dec 24 '14 at 10:30
  • @itun as Reut wrote: try multiprocessing https://docs.python.org/2/library/multiprocessing.html – Nebril Dec 24 '14 at 10:33
  • @Nebril - it's also mentioned in the docs. Atleast for python 2: *If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing* – Reut Sharabani Dec 24 '14 at 10:34
  • 1
    Praise the docs then! – Nebril Dec 24 '14 at 10:36

1 Answers1

2

From the docs:

CPython implementation detail: In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing. However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.

A a general rule of thumb:

When your bottleneck is I/O (like writing to disk...) - consider threading which allows the program to keep running elsewhere when something is blocking an execution path.

When your bottleneck is cpu power, consider multiprocessing which allows cpu utilization.

Reut Sharabani
  • 30,449
  • 6
  • 70
  • 88
  • what should I do if I cannot spawn new processes? Is there any 3rd party library to use threads in a usual sense. – itun Dec 24 '14 at 11:24
  • You should drop by the python room for advices on what to do, instead of starting an opinion-based discussion here. The chat room: http://chat.stackoverflow.com/rooms/6/python – Reut Sharabani Dec 24 '14 at 11:27