4

I have been trying to write a simple python application to implement a worker queue every webpage I found about threading has some random guy commenting on it, you shouldn't use python threading because this or that, can someone help me out? what is up with Python threading, can I use it or not? if yes which lib? the standard one is good enough?

Max
  • 4,152
  • 4
  • 36
  • 52
  • 2
    What's the aim of your "worker queue"? Are you trying to distribute CPU-intensive work to multiple processors? – Sven Marnach May 16 '11 at 09:53
  • @Sven-Marnach Not really, very simply actually, I have a bunch of Urls, and I have a method that will read and store the content of the URL, all I want to do is have 10 instances of the method run in Parallel and fetch the content of urls, the urls are stored in a python list. – Max May 16 '11 at 10:16
  • 1
    @user: If this is what you are aiming at, forget about all the comments you read and all the GIL stuff. Using the standard `threading` module is a good approach in this case. – Sven Marnach May 16 '11 at 10:24
  • @Sven-Marnach have a look at this: [link]:http://themattreid.com/wordpress/2011/01/20/simple-python-a-job-queue-with-threading/ this is what I want but the dude still responded with something about GIL! – Max May 16 '11 at 10:27
  • 1
    In the link you posted, it is not clear what the queue is meant to be used for. Your application won't be CPU-bound, so forget about the GIL. See Eli's very good answer for more details (and upvote and accept the answer if you like it). – Sven Marnach May 16 '11 at 12:14

3 Answers3

5

Python's threads are perfectly viable and useful for many tasks. Since they're implemented with native OS threads, they allow executing blocking system calls and keep "running" simultaneously - by calling the blocking syscall in a separate thread. This is very useful for programs that have to do multiple things at the same time (i.e. GUIs and other event loops) and can even improve performance for IO bound tasks (such as web-scraping).

However, due to the Global Interpreter Lock, which precludes the Python interpreter of actually running more than a single thread simultaneously, if you expect to distribute CPU-intensive code over several CPU cores with threads and improve performance this way, you're out of luck. You can do it with the multiprocessing module, however, which provides an interface similar to threading and distributes work using processes rather than threads.

I should also add that C extensions are not required to be bound by the GIL and many do release it, so C extensions can employ multiple cores by using threads.

So, it all depends on what exactly you need to do.

Eli Bendersky
  • 263,248
  • 89
  • 350
  • 412
  • Global Interpreter Lock is only true for CPython, not for Jython or IronPython. See also [wikipedia](http://en.wikipedia.org/wiki/Global_Interpreter_Lock). – Bouke Jan 22 '13 at 16:14
0
  • You shouldn't need to use threading. 95% of code does not need threads.
  • Yes, Python threading is perfectly valid, it's implemented through the operating system's native threads.
  • Use the standard library threading module, it's excellent.
Matt Joiner
  • 112,946
  • 110
  • 377
  • 526
  • @matt-joiner mate check out the first comment to this thread: [link]:http://themattreid.com/wordpress/2011/01/20/simple-python-a-job-queue-with-threading/ – Max May 16 '11 at 10:17
  • Why the downvote? @Max that thread doesn't help you whatsover. There's no guarantee threads are even scheduled or holding the GIL at the same time, let alone that they're both CPU-bound. – Matt Joiner May 19 '11 at 13:00
  • oops I am new here, I might have done it by mistake I just got the hang of voting or accepting something as an answer! – Max May 19 '11 at 13:44
0

GIL should provide you some information on that topic.

ba__friend
  • 5,783
  • 2
  • 27
  • 20