4

I want to learn how to run a function in multithreading using python. In other words, I have a long list of arguments that I want to send to a function that might take time to finish. I want my program to loop over the arguments and call the functions on parallel (no need to wait until the function finishes fromt he forst argument).

I found this example code from here:

import Queue
import threading
import urllib2

# called by each thread
def get_url(q, url):
    q.put(urllib2.urlopen(url).read())

theurls = ["http://google.com", "http://yahoo.com"]

q = Queue.Queue()

for u in theurls:
    t = threading.Thread(target=get_url, args = (q,u))
    t.daemon = True
    t.start()

s = q.get()
print s

My question are:

1) I normally know that I have to specify a number of threads that I want my program to run in parallel. There is no specific number of threads in the code above.

2) The number of threads is something that varies from device to device (depends on the processor, memory, etc.). Since this code does not specify any number of threads, how the program knows the right number of threads to run concurrently?

user9371654
  • 2,160
  • 16
  • 45
  • 78
  • Cannot understand your question but there are three threads. One main thread with two child threads created in `for` loop. – Sraw Aug 02 '18 at 15:08
  • I suspect it's not limited. In this case it's 3 I think though, for the reasons Sraw stated – sniperd Aug 02 '18 at 15:09
  • Do you want to know how many threads are running *at the same time* or just how many are defined in your code? Due to the Global Interpreter Lock, only one thread will run at a time (i.e., no "real" parallel execution). Threads will run concurrently in the sense that they split the time on the CPU and work one at a time. – GPhilo Aug 02 '18 at 15:10

1 Answers1

1

The threads are being created in the for loop. The for loop gets executed twice since there are two elements in theurls . This also answers your other two questions. Thus you end up with two threads in the program

plus main loop thread

Total 3

Bayko
  • 1,344
  • 2
  • 18
  • 25
  • But my program has hundreds of thousands up to a million of items (argument similar to the url but I read the argument from a file) that I need to loop over them. How can I manage this? – user9371654 Aug 02 '18 at 15:38
  • With a `ThreadPool` (see for example this question and the answer: https://stackoverflow.com/questions/3033952/threading-pool-similar-to-the-multiprocessing-pool ) – GPhilo Aug 02 '18 at 16:07