Python: Why different threads get their own series of values from one generator?

Question

I'm learning multithreading in Python. I want to know how to provide data to multiple threads using generators. Here's what I wrote:

  import threading

  data = [i for i in xrange(100)]

  def generator():
      for i in data:
          yield i

  class CountThread(threading.Thread):
      def __init__(self, name):
          threading.Thread.__init__(self)
          self.name = name

      def run(self):
          for i in generator():
              print '{0} {1}'.format(self.name, i)

  a = CountThread('a')
  b = CountThread('b')
  a.start()
  b.start()

I thought the list would only be iterated for once. But it seems that each thread is interating through the list independently.

output:

a 0
a 1
a 2
b 0
a 3
a 4
b 1
b 2
a 5
a 6
a 7
a 8
a 9
b 3
b 4
b 5
b 6
b 7
b 8
...
(200 lines)

What is the underlying reason for this? How can I rewrite the program so that the values in the list will only be printed once.

score 9 · Accepted Answer · edited May 23 '17 at 10:25

9

You instantiate a new generator in each thread in run function with this:

for i in generator():

each generator call returns a new instance of generator:

>>> data = [i for i in xrange(10)]
>>> a, b = generator(), generator()
>>> id(a), id(b)
(37528032, 37527952)

Here a and b have different ids, and generate identical results even without threading:

>>> list(a), list(b)
([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

Note however, that generators aren't thread safe, it is tricky to use them in threaded application. You have to worry about locking, see example, or you'll get ValueError: generator already executing error once in a while. Alternatively you can use Queue.Queue for thread communication.

edited May 23 '17 at 10:25

Community

1
1

answered Nov 18 '13 at 07:46

alko

46,136
12
94
102

I am sorry. I didnt understand the answer. Could you please explain a bit more? – thefourtheye Nov 18 '13 at 07:52
@thefourtheye tried to elaborate, see please if it is clearer now – alko Nov 18 '13 at 08:02

score 0 · Answer 2 · answered Nov 18 '13 at 09:30

Can you just use Python Queue class. I believe that is thread safe. Basically you could just use your generator to fill the Queue and then have each thread pull from it.

#!/usr/bin/python

import Queue
import threading

queueOfNumbers = Queue.Queue()

data = [i for i in xrange(100)]

def generator():
    for i in data:
        yield i

class CountThread(threading.Thread):
    def __init__(self, name, queue):
        threading.Thread.__init__(self)
        self.name = name
        self.queue = queue

    def run(self):
        i = self.queue.get()
        print '%s %s' % (self.name, i)


a = CountThread('a', queueOfNumbers)
b = CountThread('b', queueOfNumbers)
a.start()
b.start()

for num in generator():
    queueOfNumbers.put(num)

queueOfNumbers.join()

http://www.ibm.com/developerworks/aix/library/au-threadingpython/

Python: Why different threads get their own series of values from one generator?

2 Answers2

Linked