Sharing an enumerator across threads

Question

I want to call a common enumerator from different threads. When I do the following,

enum = (0..1000).to_enum
t1 = Thread.new do
  p enum.next
  sleep(1)
end
t2 = Thread.new do
  p enum.next
  sleep(1)
end
t1.join
t2.join

it raises an error:

Fiber called across threads.

when enum is called from t2 after once being called from t1.

Why is Ruby designed to not allow an enumerator (or fiber) to be called across threads, and
Is there an alternative way to provide a similar function?

I am guessing that atomicity of an operation on an enumerator/fiber is relevant here, but am not fully sure. If that is the issue, then exclusively-locking the enumerator/fiber while in use shall solve the problem, and I don't know why calling an enumerator/fiber across threads is prohibited in general. If an alternative can be provided by using locking, that would satisfy my need.

will a Queue satisfy your needs? – Uri Agassi Feb 23 '14 at 11:58 — Uri Agassi, Feb 23 '14 at 11:58
@UriAgassi If you can do it with it, it would. – sawa Feb 23 '14 at 12:47 — sawa, Feb 23 '14 at 12:47
@sawa, what did you end up doing? – Ismail Moghul Mar 23 '15 at 15:07 — Ismail Moghul, Mar 23 '15 at 15:07

Uri Agassi · Answer 1 · 2015-09-11T05:52:54.747

1

You can use Queue

queue = Queue.new
(0..1000).map(&queue.method(:push))

t1 = Thread.new do
  while !queue.empty?
    p queue.pop(true)
    sleep(0.1)
  end
end
t2 = Thread.new do
  while !queue.empty?
    p queue.pop(true)
    sleep(0.1)
  end
end
t1.join
t2.join

edited Sep 11 '15 at 05:52

answered Feb 23 '14 at 12:58

Uri Agassi

36,848
14
76
93

I want to be able to reuse it. By popping, it will change. – sawa Feb 23 '14 at 13:42
This replaces the enumerator, not the array/range/whatever... instead of `rewind` - rebuild it – Uri Agassi Feb 23 '14 at 13:53
I am trying to do this same thing in a ROR application, and I need to be able to use an ActiveRecord query (with a large number of results) as an Enumerator. If I map the whole thing to a Queue, I end up using too much ram. (I'm using the find_each method, so that it only queries in batches of 1000. If I map it to a Queue, I will have the whole table sitting in a variable, which is not practical). See [this question](http://stackoverflow.com/q/32513535/599402). – Ephraim Sep 11 '15 at 00:20
@Ephraim - the trick here is to spin up the threads _before_ filling up the queue. This way the threads will consume the results as you are requesting for them - actually working when you are waiting for the DB to bring the next batch. – Uri Agassi Sep 11 '15 at 05:31

Sharing an enumerator across threads

1 Answers1

Linked