stop a twisted reactor after a gatherResults has finished

Question

new to twisted and just trying some deferred stuff out. I have the following code that makes up a list of 100 deferred calls - each waiting a random time and returning a value. That list the prints the results and finally terminates the reactor.

However, I'm pretty sure the way I'm stopping the reactor is probably... not great.

__author__ = 'Charlie'

from twisted.internet import defer, reactor
import random

def getDummyData(x):
    """returns a deferred object that will have a value in some random seconds
    sets up a callLater on the reactor to trgger the callback of d"""
    d = defer.Deferred()
    pause = random.randint(1,10)
    reactor.callLater(pause, d.callback, (x, pause))
    return d


def printData(result):
    """prints whatever is passed to it"""
    print result


def main():
    """makes a collection of deffered calls and then fires them. Stops reactor at end"""
    deferred_calls = [getDummyData(r) for r in range(0,100)]
    d = defer.gatherResults(deferred_calls, consumeErrors = True)
    d.addCallback(printData)

    # this additional callback on d stops the reacor
    # it fires after all the delayed callbacks have printed their values
    # the lambda ignored: ractor.stop() is required as callback takes a function
    # that takes a parameter.
    d.addCallback(lambda ignored: reactor.stop())

    # start the reactor.
    reactor.run()

if __name__ == "__main__":
    main()

I'm assuming that by adding a callback:

d.addCallback(lambda ignored: reactor.stop())

to the gathered results actually adds that callback on all the deferred items?

if so then there is probably a more elegant / correct way to do it?

Cheers!

Side note: `_` is the idiom for an ignored variable. – ivan_pozdeev Apr 19 '17 at 01:37 — ivan_pozdeev, Apr 19 '17 at 01:37

score 7 · Accepted Answer · answered Nov 06 '13 at 17:22

I'm assuming that by adding a callback: d.addCallback(lambda ignored: reactor.stop()) to the gathered results actually adds that callback on all the deferred items?

This isn't the case. gatherResults returns a new Deferred. It's just like any other Deferred you might come across. Its addCallback method does the same thing as usual: adds one function that will be called at one point in one callback chain.

The reason everything is nice and regular and unspecial like this is that gatherResults takes care of all the logic necessary to only give that regular Deferred that it returns a result after all of the input Deferreds have a result.

So, feel free to use gatherResults just like you're use any other Deferred-returning API. It's not special!

That said, starting with Twisted 12.3 there's a handy utility that you might want to use for this sort of thing - twisted.internet.task.react. Here's what your main function would look like if you used it:

def main(reactor):
    """makes a collection of deffered calls and then fires them. Stops reactor at end"""
    deferred_calls = [getDummyData(r) for r in range(0,100)]
    d = defer.gatherResults(deferred_calls, consumeErrors = True)
    d.addCallback(printData)
    return d

if __name__ == "__main__":
    from twisted.internet import task
    task.react(main, [])

And notice that you could change getDummyData so that it doesn't depend on the global reactor either:

def getDummyData(reactor, x):
    """returns a deferred object that will have a value in some random seconds
    sets up a callLater on the reactor to trgger the callback of d"""
    d = defer.Deferred()
    pause = random.randint(1,10)
    reactor.callLater(pause, d.callback, (x, pause))
    return d

def main(reactor):
    """makes a collection of deffered calls and then fires them. Stops reactor at end"""
    deferred_calls = [getDummyData(reactor, r) for r in range(0,100)]
    d = defer.gatherResults(deferred_calls, consumeErrors = True)
    d.addCallback(printData)
    return d

And now your code does't need any twisted.internet.reactor imports at all.

You could also use twisted.internet.task.deferLater in getDummyData to save a bit more typing:

def getDummyData(reactor, x):
    """returns a deferred object that will have a value in some random seconds
    sets up a callLater on the reactor to trgger the callback of d"""
    pause = random.randint(1,10)
    return deferLater(reactor, pause, lambda: (x, pause))

wow - thanks for the very detailed answer. I'll dig though this tomorrow. — Transact Charlie, Nov 06 '13 at 23:25

stop a twisted reactor after a gatherResults has finished

1 Answers1

Linked