Running twisted reactor in iPython

Question

I'm aware this is normally done with twistd, but I'm wanting to use iPython to test out code 'live' on twisted code.

How to start twisted's reactor from ipython asked basically the same thing but the first solution no longer works with current ipython/twisted, while the second is also unusable (thread raises multiple errors).

https://gist.github.com/kived/8721434 has something called TPython which purports to do this, but running that seems to work except clients never connect to the server (while running the same clients works in the python shell).

Do I have to use Conch Manhole, or is there a way to get iPython to play nice (probably with _threadedselect).

For reference, I'm asking using ipython 5.0.0, python 2.7.12, twisted 16.4.1

Answer 1

Async code in general can be troublesome to run in a live interpreter. It's best just to run an async script in the background and do your iPython stuff in a separate interpreter. You can intercommunicate using files or TCP. If this went over your head, that's because it's not always simple and it might be best to avoid the hassle of possible.

However, you'll be happy to know there is an awesome project called crochet for using Twisted in non-async applications. It truly is one of my favorite modules and I'm shocked that it's not more widely used (you can change that ;D though). The crochet module has a run_in_reactor decorator that runs a Twisted reactor in a separate thread managed by crochet itself. Here is a quick class example that executes requests to a Star Wars RESTFul API, then stores the JSON response in a list.

from __future__ import print_function
import json

from twisted.internet import defer, task
from twisted.web.client import getPage

from crochet import run_in_reactor, setup as setup_crochet
setup_crochet()

class StarWarsPeople(object):
    people_id = [_id for _id in range(1, 89)]
    people = []

    @run_in_reactor
    def requestPeople(self):
        """
        Request Star Wars JSON data from the SWAPI site.
        This occurs in a Twisted reactor in a separate thread.
        """
        for _id in self.people_id:
            url = 'http://swapi.co/api/people/{0}'.format(_id).encode('utf-8')
            d = getPage(url)
            d.addCallback(self.appendJSON)

    def appendJSON(self, response):
        """
        A callback which will take the response from the getPage() request, 
        convert it to JSON, then append it to self.people, which can be 
        accessed outside of the crochet thread.
        """
        response_json = json.loads(response.decode('utf-8'))
        #print(response_json)    # uncomment if you want to see output
        self.people.append(response_json)

Save this in a file (example: swapi.py), open iPython, import the newly created module, then run a quick test like so:

from swapi import StarWarsPeople
testing = StarWarsPeople()
testing.requestPeople()

from time import sleep
for x in range(5):
    print(len(testing.people))
    sleep(2)

As you can see it runs in the background and stuff can still occur in the main thread. You can continue using the iPython interpreter as you usually do. You can even have a manhole running in the background for some cool hacking too!

References

• https://crochet.readthedocs.io/en/1.5.0/introduction.html#crochet-use-twisted-anywhere

Answer 2

While this doesn't answer the question I thought I had, it does answer (sort of) the question I posted. Embedding ipython works in the sense that you get access to business objects with the reactor running.

from twisted.internet import reactor
from twisted.internet.endpoints import serverFromString
from myfactory import MyFactory

class MyClass(object):
    def __init__(self, **kwargs):
        super(MyClass, self).__init__(**kwargs)
        server = serverFromString(reactor, 'tcp:12345')
        server.list(MyFactory(self))
        def interact():
            import IPython
            IPython.embed()
        reactor.callInThread(interact)

if __name__ == "__main__":
    myclass = MyClass()
    reactor.run()

Call the above with python myclass.py or similar.