3

for a django application I'm working on, I need to implement a two ways RPC so :

  • the clients can call RPC methods from the platform and
  • the platform can call RPC methods from each client.

As the clients will mostly be behind NATs (which means no public IPs, and unpredictable weird firewalling policies), the platform to client way has to be initiated by the client.

I have a pretty good idea on how I can write this from scratch, I also think I can work something out of the publisher/subscriber model of twisted, but I've learned that there is always a best way to do it in python.

So I'm wondering what would be the best way to do it, that would also integrate the best to django. The code will have to be able to scope with hundreds of clients in short term, and (we hope) with thousands of clients in medium/long term.

So what library/implementation would you advice me to use ? I'm mostly looking to starting points for RTFM !

zmo
  • 24,463
  • 4
  • 54
  • 90
  • As django is focused on servicing http requests, I think I have to work out an external module django will have to connect to. I also will need to encrypt the communications to avoid mitm tampering. For the actual implementation, I'm considering different implementations of pubsub mechanisms: twisted, 0mq, rabbitmq. – zmo May 19 '12 at 14:55
  • little update, I'm also looking on python websockets implementations... – zmo May 20 '12 at 13:24
  • well, after some digging, I found out about django-websockets (http://pypi.python.org/pypi/django-websocket), but after trying it out (https://bitbucket.org/jmad/tests_django/src/f4e50796311c/websocket/), I understood it is a no-go, as it will not work with wsgi. – zmo May 21 '12 at 12:07
  • I think most of the pieces are in twisted : it has xmlrpc support and websockets support... I need to find out how to glue all that together now. http://stackoverflow.com/questions/4406256/twisted-and-websockets-beyond-echo ; http://stackoverflow.com/questions/8096237/python-twisted-websocket-client ; http://twistedmatrix.com/trac/export/29073/branches/websocket-4173-2/doc/web/howto/websocket.xhtml – zmo May 21 '12 at 12:10
  • just found out and I am trying : http://blog.jupo.org/2011/08/13/real-time-web-apps-with-django-and-websockets/ – zmo May 21 '12 at 12:17
  • I'm also trying the django_socketio package, trying to adapt the example from https://github.com/stephenmcd/django-socketio/tree/master/django_socketio/example_project. I still did not understood how to create a python client that can communicate with it... – zmo May 23 '12 at 12:16
  • well, I finally got ws4py client working with ws4py server and websocksify server... but I'm still looking for a best way to go – zmo May 23 '12 at 15:19

4 Answers4

7

websocket is a moving target, with new specifications from time to time. Brave developpers implements server side library, but few implements client side. The client for web socket is a web browser.

websocket is not the only way for a server to talk to a client, event source is a simple and pragmatic way to push information to a client. It's just a never ending page. Twitter fire hose use this tricks before its specification. The client open a http connection and waits for event. The connection is kept open, and reopen if there is some troubles (connection cut, something like that). No timeout, you can send many events in one connection.

The difference between websocket and eventsource is simple. Websocket is bidirectionnal and hard to implement. Eventsource is unidirectionnal and simple to implement, both client and server side.

You can use eventsource as a zombie controller. Each client connects and reconnect to the master and wait for instruction. When instruction is received, the zombie acts and if needed can talk to its master, with a classical http connection, targeting the django app.

Eventsource keep the connection open, so you need an async server, like tornado. Django need a sync server, so, you need both, with a dispatcher, like nginx. Django or a cron like action talks to the async server, wich talks to the right zombie. Zombie talks to django, so, the async server doesn't need any peristance, it's just a hub with plugged zombies.

Gevent is able to handle such http server but there is no decent doc and examples for this point. It's a shame. I want a car, you give me a screw.

athoune
  • 231
  • 1
  • 2
2

You can also use Tornado + Tornadio + Socket.io. That's what we are using right now for notifications, and the amount of code that you should write is not that much.

from tornadio2 import SocketConnection, TornadioRouter, SocketServer
class RouterConnection(SocketConnection):
__endpoints__ = {'/chat': ChatConnection,
                 '/ping': PingConnection,
                 '/notification' : NotificationConnection
                 }

def on_open(self, info):
    print 'Router', repr(info)

MyRouter = TornadioRouter(RouterConnection)

# Create socket application
application = web.Application(
MyRouter.apply_routes([(r"/", IndexHandler),
    (r"/socket.io.js", SocketIOHandler)]),
        flash_policy_port = 843,
        flash_policy_file = op.join(ROOT, 'flashpolicy.xml'),
        socket_io_port = 3001,
        template_path=os.path.join(os.path.dirname(__file__), "templates/notification")
)

class PingConnection(SocketConnection):
     def on_open(self, info):
       print 'Ping', repr(info)

     def on_message(self, message):
       now = dt.utcnow()
       message['server'] = [now.hour, now.minute, now.second, now.microsecond / 1000]
       self.send(message)
class ChatConnection(SocketConnection):
   participants = set()
   unique_id = 0

   @classmethod
   def get_username(cls):
       cls.unique_id += 1
       return 'User%d' % cls.unique_id

   def on_open(self, info):
       print 'Chat', repr(info)

       # Give user unique ID
       self.user_name = self.get_username()
       self.participants.add(self)

   def on_message(self, message):
       pass

   def on_close(self):
       self.participants.remove(self)

   def broadcast(self, msg):
       for p in self.participants:
            p.send(msg)
Bahadir Cambel
  • 422
  • 5
  • 12
  • damn... I see your comment just after I made a full event_source implementation in tornado... and it had more locs ! (and though your answer seems cool, I don't have the nerves to try out yet another solution, and as I based the solution I coded on athoune's suggestions, I'll let him keep the 'accepted answer'). – zmo May 25 '12 at 13:03
  • too bad :) no problem. Somebody else will benefit as well. – Bahadir Cambel May 29 '12 at 22:51
1

here is a really simple solution I could came up with :

import tornado.ioloop
import tornado.web
import time

class MainHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous 
    def get(self):
        self.set_header("Content-Type", "text/event-stream")
        self.set_header("Cache-Control", "no-cache")
        self.write("Hello, world")
        self.flush()
        for i in range(0, 5):
            msg = "%d<br>" % i
            self.write("%s\r\n" % msg) # content
            self.flush()
            time.sleep(5)


application = tornado.web.Application([
    (r"/", MainHandler),
])

if __name__ == "__main__":
    application.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

and

curl http://localhost:8888

gives output when it comes !

Now, I'll just have to implement the full event-source spec and some kind of data serialization between the server and the clients, but that's trivial. I'll post an URL to the lib I'll write here when it'll be done.

zmo
  • 24,463
  • 4
  • 54
  • 90
  • here is what I came to create for event_source protocol support with tornado : https://gist.github.com/2788003 – zmo May 25 '12 at 13:02
  • full implementation for event_source in python on https://github.com/guyzmo/event-source-library – zmo May 26 '12 at 14:57
1

I've recently played with Django, Server-Sent Events and WebSocket, and I've wrote an article about it at http://curella.org/blog/2012/jul/17/django-push-using-server-sent-events-and-websocket/

Of course, this comes with the usual caveats that Django probably isn't the best fit for evented stuff, and both protocols are still drafts.

fcurella
  • 2,461
  • 2
  • 19
  • 7
  • interesting article, though until wsgi supports multithreading, it's not imaginable to go that way in production. – zmo Aug 08 '12 at 08:23