Code in Python, communicate in Node.js and Socket.IO, present in HTML

Question

You have a python script diagnosis.py that generates realtime event-based data. Using Node.js, you can launch it as a child process and capture its output and then using Socket.IO emit that to the client and present it using HTML.

Server

var util  = require('util'),
    spawn = require('child_process').spawn,
    ls    = spawn('python', ['diagnosis.py']);

var app = require('http').createServer(handler)
  , io = require('socket.io').listen(app)
  , fs = require('fs')

app.listen(80);

function handler (req, res) {
  fs.readFile(__dirname + '/index.html',
  function (err, data) {
    if (err) {
      res.writeHead(500);
      return res.end('Error loading index.html');
    }

    res.writeHead(200);
    res.end(data);
  });
}

io.sockets.on('connection', function (socket) {
    ls.stdout.on('data', function (gdata) {
      socket.emit('news', gdata.toString());
    });
});

Client

<html>
    <head>
        <script src="/socket.io/socket.io.js"></script>
        <script>
          var d = "";
          var socket = io.connect('http://localhost');
          socket.on('news', function (data) {
            d += data;
            document.getElementById('data').innerHTML = d;
            console.log(data);
          });
        </script>
    </head>
    <body>
        <div id="data"></div>
    </body>
</html>

Question

This is great and all, but what if you're looking for the same HTML-Node.js communicative power that Socket.IO provides but instead between Node.js and Python? How would you do that? There's no web server there, so Socket.IO does not make a lot of sense and communicating over bare TCP does not provide the same power/elegance. How do I achieve full duplex communication between Node.js and Python?

enter image description here

Update I answered my own question, but I'm open to an alternative approach. RPC doesn't quite do what I want though.

If that's all you're doing, why not using python directly? [SimpleHTTPServer](http://docs.python.org/2/library/simplehttpserver.html) and/or [WSGI](http://www.python.org/dev/peps/pep-3333/) might help — Jonas Schäfer, Dec 12 '12 at 21:29
I do agree with @JonasWielicki. Is this all your node.js does or is it just a simplified example of this one problem? — jdi, Dec 12 '12 at 21:39
Well, he wants to use WebSockets on the browser side, so `SimpleHTTPServer` and/or `WSGI` won't help. But http://pypi.python.org/pypi?%3Aaction=search&term=websocket&submit=search will. — abarnert, Dec 12 '12 at 21:43
@abarnert: OP actually wants socket.io specifically. But there are socket.io solutions for python as well, which means the entire server could be in python if desired. — jdi, Dec 12 '12 at 21:51
@jdi: Socket.IO is a library that implements WebSockets. And the reason I gave a link to a search for websockets at PyPI is to show that the entire server could be written in Python, just not with `SimpleHTTPServer` and/or `WSGI`. — abarnert, Dec 12 '12 at 22:01
@abarnert: I am aware that socket.io uses WebSockets as **one** of its transport types. My point is that `socket.io` is not websockets. But we make the same argument that the server can be written in python by some existing package. — jdi, Dec 12 '12 at 22:03
You're referring to the twisted/tornado route? Yes, I could do that and I could also use python for presentation (e.g. PyGTK) but it's just a matter of taste. — melhosseiny, Dec 12 '12 at 22:05
Understandable. I think the origin of the question was really just about whether you are only using node.js for the simple purposes listed in your example and whether the communication layer between node and python is even needed. But I assume the answer is that you are really doing a lot more with node.js than listed here. — jdi, Dec 12 '12 at 22:11

jdi · Answer 1 · 2012-12-12T22:49:07.797

Apache Thrift is a pretty awesome way to write RPC code between all of the major languages. You write a generic Thrift spec declaring the types and services, and then the code generator creates bindings for your desired languages. You can have apis for calling methods between your node and python code bases.

On a more generic low level approach, ZeroMQ is a message queue library that also has support for all of the major languages. With this, you can design your own solution for how you want to communicate (not just purely RPC). It has patterns for request/reply, push/pull, pub/sub, and pair. It gives you enough tools to put together a scalable system of any type.

I have used both and they are great solutions.

Just as a very rough example, the Thrift spec may be something like this. Lets say you want to communicate events from python to node.js, have it processed and get back some response:

myspec.thrift

struct Event {
    1: string message; 
}

service RpcService {
    string eventOccurred(1:Event e)
}

This defines a data structure called Event with a single string member to hold the message. Then a service called RpcService define one function called eventOccured which expects an Event as an argument, and will return a string.

When you generate this code for python and node.js, you can then use the client side code for python and the server side code for node.js

python

from myrpc import RpcService, ttypes

# create a connection somewhere in your code
CONN = connect_to_node(port=9000)

def someoneClickedSomething():
    event = ttypes.Event("Someone Clicked!")
    resp = CONN.eventOccurred(event)
    print "Got reply:", resp

node.js

// I don't know node.js, so this is just pseudo-code

var thrift = require('thrift');
var myrpc = require('myrpc.js');

var server = thrift.createServer(myrpc.RpcService, {
  eventOccurred: function(event) {
    console.log("event occured:", event.message);
    success("Event Processed.");
  },
});
server.listen(9000);

But does it support event-based realtime communication? Can I emit data from the Python server and do something on the Node.js client when it is emitted? — melhosseiny, Dec 12 '12 at 22:20
It is event based if you use it as such. Thrift, for instance, would give you method calls between the two systems. Lets say you write a thrift service called `eventOccured(event)`. When your event happens on the python side, you just call `eventOccured()` with your event object, and then the node.js side will receive that call as if it were local, and can send a return value if it wants (if it was set up in your thrift spec that it returns a result). With thrift you just define what the handler of the specific service is. In node.js that would basically be a callback for each service. — jdi, Dec 12 '12 at 22:23

score 2 · Answer 2 · answered Dec 12 '12 at 21:36

2

You can look at some messaging systems like 0mq http://www.zeromq.org/

answered Dec 12 '12 at 21:36

Vallières

1,409
13
17

score 2 · Accepted Answer · edited May 23 '17 at 11:46

I used the library inspired by this question to turn diagnosis.py into a Socket.IO client. This way I can emit the realtime data to the Node.js Socket.IO server:

socketIO.emit('gaze', ...)

And then have it do a socket.broadcast.emit to emit the data to all the Socket.IO clients (browser and diagnosis.py).

RPC is probably the more standard approach for cross-language development but I find it's a bit of an overkill to do that when the goal is to exchange data. It also does not support evented IO out of the box.

Update on Jan 2013 Since socket.broadcast.emit generates a lot of unnecessary traffic, I tried to find a better way of doing this. The solution I came up with is to use namespaces which is supported by the basic python Socket.IO client library I mentioned.

Python

self.mainSocket = SocketIO('localhost', 80)
self.gazeSocket = self.mainSocket.connect('/gaze')
self.gazeSocket.emit('gaze', ...)

To connect to the gaze namespace.

Node.js

var gaze = io.of('/gaze').on('connection', function (socket) {
    socket.on('gaze', function (gdata) {
        gaze.emit('gaze', gdata.toString());
    });
});

This emits the data received only to clients connected to the gaze namespace.

Browser

var socket = io.connect('http://localhost/gaze');
socket.on('gaze', function (data) {
    console.log(data);
});

abarnert · Answer 4 · 2012-12-12T22:15:07.213

If you're just looking for something that gives you a simple protocol on top of sockets, so you don't have to deal with buffering and delimiting messages and all that stuff, the two obvious choices (depending on what kind of data you're trying to send) are netstrings and JSON-RPC. There are multiple choices of libraries for both, in both languages, but you can end up with code as simple as this:

class MyService(Service):
    # … code to set up the newsdb
    def getnews(self, category, item):
        return self.newsdb.get(category, item)
myService = MyService(6000)

myService = Service(6000)
// … code to set up the newsdb
myService.getnews = function(category, item, callback) {
    callback(this.newsdb.get(category, item);
}

Code in Python, communicate in Node.js and Socket.IO, present in HTML

4 Answers4