3

I'm working on a web application written in Golang that needs to call a Python program/module to do some heavy work. Since that is very memory/CPU intensive, it may be on a separate machine. Since Golang and Python can't talk directly, there are 3 ways to achieve this:

  1. Just execute the python program as an OS process from Go (if on same machine) (or RPC?)
  2. Wrap Python process in a service and expose it for it to be called from Go (may be a simple CRUD like service - A Bottle/flask restful service)
  3. Have a simple pub-sub system in place to achieve this (Redis or some MQ system) - Adding Redis based caching is on the radar so maybe a good reason to go this way. Not sure.

The main thing is that the python process that takes really long to finish must "inform" the web application that it has finished. The data could either be in a file/DB or 'returned' by the process.

What could be the simplest way to achieve this in a pub/sub like environment?

UPDATE REST seems like one way but would incur the cost of implementing server side push which may or may not be easily doable with existing micro web frameworks. The pub/sub would add an additional layer of complexity for maintainability and a learning curve nevertheless. I'm not sure if an RPC like invocation could be achieved across machines. What would be a good choice in this regard?

PhD
  • 11,202
  • 14
  • 64
  • 112

2 Answers2

4

Use ZeroMQ

Python has nice pyzmq

GO bindngs exists too.

ZeroMQ code is typically short and effective.

You can even use local socket on Unix.

There are even other messaging patterns apart from pub/sub, you can use req/resp or push/pull.

Not knowing much about your applications, I can offer sample of integrating lock manager using zeromq in Python

Using ZeroRPC

There is experimental Golang lib being able communicating with Python and Node.js code via ZeroMQ as shown (for Python and Node.js) zerorpc.dotclould.com

Writing functions in Python being callable over ZeroRPC is extremely simple - in fact you do not have to write a line using zeromq (there is command line tool, which allows integrating functions with properly shaped signatures).

peeyush pathak
  • 3,663
  • 3
  • 19
  • 25
Jan Vlcinsky
  • 42,725
  • 12
  • 101
  • 98
  • From a maintainability and economic standpoint, is it worth the overhead? It's a very small use case. – PhD Jul 16 '14 at 21:41
  • @PhD What is so expensive on using ZeroMQ? Technically it is very fast, does not consume much memory and you will have difficulties finding technically more effective method. If the question goes for pub/sub, do you have any better offer? Redis is bigger and must be installed. Db needs a file. – Jan Vlcinsky Jul 16 '14 at 21:44
  • @PhD Just added link to another answer showing integration using zeromq. Just count lines needed for communication. Apart from having pyzmq library installed for Python and similar lib for Go, there is no more to set up and maintain. – Jan Vlcinsky Jul 16 '14 at 21:48
  • @PhD Added ZeroRPC solution. I am not able to evaluate, how experimental the Golang part is, but I have tried the Python part and it is awesome - just repeat the examples from (ZeroRPC README](https://github.com/dotcloud/zerorpc-python) and you will understand what I mean. – Jan Vlcinsky Jul 16 '14 at 22:01
0

For your specific pattern, simply spawning the process from Go and reading the stdout is the most efficient, there's no point adding an over head.

It highly highly depends on what your python script does, if it's one specific task then simply spawning the process and checking the exit code is more than enough, if you have to keep the script in the background at all time and communicate with it then Redis or ZeroMQ are good, and very mature on both Go and Python.

If it's on a different server then ZeroMQ/RPC or just a plain http server in python should be fine, the overhead should be minimal.

OneOfOne
  • 95,033
  • 20
  • 184
  • 185
  • It doesn't really write anything to stdout and we won't need it. I see what you mean and tend to lean on this. But there is a possibility of adding Redis to the application in the future and was wondering if I should go that route. – PhD Jul 16 '14 at 21:40
  • It highly highly depends on what your python script does, if it's one specific task then simply spawning the process and checking the exit code is more than enough, if you have to keep the script in the background at all time and communicate with it then Redis or ZeroMQ are good, and very mature on both Go and Python. – OneOfOne Jul 16 '14 at 21:52
  • It's a huge memory intensive task that will be run on a separate machine to prevent to the golang server from slowing down. So spawning a separate process may not be the best idea – PhD Jul 17 '14 at 00:26
  • Then ZeroMQ/RPC and/or just plain http server in python should be fine, the overhead should be minimal. – OneOfOne Jul 17 '14 at 01:23
  • Could you please add your last comment to the answer so that I can go ahead and accept it? – PhD Jul 22 '14 at 18:29