Integration patterns
There are few integration patterns usable for this purpose.
Messaging allows decoupling by communicating contracted messages, all the rest is left as platform specific detail, which does not affect other parties.
Other method could be e.g. shared data store - one side would store requests for processing there, another would fetch them and process.
Messaging integration pattern using ZeroMQ
ZeroMQ is a library, allowing very lightway messaging without requiring to run some big messaging application.
What you describe could be resolved this way:
- Python script is running in a loop, awaiting requests for something to do.
- Any client (being able to issue ZeroMQ request) would ask for an action and get the response
With clients being in multiple languages you shall use some cross-platform serialization, result buffers or even better JSON strings are good options.
ZeroMQ has libraries for many languages, so you shall be able to communicate.
There is great ZeroMQ Guide providing clear explanation of concepts and showing implementation in many languages.
For planning the communication infrastructure, I would start with your Python script being bound to some TCP port (being "fixed" part) and your clients connecting to it.
Python script would use REP socket (for "reply"), your clients in other languages REQ sockets.
Following samples come from huge set of examples in zguide written for many languages
Sample Python server code (hwserver.py):
import zmq
import time
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")
while True:
# Wait for next request from client
message = socket.recv()
print("Received request: %s" % message)
# Do some 'work'
time.sleep(1)
# Send reply back to client
socket.send(b"World")
Sample PHP client (hwclient.php):
<?php
/*
* Hello World client
* Connects REQ socket to tcp://localhost:5555
* Sends "Hello" to server, expects "World" back
* @author Ian Barber <ian(dot)barber(at)gmail(dot)com>
*/
$context = new ZMQContext();
// Socket to talk to server
echo "Connecting to hello world server...\n";
$requester = new ZMQSocket($context, ZMQ::SOCKET_REQ);
$requester->connect("tcp://localhost:5555");
for ($request_nbr = 0; $request_nbr != 10; $request_nbr++) {
printf ("Sending request %d...\n", $request_nbr);
$requester->send("Hello");
$reply = $requester->recv();
printf ("Received reply %d: [%s]\n", $request_nbr, $reply);
}
Integration over shared data store
Shared file or database also allows communication between processes. The problem becomes complex with locking resources, taking care, that the same task is not processed by multiple workers etc.
Conclusion
There are multiple patterns and multiple platforms to implement.
- database - you need to set it up and have it running. Might be too much for some small tasks.
- shared file - possible conflict with sharing the file from multiple processes - if multiple processes try to write, there will be possibly a conflict.
- messaging over TCP or UDP sockets - cold work, but in practice this is technically non-trivial as it has many cornercases you have to resolve (what to do, when one side gets down, reconnects...)
- messaging over web services - you could set up web server, accepting requests from clients, processing on server. This is rather easy for quick tasks, but if the task takes more then 30 seconds, you probably fail and have to turn it to background processing - using some sort of messaging.
- middle size and huge messaging systems (like Celery with RabbitMQ, Redis...) - have great tooling, could be overkill for many cases
- messaging over ZeroMQ - this is my favourite - you can set it up in few lines of code and running without too many problems around - and it resolves problems like reconnects. etc. E.g. see full working example for distributed locker
