3

I am looking at linking a few applications together (all written in different languages like C#, C++, Python) and I am not sure how to go about it.

What I mean by linking? The system I am working on consists of small programs each responsible for a particular processing task. I need to be able to transfer a data set from one application to another easily (the data set in question is not huge, probably a few megabytes) and I also need some form of way to control the current state of the operation (This is where a client-server model rings a bell)

It seems like sockets or maybe SOAP would be a universal solution but just wanted to get some opinions as to what people think about this subject.

Comments/suggestions will be appreciated, thanks!

Hamza
  • 2,313
  • 7
  • 25
  • 33

6 Answers6

2

I personally take a liking towards ØMQ. It's a library that has a familiar BSD-sockets-like interface for passing messages, but you'll find it implements interesting patterns for distributing tasks.

It sounds like you want to arrange several processes in a pipeline. ØMQ allows you to do that using push and poll sockets. (And afterwards, you'll find it's even possible to scale up across multiple processes and machines with little effort.) Take a look at the guide to get started, and the zmq_socket(3) manpage specifically for how push and pull works.

Bindings are available for all the languages you mention.

As for the contents of the message, ØMQ doesn't concern itself with that, they are just blocks of raw data. You can use any format that suits you, such as JSON, or perhaps Protocol Buffers.

What I'm not sure about is the ‘controlling state’ you mention. Are you interested in, for example, cancelling a job halfway through?

Stéphan Kochen
  • 19,513
  • 9
  • 61
  • 50
  • Thanks for the suggestion, This looks very interesting! Pipelining is essentially what I am doing although I will have to deal with forking the data into multiple sub-apps and wait for them to finish processing before I can move onto the next stage, this is where I will need some sort of state-checking. (Almost like pseudo-concurrency by means of running the data through different processes, I guess) – Hamza Dec 03 '10 at 20:18
  • Each sub-app is responsible for pushing the task to the next stage. One way to signal completion of the entire pipeline is by using pub-sub sockets, and having the last sub-app in the chain publish a notification of some sort. (For example, multiple processes may submit jobs using push sockets to the start of the pipeline, and each subscribes to completion notifications, waiting for their job's ID to come by.) This is how mongrel2 does it, if I'm not mistaken, see: http://www.igvita.com/2010/09/03/zeromq-modern-fast-networking-stack/ – Stéphan Kochen Dec 03 '10 at 20:37
  • That sounds quite similar to what I am trying to accomplish, looks like my reading material for the week is sorted. Many thanks for your suggestion! – Hamza Dec 03 '10 at 21:36
1

For C# to C# you can use Windows Communication Foundation. You may be able to use it with Python and C++ as well.

You may also want to checkout named pipes.

Nick
  • 3,217
  • 5
  • 30
  • 42
  • WCF looks interesting, although I may have to integrate some other languages that may not be supported by it. Thanks for the suggestion. – Hamza Dec 03 '10 at 20:15
0

I would think about moving to a model where you eliminate the issue by having centralized data that all of the applications look at. Keep "one source of the truth" so to speak.

Ben L.
  • 786
  • 7
  • 15
  • Unfortunately that's something outside my control, I agree that the single language approach would make my life a lot easier :) – Hamza Dec 03 '10 at 20:16
  • Doesn't have to be singular language, just singular data. – Ben L. Dec 07 '10 at 15:50
0

Most outside software has trouble linking against C++ code, due to the name-mangling algorithm it uses for its symbols. For that reason, when interfacing with programs written in other languages, it is often best to declare wrappers to things as extern "C" or inside an extern "C" { block.

T.E.D.
  • 44,016
  • 10
  • 73
  • 134
0

I need to be able to transfer a data set from one application to another easily (the data set in question is not huge, probably a few megabytes)

Use the file system.

and I also need some form of way to control the current state of the operation

Again, use the file system. A "current_state.json" file with a JSON serialized object is perfect for multiple languages to work with.

It seems like sockets or maybe SOAP would be a universal solution.

Perhaps. But it's overkill for this kind of thing. Your OS already has all the facilities you need. Just use the file system. It's very simple and very reliable.

S.Lott
  • 384,516
  • 81
  • 508
  • 779
  • Thanks for the suggestion, I have certainly considered the filesystem approach but thought maybe using a structured protocol (like SOAP) would make it easier to combine the data and control messages together. – Hamza Dec 03 '10 at 20:15
  • "easier"? How so? Provide some example in the question where it would be "easier" than the file system. I'm drawing a blank on what could possibly be simpler than ordinary files in ordinary directories. Perhaps I'm missing something. – S.Lott Dec 03 '10 at 20:16
  • I agree that nothing could be simpler than writing/reading from a file, I was referring to maintaining a coherent set of data/control messages flowing through the system all together (where a structured protocol might come in more handy). I guess with your suggested method of keeping a json file to keep track of what's going on that's not really an issue. Another thing that would be interesting to consider would be the performance of this system implemented using filesystem vs sockets. I may have to do some simple comparisons. – Hamza Dec 03 '10 at 20:27
  • @Hamza: "data/control messages flowing through the system". What does this mean? "The system I am working on consists of small programs each responsible for a particular processing task" Sounds like ordinary OS process management. Why are there messages "flowing through the system" when you have small, simple programs. Don't over-invent if you can avoid it. Simpler is always faster, cheaper and easier to maintain. Your requirements seem pretty simple. Unless your question is incomplete. – S.Lott Dec 03 '10 at 20:35
  • I guess I am just not good at explaining the requirements, point taken though, thanks :) – Hamza Dec 03 '10 at 21:30
0

There are many ways to do interprocess communication. As you said, sockets may be a universal solution. SOAP, i think, is somewhat an overkill. You may also use mailslots. I wrote C++ application using it a couple of years ago. Named pipes could be also a solution, but if you are coding on Windows, it may be difficult.

In my opinion:

  1. Sockets
  2. Mailslots

Are the best candidates.

JohnGray
  • 656
  • 1
  • 10
  • 27