0

I have a c process which needs to send a lot of c structs (roughly 10,000 per second) to a java process that needs to put the data into a matching class.

The size of the data that needs to be sent will be around 50-100 Byte per packet.

Latency is a concern as the data needs to be displayed in real time, which is why i am looking for alternatives to my current solution.

Currently im doing this with JNI and a POSIX message queue. Is there a better way than using JNI and message queues/pipes? I read somewhere that calling JNI methods often has a lot of overhead. Could this become a problem when a lot of data has to be sent?

Another solution i had in mind was to just write the data to a UNIX Socket and parse it in java.

Mark G
  • 250
  • 4
  • 10
  • Is there a need for the code in Java and the one in C to run in two distinct processes? – Samuel Audet Dec 24 '15 at 05:38
  • @SamuelAudet Yes, the data comes from a plugin for an IDS which has to process network packets as fast as possible. After processing a packet we send the decoded data to another process which does the graph handling. Furthermore the c process runs as a service while the java program should be independently startet or stopped when one desires to, without influencing the running c program. – Mark G Dec 24 '15 at 11:40
  • So, I'm guessing a memory mapped file would be the most efficiency way to do to this, and that appears to be the case: http://stackoverflow.com/a/6412333/523744 – Samuel Audet Dec 25 '15 at 08:51

3 Answers3

1

If you must process the data eventually using Java, then remove as many intermediate steps as possible.

If you can read the data directly into Java ( bypassing JNI and the C code ), then do so. Avoid the JNI, message queue and (presumably) a stage where C receives the data. The queue can't be helping latency either.

If the data starts in a C-friendly form that is unfriendly to Java, then I;d consider switching entirely to C or C++ rather than processing in Java at all.

StephenG - Help Ukraine
  • 959
  • 6
  • 12
  • The problem is that the application where the information is coming from is in c and i (my team) is merely extending it. We have to use java because except for 2 people nobody really knows how to program in c/c++ so we chose java (6 people team). The c part is responsible for decoding network packets for an intrusion detection system. This has to be as fast as possible and is done in c. Then we split off some data to the java program which builds a graph out of the data to visualize the network. – Mark G Dec 23 '15 at 04:38
  • If your goal is building a graph to visualize the results, then speed, even in real time, is unlikely to be a major issue. Have you actually encountered any real performance issue ? It's also likely that you can build the graph in C and pass a minimum of data to Java ( the graph structure, as opposed to the individual data items ). – StephenG - Help Ukraine Dec 23 '15 at 05:16
  • 1
    @MarkG *We have to use java because except for 2 people nobody really knows how to program in c/c++ so we chose java (6 people team).* If you don't know how to program in C/C++, you're going to have a very tough time coding JNI, where you not only have to know how to code in Java **and** C, but you also have to understand how both languages work. If you're already coding JNI, you know how to code C at least. – Andrew Henle Dec 23 '15 at 10:48
0

You can achieve high throughput by avoiding unnecessary copying of memory. Transfer data between C and Java through direct byte buffers. From the Java side, you can then read the data in the byte buffers and copy values into the object fields. For two processes talking to each other, you could use a memory mapped file to do this (you would use a MappedByteBuffer for this).

For something simpler but with a bit more overhead, you could simply use the stdin/stdout of each process to communicate and send data that way. Or, as you suggested, a socket is another option.

With 10,000 structs at 100 bytes each, this will be 1 MB / second of data processed. This really shouldn't be a problem on modern hardware (for one of my projects I have managed easily over 1GB / second between direct buffers and Java objects with primitive + array fields, but this was all in one process between JNI and Java).

You might want to use the simpler approach first (stdin/stdout) and see is that fast enough first before digging into using memory mapped files.

prunge
  • 22,460
  • 3
  • 73
  • 80
  • Thanks for the input. I'll try mashing toghether a test scenario and see if that works better. – Mark G Dec 23 '15 at 04:35
0

This, I think, can be solved using typical IPC methods. I would use pipes over a message queue and sockets. The overhead of message queue is going to slow the processing. And pipes are faster than sockets but not by much.

For your issue, 10,000 structs at 100 bytes per packet comes to 1 MB/s. A modern multicore processor will handle this without a problem.

byteherder
  • 331
  • 2
  • 9