Transferring data between executables

Question

I have two executables written in C++ on Windows. I generate some data in one, and want to call the other executable to process this data. I could write the data out to a file then read it in the other executable, but that seems rather expensive in terms of disk I/O. What is a better way of doing this? It seems like a simple enough question but google just isn't helping!

Let's say the data is around 100MB, and is generated in its entirety before needing to be sent (i.e. no streaming is needed).

Answers that work when mixing 32 bit and 64 bit processes gain bonus points.

named pipes, local winsockets, memory mapped files, shared memory, mail slot, registry, window messages (not good for large data), what else? take your pick. if you tell us a little more about the data (size, etc.) and the nature to which you want to share the data, we can give better advice as far as which method is best. — thang, Feb 07 '13 at 12:22
In general, this is called [inter-process communication](http://en.wikipedia.org/wiki/Inter-process_communication). — Oswald, Feb 07 '13 at 12:27
Sure, I've added a little information about the data. Originally I just wrote "large amount" but edited my question down before posting, I should really have just given some numbers :) — Ben Hymers, Feb 07 '13 at 12:27
General term for what you are looking for is *interprocess communication*, I'm sure Google can help with that. Or be more specific, explain your goals and requirements, so more technical answer might be provided here. — KBart, Feb 07 '13 at 12:28

score 5 · Accepted Answer · answered Feb 07 '13 at 14:26

5

If your processes can easily write to and read from file, just go ahead. Create the file with CreateFile and mark it as temporary & shareable. Windows uses this hint to delay physical writes, but all file semantics are still obeyed. Since your file is only 100 MB and actively in use, Windows is almost certainly able to keep its contents fully in RAM.

answered Feb 07 '13 at 14:26

MSalters

173,980
10
155
350

Ok, I didn't know this kind of caching was this reliable. Definitely the simplest solution, thanks :) – Ben Hymers Feb 08 '13 at 09:39
1

This sort of caching is just as reliable as keeping the data in RAM. Windows can page RAM to swapfile if needed. And behind the scenes, this is in fact the same mechanism. The Virtual Memory Manager keeps an association between RAM and the backing location on disk. – MSalters Jan 28 '19 at 10:40

log0 · Answer 2 · 2013-02-07T13:19:54.020

You can use Boost.MPI. It is from Boost, which has high quality standard, and the code sample seems pretty explicit:

http://www.boost.org/doc/libs/1_53_0/doc/html/mpi/tutorial.html#mpi.point_to_point

// The following program uses two MPI processes to write "Hello, world!"
// to the screen (hello_world.cpp):

int main(int argc, char* argv[])
{
  mpi::environment env(argc, argv);
  mpi::communicator world;

  if (world.rank() == 0) {
    world.send(1, 0, std::string("Hello"));
    std::string msg;
    world.recv(1, 1, msg);
    std::cout << msg << "!" << std::endl;
  } else {
    std::string msg;
    world.recv(0, 0, msg);
    std::cout << msg << ", ";
    std::cout.flush();
    world.send(0, 1, std::string("world"));
  }
  return 0;
}

Boost never ceases to amaze me! This seems like a good option, especially if I extend this to work over multiple machines, but for now I'll go with the simpler option of relying on file caching as MSalters suggests. — Ben Hymers, Feb 08 '13 at 09:43

score 1 · Answer 3 · answered Feb 07 '13 at 12:28

1

Assuming you only want to go "one direction" (that is, you don't need to get data BACK from the child process), you could use _popen(). You write your data to the pipe and the child process reads the data from stdin.

If you need bidirectional flow of data, then you will need to use two pipes, one as input and one as output, and you will need to set up a scheme for how the child process connects to those pipes [you can still set up the stdin/stdout to be the data path, but you could also use a pair of named pipes].

A third option is a shared memory region. I've never done this in Windows, but the principle is pretty much the same as what I've used in Linux [and many years back in OS/2]: 1. Create a memory region with a given name in your parent process. 2. The child process opens the same memory region. 3. Data is stored by parent process and read by child process. 4. If necessary, semaphores or similar can be used to signal completion/results ready/etc.

answered Feb 07 '13 at 12:28

Mats Petersson

126,704
14
140
227

1

I've learnt something new :) Thanks! Pipes sound like a good option but I'm guessing shared memory implies shared address space, which wouldn't work when mixing 32 and 64 bit processes. – Ben Hymers Feb 08 '13 at 09:41
As I haven't tried, I don't know, but I believe as long as you use a named shared memory, and don't exceed the address limit of 32-bit [so less than 2GB], it should work. Just remember to not put pointers within your shared memory. The memory isn't (necessarily) shared at the same address. – Mats Petersson Feb 08 '13 at 09:43
Ah right, I misunderstood the code sample. Yes, it does look like it works that way. Thanks x2! – Ben Hymers Feb 08 '13 at 09:45
Using boost::interprocess::managed_windows_shared_memory did not work properly for me when mixing 32 and 64 bit. – MShekow Jan 14 '14 at 08:01

Transferring data between executables

3 Answers3

Linked