0

I am accessing an executable file through subprocess, The exe, takes two file inputs like so,

foo.exe file1.txt file2.txt

It also accepts pipe's as inputs, so I give one of the files to the exe through a subprocess.PIPE, which works fine(can test it with a single file), however, because each process has a single stdin, I cannot supply the second file. I could pipe one input, then write, read the other, however both of the input files are generated inside the Python program, I don't want to decrease the speed of the code by writing to hard disk, rather use some other method to pipe the second file, which is on RAM. How can I achieve this ?

I am on Windows 10, with Python 3 (if the solution is system dependant).

slhck
  • 36,575
  • 28
  • 148
  • 201
Rockybilly
  • 2,938
  • 1
  • 13
  • 38

1 Answers1

0

It is rather broad for SO. As you say, out of the box only one input channel is given to a child. If you need more, it will require cooperation from the child. Some ideas: a protocol prefixing each input line (from stdin) with a file number, named pipes, TCP sockets.

But a perhaps simpler way would be to use a memory file system. That way, the child process keeps its normal interface to 2 files, but the files lie in a memory disk.

But anyway, according to this other SO post disk caching aready provided by the system could allow the child to access the file content before it even reaches the disk, so you should carefully benchmark whether RAM disks really improve performances, at least for small files.

Serge Ballesta
  • 143,923
  • 11
  • 122
  • 252
  • Thanks for the answer, I have read about named pipes, though they are useful, it seemed hard to use on Windows, I am currectly writing a temp file to the disk which does the job just for now. I was curious if there was a simple answer I have been overlooking :) – Rockybilly Nov 01 '17 at 10:15
  • For a temporary file, you probably want the `CreateFile` attribute `FILE_ATTRIBUTE_TEMPORARY` and flag `FILE_FLAG_DELETE_ON_CLOSE`, which keeps as much as possible in cache memory and guarantees the file will be deleted even if the process is terminated. For Python's `os.open`, these are respectively the mode flags `O_SHORT_LIVED` and `O_TEMPORARY`. `tempfile.NamedTemporaryFile` uses `O_TEMPORARY`, but you'd have to patch `tempfile._bin_openflags` to include `O_SHORT_LIVED`. – Eryk Sun Nov 01 '17 at 17:17