What can I use instead of a Unix pipe for inter-process communication on Windows?

Question

I have an application that launches a subprocess. The subprocess reads files to operate on from stdin. For some operations it needs an input file containing information on what to do with the files it operates on – let's call this the "control file". The name of the control file is also read from stdin. The parent application could use a temporary file as control file, but I would prefer to avoid a real, disk-backed file.

On Linux, this is simple: I can create a Unix pipe, fork, close the respective ends of the pipe before starting the subprocess, and use /dev/fd/3 (or whatever the file descriptor is) as control file name, and then write the control data to the pipe in the parent application. Alternatively, I could use a named pipe in /tmp (or whatever).

How could I achieve a similar thing on Windows? Could the strange "named pipes" Windows offers be used for this, that is, can they be read from by the usual C library fread() function? If yes, what file name do I use to access them? Or is there a better way than using named pipes?

(The subprocess is the exiftool command-line utility run in batch mode, so I don't have control over it. The parent application is written in Python.)

Unlikely to work, I'm afraid; you probably need a real file. If the disk I/O involved is an issue, RAMdisk software is available for Windows. — Harry Johnston, Aug 01 '12 at 21:09
@HarryJohnston: The disk I/O is not too much of an issue. The files `exiftool` operates upon are usually media files, so the I/O involved in writing and reading the control file is dwarved by the I/O involved in reading and writing the media files. Conceptually, the pipe is what I would use on Linux, so I hoped there would be an equivalent solution on Windows. If this is not the case, I'll simply go with the temporary file solution. — Sven Marnach, Aug 01 '12 at 22:01

asveikau · Answer 1 · 2012-08-01T21:31:22.667

Update:

@Harry Johnston points out that I misread your question - you don't want to modify the subprocess. In that case you can try calling CreateProcess and fill in the hStdInput member of STARTUPINFO with a HANDLE from CreateNamedPipe.

Previous answer:

In general the Windows CRT (C runtime, or "libc" in Unix speak) is a strange beast: it's a very "barebones" shim for the C standard library with a few extra things thrown in, not very well maintained and not exposing a lot of what Windows can do. The most natural way to write Windows software in C is the Win32 API. That said:

Could the strange "named pipes" Windows offers be used for this, that is, can they be read from by the usual C library fread() function?

Yes, I believe you can do that with _open_osfhandle, where the first parameter can be a HANDLE. This would give you an integer, with is the Windows CRT's weird mockery of a Unix file descriptor. You could then get a FILE* with _fdopen.

If yes, what file name do I use to access them?

I suppose you could attempt to generate a random one that doesn't collide. Maybe prefix it with your application's name, and take some combination of process ID and the current time? That's just something I'm throwing out there.

Or is there a better way than using named pipes?

You could use sockets in the AF_UNIX family, though it would end up being pretty similar...

Using _open_osfhandle would require the OP to modify the subprocess, which he's said he can't do. — Harry Johnston, Aug 01 '12 at 21:07
@HarryJohnston Hmm, thanks, I seem to have missed that. I'll update the answer. — asveikau, Aug 01 '12 at 21:29
Thanks for your answer. I don't quite understand how the `hStdInput` member helps with my problem. Note that I need both, the usual stdin *and* some other pipe for the control file to communicate with the subprocess. The subprocess uses C library function for file operations, and I need a file name to pass to the subprocess. — Sven Marnach, Aug 01 '12 at 22:07

Zilog80 · Answer 2 · 2021-05-11T12:58:29.363

You can use PowerShell as a batch subprocess to create named pipe and use it to build IPC between the batch and others batchs / sub systems on Windows.

Here is an example using that for quickly coloring batch output : PowerShell as a subprocess through named pipe

EDIT: Set it to message mode and make it more readable for @Bbb

EDIT2: Adding missing NOP and security warning.

Be careful regarding security issue. Use credentials for the named pipe and note that the fd 5 is readable by any console/process on the server. Use true bi-directionnal pipe or another named pipe instead of fd 5 to avoid that. Don't use this "as is" in the wild.

Example batch "code" to create a "bi-directional" named pipe through PowerShell :

::
:: Launch a PowerShell child process in the background linked to the console and 
:: earing through named pipe which name provided as an argument of this function.
::
:: Parameters :
::   [ Name] : A name for the named pipe.
:: Return :
::   0 if the child PowerShell has been successfully launched and the named pipe is available.
::   1 if it fails.
::   3 if PowerShell is not present or doesn't work.
::
:LaunchPowerShellSubProcessPipe
  SET LOCALV_NAME=
  IF NOT "%~1" == "" SET "LOCALV_NAME=%~1"
  IF "%~1" == "" ( ECHO :LaunchPowerShellSubProcessPipe need a name.& EXIT /B 1)
  powershell -command "$_" 2>&1 >NUL
  IF NOT "!ERRORLEVEL!" == "0" EXIT /B 3
  REM As properly stated by WReach [https://mathematica.stackexchange.com/questions/198187/how-to-read-a-named-pipe-on-windows], Windows named pipe have their own hand shake logic (WaitNamedPipe... {humor}Microsoft hate us all probably{/humor}
  REM To "read" the named pipe from batch in a POSIX way, we redirected reply to fd 5. Reply to the output part of the named pipe can't be read from batch directly nor with common windows tools. 
  REM Do not use [System.IO.Pipes.PipeOptions]::Asynchronous ! 
  (ECHO $CloseHandle = Add-Type 'A' -PassThru -MemberDefinition '[DllImport^^^("kernel32.dll"^^^)] public static extern bool CloseHandle^^^(IntPtr hObject^^^);'; & ^
ECHO # In case of, we close the fd beforehand & ^
ECHO $Clean = $CloseHandle::CloseHandle^^^(5^^^); & ^
ECHO $npipeServer = new-object System.IO.Pipes.NamedPipeServerStream^^^('%LOCALV_NAME%', [System.IO.Pipes.PipeDirection]::In, 1, [System.IO.Pipes.PipeTransmissionMode]::Message, [System.IO.Pipes.PipeOptions]::None, 256, 256^^^); & ^
ECHO Try { & ^
ECHO   $npipeServer.WaitForConnection^^^(^^^); & ^
ECHO   $pipeReader = new-object System.IO.StreamReader^^^($npipeServer^^^); & ^
ECHO   Try { & ^
ECHO     While^^^(^^^($msg = $pipeReader.ReadLine^^^(^^^)^^^) -notmatch 'QUIT'^^^) { & ^
ECHO       Try { & ^
ECHO         If ^^^($msg.Length -gt 0^^^) { & ^
ECHO           $disp='Server got :'+$msg; & ^
ECHO           Write-Host^^^($disp^^^); & ^
ECHO           $curdte = Get-Date -Format 'HH:mm:ss'; & ^
ECHO           $reply='Reply to '+$msg+', im here, time is '+$curdte; & ^
ECHO           $reply_flag=0; & ^
ECHO           While^^^($reply_flag -eq 0^^^) { & ^
ECHO             Try { & ^
ECHO               Write-Output^^^($reply^^^) ^^^>5; & ^
ECHO               $reply_flag=1; & ^
ECHO             } Catch [System.IO.IOException] { & ^
ECHO               # Deal as you whish with potential errors & ^
ECHO             }; & ^
ECHO           }; & ^
ECHO         } & ^
ECHO       } Finally { & ^
ECHO         $npipeServer.Disconnect^^^(^^^); & ^
ECHO         $npipeServer.WaitForConnection^^^(^^^); & ^
ECHO       } & ^
ECHO     }; & ^
ECHO   } Catch [System.IO.IOException] {  & ^
ECHO     # Deal as you whish with potential errors, you will have InvalidOperation here when the streamReader is empty & ^
ECHO   } & ^
ECHO } Finally { & ^
ECHO   If ^^^($npipeServer^^^) { & ^
ECHO     $npipeServer.Dispose^^^(^^^); & ^
ECHO   } & ^
ECHO   # We close the fd & ^
ECHO   $Clean = $CloseHandle::CloseHandle^^^(5^^^); & ^
ECHO } & ^
ECHO exit & ^
ECHO[)| START /B powershell 2>NUL >NUL
  SET /A LOCALV_TRY=20 >NUL
  :LaunchPowerShellSubProcessPipe_WaitForPipe
  powershell -nop -c "& {sleep -m 50}"
  SET /A LOCALV_TRY=!LOCALV_TRY! - 1 >NUL
  IF NOT "!LOCALV_TRY!" == "0" cmd /C "ECHO[|SET /P=|MORE 1>\\.\pipe\!LOCALV_NAME!" 2>NUL || GOTO:LaunchPowerShellSubProcessPipe_WaitForPipe
  IF "!LOCALV_TRY!" == "0" EXIT /B 1
  REM 250 ms. PowerShell isn't the fastest thing to start when dealing with fd redirection... And we need it fully up to write to pipe and read from fd 5.
  powershell -nop -c "& {sleep -m 250}"
  EXIT /B 0

And an example of implementation :

@echo off
SETLOCAL ENABLEEXTENSIONS
IF ERRORLEVEL 1 (
  ECHO Can't use extensions
  EXIT /B 1
)
::
SETLOCAL ENABLEDELAYEDEXPANSION
IF ERRORLEVEL 1 (
  ECHO Can't use expansion 
  EXIT /B 1
)
REM We create 'MyNamedPipe'
CALL:LaunchPowerShellSubProcessPipe "MyNamedPipe"
SET "LOCALV_RET=!ERRORLEVEL!"
IF NOT "!LOCALV_RET!" == "0" (
  ECHO Failed to create the named pipe... Exit code from LaunchPowerShellSubProcessPipe : !LOCALV_RET!
  EXIT /B 1
)
REM Sending something through the pipe to the PowerShell subprocess that can be used as an IPC gate for other processes
ECHO Batch send hi to the PowerShell subprocess
ECHO Hi from the batch>\\.\pipe\MyNamedPipe
REM A NOP equivalent to be sure pipe hand shake reach completion before any other comms and that nothing stay open from this CMD and the named pipe
ECHO[|SET /P=>NUL
REM At this time, the PowerShell subprocess will take few cycles to write on the console and then write the response through fd 5.
REM We can check the pipe availability in the same way it's done in :LaunchPowerShellSubProcessPipe, but a direct read will suffice here. 
REM If we've read from the pipe, it should have ben a blocked read as we've created a synchronous pipe. 
REM As we read reply from fd 5, it's not synchronous, so we need to wait until we get the reply.
:WaitForReply
SET LOCALV_REPLY=0
FOR /F "tokens=*" %%R IN ('MORE ^<5') DO (
  ECHO Batch got a reply from SubProcess : %%R
  SET LOCALV_REPLY=1
)
IF NOT "!LOCALV_REPLY!" == "1" GOTO :WaitForReply
ECHO caller is happy >\\.\pipe\MyNamedPipe
REM A NOP equivalent to be sure pipe hand shake reach completion before any other comms and that nothing stay open from this CMD and the named pipe
ECHO[|SET /P=>NUL
:WaitForReply2
SET LOCALV_REPLY=0
FOR /F "tokens=*" %%R IN ('MORE ^<5') DO (
  ECHO Batch got a 2nd reply from SubProcess : %%R
  SET LOCALV_REPLY=1
)
IF NOT "!LOCALV_REPLY!" == "1" GOTO :WaitForReply2
REM Waiting one second for visible time delta
powershell -nop -c "& {sleep -m 1000}" 
ECHO This is my leave. >\\.\pipe\MyNamedPipe
REM A NOP equivalent to be sure pipe hand shake reach completion before any other comms and that nothing stay open from this CMD and the named pipe
ECHO[|SET /P=>NUL
:WaitForReply3
SET LOCALV_REPLY=0
FOR /F "tokens=*" %%R IN ('MORE ^<5') DO (
  ECHO Batch got a 3th reply from SubProcess : %%R
  SET LOCALV_REPLY=1
)
IF NOT "!LOCALV_REPLY!" == "1" GOTO :WaitForReply3
REM We can now tell goodbye to the PowerShell subprocess
ECHO QUIT>\\.\pipe\MyNamedPipe
EXIT /B 0
::
:: Launch a PowerShell child process in the background linked to the console and 
:: earing through named pipe which name provided as an argument of this function.
::
:: Parameters :
::   [ Name] : A name for the named pipe.
:: Return :
::   0 if the child PowerShell has been successfully launched and the named pipe is available.
::   1 if it fails.
::   3 if PowerShell is not present or doesn't work.
::
:LaunchPowerShellSubProcessPipe
  SET LOCALV_NAME=
  IF NOT "%~1" == "" SET "LOCALV_NAME=%~1"
  IF "%~1" == "" ( ECHO :LaunchPowerShellSubProcessPipe need a name.& EXIT /B 1)
  powershell -command "$_" 2>&1 >NUL
  IF NOT "!ERRORLEVEL!" == "0" EXIT /B 3
  REM As properly stated by WReach [https://mathematica.stackexchange.com/questions/198187/how-to-read-a-named-pipe-on-windows], Windows named pipe have their own hand shake logic (WaitNamedPipe... {humor}Microsoft hate us all probably{/humor}
  REM To "read" the named pipe from batch in a POSIX way, we redirected reply to fd 5. Reply to the output part of the named pipe can't be read from batch directly nor with common windows tools. 
  REM Do not use [System.IO.Pipes.PipeOptions]::Asynchronous ! 
  (ECHO $CloseHandle = Add-Type 'A' -PassThru -MemberDefinition '[DllImport^^^("kernel32.dll"^^^)] public static extern bool CloseHandle^^^(IntPtr hObject^^^);'; & ^
ECHO # In case of, we close the fd beforehand & ^
ECHO $Clean = $CloseHandle::CloseHandle^^^(5^^^); & ^
ECHO $npipeServer = new-object System.IO.Pipes.NamedPipeServerStream^^^('%LOCALV_NAME%', [System.IO.Pipes.PipeDirection]::In, 1, [System.IO.Pipes.PipeTransmissionMode]::Message, [System.IO.Pipes.PipeOptions]::None, 256, 256^^^); & ^
ECHO Try { & ^
ECHO   $npipeServer.WaitForConnection^^^(^^^); & ^
ECHO   $pipeReader = new-object System.IO.StreamReader^^^($npipeServer^^^); & ^
ECHO   Try { & ^
ECHO     While^^^(^^^($msg = $pipeReader.ReadLine^^^(^^^)^^^) -notmatch 'QUIT'^^^) { & ^
ECHO       Try { & ^
ECHO         If ^^^($msg.Length -gt 0^^^) { & ^
ECHO           $disp='Server got :'+$msg; & ^
ECHO           Write-Host^^^($disp^^^); & ^
ECHO           $curdte = Get-Date -Format 'HH:mm:ss'; & ^
ECHO           $reply='Reply to '+$msg+', im here, time is '+$curdte; & ^
ECHO           $reply_flag=0; & ^
ECHO           While^^^($reply_flag -eq 0^^^) { & ^
ECHO             Try { & ^
ECHO               Write-Output^^^($reply^^^) ^^^>5; & ^
ECHO               $reply_flag=1; & ^
ECHO             } Catch [System.IO.IOException] { & ^
ECHO               # Deal as you whish with potential errors & ^
ECHO             }; & ^
ECHO           }; & ^
ECHO         } & ^
ECHO       } Finally { & ^
ECHO         $npipeServer.Disconnect^^^(^^^); & ^
ECHO         $npipeServer.WaitForConnection^^^(^^^); & ^
ECHO       } & ^
ECHO     }; & ^
ECHO   } Catch [System.IO.IOException] {  & ^
ECHO     # Deal as you whish with potential errors, you will have InvalidOperation here when the streamReader is empty & ^
ECHO   } & ^
ECHO } Finally { & ^
ECHO   If ^^^($npipeServer^^^) { & ^
ECHO     $npipeServer.Dispose^^^(^^^); & ^
ECHO   } & ^
ECHO   # We close the fd & ^
ECHO   $Clean = $CloseHandle::CloseHandle^^^(5^^^); & ^
ECHO } & ^
ECHO exit & ^
ECHO[)| START /B powershell 2>NUL >NUL
  SET /A LOCALV_TRY=20 >NUL
  :LaunchPowerShellSubProcessPipe_WaitForPipe
  powershell -nop -c "& {sleep -m 50}"
  SET /A LOCALV_TRY=!LOCALV_TRY! - 1 >NUL
  IF NOT "!LOCALV_TRY!" == "0" cmd /C "ECHO[|SET /P=|MORE 1>\\.\pipe\!LOCALV_NAME!" 2>NUL || GOTO:LaunchPowerShellSubProcessPipe_WaitForPipe
  IF "!LOCALV_TRY!" == "0" EXIT /B 1
  REM 250 ms. PowerShell isn't the fastest thing to start when dealing with fd redirection... And we need it fully up to write to pipe and read from fd 5.
  powershell -nop -c "& {sleep -m 250}"
  EXIT /B 0

Expected output :

Batch send hi to the PowerShell subprocess
Batch got a reply from SubProcess : Reply to Hi from the batch, im here, time is 14:15:34
Batch got a 2nd reply from SubProcess : Reply to caller is happy , im here, time is 14:15:34
Batch got a 3th reply from SubProcess : Reply to This is my leave. , im here, time is 14:15:35

What can I use instead of a Unix pipe for inter-process communication on Windows?

2 Answers2

Linked