Start executable as background process, redirecting stderr to file, and obtain PID, in single cmd.exe window via batch script?

Question

I have seen:

• How to start an application without waiting in a batch file?
• How can I run a program from a batch file without leaving the console open after the program starts?
• How to get PID of process just started from within a batch file?
• Is there any way to redirect stderr output from a command run with "start" in the Windows command line?

... but I still cannot really get what I want working, so here goes my question.

I have a program that basically loops forever (until interrupted with Ctrl-C), and outputs log messages to stderr; here is an example, testlogerr.c:

// based on https://stackoverflow.com/questions/26965508/infinite-while-loop-and-control-c
// can be compiled in MINGW64 with:
// gcc -g testlogerr.c -o testlogerr.exe

#include <unistd.h>
#include <stdio.h>
#include <signal.h>
#include <time.h>

volatile sig_atomic_t stop;

void inthand(int signum) {
  stop = 1;
}

int main(int argc, char **argv) {

  signal(SIGINT, inthand);
  int counter = 0;

  while(!stop) {
    fprintf( stderr, "%d: Logging line %d\n", (int)time(NULL), counter++);
    sleep(2);
  }

  printf("exiting safely\n");
  //system("pause"); // does "Press any key to continue . . ."; skip here
  return 0;
}

Now, having built this program (in MINGW64) as a Windows .exe, I would like to start it via a batch script in cmd.exe as a background process, having its stderr redirected to a log file, and obtaining its PID as a process. To do this, in Linux bash I'd simply do (see also https://unix.stackexchange.com/questions/74520/can-i-redirect-output-to-a-log-file-and-background-a-process-at-the-same-time , How to get process ID of background process?):

testlogerr > myfile.log 2>&1 &
TESTLOGERR_PID=$!
echo "testlogerr started, its PID is $TESTLOGERR_PID"

My question is: how can I do the same in a batch script, so I get only one cmd.exe window started, and I get a prompt in that window after the background process has started?

As far as I've seen from the links above, start /b would start a command in background - but then one cannot obtain the PID of the background process.

Furthermore, https://stackoverflow.com/a/59971707/6197439 recommends PowerShell, so I tried the following, e.g. as testlogerr.bat:

powershell -executionPolicy bypass -command ^
 "& {$process = start-process $args[0] -passthru -argumentlist $args[1..($args.length-1)]; exit $process.id}" ^
 testlogerr.exe 2>testlogerr.log

... however, the problem is - when I double-click this testlogerr.bat file in Windows Explorer:

• First one cmd.exe terminal window gets started, then it closes, and the testlogerr.exe gets started in another cmd.exe window
• The cmd.exe terminal window where testlogerr.exe runs, shows no cmd.exe prompt - instead, it shows stderr log messages; meaning it is running in foreground, not background
  • Another indication of foreground run, is that when I hit Ctrl-C, testlogerr.exe exits - and so does its terminal cmd.exe window
• The testlogerr.log file gets created, but its empty

So - how can I start the program as a background process, redirecting its stderr to file, obtain and print its pid, and finally show a cmd.exe terminal prompt (while the started process runs in the background) - all in a single cmd.exe terminal window?

Answer 1

In Linux shell, $! stores the last executed PID. Powershell can achieve the same using $process.id The current PowerShell code only exists with the PID though and is never displayed. Therefore change the code in the batch file to use Write-Host to display the PID (Similar to echo in bash):

@echo off
powershell -executionPolicy bypass -command "& {$process = start-process $args[0] -passthru -argumentlist $args[1..($args.length-1)]; Write-Host testlogerr started, its PID is $process.id}" testlogerr.exe 2>testlogerr.log
pause>nul

PS!! If you want both stdout and stderr in the log file, then change from 2>testlogerr.log to >testlogerr.log 2>&1

Answer 2

I think I finally got this - after a ton of failed attempts ...

First, let me note that I expected a "line buffered" redirection; however, Windows has no support for that, it either has "character buffered" I/O (serial ports), or it buffers everything (until process exits, then its output is flushed to file). However, the buffering happens at the output of the program - so to make sure my program is unbuffered, I made these changes (if you do not have control over the program you want to run in that way, see one of the referred links where they suggest using winpty for "unbuffer"ing) in testlogerr.c:

// based on https://stackoverflow.com/questions/26965508/infinite-while-loop-and-control-c
// can be compiled in MINGW64 with:
// gcc -g testlogerr.c -o testlogerr.exe

#include <unistd.h>
#include <stdio.h>
#include <signal.h>
#include <time.h>

volatile sig_atomic_t stop;

void inthand(int signum) {
  stop = 1;
}

int main(int argc, char **argv) {

  // disable output/line buffering:
  // https://stackoverflow.com/questions/8192318/why-does-delayed-expansion-fail-when-inside-a-piped-block-of-code#8194279
  // https://stackoverflow.com/questions/40487671/is-out-host-buffering
  // https://stackoverflow.com/questions/11516258/what-is-the-equivalent-of-unbuffer-program-on-windows
  // https://stackoverflow.com/questions/7876660/how-to-turn-off-buffering-of-stdout-in-c
  setvbuf(stdout, NULL, _IONBF, 0);
  setvbuf(stderr, NULL, _IONBF, 0);

  signal(SIGINT, inthand);
  int counter = 0;

  while(!stop) {
    fprintf( stderr, "%d: Logging line %d\n", (int)time(NULL), counter++);
    sleep(2);
  }

  printf("exiting safely\n");
  //system("pause"); // does "Press any key to continue . . ."; skip here
  return 0;
}

Right, so now that we have a program that writes "unbuffered" (i.e. "character buffered") to stderr, this is the batch file that works for me, testlogerr.bat - first the working portion, then for reference, everything else that did not work for me:

:: so, the only way to prevent Ctrl-C, AND run in the same cmd.exe window started by .bat, AND obtain the PID of the background process, is to use start /b - and then, retrieve the PID from the difference in started tasks ..
:: https://stackoverflow.com/questions/4677462/windows-batch-file-pid-of-last-process
@echo off
tasklist /FI "imagename eq testlogerr.exe" /NH /FO csv > task-before.txt
start /b testlogerr.exe > testlogerr.log 2>&1
tasklist /FI "imagename eq testlogerr.exe" /NH /FO csv > task-after.txt

for /f "delims=, tokens=2,*" %%A in ('"fc /L /LB1  task-before.txt task-after.txt | find /I "testlogerr.exe""') do set TESTPID=%%A
del task-before.txt
del task-after.txt
:: next command deletes double quotes (") from the string itself, so only the PID number remains: 
SET TESTPID=%TESTPID:"=%
:: now, print the PID we obtained:
echo TESTPID is %TESTPID%
:: note that at this point, the started terminal actually blocks!
:: so here we run cmd.exe one more time, so we get the command prompt shell
cmd
:: to exit this cmd shell, first you have to do `taskkill /F /PID %TESTPID%`, and only then `exit` will work
:: (otherwise it blocks) - or, just close via the X button at upper right corner of the cmd.exe window


REM Failed approaches below:


REM :: there is -RedirectStandardError for powershell Start-Process;
REM :: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/start-process?view=powershell-7.1
REM :: that means we do not have to start cmd so we have stream redirection; however,
REM :: -RedirectStandardError causes the reading via for..in..do to freeze (otherwise the below works)
REM @echo off
REM for /F "delims=" %%i IN ('powershell -command "$proc = Start-Process testlogerr.exe -RedirectStandardError testlogerr.log -NoNewWindow -passthru ; Write-Output $proc.id"') DO set i=%%i
REM echo i was %i%
REM pause


REM :: also "batch macro" https://stackoverflow.com/q/69539939/ does not work,
REM :: when RedirectStandardError is there (but works otherwise)
REM @echo off
REM call :initMacro
REM %$set% TESTPID="powershell -command "$proc = Start-Process testlogerr.exe 2>testlogerr.log -NoNewWindow -passthru ; Write-Output $proc.id"" 
REM echo TESTPID %TESTPID[0]%
REM ...


REM :: piping to SET https://stackoverflow.com/q/8192318 also does not work:
REM powershell -command "$proc = Start-Process testlogerr.exe -RedirectStandardError testlogerr.log -NoNewWindow -passthru ; Write-Output $proc.id" | set /p TESTPID=
REM echo TESTPID %TESTPID%
REM pause


REM :: must redirect to files, then https://stackoverflow.com/q/8192318 - this works:
REM :: openfiles /local on :: requires system reboot!
REM :: note: somehow test.pid here ends up being held/"used by" testlogerr.exe? yes, see https://superuser.com/q/986202 ; that means we cannot really delete it (ugh!)
REM @echo off
REM powershell -command "$proc = Start-Process testlogerr.exe -RedirectStandardError testlogerr.log -NoNewWindow -passthru ; Write-Output $proc.id" > test.pid
REM set /p TESTPID=<test.pid
REM :: openfiles /query /fo table | findstr test.pid :: access denied
REM :: del test.pid
REM echo TESTPID %TESTPID%
REM pause


REM :: like this, though, test.pid is not kept under ownership, so we can easily delete it
REM @echo off
REM powershell -command "$proc = Start-Process testlogerr.exe -RedirectStandardError testlogerr.log -NoNewWindow -passthru ; Write-Output $proc.id | Out-File -Encoding ASCII -FilePath .\test.pid" 
REM set /p TESTPID=<test.pid
REM del test.pid
REM echo TESTPID %TESTPID%
REM :: pause :: not needed anymore, no need for "Press any key to continue . . ."
REM :: note that at this point, the started terminal actually blocks!
REM :: so here we run cmd.exe one more time, so we get the command prompt shell
REM :: (however, even there, if we hit Ctrl-C, it will break our background program!)
REM cmd


REM :: as per https://superuser.com/q/1479119
REM :: like this, Ctrl-C does not kill the process by accident anymore;
REM :: however, the PID returned is for the cmd.exe, not the testlogerr.exe
REM @echo off
REM powershell -command "$proc = Start-Process -FilePath 'CMD.EXE' -ArgumentList '/C START /B testlogerr.exe' -RedirectStandardError testlogerr.log -NoNewWindow -passthru ; Write-Output $proc.id | Out-File -Encoding ASCII -FilePath .\test.pid" 
REM set /p TESTPID=<test.pid
REM del test.pid
REM echo TESTPID %TESTPID%
REM cmd


REM :: so, the only way to prevent Ctrl-C, AND run in the same cmd.exe window started by .bat, AND obtain the PID of the background process, is to use start /b - and then, retrieve the PID from the difference in started tasks ..
REM :: https://stackoverflow.com/questions/4677462/windows-batch-file-pid-of-last-process
REM ( ... here was the working code, which has now been moved at start/top of snippet)

So, what this .bat file now allows me, is that I can double click it, and I will get in the newly started cmd.exe terminal window:

TESTPID is 5532
Microsoft Windows [Version 10.0.19043.1266]
(c) Microsoft Corporation. All rights reserved.

C:\tmp>

So, the testlogerr.exe process started in the background - but if I hit Ctrl-C by accident in the newly started cmd.exe terminal, I will not shut testlogerr.exe down.

Furthermore, testlogerr.exe's log messages are piped to the testlogerr.log file (and I can confirm lines in that logfile appear one by one - if we had Linux tail, we could have seen realtime updates with tail -f testlogerr.log).

And finally, I get a cmd.exe shell prompt at end - which means, I can immediately inspect the situation:

C:\tmp>tasklist | findstr 5532
testlogerr.exe                5532 Console                    1      3,148 K

Great, that works!

Unfortunately, I thought this would allow me to start two such terminals, with two separate instances of the background process - unfortunately, if I try to do that, I get in the second terminal:

The process cannot access the file because it is being used by another process.
TESTPID is "=

But, at least I got the behavior that I wanted in the OP/question ...