Waiting for eight parallel batch scripts

Question

I have like 0 experience with coding batch files and I'm actually refering to an older post "Waiting for parallel batch scripts", and to the answer of dbenham.

@echo off
setlocal
set "lock=%temp%\wait%random%.lock"

:: Launch one and two asynchronously, with stream 9 redirected to a lock file.
:: The lock file will remain locked until the script ends.
start "" cmd /c 9>"%lock%1" one.bat
start "" cmd /c 9>"%lock%2" two.bat

:Wait for both scripts to finish (wait until lock files are no longer locked)
1>nul 2>nul ping /n 2 ::1
for %%N in (1 2) do (
  ( rem
  ) 9>"%lock%%%N" || goto :Wait
) 2>nul

::delete the lock files
del "%lock%*"

:: Launch three and four asynchronously
start "" cmd /c three.bat
start "" cmd /c four.bat

His answer works fine, but I would like to adapt his solution to run not 2 files simultaneously but 8, and after that another 8 and so on...

Can someone help me?

What I tried so far is this (for 3 blocks of 8 batch files each)

@echo off
setlocal
set "lock=%temp%\wait%random%.lock"

:: Launch 8 files asynchronously, with stream 9 redirected to a lock file.
:: The lock file will remain locked until the script ends.
start "" cmd /c 9>"%lock%1" one.bat
start "" cmd /c 9>"%lock%2" two.bat
start "" cmd /c 9>"%lock%3" three.bat
start "" cmd /c 9>"%lock%4" four.bat
start "" cmd /c 9>"%lock%5" five.bat
start "" cmd /c 9>"%lock%6" six.bat
start "" cmd /c 9>"%lock%7" seven.bat
start "" cmd /c 9>"%lock%8" eight.bat

:Wait for all scripts to finish (wait until lock files are no longer locked)
1>nul 2>nul ping /n 2 ::1
for %%N in (1 2 3 4 5 6 7 8) do (
  ( rem
  ) 9>"%lock%%%N" || goto :Wait
) 2>nul

::delete the lock files
del "%lock%*"

:: Launch 8 files asynchronously, with stream 9 redirected to a lock file.
:: The lock file will remain locked until the script ends.
start "" cmd /c 9>"%lock%1" nine.bat
start "" cmd /c 9>"%lock%2" ten.bat
start "" cmd /c 9>"%lock%3" eleven.bat
start "" cmd /c 9>"%lock%4" twelve.bat
start "" cmd /c 9>"%lock%5" thirteen.bat
start "" cmd /c 9>"%lock%6" fourteen.bat
start "" cmd /c 9>"%lock%7" fifteen.bat
start "" cmd /c 9>"%lock%8" sixteen.bat

:Wait for all scripts to finish (wait until lock files are no longer locked)
1>nul 2>nul ping /n 2 ::1
for %%N in (1 2 3 4 5 6 7 8) do (
  ( rem
  ) 9>"%lock%%%N" || goto :Wait
) 2>nul

::delete the lock files
del "%lock%*"

:: Launch three and four asynchronously
start "" cmd /c seventeen.bat
start "" cmd /c eighteen.bat
start "" cmd /c nineteen.bat
start "" cmd /c twenty.bat
start "" cmd /c twenty-one.bat
start "" cmd /c twenty-two.bat
start "" cmd /c twenty-three.bat
start "" cmd /c twenty-four.bat

But it doesn't seem to work right. Normaly a block of eight batch files will be completed after max. 3h but I waited almost 24h and he seems to be stuck in the first block...

Critical Info: When a batch file (or Job) is executed a "cmd" window opens for like 5 seconds, closes and an other programm starts which is doing the actual job of calculations for max. 3h. I looked into the Task Manager and what happens is, 3 processes are started. "standard.exe" and "eliT_DriverLM.exe" after like 20 sec and "python.exe" after 5 sec. Since the process is finished when this 3 processes are no longer running (not showing up in the Task Manager anymore), the programm should wait for the for at least the "standard.exe" to terminate.

Well, what have you tried so far to achieve your goal? please [edit] your question and provide a [mcve] of your attempts! — aschipfl, Jul 23 '19 at 19:23
There are *two* asynchronous calls `start "" cmd /c 9>"%lock%1" one.bat` and `start "" cmd /c 9>"%lock%2" two.bat`; simply write *eight* lines of this syntax pattern. Moreover, there is a code snippet of _wait until **two** lock files are no longer locked_. To wait until **eight** lock files are no longer locked, use `for %%N in (1 2 3 4 5 6 7 8) do (`… — JosefZ, Jul 23 '19 at 19:30
[Related](https://stackoverflow.com/a/44697124/2152082) and a [possible duplicate](https://stackoverflow.com/a/47657549/2152082) using another method. — Stephan, Jul 23 '19 at 20:12

Magoo · Accepted Answer · 2019-07-24T21:42:40.183

@ECHO OFF
SETLOCAL
SET "sourcedir=U:\sourcedir"
SET "filename1=%sourcedir%\q57071663.txt"
SET "lockfilename=%temp%\LOCK#_"

:: BLOCKSIZE - if positive, run in blocks of this size.
:: If negative, run max this number of parallel jobs

SET /a blocksize=-5

FOR /f "usebackqtokens=1*delims=|" %%a IN ("%filename1%") DO (
 IF "%%b"=="" (SET "jobname="&SET "jobinstr=%%a") ELSE (SET "jobname=%%a"&SET "jobinstr=%%b")
  CALL :sub
)
GOTO :EOF

:sub
SET /a absblocksize=blocksize
IF %blocksize% gtr 0 GOTO blockmode
SET /a absblocksize=-blocksize
:blockmode
FOR /L %%c IN (1,1,%absblocksize%) DO IF NOT EXIST "%lockfilename%_%%c" SET "lock=%lockfilename%_%%c"&GOTO release
:: No vacant spots, so block complete. Wait for ALL to finish
:blockwait
CALL :wait1
IF %blocksize% gtr 0 IF EXIST "%lockfilename%_*" (GOTO blockwait) ELSE (GOTO blockmode)
GOTO blockmode


:release
ECHO.>"%lock%"
START "%jobname%" CMD /c "call %jobinstr%&DEL "%lock%""

GOTO :eof

:wait1
timeout /t 1 >NUL 2>NUL
GOTO :eof

You would need to change the setting of sourcedir to suit your circumstances. The listing uses a setting that suits my system.

I used a file named q57071663.txt containing some dummy data for my testing.

%lockfile% is used temporarily and is a filename of your choosing.

The usebackq option is only required because I chose to add quotes around the source filename.

The contents of q57071663.txt are, for my demo,

Job 1 | q57071663_sub.bat 12
Job 2 | q57071663_sub.bat 4
Job 3 | q57071663_sub.bat 15
Job 4 | q57071663_sub.bat 2
Job 5 | q57071663_sub.bat
Job 6 | q57071663_sub.bat
Job 7 | q57071663_sub.bat
Job 8 | q57071663_sub.bat
Job 9 | q57071663_sub.bat 6
Job11 | q57071663_sub.bat 12
Job12 | q57071663_sub.bat 4
Job13 | q57071663_sub.bat 15
Job14 | q57071663_sub.bat 2
Job15 | q57071663_sub.bat
Job16 | q57071663_sub.bat
Job17 | q57071663_sub.bat
Job18 | q57071663_sub.bat
Job19 | q57071663_sub.bat 6

Each line is an optional job name, a separator (I chose |, obviously) and the command to be executed by start...

and q57071663_sub.bat is simply

@ECHO OFF
SETLOCAL
SET "delay=%1"
:: IF no delay defined, random 3..19 secs
IF NOT DEFINED delay SET /a delay=3+(%RANDOM% %% 17)
timeout /t %delay% >NUL 2>NUL

ie. delay for (argument) seconds, or 3 to 19 if no argument provided.

As for the main routine, the for/f splits each line from the file using the | separator, and assigns the two parts to jobname and jobinstr, then executes the subroutine :sub.

The subroutine calculates the absolute value of blocksize and looks for a missing lockfile. If it finds a lockfile missing in the range, lock gets the lock filename and release sets a lockfile and starts the job, appending an instruction to delete the lockfile when the job terminates.

if there are no vacant slots available, we wait. If blocksize is greater than 0, then we may proceed to the next block if there are no remaining lockfiles, otherwise we wait. If blocksize is negative, then we simply try again.

So - if blocksize is positive, this would run all of the jobs in the file n at a time, waiting until each block of n is finished before starting the next block. If blocksize is negative, then we run n jobs in parallel, starting the next job in the list when any job terminates. This runs n jobs at all times.

Complete revision in the light of comments and critical information now included in question.

It would appear that the actual requirement is that at most 8 instances of standard.exe are running in parallel.

@ECHO OFF
SETLOCAL
SET "sourcedir=U:\sourcedir"
SET "filename1=%sourcedir%\q57071663_c.txt"

SET /a blocksize=5

FOR /f "usebackqdelims=" %%a IN ("%filename1%") DO (
 SET "jobinstr=%%a"
 CALL :sub
)
GOTO :EOF

:sub
:: Wait 30 secs (29 + 1) to ensure last-started job is  active
timeout /t 5 >NUL 2>NUL
:delay1
timeout /t 1 >NUL 2>NUL

:: Count #times critical .exe is in tasklist

SET /a active=0
:: FOR /f %%c IN ('tasklist^|findstr /b "standard.exe"^|find /c "standard.exe"' ) DO SET active=%%c
FOR /f %%c IN ('tasklist/v^|findstr /b "cmd.exe"^|find /c "q57071663"' ) DO SET active=%%c
ECHO %active% instances active
IF %active% geq %blocksize% GOTO delay1

START "%jobinstr%" CMD /c "call %jobinstr%"

GOTO :eof

You would need to change the setting of sourcedir to suit your circumstances. The listing uses a setting that suits my system.

I used a file named q57071663_c.txt containing the same job-data as above, except that the job-name and delimiter are removed.

I used a blocksize of 5 for testing. In OP's application, substitute 8

So - read each line of job-instructions to %%a and thence to jobinstr and call the subroutine SUB.

First thing to do in sub is to wait 30 secs (I used 6 sec for my testing) - this is just to ensure that the last-started job has created its instance of standard.exe

Then count the active standard.exe instances. I've commented-out the line for standard.exe and left in my test command, which requires both cmd.exe and q57071663 to be on the same tasklist line to indicate an instance of interest.

If there are the required number of instances already active, delay 1 second and try again.

Otherwise, start the required job.

Okay I think I forgot to mention something. When a batch file (or Job) is executed a "cmd" window opens for like 5 seconds, closes and an other programm starts which is doing the actual job of calculations for max. 3h. I looked into the Task Manager and what happens is, 3 processes are started. "standard.exe" and "eliT_DriverLM.exe" after like 20 sec and "python.exe" after 5 sec. Since the process is finished when this 3 processes are no longer running, is there a way to change your code so it checks after some delay time for lets say "python.exe"? — Kuba, Jul 24 '19 at 09:51
@Kuba: that's critical info, that should be in the question itself. — Stephan, Jul 24 '19 at 16:14
The crucial issue is detecting the termination of the process. Is there a way of definitively identifying the three subsidiary processes relative to the batch that started them? Does the `python` process always terminate as the last of the three? — Magoo, Jul 24 '19 at 19:48
@Stephan your absolutely right I edited my question, thanks for mentioning it. — Kuba, Jul 24 '19 at 20:08
@Magoo As far as I know they all terminate at the same time. The thing is that the "python.exe" starts first (after like 5 seconds) but the "standard.exe" (starts after like 20 seconds) is the actual process with the most RAM usage. And as far as I know the "standard.exe" is the actual programm runing the calculations. So I would want the programm to run only eight "standard.exe" at the same time since the computer cant handle more. — Kuba, Jul 24 '19 at 20:08
@Magoo Thank you, it works like a charm. You Sir are a real life-saver! — Kuba, Jul 26 '19 at 19:04

Waiting for eight parallel batch scripts

1 Answers1