1

I have a set of activities that should be run in parallel over a set of sessions.

Each session will run in parallel all tasks and next session will not be run until all tasks in parallel are finished.

run_all_sessions.bat -> each line should be run, one at a time, and next line should not be launched until the previous one has ended all tasks inside run_batches_1_to_4_in_parallel.bat:

start /wait run_batches_1_to_4_in_parallel.bat session_id=1
start /wait run_batches_1_to_4_in_parallel.bat session_id=2
start /wait run_batches_1_to_4_in_parallel.bat session_id=3
start /wait run_batches_1_to_4_in_parallel.bat session_id=4

Were run_batches_1_to_4_in_parallel.bat is a batch file that contains the following information:

start "running_batch_1" batch1.bat %1
start "running_batch_2" batch2.bat %1
start "running_batch_3" batch3.bat %1
start "running_batch_4" batch4.bat %1
exit

The problem I have is that as there is no /wait inside run_batches_1_to_4_in_parallel.bat and then the next line in run_all_sessions.bat is run, clogging the machine as the execution of batch*.bat takes to much resources.

The solution I want to find is to modify run_batches_1_to_4_in_parallel.bat in the sense that I do not return control to run_all_sessions.bat until all batch*.bat in run_batches_1_to_4_in_parallel.bat have finished and returned their termination code. It should be noted that:

  • each batch*.bat returns a lot of stdout and stderr messages
  • each batch*.bat has a running time that is different and I have to wait until all of them are completed.
  • No pause can be added in run_batches_1_to_4_in_parallel.bat, as the run_all_sessions.bat script should run completely without user interaction.
  • an asynchronous solution is prefered over a synchronous solution that regularly checks progress, as performance is critical in the machine.

Answer https://stackoverflow.com/a/33586872/4919040 proposes to use set /P "=" but it is expected that all batch*.bat return a lot of output in stdout, so this approach is not feasible.

Answer https://stackoverflow.com/a/33585114/4919040 proposes to check tasklist and filtering but I neither want to filter by process name nor want to be regularly polling and checking whether the processes have finished.

Answer https://stackoverflow.com/a/49051876/4919040 is not useful as the pause command requires user interaction.

I have been suggested to use the following solution for run_batches_1_to_4_in_parallel.bat but I fear it does not assure that returns controls after all the batch*.bat have been finished.

start "running_batch_1" batch1.bat %1
start "running_batch_2" batch2.bat %1
start "running_batch_3" batch3.bat %1
start "running_batch_4" /wait batch4.bat %1
exit
aturegano
  • 936
  • 15
  • 30
  • `start` starts the script/program in its own independent process. The usual way for a batch script would be to `call` it, not to `start` it. `Call` will return, when the called script ends. `Start` is usually used when the secondary script *should* run in the background without interrupting the first script. – Stephan Aug 10 '23 at 08:28
  • @Stephan in `run_all_sessions.bat` the `start /wait` can be replaced by `call`, but it does not solve the problem I have. – aturegano Aug 10 '23 at 08:52
  • Maybe `powershell` should be considered in replacement of `cmd.exe`. See for instance the usage of `Start-Job`, `Get-Job` and `Wait-Job`. See also solution proposed in https://stackoverflow.com/a/71366412/4919040 – aturegano Aug 10 '23 at 10:36
  • right, but doing it in batch*.bat would. If you can't change those files for any reason, you will have to watch their execution "from outside". This results in the `tasklist` approach, which you don't want to use. – Stephan Aug 10 '23 at 11:20

2 Answers2

0

I don't think there is an OOTB solution available for that. You could try simulating a semaphore by using simple file operations.

Each sub-batch needs to create a batch-specific signal-file in a defined "semaphore"-folder when started and will delete this signal-file when finished.

Your main-batch needs to run an infinite loop after having fired the sub-batches and check if the semaphore-folder is empty. If there is at least one file in the semaphore-folder this is an indication that there is still a sub-batch active and you need to sleep and re-check after a few seconds. If the semaphore-folder is empty you may assume all sub-batches have finished and you can continue.

Halfix
  • 81
  • 1
  • 1
  • 4
  • Maybe there is no OOTB solution with `cmd.exe`. Do you know whether a solution is possible, maybe using `powershell` instead? – aturegano Aug 10 '23 at 10:28
0

First approach, use files to indicate that processing is not finished, one file for each batchfile.

if exist "batch?.bat.lock" del "batch?.bat.lock"

for /L %%N in (1,1,4) do (
    echo . > "batch%%N.LOCK"
    start "running_batch_%%N" batch%%N.bat %1
)

:Wait
if exist "batch?.bat.lock" (
   timeout /T 5
   goto :Wait
)
echo All processes are finished
EXIT /B

Change the 4 batchN.bat files to delete the corresponding lock file when it is done processing its data and exits.

An other approach without using temporary files is using the WaitFor command to send/receive signals. The batchN.bat can send a signal when finished processing for example: waitfor /SI exitN The calling batchfile uses: waitfor exitN N with values 1 - 4. With just one batchfile this would be a clean and fast solution.

But with four batchfile it gets a bit more complicated. Now the caller needs to check/wait for four different signals but waitfor can only wait for one signal at a time.
It took a bit of experimenting but this can be done and the code is not as ugly as I first imagined it would be!

Minimal code:

for /L %%N in (1,1,4) do waitfor /SI exit%%N
for /L %%N in (1,1,4) do start "running_batch_%%N" batch%%N.bat %1
for /f "tokens=*" %%A in ('start "1" /min /b waitfor exit1 ^& start "2" /min /b waitfor exit2 ^& start "3" /min /b waitfor exit3 ^& start "4" /min /b waitfor exit4') do rem
exit /b

Code with some documentation and a bit more informative

rem send signals to avoid waitfor conflicts
for /L %%N in (1,1,4) do waitfor /SI exit%%N
rem start the 4 batchfiles
for /L %%N in (1,1,4) do start "running_batch_%%N" batch%%N.bat %1
rem start the 4 waitfor signal
for /f "tokens=*" %%A in ('start "1" /min /b waitfor exit1 ^& start "2" /min /b waitfor exit2 ^& start "3" /min /b waitfor exit3 ^& start "4" /min /b waitfor exit4') do echo %%A

rem You can check from another command prompt that there are 4 waitfor running with: tasklist /fi "imagename eq waitfor.exe"
rem when the 4 batchfiles are finished each will have send its signal and the for-loop above will finish and report 4 lines: "SUCCESS: Signal received."

exit /b
OJBakker
  • 602
  • 1
  • 5
  • 6