2

So I have a batch file that is executing a simulation given some input parameters and then processing the output data via awk, R, and Python. At the moment the input parameters are passed into the simulation through some nested for loops and each iteration of the simulation will be run one after the other. I would like for the execution of the simulation to be done in parallel because at the moment there are 1,000+ cases so in my mind I could have core 1 handle sims 1-250, core 2 handle sims 251-500, etc.

In essence what I would like to do is this:

  1. Run every case of the simulation across multiple cores
  2. Once every simulation has been completed, start the output data processing

I've tried using start /affinity n simulation.exe, but the issue here is that all of the simulations will be executed simultaneously, so when it gets to the post processing calls, it errors out because the data hasn't been generated yet. There is the start /w command, but I'm not sure if that improve the simulation. One idea I've thought of is updating a variable once each simulation has been completed, then only start the post processing once the variable reaches n runs.

Here is an excerpt of what I am doing right now:

    for %%f in (1 2 3) do (
            for %%a in (4 5 6) do (
                for %%b in (7 8 9) do (
                    call :mission %%f %%a %%b
                )
            )
         )
    some gawk scripts
    some python scripts
    some r scripts
    go to :exit

:mission
   sed -e 's/text1/%1/' -e 's/text2/%2/' -e 's/text3/%3/'
   simulation.exe
   go to :exit

:exit

And here's what I was playing around with to test out some parallel processing:

start /affinity 1 C:\Users\614890\R-4.1.1\bin\Rscript.exe test1.R
start /affinity 2 C:\Users\614890\R-4.1.1\bin\Rscript.exe test2.R
start /affinity 3 C:\Users\614890\R-4.1.1\bin\Rscript.exe test3.R
start /affinity 4 C:\Users\614890\R-4.1.1\bin\Rscript.exe test4.R

C:\Users\614890\R-4.1.1\bin\Rscript.exe plotting.R
Wallabee
  • 31
  • 4
  • 1
    Please read about how SO operates. https://stackoverflow.com/help SO focuses on specific failures and eschews questions seeking opinion. – lit Sep 29 '21 at 18:53
  • I don't know if this can help you or not as idea [Multi-Threaded_Network_IP_Ports_Scanner.bat](https://pastebin.com/N0ir9w1c) – Hackoo Sep 29 '21 at 23:20
  • [Event-driven multi-thread scheme for Batch files](https://www.dostips.com/forum/viewtopic.php?f=3&t=6601&p=63390#p63390) – Aacini Sep 30 '21 at 12:08
  • Also see [Parallel execution of shell processes](https://stackoverflow.com/a/11715437/1012053) – dbenham Sep 30 '21 at 18:46
  • @dbenham: Your parallel execution control method still uses a loop and Batch commands to supervise the start and end of parallel processes... – Aacini Oct 04 '21 at 12:56
  • @Aacini - Yes it does - minimal loop CPU usage due to use of PING (or TIMEOUT), but usage none-the-less. Perfectly fine for jobs that take more than a few seconds. The pure pipe based solution is elegant, but I find it a bit harder to follow and setup. – dbenham Oct 04 '21 at 20:36

1 Answers1

0

I was actually able to accomplish this by doing the following:

setlocal
set "lock=%temp%\wait%random%.lock"

:: Launch processes asynchronously, with stream 9 redirected to a lock file.
:: The lock file will remain locked until the script ends
start /affinity 1 9>"%lock%1" Rscript test1.R
start /affinity 2 9>"%lock%2" Rscript test2.R
start /affinity 4 9>"%lock%3" Rscript test3.R
start /affinity 8 9>"%lock%4" Rscript test4.R

:Wait for all processes to finish
1>nul 2>nul ping /n 2 ::1
for %%F in ("%lock%*") do (
 (call ) 9>"%%F" || goto :Wait
) 2>nul

del "%lock%*"

Rscript plotting.R
Wallabee
  • 31
  • 4