Working around 32 window limit in MinGW (Windows batch files)

Question

I'm a computational biologist and I'm trying to run large batches of similar code with a single command, but my implementation has hit a brick wall.

I'm using the NEURON simulation environment, which uses MinGW for its Windows interface, which is where my research has shown my problem arises.

Currently, I am using a batch file to run all of these similar pieces of code, to iterate across the "collection" subfolders:

@echo off
for /D %%a in ("%cd%\all_cells\cell_*.*") do cd "%%a\sim1\" & START neuron sim.hoc

The problem arises when I have more than 32 subfolders; the additional instances won't run and will error with a "console device allocation failure: too many consoles" error.

My research has shown me that this is a known problem with Cygwin/MinGW.

However, working around this manually (ensuring that there is no more than 32 "collection" folders) is extremely time consuming when I am now dealing with hundreds of instances (each refers to a simulated cell and I want to gather statistics on hundreds of them), so I am trying to find a solution.

That said, I am terrible at writing batch files (I'm a terrible programmer who is used to scientific languages) and I can't figure out how to code around this.

It would be great if someone could help me either find a way around the 32 limit, or failing that, help me write a batch file that would do this:

-iterate over up to 32 folders -wait for the instances to finish -do it again for the next 32, until I reach the end of the folder.

I have tried using the /wait command to do them one at a time, but it still opens all 32. (And this wouldn't be ideal as I'd like to use all 16 cores I have.

Answer 1

The following is adapted from https://stackoverflow.com/a/11715437/1012053, which shows how to run any number of processes while limiting the total number run simultaneously in parallel. See that post for some explanation, though the code below is fairly well documented with comments.

I've eliminated the /O option and the code to work with PSEXEC from the original script.

The script below runs everything in one window - the output of each process is captured to a temporary lock file, and when finished, the full output of each process is typed to the screen, without any interleaving of process output. The start and end times of each process are also displayed. Of course you can redirect the output of the master script if you want to capture everything to a single file.

I've limited the total number of parallel processes to 16 - of course you can easily modify that limit.

The code will not work as written if any of your folder paths include the ! character. This could be fixed with a bit of extra code.

Other than that, the code should work, provided I haven't made any silly mistakes. I did not test this script, although the script it was derived from has been thoroughly tested.

@echo off
setlocal enableDelayedExpansion

:: Define the maximum number of parallel processes to run.
set "maxProc=16"

:: Get a unique base lock name for this particular instantiation.
:: Incorporate a timestamp from WMIC if possible, but don't fail if
:: WMIC not available. Also incorporate a random number.
  set "lock="
  for /f "skip=1 delims=-+ " %%T in ('2^>nul wmic os get localdatetime') do (
    set "lock=%%T"
    goto :break
  )
  :break
  set "lock=%temp%\lock%lock%_%random%_"

:: Initialize the counters
  set /a "startCount=0, endCount=0"

:: Clear any existing end flags
  for /l %%N in (1 1 %maxProc%) do set "endProc%%N="

:: Launch the commands in a loop
  set launch=1
  for /D %%A in ("%cd%\all_cells\cell_*.*") do (
    if !startCount! lss %maxProc% (
      set /a "startCount+=1, nextProc=startCount"
    ) else (
      call :wait
    )
    set "cmd!nextProc!=%%A"
    echo -------------------------------------------------------------------------------
    echo !time! - proc!nextProc!: starting %%A
    2>nul del %lock%!nextProc!

    cd "%%A\sim1\"

    %= Redirect the output to the lock file and execute the command. The CMD process =%
    %= will maintain an exclusive lock on the lock file until the process ends.      =%
    start /b "" cmd /c 1^>"%lock%!nextProc!" 2^>^&1 neuron sim.hoc
  )
  set "launch="

:wait
:: Wait for procs to finish in a loop
:: If still launching then return as soon as a proc ends
:: else wait for all procs to finish
  :: redirect stderr to null to suppress any error message if redirection
  :: within the loop fails.
  for /l %%N in (1 1 %startCount%) do 2>nul (
    %= Redirect an unused file handle to the lock file. If the process is    =%
    %= still running then redirection will fail and the IF body will not run =%
    if not defined endProc%%N if exist "%lock%%%N" 9>>"%lock%%%N" (
      %= Made it inside the IF body so the process must have finished =%
      echo ===============================================================================
      echo !time! - proc%%N: finished !cmd%%N!
      type "%lock%%%N"
      if defined launch (
        set nextProc=%%N
        exit /b
      )
      set /a "endCount+=1, endProc%%N=1"
    )
  )
  if %endCount% lss %startCount% (
    timeout 1 /nobreak >nul
    goto :wait
  )

2>nul del %lock%*
echo ===============================================================================
echo Thats all folks^^!

Answer 2

You could install screen or tmux in cygwin.

Then you can start all neuron instances in a screen/tmux session.
They will not open a new window, so there is no limit anymore.