Start multiple exe with arguments

Question

I have around 5000 folders each containing a dos executable and required files. Currently I am using a for loop to call the below code. it takes a long time to execute one by one as each execution takes around 5 seconds. Is there an option where I can execute all the exe files at the same time ?

Any ideas?

Thanks

I tried using

   start "" 1/ddd.exe input.dat 
   start "" 2/ddd.exe input.dat 
   start "" 3/ddd.exe input.dat 
   .
   .
   .

in a batch file. input.dat has the arguments to pass on to the exe. but the exe opens up a new window and its not taking the arguments. first argument is "2" run to certain part of the exe and second any number to exit the program after it has finished.

It's highly unlikely that you have enough memory to execute 5000 programs simultaneously, and the increase in CPU load and disk I/O would make the performance so bad that it would be unacceptable. And are you actually running *DOS*, which has been dead for decades now? DOS apps usually measure RAM in MB, not GB. — Ken White, Oct 25 '19 at 22:11
I note that you appear to be passing a file named `input.dat` to each file too. As you're not changing or setting current directories, it appears that you're also expecting 5000 executables to all work with the same file. Are you sure whether hitting the same file 5000 times is acceptable? — Compo, Oct 26 '19 at 01:20
[related](https://stackoverflow.com/questions/49549269/batch-processing-multiple-files-at-the-same-time/49551035#49551035) avoids memory and cpu overload. — Stephan, Oct 27 '19 at 09:33
The executable is really small but needs two input arguments. input.dat is for that. It requires first argument "2" so it starts a specific part of the program and any number to exit the program after. 10 batch files to do 500 each sounds like a good idea too. I just don't like 5~6 hours run time I have right now. — ash, Oct 28 '19 at 14:07
@Stephan Is there a way I can give arguments to my exe every time it executes start? — ash, Oct 28 '19 at 14:58
What should hinder you? (in the linked answer, the `REM` line actually uses arguments) — Stephan, Oct 28 '19 at 15:15
@Stephen When I use "start myprogram.exe " its not taking any arguments. I assumed it has something to do with how it opens up a separate command window every time. In the REM line you use cmd not start right? sorry if I am wrong. I am new to this. — ash, Oct 28 '19 at 15:26
I `start` a new `cmd /c` with a unique title ("MyCommand") (needed to check how many of them are already running; the `/c` makes sure, the command window stays open until the program is finished. You may want to `start /min ...` to hide them) with a program (`process.exe`) and (technically) three arguments (`"%%i"`, `-out` and `"C:\output\%%i"`) . — Stephan, Oct 28 '19 at 15:58
@Stephan This is what I ended up with : `for /D %%a in (*) do ( echo processing: %%a start /B "Name" cmd.exe "cd %%a & ddy.exe < parameters.txt" )` . I am not sure if its gonna be any faster than the one I was doing., — ash, Oct 28 '19 at 20:18
I would prefer `start /D "%%a" /min "Name" cmd.exe /c "ddy.exe < parameters.txt"`. `/B` causes them to use the same console and they may block each other. `/D` sets the working folder (no need for `cd`). And don't forget `/c` with the `cmd` command. — Stephan, Oct 28 '19 at 20:32

lit · Answer 1 · 2019-10-28T16:53:15.233

This is a -very- minimalistic script to run N commands at a time from a list. If you are on a supported Windows system, it will have PowerShell.

There is no error checking or proper help information. It writes stdout to the specified log file, but does nothing with the exit code from the command. If something fails, it would need to be identified from the log file.

To use this, put the following code into the file Invoke-JobList.ps1

Create a .csv file with the commands you want to run and a different log file name for each command. The log file name cannot be the same for multiple commands. If you have 5000 commands to process, you will probably need to write a script/program to produce it.

I provided a sample .csv file and a batch file that I used for testing. You do not need to use to.bat.

=== Get-Content .\Invoke-JobList.ps1

[CmdletBinding()]
Param (
    [Parameter(Mandatory=$true)]
    [string[]]$jobFile

    ,[Parameter(Mandatory=$false)]
    [int]$nConcurrent = 2
)

$jobs = Import-Csv -Path $jobFile
$jobHash = @{}
$nJobsRunning = 0

foreach ($job in $jobs) {
    if ($nJobsRunning -lt $nConcurrent) {
        Write-Verbose -Message "starting command $($job.command)"
        $j = Start-Job -ScriptBlock ([ScriptBlock]::Create($job.command))
        $jobHash[$j] = $job.logfile
        $nJobsRunning++
    }

    while ($nJobsRunning -ge $nConcurrent) {
        # wait for one or more jobs to state Completed
        $jobsRunning = Get-Job
        foreach ($jobRun in $jobsRunning) {
            if (($null -ne $jobHash[$jobRun]) -and ($jobRun.State -eq 'Completed')) {
                Receive-Job -Job $jobRun | Out-File -FilePath $jobHash[$jobRun]
                Remove-Job -Job $jobRun
                $jobHash.Remove($jobRun)
                $nJobsRunning--
            }
        }
    }
}

Write-Verbose -Message $($nJobsRunning.ToString() + " remaining jobs")

# Wait for all remaining jobs to complete
while ($nJobsRunning -gt 0) {
    $jobsRunning = Get-Job
    foreach ($jobRun in $jobsRunning) {
        if (($null -ne $jobHash[$jobRun]) -and ($jobRun.State -eq 'Completed')) {
            Receive-Job -Job $jobRun | Out-File -FilePath $jobHash[$jobRun]
            Remove-Job -Job $jobRun
            $jobHash.Remove($jobRun)
            $nJobsRunning--
        }
    }
}

=== Get-Content .\joblist3.csv

command,logfile
C:\src\jobs\to.bat 10,ss-001.txt
C:\src\jobs\to.bat 10,ss-002.txt
C:\src\jobs\to.bat 10,ss-003.txt
C:\src\jobs\to.bat 10,ss-004.txt
C:\src\jobs\to.bat 10,ss-005.txt
C:\src\jobs\to.bat 10,ss-006.txt
C:\src\jobs\to.bat 10,ss-007.txt

=== Get-Content .\to.bat

@ECHO OFF
SET "TO=%1"
IF "%TO%" == "" (SET "TO=5")
REM Cannot use TIMEOUT command
ping -n %TO% localhost
EXIT /B 0

Invoke it with parameters.

.\Invoke-JobList.ps1 -jobFile joblist3.csv -nConcurrent 3 -Verbose

Thanks for your answer. Sorry I am new to this. This looks very complicated to me :D — ash, Oct 28 '19 at 14:57
@ash - I am sorry it seems confusing. Save the `Invoke-JobList.ps1` file. I have added some usage directions to the answer. Does that help? — lit, Oct 28 '19 at 16:53
Thanks Lit. I think I got it to run with `for /D %%a in (*) do ( echo processing: %%a start /B "Name" cmd.exe "cd %%a & ddy.exe < parameters.txt" )` I am not sure if this is gonna be any faster. — ash, Oct 28 '19 at 20:21
@ash - How will you control how many run simultaneously? Can your machine run all 5,000 at the same time? — lit, Oct 28 '19 at 20:46
lit - I just started running it. It's using 70% CPU and 20% memory. — ash, Oct 28 '19 at 20:50
@ash - If you want to have it use more resources, increase the -nConcurrent parameter. Perhaps `-nConcurrent 5` or `-nConcurrent 10`. — lit, Oct 30 '19 at 22:04

score 0 · Accepted Answer · answered Oct 28 '19 at 20:36

You tried

for /D %%a in (*) do ( 
  echo processing: %%a 
  start /B "Name" cmd.exe "cd %%a & ddy.exe < parameters.txt"
)

I would prefer start /D "%%a" /min "Name" cmd.exe /c "ddy.exe < parameters.txt". /B causes them to use the same console and they may block each other. /D sets the working folder (no need for cd), /min minimizes the windows to keep your screen clean.
And don't forget /c with the cmd command (without, you get no parallel processes).

As a whole:

for /D %%a in (*) do ( 
  echo processing: %%a 
  start /D "%%a" /min "Name" cmd.exe /c "ddy.exe < parameters.txt"
)

Start multiple exe with arguments

2 Answers2