2

Edit: I'm trying to make a real time dashboard with information downloaded from this rest API. Once I have a solution it will be put into a batch file and deployed to non-technical users on lots of different machines, which is another reason I can't install software to make this work.

My current solution is to use a loop in the batch file. This completes in about 45 seconds - a long time for the user to wait.

I want Powershell to make all of these API requests at the same time (there are thousands of them, only a few KB is downloaded from each). I am using this answer, here: Run parallel Invoke-WebRequest jobs in PowerShell v3

The code below instead creates over a thousand jobs, which I think is one for every URL (the jobs also fail, but 1 thing at a time). How can I put all these commands into a job and get it to make all these requests at the same time, or at least 100 at a time or something? I am using Powershell 5.1.

$urlList = @(Import-CSV idscomma.txt)
for($f=0;$f -lt $urlList.Count;$f++)
{
    $remote = $urlList[$f]
    $local = 'partner' + $f
    Start-Job {Invoke-RestMethod $using:remote -Method Get -Headers @{ \"api-version\" = \"2\" ; \"Authorization\" = \"Bearer token\"} -OutFile $using:local}
}
Get-Job|Wait-Job

Edit: I also found this: Can Powershell Run Commands in Parallel? but not sure how to apply it to what I'm doing

Ne Mo
  • 198
  • 4
  • 18
  • 1
    You'll have to use the PowerShell SDK to use _thread_-based (rather than _child process_-based) concurrency - see https://serverfault.com/a/626712/176094. Note that PowerShell (Core) 7+ now comes with `Start-ThreadJob` and `ForEach-Object -Parallel`, which are much easier to use. – mklement0 Jul 15 '21 at 21:21
  • Ok, thanks. Sometimes Microsoft's documentation isn't great at explaining what is available to normal users and what isn't, or maybe I'm just not reading it carefully enough. I'm making an edit to explain the solution I have at the moment - just want something that works quicker. – Ne Mo Jul 15 '21 at 21:53

0 Answers0