Download multiple large files asynchronously with Powershell

Question

I have four large OS installation media I need to download. This will take a long time if I wait for each download to finish before moving to the next one. Before downloading, I want to check if the media is already present.

The solution is likely a combination of a hash table, test-path and invoke-webrequest but I just can't crack it.

So in pseudo something on the lines of:

Check if file1 exists
if true then download and check file2
if false check file 2
check if file 2 exists...

So check if files exists and if not, start downloading all the ones that are missing.

I'm not very experienced with PS so all help is much appreciated, thank you very much! Researching the answer was fun but I feel I'm missing a keyword here...

This answer might be useful https://stackoverflow.com/questions/4016451/can-powershell-run-commands-in-parallel. The typical solution is to map over the files, creating a thread for each one (that checks and downloads as necessary) and then maps over the threads, joining on each one. — F. P. Freely, Mar 15 '19 at 13:42
The best method is working with runspaces, but this is nothing for beginners. https://devblogs.microsoft.com/scripting/beginning-use-of-powershell-runspaces-part-1/ — f6a4, Mar 15 '19 at 14:43

score 7 · Accepted Answer · answered Mar 15 '19 at 16:12

There is a fairly simple way for async downloads using WebClient class, although it's probably not available on older version of PS. See the example below

$files = @(
 @{url = "https://github.com/Microsoft/TypeScript/archive/master.zip"; path = "C:\temp\TS.master.zip"}
 @{url = "https://github.com/Microsoft/calculator/archive/master.zip"; path = "C:\temp\calc.master.zip"}
 @{url="https://github.com/Microsoft/vscode/archive/master.zip"; path = "C:\temp\Vscode.master.zip"}
)

$workers = foreach ($f in $files) { 

$wc = New-Object System.Net.WebClient

Write-Output $wc.DownloadFileTaskAsync($f.url, $f.path)

}

# wait until all files are downloaded
# $workers.Result

# or just check the status and then do something else
$workers | select IsCompleted, Status

This is perfect. Thank you very much for your troubles! – user2749942 Mar 16 '19 at 17:22 — user2749942, Mar 16 '19 at 17:22

Garric · Answer 2 · 2023-03-07T00:59:04.730

Based on code in https://blog.ironmansoftware.com/powershell-async-method/

[void][Reflection.Assembly]::LoadWithPartialName("System.Threading")
function Wait-Task {
    param([Parameter(Mandatory, ValueFromPipeline)][System.Threading.Tasks.Task[]]$Task)
    Begin {$Tasks = @()}
    Process {$Tasks += $Task}
    End {While(-not [System.Threading.Tasks.Task]::WaitAll($Tasks, 200)){};$Tasks.ForEach({$_.GetAwaiter().GetResult()})}
}
Set-Alias -Name await -Value Wait-Task -Force
@(
    (New-Object System.Net.WebClient).DownloadFileTaskAsync("https://github.com/Microsoft/TypeScript/archive/master.zip","$env:TEMP\TS.master.zip")
    (New-Object System.Net.WebClient).DownloadFileTaskAsync("https://github.com/Microsoft/calculator/archive/master.zip","$env:TEMP\calc.master.zip")
    (New-Object System.Net.WebClient).DownloadFileTaskAsync("https://github.com/Microsoft/vscode/archive/master.zip","$env:TEMP\Vscode.master.zip")
) | await

Download multiple large files asynchronously with Powershell

2 Answers2