Windows batch file: download many small files fast

Question

I have a list of urls (urls.txt):

https://example.com/1.webp
https://example.org/bar2.webp
... 10k more

Files vary in size from 1kb to 100kb.

How can I download these files quickly on a Windows 10 machine without installing any third-party software?

I need it to be in a single file that user can double-click without installing any additional software.

It should work on any decently up-to-date Windows 10 PC. AFAIK it means the PowerShell version is 5.1.

---

Additional information.

I tried this:

powershell -Command "Invoke-WebRequest https://example.com/1.webp -OutFile 1.webp"

but it extremely slow due to sequential execution.

So far this works in PowerShell fast enough:

Get-Content .\urls.txt |ForEach-Object {
    $FileName = Split-Path -leaf $_
    Invoke-WebRequest $_ -OutFile $FileName
}

But I can't figure out how to invoke this script with a double-click on a file.

Invoking .ps1 file from a .bat file doesn't work. Error:

download.ps1 cannot be loaded because running scripts is disabled on this system.

Asking user to adjust permissions is not an option.

This works in a clickable .bat file:

powershell -command ^
Invoke-WebRequest https://example.com/1.webp -OutFile 1.webp;

But this script fails silently:

powershell -command ^
Get-Content .\urls.txt |ForEach-Object { ^
    $FileName = Split-Path -leaf $_ ^
    Invoke-WebRequest $_ -OutFile $FileName ^
} ^

Answer 1

You could try the foreach-object -parallel method for this case, i tried something simular once with multiple process starts for robocopy to get like 1000 small files (5-10kb) on another harddrive.

I will look up if i can find it again.

Edit 1: you can go over like this for example.

$allmylinks = import-csv -path "path to your csv"
foreach -parallel($link in $allmylinks){
    Invoke-WebRequest $link
}

Answer 2

"...how do I iterate over a file lines with it? Sry, I never used Windows" (that must feel like me after a Linux machine).

Open a PowerShell prompt (Start → Run → PowerShell) or just type PowerShell.exe on the command prompt.

At the PowerShell prompt, to run the task in parallel using ForEach-Object -Parallel:

1..9 |ForEach-Object -Parallel { "Invoke-WebRequest https://example.com/$_.webp" -OutFile "$_.webp" }

Where "$_" is the current item (1to9`), you might also use a list here, like:

'One', 'Two', 'Three' |ForEach-Object -Parallel { ...

In case you "need to read it directly from the file", (presuming that you want use the name in the url as your filename) you might do something like this:

Get-Content .\urls.txt |ForEach-Object -Parallel {
    $FileName = Split-Path -leaf $_
    "Invoke-WebRequest $_ -OutFile $FileName
}

Update

(based on the additional information in your question and comments in this answer)

Final steps to making you command line easy to launch for novice user, taking in account that passing "complex" commands with special characters (as newlines, spaces and double quotes) from a batch file interpreter to PowerShell is quiet a hassle as there are a lot of exceptions on the exceptions. See: these stackoverflow questions

In your case it might be simply putting your commands in a single (quoted) command line and separate each syntax with a semi-colon (;):

powershell -command "Get-Content .\urls.txt |ForEach-Object { $FileName = Split-Path -leaf $_; Invoke-WebRequest $_ -OutFile $FileName }"

But to be on the safe side (in case e.g. a powershell command/parameter requires to be quoted by itself), I would rather supply robust solution which is encoding your command line to base64 and use the -EncodedCommand parameter. See also these answers:

• running powershell as shell command having error in StartTime variable for FilterHashtable
• Pass complex arguments to powershell script through encoded command

Encoding

To encode your command line to base64:

$Command = {
    Get-Content .\urls.txt |ForEach-Object {
        $FileName = Split-Path -leaf $_
        Invoke-WebRequest $_ -OutFile $FileName
    }
}.ToString()
$Bytes = [System.Text.Encoding]::Unicode.GetBytes($Command)
[Convert]::ToBase64String($Bytes)

Download.bat

Including the encoded command line in a (single) batch file add the following command in you batch file where the base64 string is copied from the above ToBase64String conversion:

PowerShell -EncodedCommand CgAgACAAIAAgACAAIAAgACAARwBlAHQALQBDAG8AbgB0AGUAbgB0ACAALgBcAHUAcgBsAHMALgB0AHgAdAAgAHwARgBvAHIARQBhAGMAaAAtAE8AYgBqAGUAYwB0ACAAewAKACAAIAAgACAAIAAgACAAIAAgACAAIAAgACQARgBpAGwAZQBOAGEAbQBlACAAPQAgAFMAcABsAGkAdAAtAFAAYQB0AGgAIAAtAGwAZQBhAGYAIAAkAF8ACgAgACAAIAAgACAAIAAgACAAIAAgACAAIABJAG4AdgBvAGsAZQAtAFcAZQBiAFIAZQBxAHUAZQBzAHQAIAAkAF8AIAAtAE8AdQB0AEYAaQBsAGUAIAAkAEYAaQBsAGUATgBhAG0AZQAKACAAIAAgACAAIAAgACAAIAB9AAoAIAAgACAAIAA=