3

In an attempt to utilize PowerShell to automate a process of pulling down files, doing something with them, and then copying them to somewhere else, I have most of the process working. My only issue I am encountering is I cannot get invoke webrequest to download multiple files.

# Specify variables

$SVN_BASE = "website ommitted" 
$SCCM_PATH = "path omitted"
$LOKI_PATH = "path omitted"
$INSTALLER_NAME = "Firefox Setup 58.0.1.exe"
$PROJECT_FOLDER = "mozilla_firefox_rr"

# Set an alias for the executable 7zip to be called to extract files

set-alias 7z "$env:ProgramFiles\7-Zip\7z.exe"

# Create the working directory for the application

new-item -path "$($env:userprofile)\Desktop" -name $PROJECT_FOLDER -itemtype 
directory -Force

# Change the directory to the working directory 

set-location "$($env:userprofile)\Desktop\$PROJECT_FOLDER"

# Invoke-WebRequest is aka wget. Here, we are downloading the required file
# and placing it into our working directory

Invoke-WebRequest "$SVN_BASE" -outfile ".\"
Invoke-WebRequest "$LOKI_PATH/$INSTALLER_NAME" -outfile 
"$($env:userprofile)\Desktop\$PROJECT_FOLDER\$INSTALLER_NAME"

# Extract contents of executable

7z x Firefox*.exe

# Remove contents that aren't needed

Remove-item .\$INSTALLER_NAME
Remove-item "$SCCM_PATH\core" -recurse
Remove-item "$SCCM_PATH\setup.exe" -recurse

# The final step is copying the newly extracted files to the corresponding SCCM directory

copy-item ".\*" -Destination $SCCM_PATH -recurse

The line that I am hoping to utilize to do this is

Invoke-WebRequest "$SVN_BASE" -outfile ".\"

Any suggestions?

Thanks

xerxes2985
  • 35
  • 1
  • 2
  • 7
  • When you say download multiple files; can you run the `Invoke-WebRequest` multiple times (i.e. once per file's URL); or are you looking to pull back all child items under some remote directory / all linked items from a given page / something like that? – JohnLBevan Feb 02 '18 at 21:04
  • @JohnLBevan I am looking to pull back the child items under the remote directory. I assumed I can point it to each file, but that would be cumbersome when I have a directory with 50 files for example. – xerxes2985 Feb 02 '18 at 21:08
  • I don't believe that's possible in a single request; my understanding of HTTP is that each resource needs its own request (this is likely why many people use sprites when there are multiple images on a site, rather than the browser simply downloading the collection of all image files in one request). – JohnLBevan Feb 02 '18 at 21:11
  • You could pull back a list of links from the remote directory, then pull back each item... That solution's given here (i.e. first part of the answer): https://stackoverflow.com/a/27945081/361842 – JohnLBevan Feb 02 '18 at 21:12
  • ps. you can avoid using 7zip and instead use PowerShell (v5 onwards) or .net (pre PS v5) to do the compression. 7-Zip does give better compression; but if you want to avoid any effort working out the relevant path, or having any dependencies outside your control, this option may be better. https://stackoverflow.com/a/40347498/361842 – JohnLBevan Feb 02 '18 at 21:15
  • @JohnLBevan the only purpose for 7zip is to extract the contents of the installer. – xerxes2985 Feb 02 '18 at 21:56
  • @JohnLBevan I forgot one key important part of my script. The $SVN_Base variable is tied to a subversion instance. – xerxes2985 Feb 02 '18 at 21:58

2 Answers2

5

Invoke-WebRequest performs HTTP operation with Powershell.

Invoke-WebRequest can perform all HTTP methods. You can accomplish every of them using Method parameter (most popular ones are: GET, POST, PUT, DELETE).

In HTTP protocol there is no option to download all files under particular link. If you want to do it you need to 'crawl' through the page. First list the content of the given link and after some kind of parsing website perform foreach loop through links to download each file.

enter image description here

Pawel Wujczyk
  • 422
  • 3
  • 12
0

Use invoke-webrequest to find links. You can utilize regex to pull them out. Then you can use start-bitstransfer to download each find.

Robert Cotterman
  • 2,213
  • 2
  • 10
  • 19