Passing an array of URLs as an argument to Powershell

Question

I am trying to write a script that will take a text file containing URL links to documents and download them. I am having a hard time understanding how to pass the arguments and manipulate them in powershell. Here is what I got so far. I think I should be using the param method of taking an argument so I can require it for the script, but $args seemed easier on face value... A little help would be much appreciated. **UPDATE

$script = ($MyInvocation.MyCommand.Name)
$scriptName = ($MyInvocation.MyCommand.Name -replace "(.ps1)" , "")
$scriptPath = ($MyInvocation.MyCommand.Definition)
$scriptDirectory = ($scriptPath.Replace("$script" , ""))

## ##################################
## begin code for directory creation.
## ##################################

## creates a direcory based on the name of the script.
do {
    $scriptFolderTestPath = Test-Path $scriptDirectory\$scriptName -PathType container
    $scriptDocumentFolderTestPath = Test-Path $scriptFolder\$scriptName"_Script_Documents" -PathType container
    $scriptLogFolderTestPath = Test-Path $scriptFolder\$scriptName"_Script_Logs" -PathType container

    if ($scriptFolderTestPath -match "False") { 
        $scriptFolder = New-Item $scriptDirectory\$scriptName -ItemType directory
    }
    elseif ($scriptDocumentFolderTestPath -match "False") {
        New-Item $scriptFolder\$scriptName"_Script_Documents" -ItemType directory       
    }
    elseif ($scriptLogFolderTestPath -match "False") {
        New-Item $scriptFolder\$scriptName"_Script_Logs" -ItemType directory
    }
} Until (($scriptFolderTestPath -match "True") -and ($scriptDocumentFolderTestPath -match "True") -and ($scriptLogFolderTestPath -match "True"))

## variables for downloading and renaming code.
$date = (Get-Date -Format yyyy-MM-dd)

## ################################
## begin code for link downloading.
## ################################

## gets contents of the arguement variable.
Get-Content $linkList

## downloads the linked file.
Invoke-WebRequest $linkList

Resulting Errors

PS C:\Windows\system32> C:\Users\Steve\Desktop\Website_Download.ps1
cmdlet Website_Download.ps1 at command pipeline position 1
Supply values for the following parameters:
linkList: C:\Users\Steve\Desktop\linkList.txt


    Directory: C:\Users\Steve\Desktop\Website_Download


Mode                LastWriteTime     Length Name                                                                                     
----                -------------     ------ ----                                                                                     
d----        10/27/2012   3:59 PM            Website_Download_Script_Documents                                                        
d----        10/27/2012   3:59 PM            Website_Download_Script_Logs                                                             
Get-Content : Cannot find path 'C:\Users\Steve\Desktop\linkList.txt' because it does not exist.
At C:\Users\Steve\Desktop\Website_Download.ps1:42 char:1
+ Get-Content $linkList
+ ~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\Users\Steve\Desktop\linkList.txt:String) [Get-Content], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetContentCommand

Invoke-WebRequest : Could not find file 'C:\Users\Steve\Desktop\linkList.txt'.
At C:\Users\Steve\Desktop\Website_Download.ps1:45 char:1
+ Invoke-WebRequest $linkList
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.FileWebRequest:FileWebRequest) [Invoke-WebRequest], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

score 3 · Answer 1 · edited May 23 '17 at 11:48

There is no difference in passing an array of arguments in Powershell, compared to any other type of arguments. See here for how it's done. Considering you have a text file, you don't need to pass an array of arguments, and only need to pass a file name, so just a string.

I don't have any experience with Powershell 3.0, which is what you are using (judging by the presence of Invoke-WebRequest in your code), but I would start with something like this:

$URLFile = @"
http://www.google.ca
http://www.google.com
http://www.google.co.uk/
"@

$URLs = $URLFile -split "`n";
$savedPages = @();
foreach ($url in $URLs) {
    $savedPages += Invoke-WebRequest $url   
}

That is, you have a single file, all in one place, and make sure you receive your content correctly. Not sure why you would need Start-BitsTransfer, since Invoke-WebRequest will already get you page contents. Note that I did not do anything with $savedPages, so my code is effectively useless.

After that, contents of $URLFile goes into a file and you replace a call to it with

gc "Path_To_Your_File"`

If still working, introduce a $Path parameter to your script like this:

param([string]$Path)

test again, and so on. If you are new to Powershell, always start with smaller code pieces and keep growing to include all functionality you need. If you start with a big piece, chances are you will never finish.

Thanks for the reply. This isn't making sense to me though. Why can't I just pass a txt file with URLs as an argument and use the get-content to have each line as an object and pass them to the Invoke-Webrequest cmdlet? I am updating my main post with a param piece I added and the resulting errors. — Steve, Oct 27 '12 at 20:00
@Steve: I am suggesting you a starting point. Forget about external files, forget about parameters. Make sure your single file code works as expected. Yes, you would then pass a txt file with URLs as a parameter, but you don't need to do get-content on URLs. Assuming each line in the file is a URL, you would split by newline and feed into Invoke-Webrequest, which is exactly what I did in my example. — Victor Zakharov, Oct 27 '12 at 20:09
I don't like this starting point because it's more like a perl script. I shouldn't be parsing the txt file. If I wanted to do this manually it would look like this. '$links = "C:\Users\Steve\Desktop\linkList.txt" $results = "C:\Users\Steve\Desktop\Results" $crawl = get-content $links Invoke-WebRequest $crawl' — Steve, Oct 27 '12 at 20:56

score 1 · Answer 2 · answered Oct 27 '12 at 21:07

Figured this out with the link from Neolisk about handling params. Then changed some code at the end to create another variable and handle things as I normally would. Just some confusion with passing params.

## parameter passed to the script.
param (
    [parameter(Position=0 , Mandatory=$true)]
    [string]$linkList
)

## variables for dynamic naming.
$script = ($MyInvocation.MyCommand.Name)
$scriptName = ($MyInvocation.MyCommand.Name -replace "(.ps1)" , "")
$scriptPath = ($MyInvocation.MyCommand.Definition)
$scriptDirectory = ($scriptPath.Replace("$script" , ""))

## ##################################
## begin code for directory creation.
## ##################################

## creates a direcory based on the name of the script.
do {
    $scriptFolderTestPath = Test-Path $scriptDirectory\$scriptName -PathType container
    $scriptDocumentFolderTestPath = Test-Path $scriptFolder\$scriptName"_Script_Documents" -PathType container
    $scriptLogFolderTestPath = Test-Path $scriptFolder\$scriptName"_Script_Logs" -PathType container

    if ($scriptFolderTestPath -match "False") { 
        $scriptFolder = New-Item $scriptDirectory\$scriptName -ItemType directory
    }
    elseif ($scriptDocumentFolderTestPath -match "False") {
        New-Item $scriptFolder\$scriptName"_Script_Documents" -ItemType directory       
    }
    elseif ($scriptLogFolderTestPath -match "False") {
        New-Item $scriptFolder\$scriptName"_Script_Logs" -ItemType directory
    }
} Until (($scriptFolderTestPath -match "True") -and ($scriptDocumentFolderTestPath -match "True") -and ($scriptLogFolderTestPath -match "True"))

## variables for downloading and renaming code.
$date = (Get-Date -Format yyyy-MM-dd)

## ################################
## begin code for link downloading.
## ################################

## gets contents of the arguement variable.
$webTargets = Get-Content $linkList

## downloads the linked file.
Invoke-WebRequest $webTargets

It does grab the passed param and pull the invoke-webrequest. If the param has multiple lines adding 'Get-Content $linkList | ForEach-Object { Invoke-WebRequest $_ }' For whatever reason I just wasn't passing the param correctly until reading that post and changing things around. Also all my attempts to use $args failed to work correctly. — Steve, Oct 27 '12 at 21:46

Passing an array of URLs as an argument to Powershell

2 Answers2