PowerShell Get links from FTP Site

Question

I'm trying to create a list of all the links from an FTP site. The links are to download zip files.

My end goal is to analyze each link as a string and match the beginning to a set phrase. The end of each link contains a date and I have to find the newest one to download.

In this example I want to find ABC_20170323.zip out of this list:

ABC_20170323.zip
ABC_20160102.zip
EFG_20170324.zip

I need to figure out how to acquire the links before analysis. I've tried a variety of methods and the only one that has returned any information from the site is to gather the source code:

Invoke-WebRequest $sourceuri -UseBasicParsing -Credential $user

But then I find it difficult to gather all the links from there. Anyone have a method for easily getting these file download links?

Invoke-WebRequest $sourceuri -UseBasicParsin -Credential $user | Select-Object -ExpandProperty links | Where-Object {$_.href -like "*.zip"} | Select-Object -ExpandProperty href | ForEach { Start-BitsTransfer $_ C:\test } — Will Webb, Mar 24 '17 at 14:17
if you don't want them downloaded to C:\test just remove that last pipe. So omit from the `| ForEach` onwards. — Will Webb, Mar 24 '17 at 14:18
Select-Object : Property "links" cannot be found. How is this possible when I can click on it in a browser and download it? — CuriousOne, Mar 24 '17 at 14:26
It's a FTP site with webpages. I need the links from the site on a specific page. — CuriousOne, Mar 24 '17 at 14:31
Check out http://stackoverflow.com/questions/7053243/powershell-ftp-directory-listing-help-on-a-script — G42, Mar 24 '17 at 14:52

score 0 · Accepted Answer · answered Aug 16 '17 at 19:00

Okay, so I know it's been ages, but I figured out how to do it. Admittedly, it's the hard way. What ended up happening is I gathered the source code and saved it like so:

$r = Invoke-WebRequest $sourceuri -UseBasicParsing -credential $user

Then I converted it to a string and used -split to separate out the links by their html tag and what I expected the beginning to look like (in this case 'ABC'):

$c = $r.ToString() #convert to string
$datelist = @()
$f = ($c -split 'A HREF="' -split '.zip</A>') #split by html tag (and .zip)
foreach($link in $f){
    if($link -match 'ABC') { #if the beginning of the link is 'ABC'
        $datelist += ($link.substring($link.Length-8)) #isolate the date on the end
    }
} #more logic for comparing $datelist items...

Then I wrote some logic for comparing the items in $datelist (Omitted from answer) and created a variable that had all the components I needed:

$ExactLink = "ABC_$GreatestDate" + ".zip"

Then went on to download the $ExactLink I needed.

PowerShell Get links from FTP Site

1 Answers1