Only download new files from FTP directory to local folder using PowerShell

Question

I have a PowerShell script which downloads all files from an FTP directory to a local folder (Thanks to Martin Prikryl).

The FTP directory is updated with multiple new files at various times throughout the day.

I will be running my script every 30 minutes via Task Scheduler to download files from the FTP directory.

I would like the script to check and only download new files from the FTP directory to local folder.

Note: The check must be done on the FTP directory as previously downloaded files may be removed from Local folder.

Please see below for my current script;

#FTP Server Information - SET VARIABLES
$user = 'user' 
$pass = 'password'
$target = "C:\Users\Jaz\Desktop\FTPMultiDownload"
#SET FOLDER PATH
$folderPath = "ftp://ftp3.example.com/Jaz/Backup/"

#SET CREDENTIALS
$credentials = new-object System.Net.NetworkCredential($user, $pass)

function Get-FtpDir ($url,$credentials) {
    $request = [Net.WebRequest]::Create($url)
    $request.Method = [System.Net.WebRequestMethods+FTP]::ListDirectory
    if ($credentials) { $request.Credentials = $credentials }
    $response = $request.GetResponse()
    $reader = New-Object IO.StreamReader $response.GetResponseStream() 
    $reader.ReadToEnd()
    $reader.Close()
    $response.Close()
}

$Allfiles=Get-FTPDir -url $folderPath -credentials $credentials
$files = ($Allfiles -split "`n")

$files 

$webclient = New-Object System.Net.WebClient 
$webclient.Credentials = New-Object System.Net.NetworkCredential($user,$pass)
$counter = 0
foreach ($file in ($files | where {$_ -like "*.*"})){
    $source=$folderPath + $file
    $destination = Join-Path $target $file
    $webclient.DownloadFile($source, $destination)

    #PRINT FILE NAME AND COUNTER
    $counter++
    $counter
    $source
}

Thanks in advance for you help!

Martin Prikryl · Accepted Answer · 2021-07-28T10:52:38.123

In your loop, check if the local file exists, before you actually download it:

$localFilePath = $target + $file
if (-not (Test-Path $localFilePath))
{
    $webclient.DownloadFile($source, $localFilePath)
}

As you actually do not keep the local files, you have to remember what files were already downloaded in some kind of a log:

# Load log
$logPath = "C:\log\path\log.txt"

if (Test-Path $logPath)
{
    $log = Get-Content $logPath
}
else
{
    $log = @()
}

# ... Then later in the loop:

foreach ($file in ($files | where {$_ -like "*.*"})){
    # Do not download if the file is already in the log.
    if ($log -contains $file) { continue }

    # If this is a new file, add it to a log and ...
    $log += $file

    # ... download it using your code
}

# Save the log    
Set-Content -Path $logPath -Value $log

Note that your code to split listing to lines is not very reliable. For a better solution, see my answer to PowerShell FTP download files and subfolders.

Related question:
Download most recent file from FTP using PowerShell

Only download new files from FTP directory to local folder using PowerShell

1 Answers1