Can PowerShell use FTP to retrieve remote folder and subfolder directory data in a single transmission?

Question

I have a PowerShell script that checks existing timestamp data for files at a remote location to compare with files locally.

The local information is retrieved with this snippet:

$SrcEntries = Get-ChildItem $FolderLocation -Recurse

I can then get the specific information on folders and files with the following snippet:

$Srcfolders = $SrcEntries | Where-Object{$_.PSIsContainer}
$SrcFiles = $SrcEntries | Where-Object{!$_.PSIsContainer}

The remote information is currently retrieved with this snippet which is executed individually for each file:

$Credentials = New-Object System.Net.NetworkCredential($FTPUser,$FTPPassword)
$FTPrequest = [System.Net.FtpWebRequest]::Create($RemoteFilePath)
$FTPrequest.Method = [System.Net.WebRequestMethods+FTP]::GetDateTimestamp
$FTPrequest.Credentials = $Credentials
$response = $FTPrequest.GetResponse().StatusDescription

The $response is then parsed to retrieve the timestamp of the remote file.

It would seem better to retrieve the remote data the same way local data is retrieved, i.e. in a single operation as with the first snippet above to avoid the multiple round trips to the remote server.

Is it possible to get folder and file directory data recursively in a single quick FTP operation using PowerShell? How would I do that?

Answer 1

It is s not easy to implement this without any external library. Unfortunately, neither the .NET Framework nor PowerShell have any explicit support for recursively listing files in an FTP folder.

You have to implement that yourself:

• List the remote folder
• Iterate the entries, recursing into subfolders - listing them again, etc.

The first tricky part is to identify files from subfolder. There's no way to do that in a portable way with the .NET framework (FtpWebRequest). The .NET framework unfortunately does not support the MLSD command, which is the only portable way to retrieve directory listing with file attributes in FTP protocol. See also Checking if object on FTP server is file or directory.

Your options are:

• Do an operation on a file name that is certain to fail for file and succeeds for directories (or vice versa). I.e. you can try to download the "name".
• You may be lucky and in your specific case, you can tell a file from a directory by a file name (i.e. all your files have an extension, while subdirectories do not)
• You use a long directory listing (LIST command = ListDirectoryDetails method) and try to parse a server-specific listing. Many FTP servers use *nix-style listing, where you identify a directory by the d at the very beginning of the entry. But many servers use a different format. The following example uses this approach (assuming the *nix format)

function ListFtpDirectory($url, $credentials)
{
    $listRequest = [Net.WebRequest]::Create($url)
    $listRequest.Method = [System.Net.WebRequestMethods+Ftp]::ListDirectoryDetails
    $listRequest.Credentials = $credentials
    
    $lines = New-Object System.Collections.ArrayList

    $listResponse = $listRequest.GetResponse()
    $listStream = $listResponse.GetResponseStream()
    $listReader = New-Object System.IO.StreamReader($listStream)
    while (!$listReader.EndOfStream)
    {
        $line = $listReader.ReadLine()
        $lines.Add($line) | Out-Null
    }
    $listReader.Dispose()
    $listStream.Dispose()
    $listResponse.Dispose()

    foreach ($line in $lines)
    {
        $tokens = $line.Split(" ", 9, [StringSplitOptions]::RemoveEmptyEntries)
        $name = $tokens[8]
        $permissions = $tokens[0]

        if ($permissions[0] -eq 'd')
        {
            Write-Host "Directory $name"

            $fileUrl = ($url + $name)
            ListFtpDirectory ($fileUrl + "/") $credentials
        }
        else
        {
            Write-Host "File $name"
        }
    }
}

Use the function like:

$credentials = New-Object System.Net.NetworkCredential("user", "mypassword") 
$url = "ftp://ftp.example.com/directory/to/list/"
ListFtpDirectory $url $credentials

---

Though the above code will list only names. As you need timestamps too, you would have to parse out even those. That's as complicated as determining, if an entry is folder or file.

I do not have a PowerShell code for that, but see my answers to other similar questions, which show C# code. It should be easy to translate to PowerShell:

• Retrieving creation date of file (FTP)
• Parsing FtpWebRequest ListDirectoryDetails line
• C# class to parse WebRequestMethods.Ftp.ListDirectoryDetails FTP response

---

If you want to avoid troubles with parsing the server-specific directory listing formats, use a 3rd party library that supports the MLSD command and/or parsing various LIST listing formats.

For example with WinSCP .NET assembly you can list whole directory recursively with a single call to Session.EnumerateRemoteFiles:

# Load WinSCP .NET assembly
Add-Type -Path "WinSCPnet.dll"

# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
    Protocol = [WinSCP.Protocol]::Ftp
    HostName = "ftp.example.com"
    UserName = "user"
    Password = "mypassword"
}

$session = New-Object WinSCP.Session

try
{
    # Connect
    $session.Open($sessionOptions)

    # Download files
    $options =
        [WinSCP.EnumerationOptions]::EnumerateDirectories -bor
        [WinSCP.EnumerationOptions]::AllDirectories
    $fileInfos = $session.EnumerateRemoteFiles(
        "/directory/to/list", $Null, $options)
    foreach ($fileInfo in $fileInfos)
    {
        Write-Host $fileInfo.FullName
    }
}
finally
{
    # Disconnect, clean up
    $session.Dispose()
}    

Not only the code is simpler, more rubust and platform-independent. It also makes all other file attributes (size, modification time, permissions, ownership) readily available via the RemoteFileInfo class.

Internally, WinSCP uses the MLSD command, if supported by the server. If not, it uses the LIST command and supports dozens of different listing formats.

---

Though it seems that you aim to compare the local and remote timestamps to synchronize the folders.

WinSCP .NET assembly can do that with a single call using Session.SynchronizeDirectories.

You may also find these questions (answered by me) useful:

• How to compare age of local file with file on FTP server and download if remote copy is newer in PowerShell
• How do I query a file on FTP server in PowerShell to determine if an upload is required?

(I'm the author of WinSCP)