Download many small files with FTP quickly

Question

I have a working code to download many files (hundreds) from an FTP server, but it's very slow and often a timeout error appears.

This is my current way of doing the downloads:

Using ftpClient As New WebClient()
    ftpClient.Credentials = New System.Net.NetworkCredential(ftpuser, ftppassword)
    For i As Integer = 0 To directoriesDownload.Count - 1
        If directoriesDownload(i).Contains(".") Then
            If Sync_BackgroundWorker.CancellationPending = True Then
                Exit Sub
            End If
            Dim path As String = "ftp://" & ftpserver & "Datenbank/" + directoriesDownload(i).ToString()
            Dim trnsfrpth As String = config.rootpath & "ServerDownload\" + directoriesDownload(i).ToString()
            ftpClient.DownloadFile(path, trnsfrpth)
            filenametodownload = directoriesDownload(i).ToString()
            filesdownloaded += 1
            Sync_BackgroundWorker.ReportProgress(filesdownloaded)
        End If
    Next
    ftpClient.Dispose()
End Using

Is there any faster way of downloading hundreds of small files (up to 10 KB) from an FTP server in VB.NET?

It would be the best if there is an option to sign into the FTP only once instead of logging in and out for every file.

I found somebody else having the same problem but without a working result: Using FTP to download each file *WHILE* getting the file list

I also tried multithreading with a Parallel.For loop, but WebClient does not work with multithreading. Same thing if I try with ftpClient.DownloadFileAsync(New Uri(path), trnsfrpth).

Is the timeout generated by the FTP server or by something in the code you've shown? How does the time taken by your code compare to the time taken by something like [WinSCP](https://winscp.net/eng/index.php) doing the same job? — Andrew Morton, Sep 09 '18 at 15:45
Have you tried using `Parallel.ForEach` and creating a new `WebClient` inside each task? — jmcilhinney, Sep 09 '18 at 15:53

score 0 · Accepted Answer · edited Jun 20 '20 at 09:12

Is there any faster way of downloading hundreds of small files (up to 10 KB) from an FTP server in VB.NET?

...

I also tried multithreading with a Parallel.For loop, but WebClient does not work with multithreading. Same thing if I try with ftpClient.DownloadFileAsync(New Uri(path), trnsfrpth).

Multithreading is the way to go. It's not true that WebClient does not support multithreading. Why wouldn't it?

If you have a problem with implementing multithreaded FTP transfers, you should ask a question about that, rather than asking a question about other (and probably non-existent) ways.

It would be the best if there is an option to sign into the FTP only once instead of logging in and out for every file.

Your code does sign into FTP only once.

See C# - FtpWebRequest - Multiple requests over the same connection/login - What is written there about FtpWebRequest is equally true for WebClient, as WebClient uses FtpWebRequest internally.

Download many small files with FTP quickly

1 Answers1