FtpWebRequest 30 minute time out

Question

My code is experiencing a time out exception after exactly 30 minutes when downloading a large file over FTP. The server is FileZilla running on Windows. We have an SSL certificate configured with options Enable FTP over SSL/TLS support (FTPS) and Allow explicit FTP over TLS enabled. I have access to the server and the FileZilla configuration but can't see anything that might cause this behavior. Below is the source code which is running on .NET 4.6.2 on a Windows 2012 Server Machine. It can download files from the FTP server but will time out with the exception (listed below) after exactly 30 minutes if the file takes longer than 30 minutes to download.

As a test I have used FileZilla Client, running from the same client PC, to download multiple large files from the same server end point simultaneously so that the download for each file took over 30 minutes to complete. No errors occurred in this scenario.

I have searched on StackOverflow as well as Google but nothing promising turned up. If anyone has some tips on where to look (server side or client side) I would be most appreciative.

---

Application code

public class FtpFileDownloader
{
    // log4net
    private static readonly ILog Logger = LogManager.GetLogger(typeof(FtpFileDownloader));

    public void DownloadFile()
    {
        // setting the SecurityProtocol did not change the outcome, both were tried. Originally it was not set at all.
        // ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
        ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;


        ServicePointManager.ServerCertificateValidationCallback = (sender, certificate, chain, sslPolicyErrors) => true;

        const int timeout = 7200000;
        const string file = "some-existing-file";
        try
        {
            var request = (FtpWebRequest) WebRequest.Create("uri-path-to-file");
            request.KeepAlive = false;
            request.Timeout = -1;
            request.ReadWriteTimeout = timeout;

            request.Credentials = new NetworkCredential("userName", "password");
            request.UsePassive = true;
            request.EnableSsl = true;
            request.Method = WebRequestMethods.Ftp.DownloadFile;

            Logger.Debug($"Downloading '{file}'");
            using (var response = (FtpWebResponse) request.GetResponse())
            using (var sourceStream = response.GetResponseStream())
            using (var targetStream = new FileStream("some-target-on-disk", FileMode.Create, FileAccess.Write))
            {
                try
                {
                    sourceStream.CopyTo(targetStream);
                    targetStream.Flush();
                    Logger.Debug($"Finished download '{file}'");
                }
                catch (Exception exInner)
                {
                    Logger.Error($"Error occurred trying to download file '{file}'.", exInner);
                }
            }
        }
        catch (Exception ex)
        {
            Logger.Error($"Error occurred trying to dispose streams when downloading file '{file}'.", ex);
        }
    }
}

---

Application Log

ERROR FtpFileDownloader - Error occurred trying to download file 'some-existing-file'.
System.IO.IOException: Received an unexpected EOF or 0 bytes from the transport stream.
   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security._SslStream.StartFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.TlsStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.FtpDataStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)
   at System.IO.Stream.CopyTo(Stream destination)
   at FtpFileDownloader.DownloadFile

ERROR FtpFileDownloader - Error occurred trying to dispose streams when downloading file 'some-existing-file'.
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive.
   at System.Net.FtpWebRequest.SyncRequestCallback(Object obj)
   at System.Net.FtpWebRequest.RequestCallback(Object obj)
   at System.Net.CommandStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at System.IO.Stream.Dispose()
   at System.Net.ConnectionPool.Destroy(PooledStream pooledStream)
   at System.Net.ConnectionPool.PutConnection(PooledStream pooledStream, Object owningObject, Int32 creationTimeout, Boolean canReuse)
   at System.Net.FtpWebRequest.FinishRequestStage(RequestStage stage)
   at System.Net.FtpWebRequest.SyncRequestCallback(Object obj)
   at System.Net.FtpWebRequest.RequestCallback(Object obj)
   at System.Net.CommandStream.Abort(Exception e)
   at System.Net.CommandStream.CheckContinuePipeline()
   at System.Net.FtpWebRequest.DataStreamClosed(CloseExState closeState)
   at System.Net.FtpDataStream.System.Net.ICloseEx.CloseEx(CloseExState closeState)
   at System.Net.FtpDataStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at System.IO.Stream.Dispose()
   at FtpFileDownloader.DownloadFile

---

FileZilla - General Settings

 Listen on these ports: 21
  Max. number of users: 0 (infinite)
     Number of threads: 2
    Connection timeout: 120 (seconds)
   No Transfer timeout: 9000 (seconds)
           Log timeout: 60 (seconds)

---

FileZilla Server Log

23-2-2018 11:40:40 - (not logged in) (194.123.75.2)> Connected on port 21, sending welcome message...
23-2-2018 11:40:40 - (not logged in) (194.123.75.2)> 220 Welcome
23-2-2018 11:40:40 - (not logged in) (194.123.75.2)> AUTH TLS
23-2-2018 11:40:40 - (not logged in) (194.123.75.2)> 234 Using authentication type TLS
23-2-2018 11:40:40 - (not logged in) (194.123.75.2)> TLS connection established
23-2-2018 11:40:40 - (not logged in) (194.123.75.2)> USER  my-user-account
23-2-2018 11:40:40 - (not logged in) (194.123.75.2)> 331 Password required for my-user-account
23-2-2018 11:40:40 - (not logged in) (194.123.75.2)> PASS **************
23-2-2018 11:40:40 - my-user-account (194.123.75.2)> 230 Logged on
23-2-2018 11:40:40 - my-user-account (194.123.75.2)> PBSZ 0
23-2-2018 11:40:40 - my-user-account (194.123.75.2)> 200 PBSZ=0
23-2-2018 11:40:40 - my-user-account (194.123.75.2)> PROT P
23-2-2018 11:40:40 - my-user-account (194.123.75.2)> 200 Protection level set to P
23-2-2018 11:40:40 - my-user-account (194.123.75.2)> OPTS utf8 on
23-2-2018 11:40:40 - my-user-account (194.123.75.2)> 202 UTF8 mode is always enabled. No need to send this command.
23-2-2018 11:40:40 - my-user-account (194.123.75.2)> PWD
23-2-2018 11:40:40 - my-user-account (194.123.75.2)> 257 "/" is current directory.
23-2-2018 11:40:40 - my-user-account (194.123.75.2)> TYPE I
23-2-2018 11:40:40 - my-user-account (194.123.75.2)> 200 Type set to I
23-2-2018 11:40:40 - my-user-account (194.123.75.2)> PASV
23-2-2018 11:40:40 - my-user-account (194.123.75.2)> 227 Entering Passive Mode (IP-ADDRESS,245,222)
23-2-2018 11:40:40 - my-user-account (194.123.75.2)> RETR path-to-file
23-2-2018 11:40:40 - my-user-account (194.123.75.2)> 150 Opening data channel for file download from server of "/path-to-file"
23-2-2018 11:40:40 - my-user-account (194.123.75.2)> TLS connection for data connection established
23-2-2018 12:10:41 - my-user-account (194.123.75.2)> disconnected.

Notice that there is 30 minutes between the disconnect (last line) and the line before it. If it had completed the transfer with success there would also be a line that read 226 Successfully transferred "/path-to-file" before the disconnected line

---

Progress update:

• (2018.02.20) I asked our network team to check the fire wall rules but they could not find anything of interest.
• (2018.02.22) I found that the version of FileZilla Server being used is 0.9.43 beta (release date 2014-01-02 according to the change log). Although I did not find anything in the change log that suggests this behavior was ever a fixed bug I am going to upgrade to the latest version 0.9.60.2 (release date 2017-02-08) and execute the test again. Will report back within 24 hours.
• (2018.02.23) FileZilla has been updated to the latest version. This did not fix the problem, I updated the server log but it looks almost identical to the previous log with the exception being that this last transfer took place over TLS instead of SSL.
• (2018.02.23) I found the following link Timeouts on large files on the FileZilla support page. I am going to push it back to the network staff of our hosting provider to take another look.
• (2018.02.27) It turned out that the company firewall (network where the client executes) and not the hosting firewall (network where FileZilla is hosted) was the culprit. It was configured to drop idle connections after 1800 seconds. A rule has been added to override this between these 2 end points.

---

Culprit / Answer

It turned out that the company firewall (network where the client executes) and not the hosting firewall (network where FileZilla is hosted) was the culprit. It was configured to drop idle connections after 1800 seconds. A rule has been added to override this between these 2 end points.

Answer 1

Probably you should try another implementation of FTP protocol client that isn't built on top of the FtpWebRequest.

Related issues exist for a long time, they don't have a clear solution or answer. So I would try something like FluentFTP, it uses Winsock API directly. XML Documentation Comment states that DownloadFile() should handle large file downloads well:

/// <summary>
/// Downloads the specified file onto the local file system.
/// High-level API that takes care of various edge cases internally.
/// Supports very large files since it downloads data in chunks.
/// </summary>

For more information check:

• Downloading large files(~150MB) from FTP server hangs discussion on StackOverflow
• FtpWebRequest is Broken article by Matt Mitchell
• Winsock API page on Microsoft TechNet
• Source Code section in Partial FTP Downloader article by Elmue

Answer 2

Yep, I don't think there's a "mistake" in your code; it's just that the control connection dies out after 30 minutes even though the transfer connection doesn't timeout. Perhaps it won't even be necessary to alter you KeepAlive and Timeout values, just try to reuse your request every 20 minutes or so with a dummy download: that way you will reset the control connection timer.

Btw, somewhere I've read that 30 minutes is a standard timeout for FileZilla Server, which is based on 6 keep-alive configured to be sent every 300 seconds (which gives your 30 minutes experience). If you could try with another FTP/FTPS server you'd probably find out a different idle timeout and would not bang against that 30 minute limit (but a different one).

So, personally I would invest in making the code below async, so the execution flow goes on after the enclosing using and you can enter a loop where every 20 minutes you reuse your request (and its control connection) to make a dummy download. Of course FileZilla Client doesn't need a dummy download because it operates at a lower level and probably sends TCP commands to keep the control connection alive.

using (var response = (FtpWebResponse) request.GetResponse())
        using (var sourceStream = response.GetResponseStream())
        using (var targetStream = new FileStream("some-target-on-disk", FileMode.Create, FileAccess.Write))
        {
            try
            {
                sourceStream.CopyTo(targetStream);
                targetStream.Flush();
                Logger.Debug($"Finished download '{file}'");
            }
            catch (Exception exInner)
            {
                Logger.Error($"Error occurred trying to download file '{file}'.", exInner);
            }
        }

Answer 3

There's a similar issue reported in FluentFTP here and stefanolazzarato posted a work-around:

int progress = -1;
try
{
   FtpClient client = new FtpClient("HOST");
   client.Credentials = new NetworkCredential("USER", "PASSWORD");
   client.Connect();

   client.UploadFile("LOCALPATH/FILENAME", "REMOTEPATH/FILENAME",
       FtpExists.Overwrite,
       false,
       FtpVerify.None,
       new Progress<FtpProgress>(p => progress = Convert.ToInt32(p.Progress))
       );
}
catch (Exception ex)
{
   if (progress == 100 && ex is FluentFTP.FtpException && ex.InnerException != null && ex.InnerException is TimeoutException)
   {
       // Upload complete
       // LOG Info exception
   }
   else
   {
       // LOG Fatal exception
       throw;
   }
}