FTP poller occasionally hangs on file download until restart

Question

I have a Windows service that polls a remote FTP server every three seconds. It checks a directory for files, downloads any files present, and deletes those files once downloaded. Average file size is 10 KB, and rarely they will go up to the 100 KB range.

Occasionally (I have noticed no pattern), the WebClient will throw the following:

System.Net.WebException: The operation has timed out.
at System.Net.WebClient.OpenRead(Uri address)

It will do this for one or more files, usually whatever files are in the remote directory at that time. It will continue to do so indefinitely, churning on the "stuck" files at each polling interval. The bizarre part is that when I stop/start the Windows service, the "stuck" files download perfectly and the polling/downloading works again for long stretches of time. This is bizarre because I download like this:

private object _pollingLock = new object();

public void PollingTimerElapsed(object sender, ElapsedEventArgs e)
{
    if(Monitor.TryEnter(_pollingLock);
    {
        //FtpHelper lists content of files in directory
        ...

        foreach(var file in files)
        {
            using(var client = new WebClient())
            {
                client.Proxy = null;
                using(var data = client.OpenRead(file.Uri)
                {
                    //Use data stream to write file locally
                    ...
                }
            }
            //FtpHelper deletes the file
            ...
        } 
    }
    //Release the _pollingLock inside a finally
}

I would assume that a new connection is opened and closed for each file (unless .NET is doing something behind the scenes). If a file download had an issue, it would get a fresh retry on the next polling interval (in 3 sec). Why would a service restart make things work?

I've begun to suspect that the issue has something to do with caching (file or connection). Recently I tried going into Internet Explorer and clearing the cache. Approximately 30 sec or so later, all the files downloaded with no service restart. But, the next batch of files to arrive all got hung up again. I might try adding a line like this:

client.CachePolicy = new RequestCachePolicy(RequestCacheLevel.NoCacheNoStore);

or try disabling KeepAlives, but I want to get some opinions before I start trying random stuff.

So: What is causing the occasional timeouts? Why does restarting the service work? Why did clearing the cache work?

Update

I made the cache policy and keep alive change mentioned above about two weeks ago. I just now got my first timeout since then. It appears to have improved frequency, but alas, it is still happening.

Update

As requested, this is how I am kicking off the Timer:

_pollingTimer.AutoReset = true; 
_pollingTimer.Elapser += PollingTimerElapsed; 
_pollingTimer.Interval = 10000; 
_pollingTimer.Enabled = true;`

Answer 1

Looks like you are kicking off your processing using the System.Timers.Timer.Elapsed event.

One gotcha that I found is that if your Elapsed event takes longer to execute than the timer interval, your event can be called again from another thread before it has finished executing.

This is specifically mentioned in the docs:

If the SynchronizingObject property is null, the Elapsed event is raised on a ThreadPool thread. If the processing of the Elapsed event lasts longer than Interval, the event might be raised again on another ThreadPool thread. In this situation, the event handler should be reentrant.

Assuming you are indeed using a vanilla timer with AutoReset=true (its on by default), first thing to do would be address this potential issue. You can use a SynchronizingObject, alternatively you can do something like this:

//setup code
Timer myTimer = new Timer(30000);
myTimer.AutoReset = false;
....

//Elapsed handler
public void PollingTimerElapsed(object sender, ElapsedEventArgs e)
{
    //do what you currently do
    ...

    //when finished, kick off the timer again
    myTimer.Start();
}

Either way, the main thing is to ensure that your code doesn't accidentally get called simultaneously by multiple threads - if that happens there's a good chance that occasionally you'll have one thread trying to download something from the site while another thread is simultaneously deleting the file.

The things that you mentioned e.g. it only happens occasionally, that normally file sizes are small, that its fixed by a restart, etc. would point me in the direction of this being the issue.