7

An ASP.NET 3.5 webapp has to start several tasks that takes hours to complete. For obvious reasons the pages which starts these tasks cannot wait for them to finish nor will anyone want to wait that long to get a response, so the tasks must be asynchronous.

There is a Helper class to handle all of these long running tasks. The main method that schedules and executes these tasks is currently the following:

public static bool ScheduleTask(TaskDescriptor task, Action action)
{
    bool notAlreadyRunning = TasksAsync.TryAdd(task);
    if (notAlreadyRunning)
    {
        Thread worker = null;
        worker = new Thread(() => 
        {
            try { action(); }
            catch(Exception e)
            {
                Log.LogException(e, "Worker");
            }
            TasksAsync.RemoveTask(task);
            workers.Remove(worker);
        });
        workers.Add(worker);
        worker.Start();
    }
    return notAlreadyRunning;
}

On earlier implementations we've used the ThreadPool.QueueUserWorkItem approach but the result has always been the same: after aprox. 20-30 mins a Thread was being aborted exception is thrown.

Does anyone know why is this happening? or how can it be prevented?

More Info:

  • IIS standard configuration.
  • Tasks could be anything, querys to a database and/or IO operations etc.

UPDATE: Decisions

Thank you all for your responses. Now I don´t know which question to mark as answer. All of them are valid and are possible solutions to this problem. Will wait for today and mark as answer the answer with the most up votes, in case of a draw I will choose the first shown answer, typically they are ordered by most relevance.

For anyone who want´s to know the solution I choose, again due to time restrictions, was to change the IIS recycling configuration, But what I consider to be the ideal solution, based on my research and of course the answers below, is to create a "Worker Service" and use a communication solution between the ASP.NET App and the new "Worker Service" to coordinate the long running work to be done.

Andres A.
  • 1,329
  • 2
  • 21
  • 37
  • Do your IIS AppPools/workers recycle? – Chris Sinclair Apr 27 '13 at 16:45
  • The IIS configuration is the standard configuration for IIS 7. So yes, I guess. – Andres A. Apr 27 '13 at 16:50
  • It's been a while since I've been knee-deep in ASP.NET, but if memory serves, when worker processes recycle they can stop threads. Try disabling recycling and see if that's the source of the thread abortions. (not sure of a workaround if that is the case though; like I said, haven't touched _that_ particular issue) – Chris Sinclair Apr 27 '13 at 17:04
  • You can disable App Pool recycling by following this thread: http://stackoverflow.com/questions/3156925/stop-iis-7-5-application-pool-recycling but I do think that the best approach is to run this on a different App Domain as Robert suggests. – Icarus Apr 27 '13 at 17:10
  • What type of task it is ? if you want to some database related task then you should use database trigger with service broker which is totally async and you can fire trigger at any time from your application . – Hiren Dhaduk Apr 27 '13 at 17:23
  • It's a combination of database calls and IO and "processing"?, the task can have several calls to a database and some of these calls could take a while to complete. – Andres A. Apr 27 '13 at 17:37
  • Phill Haack goes deep on the subject in his [The Dangers of Implementing Recurring Background Tasks In ASP.NET](http://haacked.com/archive/2011/10/16/the-dangers-of-implementing-recurring-background-tasks-in-asp-net.aspx) post. Hope it helps – tucaz Apr 27 '13 at 17:09
  • Great post, thanks for sharing! but due to time restrictions I don't think we could do the appdomain approach. – Andres A. Apr 27 '13 at 17:33
  • Phil's post was awesome in its day, now it's obsolete, at least on WAWS. You can now use Azure WebJobs – RickAndMSFT Jan 21 '14 at 01:44

2 Answers2

6

You can start the long-running process in its own application domain.

In the past, when I've needed this capability, I create a Windows Service for this purpose. If you use WCF to connect to it, it doesn't even have to run on the IIS machine at all; you can run it on any machine on the network.

Robert Harvey
  • 178,213
  • 47
  • 333
  • 501
  • I think you're right and that would be the approach to take, but due to time restrictions I don't think we could do it. – Andres A. Apr 27 '13 at 17:34
  • I don't think you really have a choice. Phil Haack makes it pretty clear that IIS is not designed to do this. You could try spinning up a thread in a new appdomain: http://www.pcreview.co.uk/forums/start-thread-new-appdomain-t3614317.html – Robert Harvey Apr 27 '13 at 17:35
  • I did not know you could create an appdomain on the fly like that... guess I will read about that. But if you could save me a little research, What happens to the appdomain when the thread ends? basicaly how do I make sure to free the resources of the new appdomain? – Andres A. Apr 27 '13 at 17:44
  • Back in the "olden days" we used COM+ and MSMQ to do this kind of long running process. I think the WCF and service based suggestion here should work fine. You could probably still use MSMQ if you wanted to. – jfrankcarr Apr 27 '13 at 20:16
  • @jfrankcarr: It's very easy to do with .NET Remoting, but for whatever reason that technique is not fashionable anymore. WCF is the new black. – Robert Harvey Apr 27 '13 at 21:29
4

Chances are you can get this working, by upping the timeout, using a different app pool or a variety of other hacks, but your best bet is going to be to decouple the long running task from the ui and asp.net completely, and use either a service (wouldn't recommend it) or a scheduled task that polls for work to do; personally I would use something like aws sqs/sns to keep track of work to be done and a scheduled task in windows server that checks for things todo at whatever frequency make sense. The only thing the ui/asp.net then needs to do is log that fact that something needs to be done, not actually do it.

Another benefit of this message based approach is should the long running process become so long running, or so overworked, you'd have the opportunity to add more worker tasks or servers to complete those requests.

Perhaps more than you can implement for your immediate problem, but something to consider for a better long term solution.

E.J. Brennan
  • 45,870
  • 7
  • 88
  • 116