Task Persistence C#

Question

I'm having a hard time trying to get my task to stay persistent and run indefinitely from a WCF service. I may be doing this the wrong way and am willing to take suggestions.

I have a task that starts to process any incoming requests that are dropped into a BlockingCollection. From what I understand, the GetConsumingEnumerable() method is supposed to allow me to persistently pull data as it arrives. It works with no problem by itself. I was able to process dozens of requests without a single error or flaw using a windows form to fill out the request and submit them. Once I was confident in this process I wired it up to my site via an asmx web service and used jQuery ajax calls to submit request.

The site submits request based on a url that is submitted, the Web Service downloads the html content from the url and looks for other urls within the content. It then proceeds to create a request for each url it finds and submits it to the BlockingCollection. Within the WCF service, if the application is Online (i.e. Task has started) - it pulls the request using the GetConsumingEnumerable via a Parallel.ForEach and Processes the request.

This works for the first few submissions, but then the task just stops unexpectedly. Of course, this is doing 10x more request than I could simulate in testing - but I expected it to just throttle. I believe the issue is in my method that starts the task:

 public void Start()
        {
            Online = true;

            Task.Factory.StartNew(() =>
            {
                tokenSource = new CancellationTokenSource();
                CancellationToken token = tokenSource.Token;
                ParallelOptions options = new ParallelOptions();
                options.MaxDegreeOfParallelism = 20;
                options.CancellationToken = token;

                try
                {
                    Parallel.ForEach(FixedWidthQueue.GetConsumingEnumerable(token), options, (request) =>
                    {
                        Process(request);
                        options.CancellationToken.ThrowIfCancellationRequested();

                    });
                }
                catch (OperationCanceledException e)
                {
                    Console.WriteLine(e.Message);
                    return;
                }

            }, TaskCreationOptions.LongRunning);

        }

I've thought about moving this into a WF4 Service and just wire it up in a Workflow and use Workflow Persistence, but am not willing to learn WF4 unless necessary. Please let me know if more information is needed.

possible duplicate of http://stackoverflow.com/questions/6608042/parallel-foreach-loop-with-blockingcollection-getconsumableenumerable — hatchet - done with SOverflow, May 27 '12 at 13:30
The problem is likely your use of Parallel.ForEaach with GetConsumingEnumerable (a BlockingCollection), as explained here: http://blogs.msdn.com/b/pfxteam/archive/2010/04/06/9990420.aspx — hatchet - done with SOverflow, May 27 '12 at 13:45

score 1 · Accepted Answer · answered May 27 '12 at 13:31

1

The code you have shown is correct by itself.

However there are a few things that can go wrong:

If an exception occurs, your task stops (of course). Try adding a try-catch and log the exception.
If you start worker threads in a hosted environment (ASP.NET, WCF, SQL Server) the host can decide arbitrarily (without reason) to shut down any worker process. For example, if your ASP.NET site is inactive for some time the app is shut down. The hosts that I just mentioned are not made to have custom threads running. Probably, you will have more success using a dedicated application (.exe) or even a Windows Service.

answered May 27 '12 at 13:31

usr

168,620
35
240
369

I had previously thrown in a catch to just restart the system - which didn't appear to work so I removed it. From what your saying, its likely that ASP.NET is the culprit. I did set so the workers don't timeout and I moved the WCF into a different AppPool than the Web Service(asmx) in hopes it would prevent any conflict. I suppose I have alot of studying to do. I tried to host the service via net.tcp but was unable to get the endpoints to expose. It worked fine within visual stuidos - but installed as a Windows Service it would error indicating the connection was refused. – Steven Malone May 27 '12 at 14:20
When you start threads inside a hosting process the host will actively work against you. It sounds like you are building a web crawler. I'd put the crawler in a long-running Windows service and have that service communicate with a normal WCF service or asp.net website which is sending commands to the Windows service. You are avoiding all problems that way. – usr May 27 '12 at 15:33
I ended up moving the WCF service host into a Window Service thinking this might solve the issue - but same symptoms. I can add to the collection with no issues, but if it is enumerating the blockingcollection in the task when I submit additional data to the collection - it causes the service to crash. In visual studios this does not occur. It also does not occur when I use a windows form as a client and submit request while the process is running. It only started occuring when I started submitting the request(adding to the collection) via a asmx. – Steven Malone May 28 '12 at 02:46
If someone could suggest a better way to peristantly process a collection of data using TPL - that would be great! I keep hitting a wall and this process has put me back 2 weeks on this project. – Steven Malone May 28 '12 at 02:48
What does "crash" actually mean? Can you capture and post the exception? What symptoms of a crash do you see? – usr May 28 '12 at 09:52
I've been adding try/catch throughout the application and setup a function to log everything. So far, the logs appear with the same symptoms I see - it starts working, then suddenly stops and the service crashes with the following: Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.AggregateException Stack: at System.Threading.Tasks.TaskExceptionHolder.Finalize() – Steven Malone May 28 '12 at 15:12
Well, there is your problem. Your code crashes so your task exits. Makes sense? – usr May 28 '12 at 15:18
let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/11833/discussion-between-steven-malone-and-usr) – Steven Malone May 28 '12 at 17:02

score 0 · Answer 2 · answered May 28 '12 at 18:47

It turns out the cause of this issue was with the WCF Binding Configuration. The task suddenly stopped becasue the WCF killed the connection due to a open timeout. The open timeout setting is the time that a request will wait for the service to open a connection before timing out. In certain situations, it reached the limit of 10 max connection and caused the incomming connections to get backed up waiting for a connection. I made sure that I closed all connections to the host after the transactions were complete - so I gave in to upping the max connections and the open timeout period. After this - it ran flawlessly.

Task Persistence C#

2 Answers2