Queued tasks are lost in ThreadPool

Question

I have RESTful service written in c#, its used to some internal data migration (SQL to Mongo) via some internal apis.

This REST service is getting SQL server data from web service and POST those data to another service which will insert data to mongo DB,

public void GetAllCoursesAndMigrateAllCourse(string clientstring, DataServiceCollection<Semester> semesters)
        {
            DataServiceCollection<Campus.Explorer.Proxy.Models.Course> courses = null;
            try
            {

                string endpoint = campusExploreUri + clientstring;
                CampusExplorerServiceContext campusExplorerServiceContext = new CampusExplorerServiceContext(new Uri(endpoint));
                courses = new DataServiceCollection<Campus.Explorer.Proxy.Models.Course>(campusExplorerServiceContext.Courses);

                if (courses != null && courses.Count > 0 && semesters != null && semesters.Count > 0)
                    ThreadPool.QueueUserWorkItem(new WaitCallback(f => { CreateCourse(courses, semesters); }));
               int cCount =100;
                //this step is necessary because CE results are limited to 100 per call. This will ensure we get all results
                while (courses.Continuation != null)
                {
                    courses = new DataServiceCollection<Campus.Explorer.Proxy.Models.Course>(campusExplorerServiceContext.Execute<Campus.Explorer.Proxy.Models.Course>(courses.Continuation.NextLinkUri));
                    if (courses != null && courses.Count > 0 && semesters != null && semesters.Count > 0)
                    {
                        cCount = cCount + courses.Count;
                        _logger.Info("cCount " + cCount);
                        ThreadPool.QueueUserWorkItem(new WaitCallback(k => { CreateCourse(courses, semesters); }));
                    }
                }
            }

            catch (Exception ex)
            {
                _logger.Error("Error in GetAllCoursesAndMigrateAllCourse " + ex.Message);
            }
        }

Here I'm queuing all the get data to ThreadPool to process create course function. Once after get loop is over w3wp process is stopped the processing but w3wp process shows in task manager processes as idel, so because of this I'm loosing all the data are queued in Thredpool,

I have already removed the application pool recycle times and set the idle time to 0 in application pool settings.

How do I overcome this issue?

Do you mean that your process is getting exited before your ThreadPool Queue Completes its task ? — Rangesh, Jun 22 '14 at 15:33
Yes, w3wp process is getting stopped before complete all the queued tasks — KRR, Jun 22 '14 at 15:50

score 0 · Answer 1 · edited Nov 17 '17 at 11:07

0

I Assume your Process is getting exited before your ThreadPool Queue Completes its task.Guess you know that ThreadPool are background Thread(Process will not wait until these thread finish it's task),If you don't want your Process to exit and wait till the Queue Complete its task you can make use of ManualResetEvent(Set and wait).

private static ManualResetEvent reset = new ManualResetEvent(false);
/*After you Queue*/ 
reset.WaitOne()

Do enusre that you set the ManualRestEvent , In your `Case-CreateCourse()

edited Nov 17 '17 at 11:07

Afflatus

933
1
12
39

answered Jun 22 '14 at 18:49

Rangesh

728
2
12
27

I'm getting error "The number of WaitHandles must be less than or equal to 64" after applying event – KRR Jun 23 '14 at 06:15
http://stackoverflow.com/questions/2702545/workaround-for-the-waithandle-waitall-64-handle-limit Please look at this WorkAround. – Rangesh Jun 23 '14 at 06:18
May be You should Try CountDownEvent (Wait and Signal)-Assuming you use (.NET 4.0),its far more simple than ManualResetEvent http://msdn.microsoft.com/en-us/library/system.threading.countdownevent.aspx – Rangesh Jun 23 '14 at 07:32
I tried with CountDownEvent, but Wait() function ends (Isset == true) before all the background process are completed – KRR Jun 23 '14 at 09:13
Can you show the code?Where have you put your Wait() Method...I assume after the Queuing your task? – Rangesh Jun 23 '14 at 09:31
Does this process waiting problem resolve if I move to windows service application – KRR Jun 23 '14 at 10:12
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/56108/discussion-between-restless-and-krr). – Rangesh Jun 23 '14 at 12:25

Queued tasks are lost in ThreadPool

1 Answers1