-2

I have a continuous async method that is used for things like polling resources and message queues.

private async Task MonitorAsync(CancellationToken cancelToken)
{
  while (!cancelToken.IsCancellationRequested)
  {
    await Task.Delay(100);

    // Poll stuff and take action
  }
}

This is running in the UIContext (simplifies concurrency issues).

AppViewModel()
{
  MonitorAsync(); // Starts the Monitor
}

What I've observed is in certain extreme conditions, this async method will simply stop running (e.g. application stops processing messages). For example, if too much CPU-bound code runs in the UIContext. Additionally, I have a few Monitors running, and only some of them die.

I'll grant that so far I've only seen this occur in a scenario that fundamentally needs to be addressed, but I'm still concerned about the fact that it would still be possible in edge cases.

As a workaround, I'll probably need to add a timer and restart the Monitor if it appears to have died.

Some additional notes:

  • It's not throwing an exception. Hooking into UnobservedTaskException or wrapping in try/catch confirms this.
  • It's definitely not stuck on an internal await. I added a simple flag to confirm, and it gets to "await Task.Delay(100)", but it never comes back.

QUESTIONS

  1. What could cause this to happen?
  2. How would I go about debugging this? What object could I inspect to, for example, see a list of running async methods in a given context?

I suspect the Visual Studio 2015 "Tasks Window" should be listing to all the async methods, but it is blank. It says "No tasks to display." I've never seen it display anything.

MORE INFORMATION:

I'm certain it didn't throw an exception or get stuck in an indefinite wait. The symptom appears that the Task is not running any more (or is delayed for an incredibly long time). It also appears to continue running some of the Tasks, and in fact the ones are still running are newer ones.

I have a theory that, in this rare circumstance, there is preference given to the most recently created Tasks. It gives whatever limited time it can to the recently created ones, and the old ones effectively aren't run any more.

It would help if I could access a list of Tasks. Then I could confirm if that is the case. I can do that for Threads, but so far I haven't found out how to do that for Tasks.

UPDATE 6/7/2016:

It seems the Task in question is actually still running. It's just significantly delayed. For example, if I run three Tasks running this method, then recreate the edge case - two of them are running fine (await resumes after 100 ms), but one of them (the oldest one) takes anywhere from 2 seconds to 20 seconds (sometimes more). So it seems the scheduler doesn't try to fairly distribute limited processing availability.

Per recommendation, I will split into two clearly stated questions:

  1. How to access list of Tasks
  2. How scheduler should behave

PARTIAL ANSWER

1. What could cause this to happen?

I wasn't able to find out how the scheduler manages Tasks, but observations show that when the scheduler falls behind, the resulting Task scheduling isn't even close to fair. Some Tasks may be significantly delayed, and the bias does seem towards newer Tasks. For example, in one test with three identical Tasks that expect to get processing every 100 ms:

  • The most recent two were still running consistently every ~150 ms
  • The oldest one was delayed anywhere from 500 ms to 30 seconds

2. How would I go about debugging this? What object could I inspect to, for example, see a list of running async methods in a given context?

I wasn't able to figure out how to use object inspection to get a list of Tasks, but the easiest way to get a list of Tasks, or to see a queue. The "Parallel Stacks" or "Tasks" windows in VS2015 are great tools to see all Tasks, however limitations:

  • It does not list Tasks from async methods in Windows 7
  • It won't give you any information about queue or priority. You can always add your own timers to guess at it.
Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Denis P
  • 585
  • 2
  • 6
  • 17
  • In what context are you running the code? Is it a WPF-app, or webb? Can you reproduce the error in a unit test? – smoksnes Jun 02 '16 at 05:56
  • 2
    Also, I notice that you're not awaiting the MonitorAsync. This will swallow all Exceptions. Take a look at this question http://stackoverflow.com/questions/15522900/how-to-safely-call-an-async-method-in-c-sharp-without-await – smoksnes Jun 02 '16 at 06:00
  • 1
    "simplifies concurrency issues" - but looks like it's making for serious debugging issues. I'd suggest that if you *want* background activity, use `BackgroundWorker` or explicitly start threads and start *managing* your concurrency issues, rather than trying to fake concurrency by creating multiple tasks that are all competing to get access to the UI thread. – Damien_The_Unbeliever Jun 02 '16 at 06:21
  • 1
    @smoksnes - It is a WPF app. As noted in the description, it's definitely not throwing an exception, and that case is easy to cover without changing the caller to blocking (in fact, UnobservedTaskException is sufficient for a critical Monitor). That said, I do like the ContinueWith() pattern – Denis P Jun 02 '16 at 14:29
  • @Damien_The_Unbeliever Yes, the CPU bound work noted needs to be moved to another thread, and the underlying concurrency issues addressed. However, this Monitor doesn't do a lot of work, so it's not actually the source of the problem. The fact remains that, apparently, in edge circumstances, async methods can simply stop running, even after the root problem is gone. Maybe a freak occurrence happens in production code. So why does it happen, and what are debugging tools to, for example, get a list of running async methods? – Denis P Jun 02 '16 at 14:36
  • Are async methods managed as "Task" objects? Can I get a reference to the Task object? Do these Tasks have a priority? – Denis P Jun 02 '16 at 14:44
  • 2
    @denis the method returns a task. You're simply ignoring it. It will tell you if it completed, and any errors that caused it to fail. – Servy Jun 02 '16 at 14:56
  • @Servy. Of course, I realized that after posting. So I have one Monitor that stops running, and one that keeps running. I looked closer, in the watch window, the state of the "Task" objects for both look identical (other than the Id). As I noted in the description, I haven't been able to see a stack, but I know it's not stuck in an indefinite wait since I set a flag when it starts/stops the one await Delay. It would be helpful if I could see a list of Tasks on the context. Maybe older Tasks get starved out? – Denis P Jun 02 '16 at 15:30
  • @denis you don't need a list if all tasks, just look at the one task returned by this method and do something when it errors or ends to see what's going on. – Servy Jun 02 '16 at 16:08
  • @Servy As noted in my previous comment, I examined the Task that wasn't running anymore and nothing in it's state makes it looks like it stopped running or had an error. It's state looks identical to a Task that is still running. The objective behind seeing a list of all Tasks is 1) to see if it's still listed there as an active Task, 2) to see what other diagnostics might be available. – Denis P Jun 02 '16 at 17:47

2 Answers2

1

If you're not awaiting the task and never calling Wait() or Result or GetAwaiter() or ContinueWith() on it, how do you know that it even gets scheduled anywhere before it is garbage collected?

Does changing this line:

MonitorAsync(); // starts the Monitor

to this

MonitorAsync().ContinueWith(t => Console.WriteLine("monitor ended")); // starts the Monitor

cause it to start running?

Edit: This is probably not likely the problem - according to MSDN documentation "When an asynchronous method is called, it synchronously executes the body of the function up until the first await expression on an awaitable instance that has not yet completed, at which point the invocation returns to the caller.", i.e. no matter what you do with the Task, of course everything up until the first await point or the end of the method runs. I am conflating this with creating a Task with new Task(...) and then not scheduling it on a task scheduler. awaiting the Task.Delay inside should be enough to sign up the rest of the method as a continuation and ensure that it will be executed.

Jesper
  • 7,477
  • 4
  • 40
  • 57
  • Thanks for commenting. Moving the body of the async method to a continuation does result in the same behavior. The problem appears related to the scheduler. It seems that, in extreme circumstances, some of the Tasks (but not all) effectively stop being given time. – Denis P Jun 07 '16 at 18:56
  • So if you capture the `Task`: `var task = MonitorAsync();` and then check on its [`Status`](https://msdn.microsoft.com/en-us/library/system.threading.tasks.task.status(v=vs.110).aspx), which status does the task have immediately afterwards? If you start a timer and check on it periodically, does it ever move to any other status? – Jesper Jun 08 '16 at 08:00
  • Re-reading the question, I suppose my recommendation is to get off the UI context because that's where the congestion is. Kick off `MonitorAsync()` from inside a `Task.Run()` action (or use `ConfigureAwait(false)`). If you want things to run serially with no concurrency, you can create a [`ConcurrentExclusiveSchedulerPair`](https://msdn.microsoft.com/en-us/library/system.threading.tasks.concurrentexclusiveschedulerpair(v=vs.110).aspx) and use the exclusive scheduler to make sure no more than one task runs at a time. And if you do need to touch UI work, just schedule that work on the UI context – Jesper Jun 08 '16 at 08:07
  • I agree that getting off the UI context will avoid the problem. ConcurrentExclusiveSchedulerPair sounds like a good idea to simplify concurrency issues between certain complex Tasks. Thanks. I was still hoping for insight into how it works, since it seems like the scheduler isn't very fair when it's running behind. But perhaps that is the simple answer to the quesiton. – Denis P Jun 09 '16 at 20:30
  • A final note on that - be advised that if you Task.Run something even within a task scheduled on an exclusive scheduler, it will run on the thread pool, so if you don't await it, you may inadvertently cause your tasks to kick off work that keeps running into the next iteration. – Jesper Jun 13 '16 at 08:44
  • And a note on fairness. See [PreferFairness](http://blogs.msdn.com/b/pfxteam/archive/2009/07/07/9822857.aspx) - if you don't create tasks with this option, tasks further back in one thread's queue can be "stolen" by another thread, and thus execute out of order. – Jesper Jun 13 '16 at 08:47
0

I think there are only two options:

  1. The method throws an exception. You already said it doesn't, but maybe you could double check by running your application under a debugger while having "break when thrown" enabled for all CLR exceptions?
  2. The UI threads is so busy it never gets around to running your method when it should be resumed. Since this would also mean that the UI would stop responding and you didn't say that it does, this doesn't sound like a likely option to me.
svick
  • 236,525
  • 50
  • 385
  • 514
  • (1) Confirmed it is definitely not throwing an exception. (2) Is it possible that if the UI is "kind of busy" it would mostly work, and just stop running some Tasks? This is why I asked how can I debug this and get a list of all Tasks? – Denis P Jun 03 '16 at 18:48
  • @DenisP It's possible that executing a task would be delayed, but not skipped. – svick Jun 03 '16 at 19:34
  • This is one of the things that bothers me about WPF. Every once in a while, it seems like you can lose events. For example, I've actually seen it drop keystrokes typing into a TextBox. In the old win32 message pump days, that would never happen. All the key events went into a queue and you might be slow processing them, but you wouldn't miss any. – Denis P Jun 04 '16 at 23:04
  • added larger comment to original post. In summary, my theory is it's favoring the more recently created Tasks and so the older ones get delayed unfairly. Still not sure how to access a list of Tasks to confirm. – Denis P Jun 04 '16 at 23:13
  • I've confirmed that the Task (async method) that I thought stopped running, did not stop running. For example, I ran 3 instances of the same Task, and in the reproduced extreme scenario, 2 of them were running consistently but one of them took anywhere from 2s to 20s to come back from "await Task.Delay(100)". So the delaying of Tasks in an edge scenario seems not even close to fair. – Denis P Jun 07 '16 at 18:58