-1

This question talks about how the traditional queue pattern is somewhat antiquated in modern C# due to the TPL: Best way in .NET to manage queue of tasks on a separate (single) thread

The accepted answer proposes what appears to be stateless solution. It is very elegant, but, and perhaps I'm a dinosaur or misunderstand the answer... What if I want to pause the queue or save its state? What if when enqueuing a task the behaviour should be dependent on the queue's state or if queued tasks can have different priorities?

How could one efficiently implement an ordered task queue - that actually has an explicit Queue object which you can inspect and even interact with, within the Task paradigm? Supporting single/parallel processing of enqueued tasks is a benefit but for my purposes, single-concurrency is acceptable if it raises problems. I am not dealing with millions of tasks a second, in fact my tasks are typically large/slow.

I am happy to accept there are solutions with different scalability depending on requirements, and that we can often trade-off between scalability and coding effort/complexity.

Mr. Boy
  • 60,845
  • 93
  • 320
  • 589
  • 1
    A single queue is inherently not an efficient solution to running large amounts of short-lived tasks on today's multiprocessor systems, though -- which is why the framework's default task scheduler has multiple, work-stealing queues. You are of course quite free to cook up your own implementation of the concept with a `ConcurrentQueue` as the backing mechanism, complete with mechanisms for persistence, priorities and a `CancellationToken`-like structure for cooperative pausing, but I think such a thing exceeds the scope of an answer. TANSTAAFL. – Jeroen Mostert Jan 29 '20 at 13:54
  • The documentation for [`TaskScheduler`](https://learn.microsoft.com/dotnet/api/system.threading.tasks.taskscheduler) lists a sample implementation of a scheduler that does something different from the default, and a lot of background information on the default implementation. Writing your own fully customized scheduler that plays nicely with `Task`s can be problematic with external code, though, since it's hard to force everything to use your scheduler and not the default one. – Jeroen Mostert Jan 29 '20 at 13:57
  • @JeroenMostert I mean if a more traditional approach is the logical solution to my requirement I'm happy with that, some assurance this is actually OK is nice in fact because everywhere I look I'm told I MUST make everything super-parallel. I would ask, why is it assumed my tasks are large in number and small in complexity? Should I clarify that in the question? – Mr. Boy Jan 29 '20 at 13:58
  • I get that adding/removing items from a single collection requires a lock but most of us aren't writing hugely parallel code on Big Data. It irks me we are encouraged to use a sledgehammer to crush a nut – Mr. Boy Jan 29 '20 at 14:01
  • It's assumed because this is the main use case for `Task` in the framework -- or rather, `async` / `await`, which uses `Task`, is expected to give code that will have many short-lived continuations for things like asynchronous I/O, which will run for just a short while before they go into another operation again -- and there will be many such `Task`s because this approach scales quite well (as opposed to one thread per task). You can of course have long-lived `Task`s, but it's telling that tasks created with `TaskCreationOptions.LongRunning` just commandeer a thread pool thread. – Jeroen Mostert Jan 29 '20 at 14:02
  • 1
    If you are neither married to `async` / `await` nor `Task` in your actual code, custom code that just takes and queues `Action`s may conceptually be simpler. Nobody's forcing you to use `Task`, after all. – Jeroen Mostert Jan 29 '20 at 14:03
  • @JeroenMostert maybe this is the confusion then because MS' docs explicitly say TPL is the way we are supposed to approach all multithreaded code – Mr. Boy Jan 29 '20 at 14:03
  • 1
    The advice is no doubt geared towards weaning people away from naive, one `Thread` per task code with explicit synchronization and the accompanying bugs. Using `async` / `await` is obviously superior to that and appropriate for the majority of use cases, and using `Task` with completely custom scheduling is not *wrong*, per se, just perhaps a bit more clumsy than necessary. There is never a silver bullet, no matter what any docs say. – Jeroen Mostert Jan 29 '20 at 14:11
  • Sounds like a producer/consumer kind of process would work for you. Think Channels (see answer below) or [blocking collections](https://learn.microsoft.com/en-us/dotnet/standard/collections/thread-safe/blockingcollection-overview) or even [TPL Dataflow](https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/dataflow-task-parallel-library) – Peter Bons Jan 29 '20 at 14:14

1 Answers1

0

What you are describing sounds essentially like the Channel<T> API. This already exists:

it isn't explicitly a Queue<T>, but it acts as a queue. There is support for bounded vs unbounded, and single vs multiple readers/writers.

Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
  • 1
    The OP wants to interact with the queue, meaning (I suppose) that he wants to be able to add/remove items arbitrarily from the queue, inspect the queue size, perform [Peek](https://learn.microsoft.com/en-us/dotnet/api/system.collections.queue.peek) etc. These capabilities are not supported by the [Channel](https://learn.microsoft.com/en-us/dotnet/api/system.threading.channels.channel-1) class. – Theodor Zoulias Jan 30 '20 at 01:04