0

The app I'm developing is composed this way: A producer task scan the file system for text files and put a reference to them in a bag. Many consumer tasks take file refs from the bag concurrently and read the files (and do some short work with their content)

I must be able to pause and resume the whole process.

I've tried using TPL, creating a task for every file ref as they are put in the bag (in this case the bag is just a concept, the producer directly create the consumers task as it find files) but this way I don't have control over the task I create, I can't (or I don't know how to) pause them. I can write some code to suspend the thread currently executing the task but that will ruin the point of working with logical tasks instead of manully creating threads wouldn't it? I would want something like "task already assigned to phisical thread can complete but waiting logical tasks should not start until resume command"

How can I achive this? Can it be done with TPL or should I use something else?

EDIT: Your answers are all valid but my main doubt remains unanswered. We are talking about tasks, if I use TPL my producer and my many consumer will be tasks (right?) not threads (well, ok at the moment of the execution tasks will be mapped on threads). Every synchronization mechanism i've found (like the one proposed in the comment "ManualResetEventSlim") work at thread level.

E.g. the description of the Wait() method of "ManualResetEventSlim" is "Blocks the current thread until the current ManualResetEventSlim is set."

My knowledge of task is purely academic, I don't know how things works in the "real world" but it seem logical to me that I need a way to coordinate (wait/signal/...) tasks at task level or things could get weird... like... two task may be mapped on the same thread but one was supposed to signal the other that was waiting then deadlock. I'm a bit confused. This is why I asked if my app could use TPL instead of old style simple threads.

Lain
  • 2,166
  • 4
  • 23
  • 47
flagg19
  • 1,782
  • 2
  • 22
  • 27
  • 1
    Yes, you will have to create a pause mechanism yourself. Using e.g. a `ManualResetEventSlim` that the consumers repeatedly wait on, and that the producer sets/resets to pause/resume. – Alex Aug 20 '13 at 15:28
  • A normal `Task` always executes on a single thread. So any synchronization primitive that works on `Thread`s also works with `Task`s. – svick Aug 20 '13 at 17:15
  • @svick what if i start 100 tasks that execute the same code containing the Wait()? based on the accepted answer to this question http://stackoverflow.com/questions/14515207/whats-the-proper-way-to-use-a-threadpool 100 waiting thread will be created, is it good? – flagg19 Aug 21 '13 at 06:55
  • 1
    @flagg19 I think that answer is wrong (at least for .Net 4.5). But if you don't call `ThreadPool.SetMaxThreads()`, then yes, 100 threads will be eventually created. That's not ideal, so if you don't want that, you will have to do something. Either have delegates instead of `Task`s in the collection and then have a limited number of `Task`s that process them (you could use `Parallel.ForEach()` for that). Another option would be to use [a custom `TaskScheduler` with limited degree of parallelism](http://blogs.msdn.com/b/pfxteam/archive/2010/04/09/9990424.aspx). – svick Aug 21 '13 at 11:23
  • @svick thanks, i think i'll go with "have delegates instead of Tasks in the collection and then have a limited number of Tasks that process them" – flagg19 Aug 21 '13 at 14:17

2 Answers2

1

Yes, you can do that. First, you have a main thread, your application. There you have two workers, represented by threads. The first worker would be a producer and the second worker would be a consumer.

When your application starts, you start the workers. Both of them operates on the concurrency collection, the bag. Producer searches for files and puts references to the bag and consumer takes references from the bag and starts a task per reference.

When you want to signal pause, simply pause the producer. If you do that, consumer also stops working if there is nothing in the bag. If this is not a desired behaviour, you can simply define that pausing of the producer also clears the bag - backup your bag first and than clear it. This way all running tasks will finish their job and consumer will not start new tasks, but it can still run and wait for the results.

EDIT:

Based on your edit. I don't know how to achieve it the way you want, but although it is nice try to use new technologies, don't let your mind be clouded. Using a ThreadPool is also nice thing. It will take more time to start the application, but once it is running, consuming will be faster, because you already have workers ready.

It is not a bad idea, you can specify a maximum number of workers. If you create a task for every item in the bag, it will be more memory-consuming because you will still allocate and release memory. This will not happen with ThreadPool.

Ondrej Janacek
  • 12,486
  • 14
  • 59
  • 93
  • This is not exacly the answer i was looking for but thanks for the idea of clearing/backupping the bag. – flagg19 Aug 21 '13 at 06:57
  • Yes, from my point of view, it is a better solution. If you send a notification to pause the application, clear your bag and have your consumer thread pause itself using Thread.Sleep if there is nothing in the bag. But depends on your demands. With concurrency in the place, there are tons of solutions you can try to optimize your application. – Ondrej Janacek Aug 21 '13 at 08:08
0

Sure you can use TPL for this. And may be also reactive extensions and LINQ to simplify grouping and pausing/resuming the thread works.

If you have just a short job on each file, it is pretty good idea to not to disturb the handler function with cancellations. You can just suspend queueing the workers instead.

I imagine something like this:

  • You directory scanner thread puts the found files into an observable collection.
  • The consumer thread subscribes the collection changes and gets/removes the files and assigns them to workers.
Community
  • 1
  • 1
python_kaa
  • 1,034
  • 13
  • 27