Parallel tasks with a long pause

Question

I have a function which is along the lines of

   private void DoSomethingToFeed(IFeed feed)
   {
      feed.SendData(); // Send data to remote server
      Thread.Sleep(1000 * 60 * 5); // Sleep 5 minutes
      feed.GetResults(); // Get data from remote server after it's processed it
   }

I want to parallelize this, since I have lots of feeds that are all independent of each other. Based on this answer, leaving the Thread.Sleep() in there is not a good idea. I also want to wait after all the threads have spun up, until they've all had a chance to get their results.

What's the best way to handle a scenario like this?

Edit, because I accidentally left it out: I had originally considered calling this function as Parallel.ForEach(feeds, DoSomethingToFeed), but I was wondering if there was a better way to handle the sleeping when I found the answer I linked to.

@HansPassant - That's exactly what I'm trying to figure out how to avoid. — Bobson, Oct 24 '12 at 22:06
Can you use C# 5? And the question you linked to talks specifically about `Parallel.For()`, using `Thread.Sleep()` might be good enough in your case, depending on how exactly you use it and your other requirements. — svick, Oct 24 '12 at 23:28
@svick - It's the parallelizing sleep calls that I was referring to avoiding. Hmm. I thought I'd mentioned that I was already looking at `Parallel.ForEach()` when I found that thread, but I don't see it there. Must have never made it from my head to the keyboard. C# 5 is not an option, unfortunately. — Bobson, Oct 24 '12 at 23:36

score 3 · Answer 1 · edited May 23 '17 at 10:24

3

I think you should take a look at the Task class in .NET. It is a nice abstraction on top of more low level threading / thread pool management.

In order to wait for all tasks to complete, you can use Task.WaitAll.

An example use of Tasks could look like:

IFeed feedOne = new SomeFeed();
IFeed feedTwo = new SomeFeed();

var t1 = Task.Factory.StartNew(() => { feedOne.SendData(); });

var t2 = Task.Factory.StartNew(() => { feedTwo.SendData(); });

// Waits for all provided tasks to finish execution
Task.WaitAll(t1, t2);

However, another solution would be using Parallel.ForEach which handles all Task creation for you and does the appropriate batching of tasks as well. A good comparison of the two approaches is given here - where it, among other good points is stated that:

Parallel.ForEach, internally, uses a Partitioner to distribute your collection into work items. It will not do one task per item, but rather batch this to lower the overhead involved.

edited May 23 '17 at 10:24

Community

1
1

answered Oct 24 '12 at 22:04

Lasse Christiansen

10,205
7
50
79

So basically, "Too complicated for the simple `Parallel`, use `Task` instead"? – Bobson Oct 24 '12 at 22:07
1

Personally, I use `Parallel.ForEach` when I have to process a set of tasks (its faster than creating and starting multiple tasks on your own). In contexts where `Parallel.ForEach` is not suitable, I use `Task`s instead. – Lasse Christiansen Oct 24 '12 at 22:36
And how would you handle the waiting? – svick Oct 24 '12 at 23:28
If tasks are lengthy the thread-pool will be exhausted rather quickly, and you'll end up waiting a lot of time. Using the thread-pool in this scenario is not a good idea. – zmbq Oct 26 '12 at 07:45
@zmbq first of all - in the original the question it says: "I also want to wait after all the threads have spun up, until they've all had a chance to get their results." - and that is what I'm answering. In the edited question, it would be better for him to use some callbacks for each task and include some logic in that one to, e.g., fire an event when all tasks have finished. In that way you won't block anything and you'll get the result when it's ready. – Lasse Christiansen Oct 26 '12 at 10:13

score 3 · Accepted Answer · answered Oct 24 '12 at 22:22

3

Unless you have an awful lot of threads, you can keep it simple. Create all the threads. You'll get some thread creation overhead, but since the threads are basically sleeping the whole time, you won't get too much context switching.

It'll be easier to code than any other solution (unless you're using C# 5). So start with that, and improve it only if you actually see a performance problem.

answered Oct 24 '12 at 22:22

zmbq

38,013
14
101
171

So, basically, just let the threads all call `Thread.Sleep()`, and get paged out? – Bobson Oct 25 '12 at 18:38
Threads don't get "paged out", memory does. The threads will just sleep and won't participate in the context switching until they wake up. If you don't have thousands of them, this should work properly. – zmbq Oct 26 '12 at 07:44
This probably isn't the *best* answer, but I like it because it's simple. The sleeping threads won't block other threads from starting? – Bobson Oct 29 '12 at 20:29
There's no real limit on the number of threads in a process, so no. If you use a thread-pool you have a limit on the number of threads. – zmbq Oct 29 '12 at 20:47

score 1 · Answer 3 · answered Oct 24 '12 at 22:04

1

check WaitHandle for waiting on tasks.

answered Oct 24 '12 at 22:04

Mayank

8,777
4
35
60

How exactly would you use it in this case? – svick Oct 24 '12 at 23:29
as @LasseChristiansen explaind – Mayank Oct 25 '12 at 00:21

L.B · Answer 4 · 2012-10-25T14:20:20.537

-1

private void DoSomethingToFeed(IFeed feed)
{
    Task.Factory.StartNew(() => feed.SendData())
        .ContinueWith(_ => Delay(1000 * 60 * 5)
                          .ContinueWith(__ => feed.GetResults())
                     );
}

//http://stevenhollidge.blogspot.com/2012/06/async-taskdelay.html
Task Delay(int milliseconds)      
{
    var tcs = new TaskCompletionSource<object>();
    new System.Threading.Timer(_ => tcs.SetResult(null)).Change(milliseconds, -1);
    return tcs.Task;
}

edited Oct 25 '12 at 14:20

answered Oct 24 '12 at 22:35

L.B

114,136
19
178
224

Why does this answer the OP's question? Please explain it. – the Tin Man Oct 24 '12 at 22:50
@theTinMan I don't know, I just typed random chars. – L.B Oct 24 '12 at 22:55
On second thought, some explanation would be useful. Am I supposed to call this inside the `Parallel.ForEach()`? Do I do this from a regular `foreach`? Does it matter? – Bobson Oct 24 '12 at 23:16
@Bobson No need for `Parallel.ForEach` since this method already creates tasks (and the thread exexuting the `SendData` doesn't have to be the same to the one executing `GetResults`( – L.B Oct 24 '12 at 23:19
Why would you use `Task.Delay().Wait()`? If you're on .Net 4.5, you might as well use `await`. – svick Oct 24 '12 at 23:29
@svick So you are sure OP is using 4.5. Having other alternatives doesn't make this answer wrong. Other reasons to downvote? – L.B Oct 24 '12 at 23:33
@L.B - Yeah, I just realized that I'd forgotten to have the question say I'd started out by wrapping that function in a `Parallel.ForEach()`, before wondering if there was a better way. So I was looking at it in a context no one else knew anything about. Just calling it for each feed makes sense to me. (And no, I wish I could switch, but I'm still supporting XP and Server 2003.) – Bobson Oct 24 '12 at 23:39
@L.B You seem to be sure he is on 4.5, `Task.Delay()` is not there in .Net 4.0. Also, I don't see why are you using all those `ContinueWith()`, doing everything in a single `Task` would work the same, with less code and less overhead. – svick Oct 24 '12 at 23:45
@svick It seems that you don't read comments. `doing everything in a single Task would work the same` **:** With this code `SendData` and `GetResults` may be executed in different threads. – L.B Oct 24 '12 at 23:48
No `Task.Delay()` for me, sadly. Is there a 4.0 equivalent? – Bobson Oct 25 '12 at 13:27

Parallel tasks with a long pause

4 Answers4

Linked