12

Is there any way to use Task Parallel Library in multi computer scenarios ?

I mean if i have huge number of tasks , can i schedule it over LAN in number of servers ?

Theodor Zoulias
  • 34,835
  • 7
  • 69
  • 104
TalentTuner
  • 17,262
  • 5
  • 38
  • 63

3 Answers3

7

The TPL is geared towards single computer, multiple processor core scenarios.

If you want to work across multiple systems, you'll need to use some type of clustering software, such as MPI (usable in .NET directly via MPI.NET) or one of the many options based on Windows HPC.

That being said, the TPL is very useful on each of the nodes of the cluster. It can be used to have each cluster node scale well across the cores available on that node.

Conrad
  • 2,197
  • 28
  • 53
Reed Copsey
  • 554,122
  • 78
  • 1,158
  • 1,373
  • FWIW I do remember hearing at a BUILD conference that (at least at the time) the plan was to eventually introduce "agents" that would allow for distributed tasks. However, I'm having trouble finding anything on that now... – Jeff May 28 '13 at 17:03
  • @Jeff Were you thinking of TPL Dataflow (https://nuget.org/packages/Microsoft.Tpl.Dataflow), perhaps? Or maybe the Azure Actor Framework? https://actorfx.codeplex.com/ – Reed Copsey May 28 '13 at 17:12
  • Funny, someone pointed out the Actor fx at work today. I think that must have been what I was thinking of, thanks! – Jeff May 31 '13 at 00:23
  • @ReedCopsey I think we can use akka.net to run process into multiple systems using TPL also. https://blog.jayway.com/2013/11/15/an-actor-model-implementation-in-c-using-tpl-dataflow/#comment-460430. Reed any suggestions? – virender Aug 28 '17 at 07:04
  • 2
    @virender Akka.Net would be my suggestion today (this was 7 years old) - TPL Dataflow is single computer/proc, though. – Reed Copsey Aug 28 '17 at 18:17
3

No TPL focuses on local threads within a process. There are however existing projects that do tackle this area.

http://research.microsoft.com/en-us/projects/dryad/

And you can take a look at the answers to this SO question

Any good distributed agent/service models for .NET?

Community
  • 1
  • 1
Chris Taylor
  • 52,623
  • 10
  • 78
  • 89
3

The default settings for TPL is to schedule tasks on the .NET thread pool and as such this is aimed at parallelism within a single process. However, you can implement your own TaskScheduler, which handles how tasks are actually run, so at least in theory you could move beyond the current scope. While this could be done, I am sure there are better options available as Reed Copsey and Chris Taylor point out.

Brian Rasmussen
  • 114,645
  • 34
  • 221
  • 317