17

So let's say I have some computers at home.

Let's also say that I have some algorithm I want to run, that generally takes a lot of time to solve. It can be divided in how many parts I want, so I could run part of it in one machine, part of it in another, etc, and in the end i'd just need to merge the results in one of the computers.

My question is if there is any easy and straightforward way in .NET of making use of several computers to do this kind of computations. If yes, how is it called? I don't mean having to code all the IPC code by myself, something similar to BCL's Tasks but that'd allow me to send "work" to other computers, via an IP or something.

Thanks!

devoured elysium
  • 101,373
  • 131
  • 340
  • 557

4 Answers4

7

In general distributed computing, there are two models: Message Passing and Shared Memory

It sounds like you're interested in Message Passing, because it typically involves multiple computers which communicate over the network. Shared memory uses the memory system to communicate between processes. From my (limited) experience in distributed computing, almost everyone who uses Message Passing as their model has moved to MPI. MPI is a framework that has been implemented in many languages. There are some pretty strong .NET MPI implementations out there.

This whole post assumes you're doing "distributed computing" in a HPC scenario. In such a scenario, you would have many powerful computers, connected over an inter-connection network, in a server room (or similar). If you're looking for some kind of distributed computing between seperated physical machines (such as folding@home), you'll need to find something else or roll your own solution. Therefore, as Leniel Macaferi mentions in his response, you'll need to ensure your data center is running some version of Microsoft HPC Server. Most HPC clusters run some flavor of linux, which will obviously prevent you from deploying a .NET based solution.

Jared Harding
  • 4,942
  • 2
  • 17
  • 14
  • So Microsoft HPC Server runs on Linux? And how would that prevent you from deploying something .Net based? there is always mono – Earlz Jan 09 '11 at 22:26
  • 1
    I'm saying that most HPC clusters run linux, so you need to check with the administrator of your cluster. You need to install mono or get allocated some time on a Microsoft HPC Server cluster to run a .NET solution on an HPC cluster. Microsoft HPC Server only has about 1% marketshare in the HPC space from the numbers I've seen, so it's somewhat of a rarity that you'll find an HPC cluster set up to run a .NET MPI solution without doing some communication with the administrators. – Jared Harding Jan 09 '11 at 22:43
3

That's a perfect match for Microsoft HPC Server distro as it is tuned specially to address this kind of situation.

I don't know anything that can do it purely based on .NET. That's possible of course, but it would require some coding.

Although not what you want, I'd recommend you read about the .NET Task Parallel Library that helps you divide your workload automatically across the computer's available CPUs (cores) quickly and painlessly.

Interesting articles related to distributed computing:

C# Remoting and Distributed Computing

Legion: Build your own virtual super computer with Silverlight

Windows Communication Foundation (WCF)

Leniel Maccaferri
  • 100,159
  • 46
  • 371
  • 480
  • 1
    If you already have the algorithm running in .NET it shouldn't take to long to port it to WCF and run it on as many machines as you like and then just have have some master (server) responsible for sending the "jobs" and combine all the results back.You could then have the clients all running .net Parallel Library – Gaven Jan 09 '11 at 22:36
1

Microsoft has in their research labs a Map reduce implementation of Linq called Dryad Linq (currently the beta is only available on HPC servers) but this looks very interesting and does what you are looking for.

Tim Jarvis
  • 18,465
  • 9
  • 55
  • 92
0

Microsoft HPC Server requires a server OS. From the sounds of it, you are not running a server OS. Assuming you are running a desktop version of Windows with .NET 4.0 installed, you could try one of these options:

  1. You could manually split up the work and execute the pieces manually.

  2. You could use something MPAPI and code up your own nodes and workers.

  3. You could write a console application and use DuoVia.MpiVisor to distribute and execute on your workstations. (Full disclosure: I am the author of MpiVisor)

Doing distributed computing in a casual fashion without a lot of fuss is not as easy as one might thing, but either of these two options should get you there.

Tyler Jensen
  • 795
  • 4
  • 9