2

Im interested in programming a Project which distributes a certain computation on large files throughout several computers. The need for distributed computing arises from the crashy and unstable nature of the software I'm using to do the actual computing - so it might crash on some computers but others will surely do the job. The ideas I have so far Include: -Using several servers, each pulling a task from a master server whenever its possible -Using VMwares -Using a load-balancing Cluster

What is more suited for the job? Any other ideas I should be aware of?

Also, If you can recommend any reliable distributed computing C# framework, it will be helpful.

Hans
  • 21
  • 1
  • possible duplicate of http://stackoverflow.com/questions/1755180/distributed-computing-framework-net-specifically-for-cpu-instensive-operatio – Craig Stuntz Apr 22 '10 at 17:59
  • Im not interested in CPU intensive computations but rather in a mechanism that will ensure I get all of my computations done. – Hans Apr 25 '10 at 07:50

3 Answers3

1

haven't used any of these myself (yet), but I bookmarked this question a little while ago. some good suggestions there.

Community
  • 1
  • 1
toasteroven
  • 2,700
  • 3
  • 26
  • 35
0

Have you looked at Hadoop MapReduce? It's an open-source implementation of Google's MapReduce framework. Though it's Java and not C#, it sounds like it could be perfectly suited to your scenario; the master server automatically handles load balancing and fault-tolerance in a distributed environment.

tzaman
  • 46,925
  • 11
  • 90
  • 115
  • 1
    Map reduce is definitely the way to go for this. Implementations may be a bit harder to find on the .net framework though. – James Westgate Apr 22 '10 at 17:39
0

I would check out Appistry CloudIQ Platform. It links together multiple machines into a single computing framework identified by a unified address. Your client simply submits jobs to the unified address, and the framework distributes jobs to individual machines. It also monitors task execution, and can automatically restart failed jobs. So if your application is prone to crashing, this framework could be ideal. Rather than submitting the same job to multiple machines (and wasting CPU) to cover the failure case, just submit it once, and let the framework handle restarting the jobs that actually fail. I would consider it ideal for your reliability concerns.

Brett McCann
  • 2,469
  • 2
  • 27
  • 44