23

I am currently researching the options that are available (both Open Source and Commercial) for developing a distributed application.

"A distributed system consists of multiple autonomous computers that communicate through a computer network." Wikipedia

The application is focused on distributing highly cpu intensive operations (as opposed to data intensive) so I'm sure MapReduce solutions don't fit the bill.

Any framework that you can recommend ( + give a brief summary of any experience or comparison to other frameworks ) would be greatly appreciated.

Andrey Rubshtein
  • 20,795
  • 11
  • 69
  • 104
holsee
  • 1,974
  • 2
  • 27
  • 43
  • I am actively working on this subject so I've no shame about adding my own answers to this question as well, all for the greater good! – holsee Nov 18 '09 at 13:04
  • I have made all my answers Community Wiki (seems only right) in order to allow further notes of interest to be added as the community sees fit. – holsee Nov 24 '09 at 17:50
  • So a few years down the road, is MPAPI still the correct answer you think? – Andreas Jan 09 '13 at 12:55
  • Andreas, I'm not sure tbh. MPAPI still works, probably better options out there now. It would not be hard to roll your own, that's what I would do faced same problem now. It is just work scheduling and message passing. – holsee Feb 01 '13 at 08:31
  • 2
    I think this shouldn't be closed. as it's a real question. – ariso Feb 04 '16 at 03:03
  • akka.net is only good solution that i found so for – Gomes Aug 08 '16 at 20:04

8 Answers8

10

MPAPI Framework - http://www.codeplex.com/MPAPI

My Thoughts

  • Seems to be one of the simpler solutions.
  • I see no evidence of failover support.
  • Is Open Source =] and is MONO Compatible YAY!
  • Uses the "RemotingLite" framework hense the mono compatibility.
  • Writing single-computer or multiple-computer (cluster) applications are as equally simple.
  • Developer's philosophy is "Performance is Paramount". So its safe to say he has optimised.
  • Continuously updated framework.
holsee
  • 1,974
  • 2
  • 27
  • 43
  • I really like MPAPI. It did nearly everything I wanted but not all, so I created [DuoVia.MpiVisor](http://duovia.net/using-mpivisor-code-sample/) inspired by Frank's work. For me it's easier to use but that is probably because I wrote it. ;-) – Tyler Jensen Apr 24 '13 at 16:14
4

I was one of the lead developers on Alchemi. We've had to change course are start fresh, and the result is the Utilify Platform. Conceptually similar to Alchemi, the Utilify platform builds on the lessons learnt from Alchemi and provides a more flexible programming model to build distributed .NET applications on a Windows network.

Please visit www.utilify.com for more information. Currently, there is a free beta version available for download.

Regards, Krishna.

Krishna
  • 2,997
  • 1
  • 23
  • 31
  • I tried to install Utilify Manager and got the error. So I could not proceed further. The error said the package is damaged. – dotcoder Jan 04 '12 at 10:06
3

Appistry CloudIQ Platform http://www.appistry.com

  • Will make hundreds of machines appear as one computing resource.
  • Currently used in production at FedEx to run routing algorithms
  • Allows you to write apps in C,C++, Java and .Net
  • No single point of failure
  • Auto recover/retry of failed tasks
Brett McCann
  • 2,469
  • 2
  • 27
  • 44
  • 2
    How's this work specifically? Is it like a network traffic load balancer? We used appliedalgo.com - which does scheduling, execution management (persistence+tracking of execution history/parameters/results), load balancing. Their load balancing implementation calculates "Host Score" based on CPU/Memory usage and Disk Activity (With override too), and simple "Extension API" accessible from both .NET and Java clients. Most importantly, it's a completely standalone server app with its own GUI so you can recall execution param and results and make quick comparisons. – Swab.Jat Jan 02 '14 at 07:04
  • 1
    About appliedalgo.com, you said they supports both Java and .NET but it only runs on Windows? Also, what about C++? And Linux? –  Jan 02 '14 at 07:06
  • 1
    Our firm only has jobs coded in .NET, Java and some in SQL, nothing in C++ and I'm not sure if they supports clients in C++. With AppliedAlgo, you code your batches via "Extension API" - WCF based interface exposed via WsHttpBinding for .NET and BasicHttpBinding for Java. There's no mention of C++ however. Further, their load balancing works by distributing to connected nodes (NodeController, which only runs on Windows, not Linux) – Swab.Jat Jan 02 '14 at 07:16
  • 2
    No I think perhaps you're using an earlier version. I says on their website they do support "unmanaged C++", but yes Windows only. –  Jan 17 '14 at 03:57
  • 1
    This may be helpful - https://appliedalgo.com/appliedalgoweb/Doc/CompetitiveAnalysis/AppliedAlgo.Scheduler_LoadBalancer.FeatureComparison.htm – Swab.Jat Jan 21 '14 at 11:44
  • 2
    Applied Algo ETL Suite exposes "Extension API" over WCF (Any job built in language which can talk to WCF can communicate with it) and RabbitMQ (RabbitMQ has API for many languages such as Perl/Python...etc). They bundled sample for unmanaged C++/.NET and Java. But it's not limited to these. –  Feb 08 '14 at 16:36
2

Alchemi ".NET Grid Computing Framework"

http://www.gridbus.org/~alchemi/publications.html

http://sourceforge.net/projects/alchemi/files/

Alchemi is a .NET grid computing framework that allows you to painlessly aggregate the computing power of intranet and Internet-connected machines into a virtual supercomputer (computational grid) and to develop applications to run on the grid.

Quick Notes:

  • Stopped being updated in 2007 version 1.0.6 (.NET 2.0).
  • Well documented formally done so.
  • Uses XML to define jobs. (It was all the rage in 2007 to use XML for everything)
  • Relatively complex compared to MPAPI, but don't take this as a big negative.
  • Would take a good bit of time to get up to speed.
  • Is Open Source =).
  • Is Extendible and can be used in conjunction with other cluster technologies via a Gridbus broker (http://www.gridbus.org)
  • Seems widely used (see publications http://www.gridbus.org/~alchemi/publications.html) =)
holsee
  • 1,974
  • 2
  • 27
  • 43
1

Azure Grid - http://azuregrid.codeplex.com/

Jakub Šturc
  • 35,201
  • 25
  • 90
  • 110
holsee
  • 1,974
  • 2
  • 27
  • 43
1

DryadLINQ is a Microsoft Research project to allow LINQ queries to be distributed among multiple computers, rather than just multiple cores as Parallel LINQ does.

Blog post with a brief example.

Joel Mueller
  • 28,324
  • 9
  • 63
  • 88
0

NGrid - http://ngrid.sourceforge.net/

Overview

NGrid is an open source (LGPL) grid computing framework written in C#.

  • platform independent via the Mono project.
  • a transparent multithread programming model for grid programming.
  • a physical grid framework & some grid implementations.
  • common utilities both for grid programming or grid implementations.
  • last updated May 2008
John Rayner
  • 3,485
  • 21
  • 13
holsee
  • 1,974
  • 2
  • 27
  • 43
0

AWS SDK for .NET - http://aws.amazon.com/sdkfornet/

I think this cloud offering and .NET api may also fit the bill. Amazon offer high performance machines (see price list) which would allow for delegation a execution of highly cpu intensive opperations on a large scale.

holsee
  • 1,974
  • 2
  • 27
  • 43