3

Is there any Data Mining library, which is using (or can be used by) MPI (Massage Passing Interface)? I am looking for something similar to Apache Mahout but which can easily be integrated in a MPI environment.

The reason why I want to use MPI is that the configuration (compared to Hadoop) is easy. Or does it not make sense to use MPI in a Data Mining scenario?

Has QUIT--Anony-Mousse
  • 76,138
  • 12
  • 138
  • 194
mateuszk87
  • 143
  • 11

3 Answers3

4

There is no reason why MPI (which is a concept, not a software itself!) necessarily is easier to install than Hadoop/Mahout. Indeed, the latter two currently are a mess, in particular because of their Java library chaos. Apache Bigtop tries to make them easier to install, and once you've figured out some basics it's quite ok.

However:

  • If your data is small (i.e. it can be processed on a single node), don't install a cluster solution, you pay for the overhead. Hadoop does not make much sense on single hosts. Use Weka, ELKI, RapidMiner, KNIME or whatever.
  • If your data is large, you will want to minimize data transfer. And this is where the strength of Hadoop/Mahout lies, minimizing data transfer. A typical message passing API cannot scale the same way for data-heavy operations.

There are some efforts such as Apache Hama that are quite similar to MPI stuff IMHO. It is based on messages, however they are bulk-processed via barrier synchronization. It might also have some message aggregation prior to sending to reduce traffic.

Has QUIT--Anony-Mousse
  • 76,138
  • 12
  • 138
  • 194
  • Maybe there is some sort of misunderstanding - MPI is a standard and there are implementations like OpenMPI. I had something in mind like: http://www.ism.ac.jp/IWSML2012/r1.pdf . But I couldn't find such libraries yet :( – mateuszk87 Oct 04 '12 at 09:09
2

I strongly recommend graphlab. Currently graphlab, a Distributed Graph-Parallel API, has toolkits including

  • topic modeling
  • collaborative filtering
  • clustering
  • graphical model

http://docs.graphlab.org/toolkits.html

GraphLab is a graph-based, high performance, distributed computation framework written in C++. While GraphLab was originally developed for Machine Learning tasks, it has found great success at a broad range of other data-mining tasks; out-performing other abstractions by orders of magnitude.

GraphLab Features:

A unified multicore and distributed API: write once run efficiently in both shared and distributed memory systems Tuned for performance: optimized C++ execution engine leverages extensive multi-threading and asynchronous IO Scalable: GraphLab intelligently places data and computation using sophisticated new algorithms HDFS Integration: Access your data directly from HDFS Powerful Machine Learning Toolkits: Turn BigData into actionable knowledge with ease

Community
  • 1
  • 1
greeness
  • 15,956
  • 5
  • 50
  • 80
0

this idea doesn't make sense and I think you have some misconceptions, MPI is more for tightly coupled systems and i'm 99% sure won't send messages to an external location, you can however process or analyze the data with MPI much more quickly (depending on your hardware). My 2 cents is that you are better off using one of the AMQP protocol open source implementations ,I would say zeromq is your best bet and then processing all the data you get in R or python or if your data set is very very large MPI. Another option is that you can call serial libraries on different machines connected and running MPI given they all are connected to the internet seperately. R is real easy to call with MPI so is python.

Community
  • 1
  • 1
pyCthon
  • 11,746
  • 20
  • 73
  • 135