Implementing a MapReduce skeleton in Erlang

Question

I am fairly new to both parallel programming and the Erlang language and I'm struggling a bit.

I'm having a hard time implementing a mapreduce skeleton. I spawn M mappers (their task is to map the power function into a list of floats) and R reducers (they sum the elements of the input list sent by the mapper).

What I then want to do is to send the intermediate results of each mapper to a random reducer, how do I go about linking one mapper to a reducer? I have looked around the internet for examples. The closest thing to what I want to do that I could find is this word counter example, the author seems to have found a clever way to link a mapper to a reducer and the logic makes sense, however I have not been able to tweak it in order to fit my particular needs. Maybe the key-value implementation is not suitable for finding the sum of a list of powers?

Any help, please?

Answer 1

Just to give an update, apparently there were problems with the algorithm linked in the OP. It looks like there is something wrong with the sychronization protocol, which is hinted at by the presence of the call to the sleep() function (ie. it's not supposed to be there).

For a good working implementation of the map/reduce framework, please refer to Joe Armstrong's version in the Programming Erlang book (2nd ed).

Armstrong's version only uses one reducer, but it can be easily modified for more reducers in order to eliminate the bottleck. I have also added a function to split the input list into chunks. Each mapper will get a chunk of data.