I've found this discussion: MongoDB: Terrible MapReduce Performance. Basically it says try to avoid Mongo's MR queries as it single-threaded and not supposed to be for real-time at all. 2 years has passed, and I wonder what has been changed since the time. Now we have MongoDb 2.2. I heard MRs are now multi-threaded. Please share your ideas over MR usage for real-time requests like fetching data for web application frequent http requests. Is it able to effectively use indexes?
Asked
Active
Viewed 9,552 times
29
-
Meybe aggregation framework will be change something, right now map reduce is dam slow – user956584 Oct 01 '12 at 18:13
-
Any idea why is that? Is it still single-threaded? – YMC Oct 01 '12 at 18:17
-
1http://stackoverflow.com/a/9434093/979474 or http://stackoverflow.com/a/11749738/979474 – user956584 Oct 01 '12 at 18:37
1 Answers
56
Here is the current state of functionality for Map/Reduce in MongoDB
1) Most of the performance limitations for Map/Reduce still remain in MongoDB version 2.2. The Map/Reduce engine still requires that every record get converted from BSON to JSON, the actual calculations are performed using the embedded JavaScript engine (which is slow), and there still is a single global JavaScript lock, which only allows a single JavaScript thread to run at a single time.
There have been some incremental improvements to Map/Reduce for sharded clusters. Most notably, the final Reduce operation is now distributed across multiple shards, and the output is also sharded in parallel.
I would not recommend Map/Reduce for real-time aggregation in MongoDB version 2.2
2) Starting with MongoDB 2.2, there is now a new Aggregation Framework. This is a new implementation of aggregation operations, written in C++, and tightly integrated into the MongoDB framework.
Most Map/Reduce jobs can be rewritten to use the Aggregation Framework. They usually run faster (20x speed improvement vs. Map/Reduce is common in version 2.2), they make full use of the existing query engine, and you can run multiple Aggregation commands in parallel.
If you have real-time aggregation requirements, the first place to start is with the Aggregation Framework. For more information about the aggregation framework, take a look at these links:
- http://www.10gen.com/presentations/mongonyc-2012/new-aggregation-framework
- http://docs.mongodb.org/manual/reference/aggregation/
3) There have been significant improvements in Map/Reduce in MongoDB version 2.4. The SpiderMonkey JavaScript engine has been replaced by the V8 JavaScript engine, and there is no longer a global JavaScript lock, which means that multiple Map/Reduce threads can run concurrently.
The Map/Reduce engine is still considerably slower than the aggregation framework, for two main reasons:
The JavaScript engine is interpreted, while the Aggregation Framework runs compiled C++ code
The JavaScript engine still requires that every document being examined get converted from BSON to JSON; if you're saving the output in a collection, the result set must then be converted from JSON back to BSON
There are no significant changes in Map/Reduce between 2.4 and 2.6.
I still do not recommend using the Map/Reduce for real-time aggregation in MongoDB version 2.4 or 2.6.
4) If you really need Map/Reduce, you can also look at the Hadoop Adaptor. There's more information here:
William Z
- 10,989
- 4
- 31
- 25
-
3The upcoming 2.4 version has another javascript engine (V8) that should improve performance. For details see here: http://docs.mongodb.org/manual/release-notes/2.4/ – Kamarey Feb 26 '13 at 08:52