How does Apache Crunch pipeline generate map reduce jobs?

Question

I'm new to hadoop pipeline framework like Crunch/Cascading. I was wondering at the bottom of those framework, do they generate original mapper and reducer class , like original MapReduce program ? From the Crunch source code, I didn't find the code to convert the pipeline to original MapReduce classes.

Answer 1

You can visualize the MapReduce plan crunch has created using the below lines in your code. copy the DOT code created in pipeline.dot into graphviz to see plan.

String dot = pipeline.getConfiguration().get("crunch.planner.dotfile");
Files.write(dot, new File("pipeline.dot"), Charsets.UTF_8);