PIG script: reducer preempted to make room for pending map attempts

Question

I run multiples instances of the same PIG script with different parameters.

When the cluster is under heavy load, Map Reduce jobs logs show lots of reducers preempted, which takes times:

Reducer preempted to make room for pending map attempts Container killed 
by the ApplicationMaster. Container killed on request. Exit code is 143 
Container exited with a non-zero exit code 143 

How to avoid such preemptions issue to minimize script execution time ?

Answer 1

Have a look at this link.

In short, you can use one of below commands

SET mapred.reduce.slowstart.completed.maps 1.00;
SET mapreduce.job.reduce.slowstart.completedmaps 1.00;

depending on your Hadoop version (latter on is for Hadoop 2.4.1+). It will start reducers once all mappers art done.