1
public void setup(Context context) throws IOException, InterruptedException{
            Configuration conf = context.getConfiguration();
            org.apache.hadoop.mapreduce.Cluster cluster = new org.apache.hadoop.mapreduce.Cluster(conf);
            Job currentJob = cluster.getJob(context.getJobID());
            mapperCounter = currentJob.getCounters().findCounter(TEST).getValue();  
        }

I wrote the following code to get the value of a counter that I am incrementing in my mapper function. The problem is that the currentJob returned by cluster.getJob is turning out to be null. Does anyone know how I can fix this?

My question is different cause I am trying to access my counter in the reducer not after all the map reduce tasks are done. This code that I have pasted here belongs in my reducer class.

Aditya
  • 1,240
  • 2
  • 14
  • 38
  • I don't have much experience with `org.apache.hadoop.mapreduce.Cluster`, but are you sure the job is actually running? – Ben Watson Dec 14 '15 at 20:26
  • I am not sure I get what you are saying, but I am running this code inside of my reducer task which is inside of the cluster's job task so the job must be running right? – Aditya Dec 14 '15 at 20:30
  • I'm assuming you got this code from http://diveintodata.org/2011/03/15/an-example-of-hadoop-mapreduce-counter/, but it does say that code is only for finished jobs. I have never retrieved a Counter that way. There's always a certain amount of trial and error with Counters thanks to the icky new API and its lack of transparency, but might I suggest looking at http://lintool.github.io/Cloud9/docs/content/counters.html? – Ben Watson Dec 14 '15 at 20:39
  • Possible duplicate of [How to access Hadoop counters values via API?](http://stackoverflow.com/questions/27325536/how-to-access-hadoop-counters-values-via-api) – Ben Watson Dec 14 '15 at 20:40

1 Answers1

0

It seems that cluster.getJob(context.getJobID()); does not work in hadoop's Standalone Operation.

Try running your Program with YARN in hadoop's Single Node Cluster mode like described in the documentation: https://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/SingleCluster.html#Pseudo-Distributed_Operation

dreua
  • 527
  • 2
  • 15