1

I have seen several questions related to this in stackoverflow but they did n't solve my problem.

While running job with 90 Mb file i am getting LeaseExpiredException

13/11/12 15:46:41 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/11/12 15:46:42 INFO input.FileInputFormat: Total input paths to process : 1
13/11/12 15:46:43 INFO mapred.JobClient: Running job: job_201310301645_25033
13/11/12 15:46:44 INFO mapred.JobClient:  map 0% reduce 0%
13/11/12 15:46:56 INFO mapred.JobClient: Task Id : attempt_201310301645_25033_m_000000_0, Status : FAILED
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): Lease mismatch on /user/hdfs/in/map owned by DFSClient_NONMAPREDUCE_-1622335545_1 but is accessed by DFSClient_NONMAPREDUCE_-1561990512_1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2459)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2437)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:2503)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2480)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:556)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:337)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44958)
    at org.
attempt_201310301645_25033_m_000000_0: SLF4J: Class path contains multiple SLF4J bindings.
attempt_201310301645_25033_m_000000_0: SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201310301645_25033_m_000000_0: SLF4J: Found binding in [jar:file:/tmp/hadoop-mapred/mapred/local/taskTracker/hdfs/jobcache/job_201310301645_25033/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201310301645_25033_m_000000_0: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
13/11/12 15:47:02 INFO mapred.JobClient: Task Id : attempt_201310301645_25033_m_000000_1, Status : FAILED
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): Lease mismatch on /user/hdfs/in/map owned by DFSClient_NONMAPREDUCE_-1622335545_1 but is accessed by DFSClient_NONMAPREDUCE_-1662926329_1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2459)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2262)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2175)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:299)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44954)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java
attempt_201310301645_25033_m_000000_1: SLF4J: Class path contains multiple SLF4J bindings.
attempt_201310301645_25033_m_000000_1: SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201310301645_25033_m_000000_1: SLF4J: Found binding in [jar:file:/tmp/hadoop-mapred/mapred/local/taskTracker/hdfs/jobcache/job_201310301645_25033/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201310301645_25033_m_000000_1: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
attempt_201310301645_25033_m_000000_1: log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient).
attempt_201310301645_25033_m_000000_1: log4j:WARN Please initialize the log4j system properly.
attempt_201310301645_25033_m_000000_1: log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
13/11/12 15:47:10 INFO mapred.JobClient: Task Id : attempt_201310301645_25033_m_000001_0, Status : FAILED
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hdfs/in/map: File is not open for writing. Holder DFSClient_NONMAPREDUCE_-1622335545_1 does not have any open files.
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2452)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2437)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:2503)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2480)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:556)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:337)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44958)

Why is it so? First part of my mapper code is 

public void map(Object key, Text value, Context context)
            throws IOException, InterruptedException {

Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);

 Path inputfile = new Path("in/map");
 BufferedWriter getdatabuffer = new BufferedWriter(new OutputStreamWriter(fs.create(inputfile)));
   getdatabuffer.write(value.toString());
   getdatabuffer.close();
Path Attribute = new Path("in/Attribute");
        int row =0;
        BufferedReader read = new BufferedReader(new InputStreamReader(fs.open(inputfile)));
        String str = null;
        while((str = read.readLine())!=null){

         row++;                     //total row count
         StringTokenizer st =new StringTokenizer(str," ");
         col = st.countTokens();
          }
        read.close();
...........
...........
.............
............

Further computation is based on the above "map" file.

Why this happens? I think it is unable to write into in/map for several times.

And I am not deleting any of the file. How to get rid of this?

Any Suggestions?

EDIT : Nov 15

When i examined this in/map is not created in my cluster. Why it is not getting created? I think that is why it is getting a LeaseExpiredException

Scenario

I am having a inputfile of 1GB.The contents will be like

file1.txt
0 0 6
3 4 8
5 9 3
12 4 6
8 7 8
9 8 1
6 12 0
10 8 0
8 5 1
14 8 1

I need to find out Atranspose*A where A[][] is the input data from the file.

So my logic is like:

What ever data is coming into mapper i will be finding Atranspose*A and in Reducer i will be calculating the sum of all Atran*A computed from each mapper. So that i will be able to get Atranspose *A of a file1.txt.

For that i thought of writing the each mapper data into a file and then get then into A[][] array and find ATrn*A.

USB
  • 6,019
  • 15
  • 62
  • 93

1 Answers1

2

Looks like you have potentially multiple mappers trying to write to the same file in HDFS:

Path inputfile = new Path("in/map");
BufferedWriter getdatabuffer = new BufferedWriter(new OutputStreamWriter(fs.create(inputfile)));
getdatabuffer.write(value.toString());
getdatabuffer.close();

If you have more than a single map task executing this code, you're going to have the issues you're seeing.

You shouldn't really write to HDFS directly in your mapper code, and the logic you have will overwrite the file in every mapper, for each input value - can you explain what you're trying to achieve?

Chris White
  • 29,949
  • 4
  • 71
  • 93
  • Thx for ur reply Chris White – USB Nov 14 '13 at 04:51
  • For each input value i am doing the same thing. I will explain: – USB Nov 14 '13 at 05:07
  • I am facing same kind of issue. Are you able to find any workaround? As per Chris, we can't write directly to HDFS, if we have multiple mapper involved. Is there any workaround? I have lot of zip compressed files and I am trying to decompress them and write it into HDFS. How we can implement this? I am wrote map only job. http://stackoverflow.com/questions/32799677/hadoop-leaseexpiredexception – Ajay Oct 21 '15 at 08:40
  • @Chris I am also facing similar issue, but not able to understand as per what logs is explaining. I also moved logs to DEBUG mode. Here is question link - http://stackoverflow.com/questions/39547001/why-hive-staging-file-is-missing-in-aws-emr http://stackoverflow.com/questions/39619130/why-hdfs-throwing-leaseexpiredexception-in-hadoop-cluster-aws-emr Please suggest some way to resolve this, I spent days on this, but failed to debug. – devsda Sep 22 '16 at 12:21