Override hadoop

Question

I'm running an EMR Activity inside a Data Pipeline analyzing log files and I get the following error when my Pipeline fails:

Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://10.211.146.177:9000/home/hadoop/temp-output-s3copy-2013-05-24-00 already exists at org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:121) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:944) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:905) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:905) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:879) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1316) at com.valtira.datapipeline.stream.CloudFrontStreamLogProcessors.main(CloudFrontStreamLogProcessors.java:216) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:187)

I've tried delete that folder by adding:

FileSystem fs = FileSystem.get(getConf()); fs.delete(new Path("path/to/file"), true); // delete file, true for recursive

but it does not work. Is there a way to override the FileOutputFormat method from Hadoop in java? Is there a way to ignore this error in java?

possible duplicate of [Deleting file/folder from Hadoop](http://stackoverflow.com/questions/16797358/deleting-file-folder-from-hadoop) — Charles Menguy, May 29 '13 at 19:29
Yes, but that does not work. Any ideas on how to ignore the error? — cevallos.valtira, May 29 '13 at 19:49

Tejas Patil · Accepted Answer · 2013-05-30T01:10:06.283

The path to the file to be deleted changes as the output directory uses date for naming. There are 2 ways to delete it:

Over shell, try this:

hadoop dfs -rmr hdfs://127.0.0.1:9000/home/hadoop/temp-output-s3copy-*

To do it via Java code:

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FileUtil;
import org.apache.hadoop.fs.Path;
import org.mortbay.log.Log;

public class FSDeletion {

  public static void main(String[] args) {

    try {
      Configuration conf = new Configuration();
      FileSystem fs = FileSystem.get(conf);

      String fsName = conf.get("fs.default.name", "localhost:9000");
      String baseDir = "/home/hadoop/";
      String outputDirPattern = fsName + baseDir + "temp-output-s3copy-";

      Path[] paths = new Path[1];
      paths[0] = new Path(baseDir);

      FileStatus[] status = fs.listStatus(paths);
      Path[] listedPaths = FileUtil.stat2Paths(status);
      for (Path p : listedPaths) {
        if (p.toString().startsWith(outputDirPattern)) {
          Log.info("Attempting to delete : " + p);
          boolean result = fs.delete(p, true);
          Log.info("Deleted ? : " + result);
        }
      }

      fs.close();
    } catch (IOException e) {
      e.printStackTrace();
    }
  }
}

I contacted AWS support and it seemed that the problem was that the log files I was analyzing were very big and that created an issue with memory. I added to my pipeline definition "masterInstanceType" : "m1.xlarge" in the EMRCluster section and it worked. Thanks — cevallos.valtira, May 30 '13 at 14:19

Override hadoop

1 Answers1