1

I am trying to execute hadoop fs -put <source> <destination> from Java code. When I execute this command directly from the terminal, it works fine but when I try to execute this command from within the Java code using 

String[] str = {"/usr/bin/hadoop","fs -put", source, dest};
Runtime.getRuntime().exec(str);

I get error as Error: Could not find or load main class fs. I tried to execute some non-hadoop commands like ls,mkdir commands from Java and they worked fine but the hadoop commands are not getting executed even though they work fine from the terminal.

What could be the possible reason for this and how can I solve it?

JAVA API TRY: I tried to use java api to perform the copy operation but I get error. The Java code is :

        String source = "/home/tmpe/file1.csv";
        String dest = "/user/tmpe/file1.csv";
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", "hdfs://node1:8020");
        FileSystem fs = FileSystem.get(conf);
        Path targetPath = new Path(dest);
        Path sourcePath = new Path(source);
        fs.copyFromLocalFile(false,true,sourcePath,targetPath);

The error which I get is:

Exception in thread "main" java.io.IOException: Mkdirs failed to create /user/tmpe
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:378)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:564)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:545)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:452)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:229)
    at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1230)

I have already created /user/tmpe folder and it has full read-write permissions but still this error comes. I am unable to get the issue resolved

Jason Donnald
  • 2,256
  • 9
  • 36
  • 49

2 Answers2

1

I guess you probably do not have a HADOOP_HOME environment variable set.

But since you're in Java, why on earth would you want to do a haddop fs -put in an external process when the Java API is even more friendly than the shell ?

Costi Ciudatu
  • 37,042
  • 7
  • 56
  • 92
  • Can you provide me an example of how can I use Java API to execute copy? Sorry I am a bit new to hadoop and as such am not sure how to do it. And what api will I need to get from `maven`? – Jason Donnald Oct 27 '14 at 01:08
  • and how can I set `HADOOP_HOME`? – Jason Donnald Oct 27 '14 at 01:12
  • @JasonDonnald The "Java API" link that I provided should be enough to get you started. `maven` can only give you dependencies, not APIs. As for the `HADOOP_HOME` variable (if you really want to stick to this approach), this should be a useful resource: http://stackoverflow.com/questions/580085/is-it-possible-to-set-an-environment-variable-at-runtime-from-java – Costi Ciudatu Oct 27 '14 at 01:19
  • I tried to use the Java API and when I execute my `.jar` file I get error as `java.io.IOException: Mkdirs failed to create /user/tmpe`. The Java code inside the jar is mentioned above in the post alongwith detail error description – Jason Donnald Oct 27 '14 at 13:22
  • I am seeing the same issue? How did you fix it? – vikky.rk Jan 02 '15 at 23:05
  • Same here. Any fix suggestions? ^ – usamazf Dec 14 '17 at 10:23
0

Came across old post but if you haven't tried already, execute it with hadoop jar app_name.jar instead of java -jar. This way if classpath of your jar does not have all hadoop jars it will fetch the jars predefined in $HADOOP_CLASSPATH.

Andreas
  • 2,455
  • 10
  • 21
  • 24
msashish
  • 277
  • 2
  • 6
  • 18