Reading data from HDFS - my program can't find the path

Question

I'm trying to read the contents of a file from HDFS. My code is below -

package gen;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

public class ReadFromHDFS {

 public static void main(String[] args) throws Exception {

  if (args.length < 1) {
   System.out.println("Usage: ReadFromHDFS <hdfs-file-path-to-read-from>");
   System.out.println("Example: ReadFromHDFS 'hdfs:/localhost:9000/myFirstSelfWriteFile'");
   System.exit(-1);
  } 

  try {
   Path path = new Path(args[0]);
   FileSystem fileSystem = FileSystem.get(new Configuration());
   BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(fileSystem.open(path)));
   String line = bufferedReader.readLine();
   while (line != null) {
    System.out.println(line);
    line = bufferedReader.readLine();
   }
  } catch (IOException e) {
   e.printStackTrace();
  }
 }
}

However, I can't figure out how to give this program the path to my HDFS directory. I have tried -

java -cp <hadoop jar:myjar> gen.ReadFromHDFS <path>

where with path I tried referencing the directory directly (what I see when I do hadoop fs -ls), the file inside the directory, adding hdfs:/localhost, hdfs:/ and none of them work. Can any one help me with how exactly I should pass the path of my folder to HDFS? For example, when I give the path directly (with no prefix) it says that the file does not exist.

Edit: None of the solutions so far seem to work for me. I always get the exception -

  java.io.FileNotFoundExceptoin: File <filename> does not exist.
  at org.apache.hadoop.fs.getFileSystem.getFileStatus(RawLocalFileSystem.java:361)

It seems to be trying to find the file locally.

score 2 · Answer 1 · answered Jul 28 '14 at 06:16

try

FileSystem fileSystem = FileSystem.get(new Configuration());
Path path = new Path(fileSystem.getName() + "/" + args[0]);
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(fileSystem.open(path)));
String line = bufferedReader.readLine();

and give file path in HDFS as (with no prefix)

"/myFirstSelfWriteFile"

do not include "hdfs:/localhost"

score 0 · Answer 2 · answered Jul 28 '14 at 00:57

0

You need to be using the classes in package org.apache.hadoop.fs (FileSystem, FSDataInputStream, FSDataOutputStream and Path). There are several articles out there, but I'd use this one from the Hadoop Wiki

answered Jul 28 '14 at 00:57

Chris Gerken

16,221
6
44
59

score 0 · Answer 3 · answered Jul 28 '14 at 03:59

0

Looks like you are missing one / in your path, should give two /'s after filesystem. Try specifying the following path

hdfs://localhost:9000/myFirstSelfWriteFile

answered Jul 28 '14 at 03:59

SachinJose

8,462
4
42
63

Reading data from HDFS - my program can't find the path

3 Answers3

Linked