How to upload and download file to hdfs using Java code

Question

i am newbie to hadoop, and trying to upload and download files to hdfs via. Java code. which should behave as

Data Uploading:

 hadoop fs -put or -copyFromLocal filename directoryName

and Data Downloading

  hadoop fs -get or -copyToLocal filename directoryName

from hdfs. i need this one because datasets contain image, audio, video etc file. above command works fine with all type of data, if i try using Java i/o reader code , it is working fine for text files , but not for images, video. docx etc..

pls any help here.

Edited Here:

public static void main(String[] args) throws IOException {
        // TODO Auto-generated method stub

        Configuration conf=new Configuration();
        FileSystem fs = FileSystem.get(conf);

        Path path=new Path("data");
        Path file=new Path(path,"screenshots.png");

        BufferedImage image = ImageIO.read(new File("/home/hduser/Desktop/screenshots.png"));
        if (!fs.exists(path))
          throw new IOException("Output not found!");

        ImageIO.write(image, "png", fs.open(path));


    }

As asked i have edited here code that i am using to upload image file to hdfs. here ImageIO.write is not accepting arguement fs.open(path) , because is asking for file, but i have to give path here as to read and to write to hdfs we need to give path only. Actually i am in need of a method to upload and download file from hdfs using code for all type of data, so i should not write code and use plugins for all type of file.

Please show the code that you're trying to use and tell us exactly why you think it doesn't work. — yole, Dec 24 '14 at 08:58
if you are familiar Spring then you can use [Spring Hadoop](http://projects.spring.io/spring-hadoop/), you can [see sample example here](https://github.com/spring-projects/spring-hadoop-samples/tree/master/mapreduce) which will move the input files into hdfs using groovy file and runs your Map-Reduce as a job — Prasad Khode, Dec 24 '14 at 10:12

score 2 · Accepted Answer · answered Dec 24 '14 at 09:32

2

ImageIO.write can take an OutputStream as well as a File. However, fs.open is returning an InputStream because it is for reading files only.

You need to call:

ImageIO.write(image, "png", fs.create(file));

The create method will return an OutputStream which ImageIO can write to.

http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/FileSystem.html

answered Dec 24 '14 at 09:32

Paolo

22,188
6
42
49

Hi Paolo, it works.. can you please tell, what should i use to upload all types of file to hdfs and download from hdfs, without writing code for all types of file format. +1 – Raju Sharma Dec 24 '14 at 09:43
Just open whatever file you want using the standard Java FileInputStream, read it and write the output to your HDFS outputstream. This may help: http://stackoverflow.com/questions/43157/easy-way-to-write-contents-of-a-java-inputstream-to-an-outputstream – Paolo Dec 24 '14 at 09:45

score 0 · Answer 2 · answered Dec 24 '14 at 09:40

If path is already exists than you will overwrite that file with your image. I think that you want to save your image into some existing folder in HDFS. In that case you need to write your image to new Path(path, "SomeImageName.png");.
You don't need to use ImageIO to copy the image from local file system to HDFS. Try to use copyFromLocalFile method of FileSystem:

fs.copyFromLocalFile(new Path("/home/hduser/Desktop/screenshots.png"), path);

How to upload and download file to hdfs using Java code

2 Answers2