How to append to an existing file in a Hadoop user program?

Question

I have a Hadoop program in which when the mapping and reducing phases are done, I need to append to an existing file (which is already on HDFS). How can I do that?

Answer 1

it's already supported to append a file on hdfs after hadoop 0.20.2, more information is available here1 and here2

An append example i found may help you:

FSDataOutputStream stm = fs.create(path, true,  
              conf.getInt("io.file.buffer.size", 4096),  
              (short)3, blocksize);  
String a = make(1000);  
stm.write(a.getBytes());  
stm.sync();  

Answer 2

You can use append method of HDFS,

check the file is exists on not, if exists append the new content in the same file.

for example:-

       FileSystem hdfs;
       FSDataOutputStream writeInFile;
       Path file;

       if (hdfs.exists(file)) {
            System.out.println("file exists");
            writeInFile = hdfs.append(file);
            writeInFile.writeBytes(data);
        }
        else {
            System.out.println("new file");
            writeInFile = hdfs.create(file, true);
            writeInFile.writeBytes(data);
        }