What is the best way to write and append a large file in java

Question

I have a java program that sends a series of GET requests to a webservice and stores the response body as a text file.

I have implemented the following example code (filtered much of the code to highlight the concerned) which appends the text file and writes as a new line at the EOF. The code, however, works perfectly but the performances suffers as the size of the file grows bigger.

The total size of data is almost 4 GB and appends about 500 KB to 1 MB of data on avg.

do
{
    //send the GET request & fetch data as string
    String resultData = HTTP.GET <uri>;

    // buffered writer to create a file 
    BufferedWriter writer = new BufferedWriter(new FileWriter(path, true));

    //write or append the file
    writer.write(resultData + "\n");
}
while(resultData.exists());

These files are created on daily basis and moved to hdfs for hadoop consumption and as a real-time archive. Is there a better way to achieve this?

Why are you re-opening the writer for each individual request? Just open it once, before the do-while loop. Don't forget to close it after the do-while loop. — Gimby, May 22 '15 at 14:19

K139 · Accepted Answer · 2015-05-22T14:35:41.350

1) You are opening a new writer every time, without closing the previous writer object.

2) Don't open the file for each write operation, instead open it before the loop, and close it after the loop.

BufferedWriter writer = new BufferedWriter(new FileWriter(path, true));
do{
          String resultData = HTTP.GET <uri>;
          writer.write(resultData + "\n");
}while(resultData.exists());
writer.close();

3) Default buffered size of BufferedWriter is 8192 characters, Since you have 4 GB of data, I would increase the buffer size, to improve the performance but at the same time make sure your JVM has enough memory to hold the data.

BufferedWriter writer = new BufferedWriter(new FileWriter(path, true), 8192 * 4);
do{
          String resultData = HTTP.GET <uri>;
          writer.write(resultData + "\n");
}while(resultData.exists());
writer.close();

4) Since you are making a GET web service call, the performance depends on the response time of webservice also.

I want to open a file every time for safety (in case power is gone), so what is the solution for this case? Does `FileWriter` read all the file just to add a new content at the end? — user924, Mar 01 '23 at 16:40

score 0 · Answer 2 · edited May 23 '17 at 12:07

According to this answer Java difference between FileWriter and BufferedWriter what you are doing right now is inefficient.

The code you provided is incomplete. Brackets are missing, no close statement for the writer. But if I understand correctly for every resultData you open a new buffered writer and call write once . This means that you should use the FileWriter directly, since the way you are doing it, the buffer is just an overhead.

If what you want it to get data in a loop and write them in a single file, then you should do something like this

try( BufferedWriter writer = new BufferedWriter(new FileWriter("PATH_HERE", true)) ) {
    String resultData = "";

    do {
        //send the GET request & fetch data as string
        resultData = HTTP.GET <uri>;

        //write or append the file
        writer.write(resultData + "\n");

    } while(resultData != null && !resultData.isEmpty());

} catch(Exception e) {
    e.printStackTrace();
}

The above uses try with resources, which will handle closing the writer after exiting the try block. This is available in java 7.

What is the best way to write and append a large file in java

2 Answers2