How to append/write huge data file text in Java

Question

I have a database with 150k records. I want to write this to file as fast as possible. I've tried many approaches, but all seem slow. How do I make this faster?

I read these records in blocks of 40k. So first I read 40k then another 40k and so on.

After reading the records, this process returns a StringBuilder which contains 40k lines. Then we write this StringBuilder to a file.

private static void write(StringBuilder sb, Boolean append) throws Exception {
    File file = File.createTempFile("foo", ".txt");

    FileWriter writer = new FileWriter(file.getAbsoluteFile(), append);
    PrintWriter out = new PrintWriter(writer);
    try {
        out.print(sb);           
        out.flush();
        writer.flush();
    } finally {
        writer.close();
        out.close();
    }
}

I read this other example but it is equally slow: Fastest way to write huge data in text file Java

I also tried it with NIO api:

private static void write(StringBuilder sb, Boolean append)) throws Exception {
    FileChannel rwChannel = new FileOutputStream("textfile.txt", true).getChannel();
    ByteBuffer bb = ByteBuffer.wrap(sb.toString().getBytes("UTF-8"));
    rwChannel.write(bb);
    rwChannel.close();
}

Which is the best method to write/append huge data into file?

I believe the problem is with `out.print(sb)`. This forces `toString()` on the builder, and that in turn involves copying the underlying character array. If you need speed, try simple rewriting using a char array buffer and raw byte streams. — Marcin Łoś, Sep 10 '13 at 12:17
It's the network communication part that is slowing you down, find a way to avoid transmitting raw data across a network. Perhaps consider file compression to speed up the transmission phase. — Eric Leschinski, Mar 21 '15 at 22:39

score 11 · Accepted Answer · answered Sep 10 '13 at 14:13

You don’t need a PrintWriter here. If you have whatever kind of Writer (e.g. a FileWriter) you can simply invoke append(sb) on it. And you don’t need to flush, close implies flushing.

private static void write(StringBuilder sb, Boolean append) throws Exception {
  File file = File.createTempFile("foo", ".txt");

  try(FileWriter writer = new FileWriter(file.getAbsoluteFile(), append)) {
      writer.append(sb);
  }
}

On my system I encountered a small performance improvement using a Channel rather than an OutputStream:

private static void write0a(StringBuilder sb, Boolean append) throws Exception {
  File file = File.createTempFile("foo", ".txt");

  try(Writer writer = Channels.newWriter(new FileOutputStream(
      file.getAbsoluteFile(), append).getChannel(), "UTF-8")) {
    writer.append(sb);
  }
}

However these are only slight improvements. I don’t see much possibilities here as all the code ends up calling the same routines. What could really improve your performance is keeping the Writer alive during the invocations and not flushing every record.

score 6 · Answer 2 · answered Sep 10 '13 at 12:39

If you have a huge amount of data, it's better that you don't store it to StringBuilder and then write it to file at once.

This is the best scenario:

1) Before you start process on the data create FileInputStream

FileOutputStream fos = new FileOutputStream("/path/of/your/file");

2) Create and OutputStreamWriter from this file

OutputStreamWriter w = new OutputStreamWriter(fos, "UTF-8");

3) Create BufferedWriter (Improve file writing performance)

BufferedWriter bw = new BufferedWriter(w);

4) Pass bw to your process function and then flush/close

bw.flush();
bw.close();

The functionality of StringBuilder and BufferedWriter is almost same, So you do not need to change your code so much. The only negative point of this scenario is that, your process will involve all the time that the data are writing to file, but if you don't process the data in different thread, it is not an issue.

In this way, it doesn't matter how large data is it

score 1 · Answer 3 · answered Sep 10 '13 at 12:26

You are using a FileWriter (or a FileOutputStream in the second example). These are not buffered! So they write single chars resp. bytes to the disk.

That means, you should wrap the FileWriter in a BufferedWriter (or the FileOutputSystem in a BufferedOutputSystem).

private static void write(StringBuilder sb, Boolean append) throws Exception {
    File file = File.createTempFile("foo", ".txt");
    Writer writer = new BufferedWriter(new FileWriter(file.getAbsoluteFile(), append));
    PrintWriter out = new PrintWriter(writer);
    try {
        out.print(sb);           
        out.flush();
        writer.flush();
    } finally {
        writer.close();
        out.close();
    }
}

Nope, from `FileWriter`'s javadoc: `The constructors of this class assume that the default character encoding and the default byte-buffer size are acceptable` — Marcin Łoś, Sep 10 '13 at 12:37
Oh, yes. You are right. I had FileOutputStream in mind (and wrongly assumed, that FileWriter behaves the same). Hmm ... — Seelenvirtuose, Sep 10 '13 at 12:53

score 0 · Answer 4 · edited Mar 21 '15 at 22:37

0

You are opening the file, writing one line, then closing it. It's the opening and closing that takes the time here. Find a way to keep the output file open.

edited Mar 21 '15 at 22:37

Eric Leschinski

146,994
96
417
335

answered Sep 10 '13 at 12:50

user207421

305,947
44
307
483

score -1 · Answer 5 · answered Sep 10 '13 at 12:55

-1

Did you try Apache IO, is the performance still the same?

answered Sep 10 '13 at 12:55

Amr Gawish

2,015
1
17
29

Try it from here[http://commons.apache.org/proper/commons-io/download_io.cgi] and tell me – Amr Gawish Sep 10 '13 at 14:05

How to append/write huge data file text in Java

5 Answers5