java heap space error while writing to a file

Question

I am trying to write a huge data around 64000 records at a time to a file. I am getting exceptions that I attached bellow.

the code that I used to write is

Path outputpath = Paths.get("file1.json");
try (BufferedWriter writer = Files.newBufferedWriter(outputpath, StandardCharsets.UTF_8, WRITE)) {
 writer.write(jsonObject.toString());
} catch (Exception e) {
//error msg
}

Here my "jsonObject" is nothing but a json Array which contains 65000 rows .

Can you please help me to write this to my file in an efficient way ,so that I can avoid that heap Space Error.

Don't create an in memory representation of your JSON, stream it to the `writer`. — M. le Rutte, Oct 18 '17 at 11:17
you could try with that solution: https://stackoverflow.com/questions/1062113/fastest-way-to-write-huge-data-in-text-file-java — the hand of NOD, Oct 18 '17 at 11:20
Give more memory to your JVM when you start it, look at the -Xmx command line option. Or try to stream like MleRutte suggested — Erwin Bolwidt, Oct 18 '17 at 11:21

score 1 · Answer 1 · answered Oct 18 '17 at 12:02

You've cut your stacktrace a little bit short, but I'll assume the exception happens in jsonObject.toString().

Basically, you have to decide between two things: either allocate more memory or break the big operation into several smaller ones. Adding memory is quick and simple, but if you expect even more data in the future, it won't solve your problem forever. As others have mentioned, use -Xmx and/or -Xms on java command line.

The next thing you could try is to use a different JSON library. Perhaps the one you are using now is not particularly suited for large JSON objects. Or there might be a newer version.

As the last resort, you can always construct the JSON yourself. It's a string after all, and you already have the data in memory. To be as efficient as possible, you don't even need to build the entire string at once, you could just go along and write bits and pieces to your BufferedWriter.

score -1 · Answer 2 · answered Oct 18 '17 at 11:34

You can try to iterate through your json object:

Iterator<String> keys = (Iterator<String>) jsonObject.keys();
while (keys.hasNext()) {
    String key = keys.next();
    JSONObject value = jsonObject.getJSONObject(key);
    writer.write(value.toString());
}

PS. You need to check your json object's structure.

java heap space error while writing to a file

2 Answers2