Parse large JSON file with JSON Simple (OutOfMemoryError)

Question

I am trying to parse large JSON file with JSON Simple and i am getting out of memory errors. I am on Windows 10 and my laptop has an 8gb RAM. The file is 250mb, i will also need to parse a 2gb file. I also tried with StrinBuilder but then i am getting memory errors on StringBuilder. Here is my code with StringBuilder:

    BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream("myfile.json")));
    String line = null;
    StringBuilder sb= new StringBuilder("");
    while( (line = br.readLine())!= null ){
        sb.append(line);
    }
    JSONParser parser = new JSONParser();
    Object obj=null;
    try {
        obj = parser.parse(sb.toString());  
    }catch (Exception e) {

    }

and here is the code without StringBuilder:

JSONParser parser = new JSONParser();
        Object obj=null;
        try {
            obj = parser.parse(new FileReader("myfile.json"));  
        }catch (Exception e) {

        }

The error

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded at org.json.simple.parser.Yylex.yylex(Unknown Source) at org.json.simple.parser.JSONParser.nextToken(Unknown Source) at org.json.simple.parser.JSONParser.parse(Unknown Source) at org.json.simple.parser.JSONParser.parse(Unknown Source)

Here `StringBuilder` object holds all the data during iteration. so that all contents in the file is in memory. you can follow the link that show's how to read a big size file. http://stackoverflow.com/questions/2356137/read-large-files-in-java — Ataur Rahman Munna, Apr 24 '16 at 09:28
I suggest you use an event driver pasrer instead of loading the entire JSON as a DOM. This should allow you to process much larger messages. — Peter Lawrey, Apr 24 '16 at 09:29
@ataurRahmanMunna i tries what you told and the StringBuilder is fine but i am getting out of memory error on json parser again. — Iraklis Bekiaris, Apr 24 '16 at 09:43
@PeterLawrey do you have some documentation that i can read? I am not sure what you refering to. Thanks! — Iraklis Bekiaris, Apr 24 '16 at 09:44
Windows 10 killed my machine while I was adding a link. I will look for it again. — Peter Lawrey, Apr 24 '16 at 09:54
In short you can use GSON or Jackson to start with. http://stackoverflow.com/questions/9390368/java-best-approach-to-parse-huge-extra-large-json-file We also have a streaming parser, but it is more designed for low latency. — Peter Lawrey, Apr 24 '16 at 09:55
Ok, thank you very much!! I will rake tour advice under concideration. — Iraklis Bekiaris, Apr 24 '16 at 09:57
@PeterLawrey I solved the problem with jackson library :). Thanks again Peter. Have a good day — Iraklis Bekiaris, Apr 24 '16 at 10:05

score 3 · Accepted Answer · edited Jan 16 '21 at 10:56

3

If you are open to use other Json parser then you can try Jackson's Streaming API which can be used to parse huge JSON upto even giga bytes of size.It can be used to process huge files without loading them completely in memory.It allows get the data you want and ignore what you don't want also

Read more: https://github.com/FasterXML/jackson-docs/wiki/JacksonStreamingApi

edited Jan 16 '21 at 10:56

Gabriel Silveira

3
4

answered Apr 24 '16 at 10:04

tyagi

265
2
10

1

I was able to solve the problem with the help of Peter in the question comments. This is exactly what i used in the end! Thanks anyway!! – Iraklis Bekiaris Apr 24 '16 at 10:16

Zayn Korai · Answer 2 · 2019-02-22T00:50:44.303

There are some excellent libraries for parsing large JSON files with minimal resources. One is the popular GSON library. It gets at the same effect of parsing the file as both stream and object. It handles each record as it passes, then discards the stream, keeping memory usage low.

Support arbitrarily complex objects (with deep inheritance hierarchies and extensive use of generic types)

Look at this Detailed Tutorial for GSON approach,to solve it problem.

Parse large JSON file with JSON Simple (OutOfMemoryError)

2 Answers2