0

I am reading a JSON file of roughly 6 GB using Java. The goal is to import all the JSON data into a MySQL database. The program runs fine for few initial records but then throws a java.lang.OutOfMemoryError with message Java heap space error. I am using the JsonParser class to read the file and using nextToken() to read the next JSON token. The system on which the program is running has 8 GB RAM.

How do I insert all the data into the the MySQL database from the JSON dataset?

MC Emperor
  • 22,334
  • 15
  • 80
  • 130
Faiz
  • 1
  • 3
    I think the reason that you're getting the `java.lang.OutOfMemoryError` is because the code is trying to parse the entire file at once. Look to see if the API you're using supports streaming, Which `JsonParser` class are you using? Most support streaming, have a look at the documentation. – Robert Bain Sep 21 '15 at 20:32
  • @ Robert i am using jsonparser class in jackson api – Faiz Sep 21 '15 at 20:40
  • 1
    How much memory are you allocating to the Java Heap? Last I checked the defaults were no where *near* 6GB; if the hardware you're running on has only 8GB you're likely going to need to do something fancier than loading the whole thing into memory at once. http://stackoverflow.com/questions/1434779/maximum-java-heap-size-of-a-32-bit-jvm-on-a-64-bit-os – Nathaniel Ford Sep 21 '15 at 22:06

1 Answers1

1

The fact that the system has 8 GB of RAM doesn't necessarily mean that the JVM can allocate that amount of memory. The total space the JVM depends on several factors, including operating system, Java configuration and JVM overhead.

I'm guessing that you are reading the file at once, which requires the JVM to allocate roughly 6 GB of memory in order to store all bytes into memory. The JVM fails to allocate that amount of memory and throws an OutOfMemoryError.

If the abovementioned assumption is correct, then you might want process the input stream instantly while reading it. Jackson and Gson might do the job for you.

More about the Java heap size: http://javarevisited.blogspot.nl/2011/05/java-heap-space-memory-size-jvm.html.

Community
  • 1
  • 1
MC Emperor
  • 22,334
  • 15
  • 80
  • 130