2

I am using NetBeans 7.4 on Windows 7.

When I tried to read a JSON file with the following call,

String jsonTxt = IOUtils.toString(new FileInputStream(inputFileName), "UTF-8");

I received an OutOfMemoryError exception. I checked and saw that the file was over 28 GB so there is probably no way I can read it all into memory at once.

Is there a way to read in only part of the JSON file so that I do not run out of memory? There does not appear to be a suitable method in IOUtils.

OtagoHarbour
  • 3,969
  • 6
  • 43
  • 81
  • 2
    See http://stackoverflow.com/questions/444380/is-there-a-streaming-api-for-json – FatalError Sep 22 '14 at 21:18
  • 2
    +1 for the streaming json parser. But 28GB? It's likely that you have a design problem here. You might want to think about breaking it up into multiple files if possible, or finding some other, better way to store it. – GreyBeardedGeek Sep 22 '14 at 21:25

1 Answers1

1

You can't read parts of a JSON file. The JSON structure depends on its elements and their level of nesting.

Most JSON parsing libraries have an input method that will accept an InputStream (or Reader) instead of a String. You should try those. Note that if your mapped object structure is too big, you'd still have issues.

You should consider reviewing what you are storing and why you're storing it in JSON.

Sotirios Delimanolis
  • 274,122
  • 60
  • 696
  • 724