3

How to parse huge JSON file in Android?

When I parse such file I see on something like this on logs:

04-05 15:55:46.490: DEBUG/dalvikvm(3847): GC freed 12159 objects / 557744 bytes in 142ms
04-05 15:55:46.490: INFO/global(3847): Default buffer size used in BufferedReader constructor. It would be better to be explicit if an 8k-char buffer is required.
04-05 15:55:46.650: DEBUG/dalvikvm(3847): GC freed 56 objects / 100176 bytes in 115ms
04-05 15:55:46.660: INFO/dalvikvm-heap(3847): Grow heap (frag case) to 4.207MB for 110626-byte allocation
04-05 15:55:46.760: DEBUG/dalvikvm(3847): GC freed 0 objects / 0 bytes in 104ms
04-05 15:55:46.890: DEBUG/dalvikvm(3847): GC freed 3 objects / 73856 bytes in 115ms
04-05 15:55:47.050: DEBUG/dalvikvm(3847): GC freed 7 objects / 110872 bytes in 113ms
04-05 15:55:47.050: INFO/dalvikvm-heap(3847): Grow heap (frag case) to 4.426MB for 248896-byte allocation
04-05 15:55:47.150: DEBUG/dalvikvm(3847): GC freed 0 objects / 0 bytes in 104ms
04-05 15:55:47.310: DEBUG/dalvikvm(3847): GC freed 8 objects / 166232 bytes in 106ms
04-05 15:55:47.310: INFO/dalvikvm-heap(3847): Grow heap (frag case) to 4.624MB for 373340-byte allocation
04-05 15:55:47.420: DEBUG/dalvikvm(3847): GC freed 0 objects / 0 bytes in 106ms
04-05 15:55:47.580: DEBUG/dalvikvm(3847): GC freed 10 objects / 249288 bytes in 111ms
04-05 15:55:47.590: INFO/dalvikvm-heap(3847): Grow heap (frag case) to 4.920MB for 560006-byte allocation
04-05 15:55:47.690: DEBUG/dalvikvm(3847): GC freed 0 objects / 0 bytes in 98ms
04-05 15:55:47.810: DEBUG/dalvikvm(3847): GC freed 12 objects / 373792 bytes in 88ms
04-05 15:55:47.910: DEBUG/dalvikvm(3847): GC freed 4 objects / 560088 bytes in 87ms
04-05 15:55:48.010: DEBUG/dalvikvm(3847): GC freed 5 objects / 128 bytes in 88ms
04-05 15:55:48.010: INFO/dalvikvm-heap(3847): Grow heap (frag case) to 5.350MB for 461302-byte allocation
04-05 15:55:48.130: DEBUG/dalvikvm(3847): GC freed 0 objects / 0 bytes in 120ms
04-05 15:55:48.990: DEBUG/dalvikvm(3847): GC freed 10419 objects / 1383992 bytes in 109ms
04-05 15:55:49.720: DEBUG/dalvikvm(3847): GC freed 8769 objects / 384360 bytes in 115ms
04-05 15:55:50.550: DEBUG/dalvikvm(3847): GC freed 10472 objects / 454880 bytes in 129ms
04-05 15:55:51.590: DEBUG/dalvikvm(3847): GC freed 12663 objects / 554440 bytes in 147ms

Is there a pull parser for Json files? How to deal with such thing?

pixel
  • 24,905
  • 36
  • 149
  • 251
  • 2
    That's your problem right there: HUGE file, Android (which means mobile). Rethink why you need to do this in the first place, and then avoid having to do so much work on a *mobile* device. – Stephen Chung Apr 05 '11 at 14:03
  • 1
    There are pull parsers for XML files, so huge JSON file shouldn't be an issue. – pixel Apr 05 '11 at 14:06
  • 1
    I too usually get that `INFO/global(3847): Default buffer size used in BufferedReader constructor. It would be better to be explicit if an 8k-char buffer is required.` message – Mithun Sreedharan Apr 05 '11 at 14:07
  • @Mithun P: Buffer size is not a problem. I know it's normal, but what about huge file? – pixel Apr 05 '11 at 14:28

2 Answers2

7

Take a look at the Jackson JSON parser, it's faster than the one in Android, and faster than GSON, and supports streaming.

http://jackson.codehaus.org/

I do however reiterate the point made by others that your JSON is HUGE and that does suggest a non-optimal design. e.g. if the connection is wonky and it fails a few times you could cause your user to incur additional data costs on a mobile data plan. Always better to break it down where possible so failure doesn't involve requesting the whole lot again.

Ollie C
  • 28,313
  • 34
  • 134
  • 217
3

You might look into JsonStreamParser in gson if you are looking for an interface that streams json objects instead of parsing everything immediately.

Matthew
  • 44,826
  • 10
  • 98
  • 87