Handle 47gb JSON File

Asked Jan 15 '20 at 14:13

Active Jan 15 '20 at 14:20

Viewed 81 times

I have a 47gb JSON file which has the following form

{
  "data": {
    "Key_A": {small JSON ~50mb},
    "Key_B": {small JSON ~50mb},
    "Key_C": {large JSON ~47gb}
  }
}

The exact structure and content of file.data.Key_C is unknown and I'd like to analyze it (e.g. how many key:val, get the size of the items, etc).

My idea was reading line by line unfortunately I am unable to: The following script was not able to terminate (I had to abort when monitoring the session memory usage).

f = open(file_name, 'rb')
for line in f:
    if 'Key_C' in file_name.decode('cp1252'):  # Windows encoding
        break

f.close()

If the JSON is unindented (I do not know if that is the case) reading it line by line might not be feasible. Streaming char-by-char might be a solution. I am pretty much stuck here.

asked Jan 15 '20 at 14:13

user101

1

A streaming JSON parser? There are a few old questions about that e.g. [this one](https://stackoverflow.com/questions/54560154/streaming-json-parser). – Rup Jan 15 '20 at 14:15
1

Why would indentation make *any* difference? – Scott Hunter Jan 15 '20 at 14:18
@ScottHunter I thought no indent would mean the whole JSON is in one line? – user101 Jan 15 '20 at 14:37
Indentation is a property of each individual line, not how *many* lines there are. – Scott Hunter Jan 15 '20 at 15:06

Handle 47gb JSON File

0 Answers0