3

I'm writing a Python script for JSON file processing. It's basically divided in three procedures: load, encoding and printing. While I've easily created progress bars for encoding and printing using tqdm, I can't figure out how to do it for load.

I've thoroughly read tqdm docs and searched even here, but no clues

import json
from tqdm import tqdm, trange
from iso3166 import countries

geo_json_events_path = r'/cygdrive/c/elastic/gtd.geojson'

with open(geo_json_events_path) as f:
    data = json.load(f)

I expect the progress bar for json.load() method shown.

Oblivion
  • 7,176
  • 2
  • 14
  • 33
Gian Piero D' Amico
  • 89
  • 1
  • 6

2 Answers2

5

Got it!

Using the object_hook argument in json.load() method. I had already tried using this with a simple counting function that returned a value three times bigger than the number of lines in the source file, so I assumed it was wrong. After undertanding that my function was executed for each dictionary returned by json.load I realized that there are three dictionaries per line in my file so my function was OK, I just had to find the right nesting level in the json data to get the desired iteration. This is the outcome:

from tqdm import tqdm
import json

def hook(obj):
    value = obj.get("features")
    if value:
    pbar = tqdm(value)
    for item in pbar:
        pass
        pbar.set_description("Loading")
    return obj

f = open('/cygdrive/c/elastic/gtd_tst.geojson')
docs = json.load(f, object_hook=hook)
for doc in docs:
    print(doc)

The first answer in this post was really helpful to find the solution : Python decode nested JSON in JSON

Gian Piero D' Amico
  • 89
  • 1
  • 6
0

Here's a slightly more general solution to this for importing json files that have multiple objects:

def hook(obj):
    for key, value in obj.items():
        pbar = tqdm(value)
        if type(value) is list:
            for _ in pbar:
                pbar.set_description("Loading " + str(key))
    return obj

f = open(r"C:\my_json_file.json")
my_json_file = json.load(f, object_hook=hook)

Although it still doesn't show the total progress if you have nested objects, just the progress of importing the current object.

Sean Dever
  • 101
  • 1