How to speed up process of loading and reading JSON files in Python?

Question

I am running a script (in multiprocessing mode) that extract some parameters from a bunch of JSON files but currently it is very slow. Here is the script:

from __future__ import print_function, division
import os
from glob import glob
from os import getpid
from time import time
from sys import stdout
import resource
from multiprocessing import Pool
import subprocess
try:
    import simplejson as json
except ImportError:
    import json


path = '/data/data//*.A.1'
print("Running with PID: %d" % getpid())

def process_file(file):
    start = time()
    filename =file.split('/')[-1]
    print(file)
    with open('/data/data/A.1/%s_DI' %filename, 'w') as w:
        with open(file, 'r') as f:
            for n, line in enumerate(f):
                d = json.loads(line)
                try:

                    domain = d['rrname']
                    ips = d['rdata']
                    for i in ips:
                        print("%s|%s" % (i, domain), file=w)
                except:
                    print (d)
                    pass

if __name__ == "__main__":
    files_list = glob(path)
    cores = 12
    print("Using %d cores" % cores)
    pp = Pool(processes=cores)
    pp.imap_unordered(process_file, files_list)
    pp.close()
    pp.join()

Does any body know how to speed this up?

Answer 1

swith from

import json 

to

import ujson

https://artem.krylysov.com/blog/2015/09/29/benchmark-python-json-libraries/

or switch to orjson

import orjson 

https://github.com/ijl/orjson

Answer 2

First, find out where your bottlenecks are.

If it is on the json decoding/encoding step, try switching to ultrajson:

UltraJSON is an ultra fast JSON encoder and decoder written in pure C with bindings for Python 2.5+ and 3.

The changes would be as simple as changing the import part:

try:
    import ujson as json
except ImportError:
    try:
        import simplejson as json
    except ImportError:
        import json

I've also done a simple benchmark at What is faster - Loading a pickled dictionary object or Loading a JSON file - to a dictionary?, take a look.

Answer 3

I updated the script a bit to try different experiments and found that yes, json parsing is cpu bound. I got 28MB/s, which is better than your .04Gig per minute (> 1 MB/s), so not sure what's going on there. When skipping the json stuff and just writing to the file, I got 996 MB/s.

In the code below, you can generate a dataset with python slow.py create and test several scenarios by changing the code marked todo:. My dataset was only 800 MB, so I/O was absorbed by the RAM cache (run it twice to make sure that the files to read have been cached).

I was surprised that json decode is so cpu intensive.

from __future__ import print_function, division
import os
from glob import glob
from os import getpid
from time import time
from sys import stdout
import resource
from multiprocessing import Pool, cpu_count
import subprocess

# todo: pick your poison
#import json
#import ujson as json
import simplejson as json

import sys

# todo: choose your data path
#path = '/data/data//*.A.1'
#path = '/tmp/mytest'
path = os.path.expanduser('~/tmp/mytest')

# todo: choose your cores
#cores = 12
cores = cpu_count()

print("Running with PID: %d" % getpid())

def process_file(file):
    start = time()
    filename =file.split('/')[-1]
    print(file)
    with open(file + '.out', 'w', buffering=1024*1024) as w:
        with open(file, 'r', buffering=1024*1024) as f:
            for n, line in enumerate(f):

                # todo: for pure bandwidth calculations
                #w.write(line)
                #continue

                try:
                    d = json.loads(line)
                except Exception, e:
                    raise RuntimeError("'%s' in %s: %s" % (str(e), file, line))
                try:

                    domain = d['rrname']
                    ips = d['rdata']
                    for i in ips:
                        print("%s|%s" % (i, domain), file=w)
                except:
                    print (d, 'error')
                    pass
    return os.stat(file).st_size

def create_files(path, files, entries):
    print('creating files')
    extra = [i for i in range(32)]
    if not os.path.exists(path):
        os.makedirs(path)
    for i in range(files):
        fn = os.path.join(path, 'in%d.json' % i)
        print(fn)
        with open(fn, 'w') as fp:
            for j in range(entries):
                json.dump({'rrname':'fred', 
                     'rdata':[str(k) for k in range(10)],
                     'extra':extra},fp)
                fp.write('\n')


if __name__ == "__main__":
    if 'create' in sys.argv:
        create_files(path, 1000, 100000)
        sys.exit(0)
    files_list = glob(os.path.join(path, '*.json'))
    print('processing', len(files_list), 'files in', path)
    print("Using %d cores" % cores)
    pp = Pool(processes=cores)
    total = 0
    start = time()
    for result in pp.imap_unordered(process_file, files_list):
        total += result
    pp.close()
    pp.join()
    delta = time() - start
    mb = total/1000000
    print('%d MB total, %d MB/s' % (mb, mb/delta))

Answer 4

For installation:

pip install orjson 

For import:

import orjson as json

This works especially if you want to dump or load arrays of large size.