Fastest and most efficient way to save and load a large dict

Question

I have a problem. I have a huge dict. I want to save and load this huge dict. But unfortunately I got an MemoryError. The dict should not be too big. What is read out of the database is around 4GB. I would now like to save this dict and read it out. However, it should be efficient (not consume much more memory) and not take too long.

What options are there at the moment? I can't get any further with pickle, I get a memory error. I have 200GB of free disk space left.

I looked at Fastest way to save and load a large dictionary in Python and some others questions and blogs.

import pickle
from pathlib import Path

def save_file_as_pickle(file, filename, path=os.path.join(os.getcwd(), 'dict')):
    Path(path).mkdir(parents=True, exist_ok=True)
    pickle.dump( file, open( os.path.join(path, str(filename+'.pickle')), "wb" ))

save_file_as_pickle(dict, "dict")

[OUT]

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<timed eval> in <module>

~\AppData\Local\Temp/ipykernel_1532/54965140.py in save_file_as_pickle(file, filename, path)
      1 def save_file_as_pickle(file, filename, path=os.path.join(os.getcwd(), 'dict')):
      2     Path(path).mkdir(parents=True, exist_ok=True)
----> 3     pickle.dump( file, open( os.path.join(path, str(filename+'.pickle')), "wb" ))

MemoryError: 

What worked, but took 1 hour and 26GB space disk is used

with open('data.json', 'w', encoding='utf-8') as f:
    json.dump(dict, f, ensure_ascii=False, indent=4)

---

I looked up how big my dict is in bytes. I came across this question How to know bytes size of python object like arrays and dictionaries? - The simple way and it shows that the dict is only 8448728 bytes.

import sys
sys.getsizeof(dict)
[OUT] 8448728

---

What my data looks like (example)

{
'_key': '1',
 'group': 'test',
 'data': {},
 'type': '',
 'code': '007',
 'conType': '1',
 'flag': None,
 'createdAt': '2021',
 'currency': 'EUR',
 'detail': {
        'selector': {
            'number': '12312',
            'isTrue': True,
            'requirements': [{
                'type': 'customer',
                'requirement': '1'}]
            }
        }   

 'identCode': [],
 }

Answer 1

The memory error occurs when your RAM (not your hard disk filesystem) cannot hold the serialized form of the dict data. Serialization requires storing all kinds of metadata about the data in keys and values, searching and removing duplicate referenced objects, any properties and attributes of data types (especially database types not part of built-in Python types) all done in RAM memory first before even writing a single byte into the file. Since json produced 26GB just for the data values, I'd have to assume all the metadata added on top of that would have increased the memory size of the serialized form.

Compression doesn't help since the serialized data must be in non-compressed form before doing any compression. It only saves the disk space, not RAM memory.

JSON may have worked because it starts to stream data as it is read, instead of converting to JSON all in memory. Or it could be that JSON form without all the extraneous metadata info can be held in your RAM just fine.

If you want to optimize and solve without using JSON, here is one solution:

• Create custom dict reader from database that casts common data types to built-in Python types or your own custom lean data types, rather than what the default database reader provides using its own types.
• Create custom serialization/de-serialization method for your data type that only handles data that needs to be stored, and even (de)compress the data on the fly in the (de)serialization method.

A hardware solution is of course to increase your RAM memory and optionally your hard disk.

Another solution is try this in Linux, which tends to have better memory optimization than Windows.

Answer 2

There are two ways to make the pickling more performant:

1. disabling the Garbage Collector while pickling for a speedup
2. using gzip to generate a compressed output file

Give this a try:

import gc
import gzip
import os
import pickle
from pathlib import Path


def save_file_as_pickle(file, filename, path=os.path.join(os.getcwd(), "dict")):
    Path(path).mkdir(parents=True, exist_ok=True)
    file_path = os.path.join(path, str(filename + ".pickle"))

    gc.disable()
    try:
        gc.collect()
        with gzip.open(file_path, "wb") as fp:
            pickle.dump(file, fp)
    finally:
        gc.enable()


save_file_as_pickle(my_dict, "dict")

Answer 3

I would consider trying out some new formats although I am not 100% sure that they are better.

stack overflow answers

For HDF5, I would what to try out dict to hdf5 library to see if it works.

import hdfdict
import numpy as np


d = {
    'a': np.random.randn(10),
    'b': [1, 2, 3],
    'c': 'Hallo',
    'd': np.array(['a', 'b']).astype('S'),
    'e': True,
    'f': (True, False),
}
fname = 'test_hdfdict.h5'
hdfdict.dump(d, fname)
res = hdfdict.load(fname)

print(res)

Answer 4

If nothing else works you might consider to split the dataset and save it in chunks. You can use threading or you can rewrite the code below to do it serial. I assumed that your dictionary is a list of dictionaries if its a dictionary of dictionaries you need to adjust the code accordingly. Als note that this example also needs to be adjusted as, depending on how you chose the step size, the last entries might not be saved or loaded.

import pickle
import threading
    
# create a huge list of dicts
size = 1000000
mydict_list = [{'_key':f'{i}','group': 'test'} for i in range(size)]

# try to save it as full file just to see how large it is
#with open(f'whole_list.pkl', 'wb') as f:
#    pickle.dump(mydict_list, f)


# define function to save the smaller parts
def savedata(istart,iend):
    tmp = mydict_list[istart:iend]
    with open(f'items_{istart}_{iend}.pkl', 'wb') as f:
        pickle.dump(tmp, f)

# define function to load the smaller parts
def loaddata(istart,iend):
    tmp = mydict_list[istart:iend]
    with open(f'items_{istart}_{iend}.pkl', 'rb') as f:
        results[f'{istart}_{iend}'] = pickle.load(f)

# define into how many chunks you want to split the file
steps = int(size/10)

# split the list and save it using threading
results = {}
threads={}
for i in  [i for i in range(0,len(mydict_list),steps)]:
    threads[i]=None

for i in [i for i in  range(0,len(mydict_list),steps)]:
    print(f'processing: {i,i+steps}')
    threads[i] = threading.Thread(target=savedata, args=(i,i+steps,))
    threads[i].start()

for i in [i for i in range(0,len(mydict_list),steps)]:
    threads[i].join()


# load the list using threading
threads={}
for i in  [i for i in range(0,len(mydict_list),steps)]:
    threads[i]=None

for i in [i for i in  range(0,len(mydict_list),steps)]:
    print(f'processing: {i,i+steps}')
    threads[i] = threading.Thread(target=loaddata, args=(i,i+steps,))
    threads[i].start()

for i in [i for i in range(0,len(mydict_list),steps)]:
    threads[i].join()

Answer 5

TL;DR

The main issue here is the lack of streaming-like data format. I recommend reading and writing jsonl format, but keep working with your regular dict. Try the 2 options:

1. gzip + jsonl, using the file api (faster write)
2. clear jsonl, using the mmap api (faster read)

Full details below:

---

JSON Lines Format

The idea is to provide as close as posible format to json, while staying split-able.

This allows for a line by line, or a block(of lines) by block multiprocessing, locally or distributed. (a common big-date practice might be storign on HDFS, processing by Spark, for ex').

It goes nicely with the gzip compression, which is split-friendly by itself - allowing for sequential reads and writes.

We'll wrap the read and write so that the application will be agnostic to it, and could still deal with the common dict.

A data simulator

I created 1M dict entries from your sample, with varying keys, currency and year (to challenge the gzip compression a bit). I used a macbook pro m1.

import json
import gzip
import mmap
import subprocess

d = {}
years = { 0: 2019, 1: 2020, 2:2121 }
currencies = { 0: 'EUR', 1: 'USD', 2: 'GBP' }
n = int(1e6)

for i in range(n):
    rem = i % 3
    d[i] = {
        '_key': str(i),
        'group': 'test',
        'data': {},
        'type': '',
        'code': '007',
        'conType': '1',
        'flag': None,
        'createdAt': years[rem],
        'currency': currencies[rem],
        'detail': {
            'selector': {
                'number': '12312',
                'isTrue': True,
                'requirements': [{
                    'type': 'customer',
                    'requirement': '1'}]
                }
            },
        'identCode': [],
    }

Option #1 - gzip file api

For the 1M dataset it took ~10s to write and ~6s to read again.

file_name_jsonl_gz = './huge_dict.jsonl.gz'

# write
with gzip.open(file_name_jsonl_gz, 'wt') as f:
    for k, v in d.items():
        f.write(f'{{"{k}":{json.dumps(v)}}}\n') # from k, v pair into a json line

# read again
_d = {}
with gzip.open(file_name_jsonl_gz, 'rt') as f:
    for line in f:
        __d = json.loads(line)
        k, v = tuple(__d.items())[0] # from a single json line into k, v pair
        _d[k] = v

# test integrity
json.dumps(d) == json.dumps(_d)

True

Option #2 - mmap api

For the 1M dataset it took ~5s to write and ~8s to read again.

The Memory Mapped File is a potentially very strong technique for robust-ing our IO. The basic idea is mapping [huge] files into the virtual-memory system, allowing partial / fast / concurrent reads and writes. So, good for both huge files (that can't be fitted into memory) and a performance boost.

The code is more cumbersome, and not always the fastest, but you can further tweak it for your needs. There are so many details about it, so I recommend reading more on wiki and python api, not to overwhelm the answer here.

file_name_mmap_jsonl = './huge_dict_mmap.jsonl'
# an initial large empty file (hard to estimate in advance)
# change the size for your actual needs
subprocess.Popen(['truncate', '-s', '10G', file_name_mmap_jsonl])

pos_counter = 0
with open(file_name_mmap_jsonl, mode='r+', encoding="utf-8") as f:
    # mmap gets its file descriptor from the file object
    with mmap.mmap(fileno=f.fileno(), length=0, access=mmap.ACCESS_WRITE) as mm:
        buffer = []
        for k, v in d.items():
            s = f'{{"{k}":{json.dumps(v)}}}\n' # from k, v pair into a json line
            b = s.encode()
            buffer.append(b)
            pos_counter += len(b)

            # using buffer; not to abuse the write for every line
            # try and tweak it further
            if len(buffer) >= 100:
                mm.write(b''.join(buffer))
                buffer = []
        
        mm.write(b''.join(buffer))
        mm.flush()

# shrink to the excat needed size
subprocess.Popen(['truncate', '-s', str(pos_counter), file_name_mmap_jsonl])
# read again
_d = {}
with open(file_name_mmap_jsonl, mode='r+', encoding="utf-8") as f:
    with mmap.mmap(fileno=f.fileno(), length=0, access=mmap.ACCESS_READ) as mm:
        while True:
            line = mm.readline()
            if len(line) == 0: # EOF
                break
            __d = json.loads(line)
            k, v = tuple(__d.items())[0] # from a json line into k, v pair
            _d[k] = v

# test integrity
json.dumps(d) == json.dumps(_d)

True

There was also a 3rd Option: mmap + gzip, but the write was slow and there were issues with decompressing back the lines. I recommend pursuing this, though - this will allow for a much smaller file size on disk.