How to reduce time to load a pickle file in python or map a pickle file to disk

Question

I want to load pickle file of size 4.23GB. I use below code to load the data:

import _pickle as cPickle
def read_pickle(file):
    try:
        with open(file, "rb") as input_file:
            data = cPickle.load(input_file)
            return data
    except Exception as e:
        print("Error in reading data from pickle file",e)

SyStem Configuration: 16Cores 32GB RAM

Output:

%time data=read_pickle(file)

CPU times: user 5.79 s, sys: 1.21 s, total: 7 s
Wall time: 7 s

As multiple user are using this in code so I want to load this file once in my code and use its return data each and every time is there anyway to map this file to disk to avoid load file every time or reduce loading time.

have you tried storing and reading from hdf5 It may be faster than pickle . https://stackoverflow.com/questions/37928794/which-is-faster-for-load-pickle-or-hdf5-in-python — Prayalankar Ashutosh, Feb 09 '21 at 06:56
yes I tried with hdf5 but loading and storing data from cPickle is much faster than hdf5 — Sweety Tripathi, Feb 09 '21 at 08:02

score 0 · Answer 1 · answered Feb 09 '21 at 07:55

0

what kind of data are stored in the file? If it contains only data i suggest to find an alternative. If it contains instances or other data, try to alter dunder methods getstate and setstate in order to avoid storing useless data like raw data, temporary data structures and so on.

answered Feb 09 '21 at 07:55

Glauco

1,385
2
10
20

I stored dictionary kind data. – Sweety Tripathi Feb 09 '21 at 08:02
If they are numerical data you can try to use numpy structures. binary format of numpy is incredibly fast. on the other hand ujson format can perform very well on basic data types – Glauco Feb 09 '21 at 08:44

score 0 · Answer 2 · answered Feb 09 '21 at 09:50

7 seconds is decent time to read the 4 GB file back to RAM and recreate the structures you have.

For your query about serving multiple users you might want to look at Redis (or any other in-memory key-value data structure server) to hold the data and serve the users from there rather than recreating from pickle file for each user.

How to reduce time to load a pickle file in python or map a pickle file to disk

2 Answers2