Dumping data with pickle

Question

I am very confused about how to use pickle, I'm trying to make the computation a lot faster by storing the values that have been computed, and creating a pickle file for the ones computed. When a certain function hasn't been computed yet I want to open the pickle file and create a new input of the newly computed function. Here's my atttempt:

try:
    with open('EinsteinBbb.pickle','rb') as f:
        Bbb = pickle.load(f)
except:
    Bbb = dict()
def EinsteinBbb(nl, ll, nu, lu):

    global Bbb
    try:
        Bbbfile=pickle.load(open('EinsteinBbb.pickle','rb'))
    # Check if Bbb[(nl, ll, nu, lu)] needs to be computed
        if True: #TODO: replace with the correct condition
            print('Computing Bbb[{0}, {1}, {2}, {3}]'.format(nl, ll, nu, lu))
    except:   
        Bbb[(nl, ll, nu, lu)] = nl*ll*nu*lu
        with open('EinsteinBbb.pickle','wb') as f:
            pickle.dump(Bbb, f)
        with open('EinsteinBbb.pickle','rb') as f:
            EinsteinBbb_read = pickle.load(f)
        # TODO: Open 'EinsteinBbb.pickle' with write mode and dump Bbb so that it does not need to be recomputed
    return Bbb[(nl, ll, nu, lu)]
print(EinsteinBbb(1,2,3,4))

When I try for example print(EinsteinBbb(2,1,2,0)) I get TypeError: string indices must be integers I feel like my code is very far from being right but any advice would be incredibly helpful!

Edit: Creating a minimal reproducible made the TypeError disappear, and I think it had to do with where the code was stored, but now I get KeyError for any key other than the one I ran the code with the first time.

Any further explanations? Pickle usage in your code looks like it should be. Where is the problem? Have you debugged it? Errata : Here is error `pickle.dump('EinsteinBbb.pickle', f)`. Should be `pickle.dump(Bbb, f)` because i assume you want to save Bbb variable. — s.paszko, Apr 22 '22 at 12:53
Right! I probably should've mentioned what goes wrong, well when I try for example `print(EinsteinBbb(2,1,2,0))` I get `TypeError: string indices must be integers` — eyad mardini, Apr 22 '22 at 12:55
Exactly, because instead of saving `Bbb` you save string `'EinsteinBbb.pickle'` inside pickle file. — s.paszko, Apr 22 '22 at 12:56
Ohhh so it should be `pickle.dump(Bbb[(nl, ll, nu, lu)], f)`? — eyad mardini, Apr 22 '22 at 12:58
@s.paszko Changing it to `pickle.dump(Bbb,f)` Is most definitely a correct step but the TypeError however, remains. — eyad mardini, Apr 22 '22 at 13:04
You need to `pickle.dump(Bbb, f)` if you want to save `Bbb`. Your code is currently pickling a string as @s.paszko pointed out. — martineau, Apr 22 '22 at 13:05
@martineau I've done that now but still get the same error message, but I will keep the change,any other suggestions? — eyad mardini, Apr 22 '22 at 13:22
You haven't made that change in the current version of you code (which still has `pickle.dump('EinsteinBbb.pickle', f)`). As far as I can tell, the `Bbb` variable isn't defined anywhere either. Please provide a runnable [mre]. — martineau, Apr 22 '22 at 13:26
@martineau I have created a minimal reproducible, which solved the problem but created a new one, I've edited the question to show you what I mean. — eyad mardini, Apr 22 '22 at 13:53

martineau · Accepted Answer · 2022-04-23T20:15:55.893

I think the following does what you want (including all your TODOs as well as not producing a KeyError for keys not already in the Bbb dictionary). It uses exceptions to determine when it can initialize the dictionary from the existing pickled cache file, as well as when an attempt is being made to access a non-existent entry so an new entry for it gets added.

I put in number of print calls to enable you to see what it's doing.

Tip: Most of the time you should avoid using "bare" except: statements when handling exceptions because they can hide things you would probably like to allow to occur, like SyntaxErrors or SIGINT keyboard (Ctrl + C) signals, and therefore would not want suppressed. In situations where you're not exactly sure what to expect, use except Exception: instead to avoid the problem of catching too much (because it won't include them).

import pickle


PICKLE_FILEPATH = 'EinsteinBbb.pickle'

# Load or create cache dictionary
try:
    with open(PICKLE_FILEPATH, 'rb') as f:
        Bbb = pickle.load(f)
    print('Bbb dictionary read from file.')
except Exception:
    print('Creating Bbb dictionary.')
    Bbb = dict()

def EinsteinBbb(nl, ll, nu, lu):
    try:
        return_value = Bbb[(nl, ll, nu, lu)]
        print(f'Using existing entry for ({nl}, {ll}, {nu}, {lu}) in Bbb.')
    except KeyError:
        print(f'Adding entry ({nl}, {ll}, {nu}, {lu}) to Bbb.')
        return_value = Bbb[(nl, ll, nu, lu)] = nl*ll*nu*lu
        # Overwrite pickle file with updated dictionary.
        with open(PICKLE_FILEPATH, 'wb') as f:
            pickle.dump(Bbb, f)

    return return_value

print(f'{EinsteinBbb(1,2,3,4)=}')
#print(f'{EinsteinBbb(2,3,4,5)=}')

Thank you very much! I'm a little confused on what you did to solve the KeyError? Was it explicitily doing except KeyError that solved it? — eyad mardini, Apr 22 '22 at 22:34
Yes, the `except KeyError:` is how. I could have explicitly checked if the key was present or not before attempting to use it, which is referred to as **LBYL** which stands for "Look before you leap", however I chose instead to use what's known as **EAFP** which stands for "It's easier to ask forgiveness than permission" because that approach is considered to be "more pythonic" (and is often shorter). See my answer to [Determine whether a key is present in a dictionary](https://stackoverflow.com/questions/3733992/determine-whether-a-key-is-present-in-a-dictionary). — martineau, Apr 22 '22 at 23:44

Dumping data with pickle

1 Answers1