Pickle dump replaces current file data

Question

When I use pickle, it works fine and I can dump any load.

The problem is if I close the program and try to dump again, it replaces the old file data with the new dumping. Here is my code:

import pickle
import os
import time


dictionary = dict()


def read():
    with open('test.txt', 'rb') as f:
        a = pickle.load(f)
    print(a)
    time.sleep(2)


def dump():
    chs = raw_input('name and number')
    n = chs.split()
    dictionary[n[0]] = n[1]
    with open('test.txt', 'wb') as f:
        pickle.dump(dictionary, f)


Inpt = raw_input('Option : ')
if Inpt == 'read':
    read()
else:
    dump()

Answer 1

When you open a file in w mode (or wb), that tells it to write a brand-new file, erasing whatever was already there.

As the docs say:

The most commonly-used values of mode are 'r' for reading, 'w' for writing (truncating the file if it already exists), and 'a' for appending…

In other words, you want to use 'ab', not 'wb'.

---

However, when you append new dumps to the same file, you end up with a file made up of multiple separate values. If you only call load once, it's just going to load the first one. If you want to load all of them, you need to write code that does that. For example, you can load in a loop until EOFError.

---

Really, it looks like what you're trying to do is not to append to the pickle file, but to modify the existing pickled dictionary.

You could do that with a function that loads and merges all of the dumps together, like this:

def Load():
    d = {}
    with open('test.txt', 'rb') as f:
        while True:
            try:
                a = pickle.load(f)
            except EOFError:
                break
            else:
                d.update(a)
    # do stuff with d

But that's going to get slower and slower the more times you run your program, as you pile on more and more copies of the same values. To do that right you need to load the old dictionary, modify that, and then dump the modified version. And for that, you want w mode.

However, a much better way to persist a dictionary, at least if the keys are strings, is to use dbm (if the values are also strings) or shelve (otherwise) instead of a dictionary in the first place.

Answer 2

Opening a file in "wb" mode truncates the file -- that is, it deletes the contents of the file, and then allows you to work on it.

Usually, you'd open the file in append ("ab") mode to add data at the end. However, Pickle doesn't support appending, so you'll have to save your data to a new file (come up with a different file name -- ask the user or use a command-line parameter such as -o test.txt?) each time the program is run.

On a related topic, don't use Pickle. It's unsafe. Consider using JSON instead (it's in the standard lib -- import json).