How to implement a persistent Python `list`?

Question

I'm trying to make an object act like a built-in list, except that its value be saved once modified.

The implementation I come up with is wrapping a list in a PersistentList class. For every access to method that may change the list, the wrapper delegates to the wrapped list, and save it to a key-value database after it's invoked.

Code:

class PersistentList(object):
    def __init__(self, key):
        self.key = key 
        self._list = db.get(key, []) 

    def __getattr__(self, name):
        attr = getattr(self._list, name)
        if attr:
            if attr in ('append', 'extend', 'insert', 'pop',
                'remove', 'reverse', 'sort'):
                attr = self._autosave(attr)
            return attr
        raise AttributeError

    def _autosave(self, func):
        @wraps(func)
        def _(*args, **kwargs):
            ret = func(*args, **kwargs)
            self._save()
            return ret 
        return _

    def _save(self):
        db.set(self.key, self._list)

There are several problems with this implementation:

1. I have to decorate methods like append every time they are accessed, is there a better way to decorate multiple methods of some object?

2. Operations like l += [1,2,3] don't work because I haven't implemented the iadd method.

What can I do to simplify this?

Answer 1

I like @andrew cooke's answer but I see no reason why you can't derive directly from a list.

class PersistentList(list):
    def __init__(self, *args, **kwargs):
        for attr in ('append', 'extend', 'insert', 'pop', 'remove', 'reverse', 'sort'):
            setattr(self, attr, self._autosave(getattr(self, attr))
        list.__init__(self, *args, **kwargs)
    def _autosave(self, func):
        @wraps(func)
        def _func(*args, **kwargs):
            ret = func(*args, **kwargs)
            self._save()
            return ret 
        return _func

Answer 2

Here is a way to avoid having to decorate every list method. It makes PersistentList a context manager, so you can use the

with PersistentList('key', db) as persistent:
    do_stuff()

syntax. Admittedly, this does not cause the _save method to be called after each list operation, only when you exit the with-block. But I think it gives you enough control to save when you want to save, especially since the __exit__ method is guaranteed to be executed no matter how you leave the with-block, including if it happens because of an exception.

You might be an advantage that _save is not called after every list operation. Imagine appending to the list 10,000 times. So many individual calls to db.set (a database?) could be quite time-consuming. I would be better, at least from a performance point of view, to make all the appends and the save once.

---

class PersistentList(list):
    def __init__(self, key, db):
        self.key = key
        self.extend(db.get(key, []))
    def _save(self):
        # db.set(self.key, self)
        print('saving {x}'.format(x = self))
    def __enter__(self):
        return self
    def __exit__(self,ext_type,exc_value,traceback):
        self._save()

db = {}
p = PersistentList('key', db)

with p:
    p.append(1)
    p.append(2)

with p:
    p.pop()
    p += [1,2,3]

# saving [1, 2]
# saving [1, 1, 2, 3]

Answer 3

Here's an answer that's a lot like @unutbu's, but more general. It gives you a function you can call to sync your object to disk, and it works with other pickle-able classes besides list.

with pickle_wrap(os.path.expanduser("~/Desktop/simple_list"), list) as (lst, lst_sync):
    lst.append("spam")
    lst_sync()
    lst.append("ham")
    print(str(lst))
    # lst is synced one last time by __exit__

Here's the code that makes that possible:

import contextlib, pickle, os, warnings

def touch_new(filepath):
    "Will fail if file already exists, or if relevant directories don't already exist"
    # http://stackoverflow.com/a/1348073/2829764
    os.close(os.open(filepath, os.O_WRONLY | os.O_CREAT | os.O_EXCL))

@contextlib.contextmanager
def pickle_wrap(filepath, make_new, check_type=True):
    "Context manager that loads a file using pickle and then dumps it back out in __exit__"
    try:
        with open(filepath, "rb") as ifile:
            result = pickle.load(ifile)
        if check_type:
            new_instance = make_new()
            if new_instance.__class__ != result.__class__:
                # We don't even allow one class to be a subclass of the other
                raise TypeError(("Class {} of loaded file does not match class {} of "
                    + "value returned by make_new()")
                    .format(result.__class__, new_instance.__class__))
    except IOError:
        touch_new(filepath)
        result = make_new()
    try:
        hash(result)
    except TypeError:
        pass
    else:
        warnings.warn("You probably don't want to use pickle_wrap on a hashable (and therefore likely immutable) type")

    def sync():
        print("pickle_wrap syncing")
        with open(filepath, "wb") as ofile:
            pickle.dump(result, ofile)

    yield result, sync
    sync()

Answer 4

I know it's not pretty or clever, but I would just write the individual methods out...

class PersistentList(object):
   ...

   def append(self, o):
      self._autosave()
      self._list.append(o)

   ...etc...