Python lazy list

Question

I would like create my own collection that has all the attributes of python list and also knows how to save/load itself into/from a database. Also I want to make the load implicit and lazy, as in it doesn't happen at the point of creation of the list, but waits until its first used.

Is there a single__xxx__method I can override to load the list on first usage of any list property (such aslen,getitem,iter...etc) without having to override them all?

Answer 1

Not a single, but 5 is enough:

from collections import MutableSequence

class Monitored(MutableSequence):
    def __init__(self):
        super(Monitored, self).__init__()
        self._list = []

    def __len__(self):
        r = len(self._list)
        print "len: {0:d}".format(r)
        return r

    def __getitem__(self, index):
        r = self._list[index]
        print "getitem: {0!s}".format(index)
        return r

    def __setitem__(self, index, value):
        print "setitem {0!s}: {1:s}".format(index, repr(value))
        self._list[index] = value

    def __delitem__(self, index):
        print "delitem: {0!s}".format(index)
        del self._list[index]

    def insert(self, index, value):
        print "insert at {0:d}: {1:s}".format(index, repr(value))
        self._list.insert(index, value)

The correct way of checking if something implements the whole list interface is to check if it is a subclass of MutableSequence. The ABCs found in the collections module, of which MutableSequence is one, are there for two reasons:

1. to allow you to make your own classes emulating internal container types so that they are usable everywhere a normal built-in is.

2. to use as argument for isinstance and issubclass to verify that an object implements the necessary functionality:

>>> isinstance([], MutableSequence)
True
>>> issubclass(list, MutableSequence)
True

Our Monitored class works like this:

>>> m = Monitored()
>>> m.append(3)
len: 0
insert at 0: 3
>>> m.extend((1, 4))
len: 1
insert at 1: 1
len: 2
insert at 2: 4
>>> m.l
[3, 1, 4]
>>> m.remove(4)
getitem: 0
getitem: 1
getitem: 2
delitem: 2
>>> m.pop(0)   # after this, m.l == [1]
getitem: 0
delitem: 0
3
>>> m.insert(0, 4)
insert at 0: 4
>>> m.reverse()   # After reversing, m.l == [1, 4]
len: 2
getitem: 1
getitem: 0
setitem 0: 1
setitem 1: 4
>>> m.index(4)
getitem: 0
getitem: 1
1

Answer 2

Not exactly. For emulating things other than lists, there's __getattribute__, but unfortunately Python doesn't consider operators like x[y] or x(y) to be exactly the same as x.__getitem__(y) or x.__call__(y). Operators like that are attributes of the class, not attributes of the instance, as you can see here:

>>> class x(object):
...     def __getattribute__(self, o):
...         print o
... 
>>> x()[3]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'x' object does not support indexing

However, you can take advantage of Python's dynamic nature to effectively eliminate that distinction. If your main concern is to save yourself typing, and to produce less code that needs maintaining, you can do something like this:

class override(object):
    def __init__(self, methodName):
        self.methodName = methodName

    def __get__(self, oself, cls):
        oself._load(self.methodName)
        return getattr(super(oself.__class__, oself), self.methodName)

class LazyList(list):
    def _load(self, name):
        print 'Loading data for %s...' % (name,)

    for methodName in set(dir(list)) - set(dir(object)):
        locals()[methodName] = override(methodName)

You probably don't want to use dir() in real life, but a suitable fixed list of strings could work as a substitute.

Answer 3

__getattribute__

Answer 4

No, there isn't.

Answer 5

There isn't a single method. You have to redefine quite a lot of them. MutableSequence seems to be the modern way of doing it. Here is a version that works with Python 2.4+::

class LazyList(list):
    """List populated on first use."""
    def __new__(cls, fill_iter):

        class LazyList(list):
            _fill_iter = None

        _props = (
            '__str__', '__repr__', '__unicode__',
            '__hash__', '__sizeof__', '__cmp__', '__nonzero__',
            '__lt__', '__le__', '__eq__', '__ne__', '__gt__', '__ge__',
            'append', 'count', 'index', 'extend', 'insert', 'pop', 'remove',
            'reverse', 'sort', '__add__', '__radd__', '__iadd__', '__mul__',
            '__rmul__', '__imul__', '__contains__', '__len__', '__nonzero__',
            '__getitem__', '__setitem__', '__delitem__', '__iter__',
            '__reversed__', '__getslice__', '__setslice__', '__delslice__')

        def lazy(name):
            def _lazy(self, *args, **kw):
                if self._fill_iter is not None:
                    _fill_lock.acquire()
                    try:
                        if self._fill_iter is not None:
                            list.extend(self, self._fill_iter)
                            self._fill_iter = None
                    finally:
                        _fill_lock.release()
                real = getattr(list, name)
                setattr(self.__class__, name, real)
                return real(self, *args, **kw)
            return _lazy

        for name in _props:
            setattr(LazyList, name, lazy(name))

        new_list = LazyList()
        new_list._fill_iter = fill_iter
        return new_list