4

So I have a django project which was built on 1.6.5 and now I'm migrating it to 1.9.5. I successfully migrated it to 1.7.0 and then to 1.8.0. When doing so from 1.8.0 to 1.9.0, I had to replace SortedDict by collections.OrderedDict. Now I'm encountering this error when I do python manage.py test:

    File "forum/models/base.py", line 134, in iterator
    key_list = [v[0] for v in self.values_list(*values_list)]
  File "venv/mystuff/lib/python2.7/site-packages/django/db/models/query.py", line 258, in __iter__
    self._fetch_all()
  File "venv/mystuff/lib/python2.7/site-packages/django/db/models/query.py", line 1074, in _fetch_all
    self._result_cache = list(self.iterator())
  File "forum/models/base.py", line 134, in iterator
    key_list = [v[0] for v in self.values_list(*values_list)]
  File "venv/mystuff/lib/python2.7/site-packages/django/db/models/query.py", line 258, in __iter__
    self._fetch_all()
  File "venv/mystuff/lib/python2.7/site-packages/django/db/models/query.py", line 1074, in _fetch_all
    self._result_cache = list(self.iterator())
  File "forum/models/base.py", line 134, in iterator
    key_list = [v[0] for v in self.values_list(*values_list)]
  File "venv/mystuff/lib/python2.7/site-packages/django/db/models/query.py", line 725, in values_list
    clone = self._values(*fields)
  File "venv/mystuff/lib/python2.7/site-packages/django/db/models/query.py", line 671, in _values
    clone = self._clone()
  File "venv/mystuff/lib/python2.7/site-packages/django/db/models/query.py", line 1059, in _clone
    query = self.query.clone()
  File "venv/mystuff/lib/python2.7/site-packages/django/db/models/sql/query.py", line 298, in clone
    obj._annotations = self._annotations.copy() if self._annotations is not None else None
  File "/opt/python/python-2.7/lib64/python2.7/collections.py", line 194, in copy
    return self.__class__(self)
  File "/opt/python/python-2.7/lib64/python2.7/collections.py", line 57, in __init__
    self.__update(*args, **kwds)
  File "venv/mystuff/lib64/python2.7/abc.py", line 151, in __subclasscheck__
    if subclass in cls._abc_cache:
  File "venv/mystuff/lib64/python2.7/_weakrefset.py", line 72, in __contains__
    wr = ref(item)
RuntimeError: maximum recursion depth exceeded

The solutions on other similar questions ask to upgrade python to 2.7.5 but I'm already running it on 2.7.11.

EDIT: forum/models/base.py

def iterator(self):
    cache_key = self.model._generate_cache_key("QUERY:%s" % self._get_query_hash())
    on_cache_query_attr = self.model.value_to_list_on_cache_query()

    to_return = None
    to_cache = {}

    with_aggregates = len(self.query.aggregates) > 0
    key_list = self._fetch_from_query_cache(cache_key)

    if key_list is None:
        if not with_aggregates:
            values_list = [on_cache_query_attr]

            if len(self.query.extra):
                values_list += self.query.extra.keys()

            key_list = [v[0] for v in self.values_list(*values_list)] #Line 134
            to_cache[cache_key] = (datetime.datetime.now(), key_list)
        else:
            to_return = list(super(CachedQuerySet, self).iterator())
            to_cache[cache_key] = (datetime.datetime.now(), [
                (row.__dict__[on_cache_query_attr], dict([(k, row.__dict__[k]) for k in self.query.aggregates.keys()]))
                for row in to_return])
    elif with_aggregates:
        tmp = key_list
        key_list = [k[0] for k in tmp]
        with_aggregates = [k[1] for k in tmp]
        del tmp

    if (not to_return) and key_list:
        row_keys = [self.model.infer_cache_key({on_cache_query_attr: attr}) for attr in key_list]
        cached = cache.get_many(row_keys)

        to_return = [
            (ck in cached) and self.obj_from_datadict(cached[ck]) or ToFetch(force_unicode(key_list[i])) for i, ck in enumerate(row_keys)
        ]

        if len(cached) != len(row_keys):
            to_fetch = [unicode(tr) for tr in to_return if isinstance(tr, ToFetch)]

            fetched = dict([(force_unicode(r.__dict__[on_cache_query_attr]), r) for r in
                          models.query.QuerySet(self.model).filter(**{"%s__in" % on_cache_query_attr: to_fetch})])

            to_return = [(isinstance(tr, ToFetch) and fetched[unicode(tr)] or tr) for tr in to_return]
            to_cache.update(dict([(self.model.infer_cache_key({on_cache_query_attr: attr}), r._as_dict()) for attr, r in fetched.items()]))

        if with_aggregates:
            for i, r in enumerate(to_return):
                r.__dict__.update(with_aggregates[i])


    if len(to_cache):
        cache.set_many(to_cache, 60 * 60)

    if to_return:
        for row in to_return:
            if hasattr(row, 'leaf'):
                row = row.leaf

            row.reset_original_state()
            yield row

django/db/models/query.py:

def _fetch_all(self):
    if self._result_cache is None:
        self._result_cache = list(self.iterator()) #Line 1074
    if self._prefetch_related_lookups and not self._prefetch_done:
        self._prefetch_related_objects()

django/db/models/query.py

def __iter__(self):
    """
    The queryset iterator protocol uses three nested iterators in the
    default case:
        1. sql.compiler:execute_sql()
           - Returns 100 rows at time (constants.GET_ITERATOR_CHUNK_SIZE)
             using cursor.fetchmany(). This part is responsible for
             doing some column masking, and returning the rows in chunks.
        2. sql/compiler.results_iter()
           - Returns one row at time. At this point the rows are still just
             tuples. In some cases the return values are converted to
             Python values at this location.
        3. self.iterator()
           - Responsible for turning the rows into model objects.
    """
    self._fetch_all() #Line 258
    return iter(self._result_cache)

UPDATE: My Django Log is showing this:

/forum/settings/base.py TIME: 2017-06-27 06:49:53,410 MSG: base.py:value:65 Error retrieving setting from database (FORM_EMPTY_QUESTION_BODY): maximum recursion depth exceeded in cmp
/forum/settings/base.py TIME: 2017-06-27 06:49:53,444 MSG: base.py:value:65 Error retrieving setting from database (FORM_MIN_NUMBER_OF_TAGS): maximum recursion depth exceeded
Priyansh Agrawal
  • 330
  • 2
  • 19
  • Could you provide `manage.py` snippet responsible for error? – custom_user Jun 27 '17 at 10:11
  • 1
    Could you provide the complete method for `forum/models/base.py", line 134`. It looks like that recurses. –  Jun 27 '17 at 10:13
  • @Melvyn I've edited the question with those methods. – Priyansh Agrawal Jun 27 '17 at 10:20
  • Try doing a git bisect between Django 1.9 and 1.8 to figure out the changes to Django that cause your code to stop working. This might give you a hint how to fix your method. – Alasdair Jun 27 '17 at 11:04
  • @Alasdair are you suggesting to do a bisect of my project or django? Can I do a bisect even if I didn't commit when changing from 1.7 to 1.8? – Priyansh Agrawal Jun 27 '17 at 11:24
  • I meant bisect Django, since it is the change from Django 1.8 to 1.9 that is causing you problems. Anyway... looks like @Melvyn has done it for you. – Alasdair Jun 27 '17 at 11:41

1 Answers1

3

So you call values_list() from your iterator, which clones the QuerySet, which iterates the QuerySet, which calls your iterator, which clones the Queryset...

The API changed there in this commit. It should provide you with ample information to reimplement your Queryset.

On a different note, it looks like Django has implemented query caching itself, so before refactoring you might take a look if your CachedQuerySet has become obsolete.

  • Excuse me for being a noob, but it seems like to use that commit, I'll have to upgrade django to 1.11.2 and I currently don't want that. Also, if I decide to change the QuerySet, I'll be changing official django code, which I don't want. Is there something wrong with my method in models/base.py or can I change something in it to make the code work? – Priyansh Agrawal Jun 27 '17 at 11:50
  • 2
    No, github isn't very clear. This got introduced in 1.9 alpha 1. So it's in 1.9 you're currently testing. It lists the changes that made your work not function anymore. The green is the new code and you can see that the `iterator` method now is supposed to return an iterator on a fixed set of values to avoid this infinite recursion. –  Jun 27 '17 at 11:58
  • Looking at my current iterator method provided in the question, what changes do you suggest so that it would be able to return an iterator on a fixed set of values? – Priyansh Agrawal Jun 27 '17 at 12:05