8

I am trying to figure out if it's possible to do view post-processing on my queryset before rendering it in a django template that uses django-endless-pagination for infinite scroll.

I have view-specific logic that omits certain results from the queryset based on context, as well as adding attributes to the objects in the list for use by the templates. This logic cannot be executed via SQL as it is not inherent to the model. It must be done in python.

With django-endless-pagination and other pre-rolled django pagination modules, all the logic seems to be executed by templatetags, thus preventing the ability to do business logic before the rendering stage (which is a django tenet).

Because my view logic runs through the result set before the template tags execute, I'm losing the optimizations offered by this module (like SQL queries with paging e.g. limit 20; offset 20). My code traverses the entire unpaged result list every time the user pages, bypassing the lazy pagination benefits offered by the template tag.

Short of moving my code into the pagination module directly (which I'd rather not do and would require adding a bunch of extra data into the request context for use in a tag), is there any alternative?

Thanks!

randalv
  • 900
  • 9
  • 19

1 Answers1

5

If you look lazy_paginate tag use LazyPaginator class to process the queryset. You can override that class to serve your purpose. In order to do that you need to write Custom Template Tag. More instructions in the code comments.

*my_app/templatetags/custom_pagination_tags.py*

from django import template
from endless_pagination.templatetags.endless import paginate
from endless_pagination.paginators import LazyPaginator

register = template.Library()

Class CustomLazyPaginator(LazyPaginator):

    def page(self, number):
        page_obj = super(CustomLazyPaginator, self).page(number)
        # page function returns page object from that you could access object_list
        object_list = page_obj.object_list
        # Do some processing here for your queryset
        # Do not remove elements otherwise you will put your self in trouble
        # Just add some values in objects as you wanted to
        page_obj.object_list = object_list # override here
        return page_obj

@register.tag
def custom_lazy_paginate(parser, token):
    return paginate(parser, token, paginator_class=CustomLazyPaginator)

Now in template load your custom template tags and use that instead:

{% load custom_pagination_tags %}

{% custom_lazy_paginate queryset %}

Difficult: First Approach To Access Request Context In CustomLazyPaginator Class

Yes there is a way to pass the request context, but in order to do that you need to override paginate tag and also the render method of PaginateNode as you can see here when it calls the paginator_class it does not pass any context information. Below are the steps to achieve that:

Add __init__ method in CustomLazyPaginator:

def __init__(self, *args, **kwargs):
    self.context = kwargs.pop('context', None)
    super(CustomLazyPaginator, self).__init__(*args, **kwargs)

Copy the paginate tag and change the return statement from PaginateNode(paginator_class, objects, **kwargs) to CustomPaginateNode(paginator_class, objects, **kwargs) we will write CustomPaginateNode below.

from endless_pagination.templatetags.endless import PAGINATE_EXPRESSION

@register.tag
def paginate(parser, token, paginator_class=None):
    # Validate arguments.
    try:
        tag_name, tag_args = token.contents.split(None, 1)
    except ValueError:
        msg = '%r tag requires arguments' % token.contents.split()[0]
        raise template.TemplateSyntaxError(msg)

    # Use a regexp to catch args.
    match = PAGINATE_EXPRESSION.match(tag_args)
    if match is None:
        msg = 'Invalid arguments for %r tag' % tag_name
        raise template.TemplateSyntaxError(msg)

    # Retrieve objects.
    kwargs = match.groupdict()
    objects = kwargs.pop('objects')

    # The variable name must be present if a nested context variable is passed.
    if '.' in objects and kwargs['var_name'] is None:
        msg = (
            '%(tag)r tag requires a variable name `as` argumnent if the '
            'queryset is provided as a nested context variable (%(objects)s). '
            'You must either pass a direct queryset (e.g. taking advantage '
            'of the `with` template tag) or provide a new variable name to '
            'store the resulting queryset (e.g. `%(tag)s %(objects)s as '
            'objects`).'
        ) % {'tag': tag_name, 'objects': objects}
        raise template.TemplateSyntaxError(msg)

    # Call the node.
    return CustomPaginateNode(paginator_class, objects, **kwargs)

Remove the following import which we call earlier to avoid calling original paginate function:

from endless_pagination.templatetags.endless import paginate

Override the render method of PaginateNode to pass context to our CustomLazyPaginator class:

from endless_pagination.templatetags.endless import PaginateNode
from endless_pagination import (
    settings,
    utils,
)

class CustomPaginateNode(PaginateNode):
    def render(self, context):
        # Handle page number when it is not specified in querystring.
        if self.page_number_variable is None:
            default_number = self.page_number
        else:
            default_number = int(self.page_number_variable.resolve(context))

        # Calculate the number of items to show on each page.
        if self.per_page_variable is None:
            per_page = self.per_page
        else:
            per_page = int(self.per_page_variable.resolve(context))

        # Calculate the number of items to show in the first page.
        if self.first_page_variable is None:
            first_page = self.first_page or per_page
        else:
            first_page = int(self.first_page_variable.resolve(context))

        # User can override the querystring key to use in the template.
        # The default value is defined in the settings file.
        if self.querystring_key_variable is None:
            querystring_key = self.querystring_key
        else:
            querystring_key = self.querystring_key_variable.resolve(context)

        # Retrieve the override path if used.
        if self.override_path_variable is None:
            override_path = self.override_path
        else:
            override_path = self.override_path_variable.resolve(context)

        # Retrieve the queryset and create the paginator object.
        objects = self.objects.resolve(context)
        paginator = self.paginator(
            objects, per_page, first_page=first_page, orphans=settings.ORPHANS,
            context=context) # <--- passing context here

    # Normalize the default page number if a negative one is provided.
    if default_number < 0:
        default_number = utils.normalize_page_number(
            default_number, paginator.page_range)

    # The current request is used to get the requested page number.
    page_number = utils.get_page_number_from_request(
        context['request'], querystring_key, default=default_number)

    # Get the page.
    try:
        page = paginator.page(page_number)
    except EmptyPage:
        page = paginator.page(1)

    # Populate the context with required data.
    data = {
        'default_number': default_number,
        'override_path': override_path,
        'page': page,
        'querystring_key': querystring_key,
    }
    context.update({'endless': data, self.var_name: page.object_list})
    return ''

Simple: Second Approach To Access Request Context In CustomLazyPaginator Class

Just install django-contrib-requestprovider and add it in middleware in django's settings.py and access current request any where you want as:

from gadjo.requestprovider.signals import get_request

http_request = get_request()
Aamir Rind
  • 38,793
  • 23
  • 126
  • 164
  • 1
    Thanks Aamir. That allows me to move the processing to the tag, but still doesn't allow me to access dynamic context variables that are set in my view and determine what I need to do in the processing. Looks like the base classes would need rewriting to either support passing in "with x=True y=5" type variables to the tag or in some way making the context accessible to read from within subtags. Do you see another way? – randalv Nov 20 '13 at 19:59
  • I have added two approaches to access request context. Have a look. I have not tested my code. You may have some missing imports from endless_pagination. But i have tried to cover up as much as possible. This is full implementation if you play with it you will understand what is happening. – Aamir Rind Nov 21 '13 at 14:19
  • This is not an ideal solution. Having to rewrite a lot of the base endless-pagination code adds technical debt and complexity. Your approach #1 works but has a few bugs still. Can't pass context as a kwarg to self.paginate unless you pop it off in constructor before calling super() or set it after the constructor is called, or there's an exception. Also have to replace 'paginator_class' with 'CustomLazyPaginator' in CustomPaginateNode call). This module is just not set up for easy extendibility. I'll accept your solution because it works (with changes I mentioned) and you answered my question. – randalv Nov 21 '13 at 21:53
  • I told you i wrote this code on the fly. I have not tested it so yes there will be some errors. But my aim was to give you a general idea. Unfortunately this is the only way to do it as django endless pagination module is not easy to extend. Thanks – Aamir Rind Nov 22 '13 at 07:20