0

I have been trying to dynamically construct list of filters based on some function arguments. I build a list of predicates using Q objects and finally I construct one predicate with:

filters = []
... some code appending Q objects to filters ...
combined_filter = reduce(and_, filters)

Then I query my database objects with:

MyModel.objects.filter(combined_filter)

I have noticed a bit odd behaviour when I try to combine Q objects with operator.and_.

For example comparing output of:

items = Item.objects.filter(and_(is_metal, ~is_wood))
print([i.name for i in items])

items = Item.objects.filter(is_metal and ~is_wood)
print([i.name for i in items])

items = Item.objects.filter(is_metal, ~is_wood)
print([i.name for i in items])

I get:

['Table', 'Container']
['Container']
['Table', 'Container']

What is the reason for different behaviour between and_ and and?

My expected output is to get just ['Container'] (see below for full example, "Container" is the only thing with only "metal" as material, "Table" should be excluded because it also has "wood").

Followup question would be: how do I get behaviour of and when using reduce?

My django version is 2.0.7 I have reproduced that exact problem on https://repl.it/repls/AgonizingBossyEfficiency

In case above link dies all code I've modified is below:

models.py:

from django.db import models

class Material(models.Model):
  name = models.CharField(max_length=50)

class Item(models.Model):
  name = models.CharField(max_length=50)
  materials = models.ManyToManyField(Material)

views.py:

from django.shortcuts import render
from django.db import transaction
from django.db.models import Q
from operator import and_
from .models import Material, Item

# Create your views here.
def home(request):
    with transaction.atomic():
      metal = Material(name="metal")
      metal.save()
      wood = Material(name="wood")
      wood.save()

      table = Item(name="Table")
      table.save()
      table.materials.add(metal, wood)
      table.save()

      chair = Item(name="Chair")
      chair.save()
      chair.materials.add(wood)
      chair.save()

      container = Item(name="Container")
      container.save()
      container.materials.add(metal)
      container.save()

      is_metal = Q(materials__name__in = ('metal',))
      is_wood = Q(materials__name__in = ('wood',))

      items = Item.objects.filter(and_(is_metal, ~is_wood))
      print([i.name for i in items])

      items = Item.objects.filter(is_metal and ~is_wood)
      print([i.name for i in items])

      items = Item.objects.filter(is_metal, ~is_wood)
      print([i.name for i in items])

      raise Exception('nope')
    return render(request, 'main/index.html')
Ilja Everilä
  • 50,538
  • 7
  • 126
  • 127
elmo
  • 1,189
  • 1
  • 10
  • 35
  • 1
    Somewhat related: https://stackoverflow.com/questions/19579372/python-django-what-is-the-difference-between-the-and-operator-and-the-o. – Ilja Everilä Apr 28 '19 at 18:00

1 Answers1

0

As pointed out by Ilja my question is effectively a duplicate.

My main problem was thinking that operator.and_ applies logical and. However that is not the case, operator.and_ is a bitwise operator.

If I compare behaviour of operator.and_(is_metal, is_wood) to is_metal & is_wood (as opposed to is_metal && is_wood == is_metal and is_wood) I do get the same results.

So my problem was that I was using bitwise and where I wanted logical and.

elmo
  • 1,189
  • 1
  • 10
  • 35
  • 2
    Careful there. The [documented way](https://docs.djangoproject.com/en/2.2/topics/db/queries/#s-complex-lookups-with-q-objects) of combining Q objects is with bitwise and, i.e. `&` or `_and` which are identical as you correctly concluded. However, as described in the question marked as a duplicate, when using `and`, you end up with the last Q object only, i.e. `filter(is_metal and ~is_wood)` is equivalent to `filter(~is_wood)` or even `filter('hi there' and ~is_wood)`. This is hardly what you want. – Endre Both Apr 28 '19 at 18:52
  • 1
    The reason you got the expected results with the wrong `and` and unexpected results with the correct `_and` lies in the intricacies of correctly defining a multi-criteria filter for the many side of an x-to-many relationship - this is what you should be [looking](https://stackoverflow.com/questions/54841016/filtering-on-many-to-many-relations-that-fulfill-a-set-of-criteria/54844688#54844688) [into](https://stackoverflow.com/questions/55540927/django-orm-not-generating-correct-sql-for-many-to-many-not-in/55542110#55542110). – Endre Both Apr 28 '19 at 19:01
  • Thanks @EndreBoth. Both comments are useful. I've worked out first myself after a bit of pondering and then I realised my issue with many-to-many query. I still find `filter` and `Q.__and__` quite confusing. – elmo Apr 28 '19 at 19:08