MongoDB using an OR clause in mongoengine

Question

I'm using python's mongoengine to query MongoDB, and have loved it for the most part, but I'm having an issue with an advanced query.

Here's my model

class ContentItem(Document):
    account = ReferenceField(Account)
    creator = ReferenceField(User)
    public = BooleanField(default=False) 
    last_used = DateTimeField(default=datetime.now)

I would like to make a query for all ContentItem's that are of a particular account, and are either created by the logged in user or are public. Here's the query I wrote

query = ContentItem.objects.filter( (Q(account=account) & Q(public=True)) |  (Q(account=account) & Q(creator=logged_in_user)) ).order_by('-last_used')

or:

query = ContentItem.objects.filter( Q(account=account) & ( Q(public=True) |  Q(creator=logged_in_user) ) ).order_by('-last_used')

But these seem to be XOR where if either public, or the creator but not both. Is this expected?

Am I overlooking something? Should I do this directly with mongodb instead of mongoengine?

My current workaround is to make two different queries and combine the results, but as the # of Content Items gets larger the result is taking a long time to come back because I need to get all items before I can order them, thereby losing all the benefit of (django) paginated results.

score 12 · Answer 1 · edited Jul 01 '20 at 20:58

12

MongoEngine docs say otherwise. Please check this:

edited Jul 01 '20 at 20:58

Gianfranco P.

10,049
6
51
68

answered May 31 '19 at 20:10

Richil

141
1
3

8

Mongoengine maintainer here, accepted answer is wrong, this is the correct answer – bagerard Aug 15 '19 at 08:33

score 6 · Answer 2 · answered Nov 17 '15 at 13:10

The correct way to do the query is to use bitwise operations | and & the way you wrote it in your question:

query = ContentItem.objects.filter( (Q(account=account) & Q(public=True)) |  (Q(account=account) & Q(creator=logged_in_user)) ).order_by('-last_used')

Note: using the standard Python boolean operators and and or will not work. This is explained in the MongoEngine documentation.

score 5 · Accepted Answer · answered Nov 21 '11 at 21:34

5

The mongoengine documentation is apparently incorrect in this case. Instead of using the bitwise operators "&" and "|", you should use the standard operators "and" and "or".

So your first query becomes:

query = ContentItem.objects.filter( (Q(account=account) and Q(public=True)) or  (Q(account=account) and Q(creator=logged_in_user)) ).order_by('-last_used')

answered Nov 21 '11 at 21:34

apiguy

5,282
1
23
24

that did the trick. i added a note on the github account to fix the documentation https://github.com/hmarr/mongoengine/issues/363 – MattoTodd Nov 21 '11 at 21:44
This is NOT the case; see the github issue referenced above. I tried this, but using `or` instead of `|` will not apply the filter. Using `&` and `|` works just fine for me. – Paul Aug 27 '12 at 12:33
2

@Paul - perhaps you did not notice that this post was 9 months old, and that the bug has since been fixed? – apiguy Aug 28 '12 at 01:58
8

@apiguy Nope. You were wrong at the first place. It's impossible to override the `and`/`or` in Python, instead you can override `|`/`&`. The source code of Q (mongoengine/queryset/visitor.py )and its history confirms that – Leonardo.Z Nov 05 '13 at 07:56
5

You have to use bitwise operators. You cannot use or, and to combine queries as Q(a=a) or Q(b=b) is not the same as Q(a=a) | Q(b=b). As Q(a=a) equates to true Q(a=a) or Q(b=b) is the same as Q(a=a). Docs -> https://mongoengine-odm.readthedocs.org/guide/querying.html#advanced-queries – Adil Abbasi Aug 03 '15 at 06:33

score 2 · Answer 4 · edited Aug 28 '20 at 14:32

https://github.com/MongoEngine/mongoengine/blob/master/tests/queryset/transform.py

from line 134:

def test_raw_query_and_Q_objects(self):
    query = Foo.objects(__raw__={'$nor': [{'name': 'bar'}]})._query
    self.assertEqual(query, {'$nor': [{'name': 'bar'}]})

    q1 = {'$or': [{'a': 1}, {'b': 1}]}
    query = Foo.objects(Q(__raw__=q1) & Q(c=1))._query
    self.assertEqual(query, {'$or': [{'a': 1}, {'b': 1}], 'c': 1})

score 2 · Answer 5 · edited Feb 16 '20 at 11:46

2

you are probably import the wrong Q

from mongoengine.queryset.visitor import Q as mongo_Q

from django.db.models import Q as normal_Q

edited Feb 16 '20 at 11:46

Ghasem

14,455
21
138
171

answered Nov 28 '17 at 15:07

user3484569

71
5

MongoDB using an OR clause in mongoengine

5 Answers5

Linked