0

The NDB Datastore forbids multiple inequality queries on different properties. To get around this, I thought that the solution might be to combine the results of multiple independent queries. I found this 2011 question which recommends geohashing, with which I am not familiar. So, perhaps there is a better solution today.

Consider these two queries:

q1 = User.query(User.age < 18).fetch()
q2 = User.query(User.city != 'New York City').fetch()

I attempt to join them like this: 

results = set(q1).intersection(q2)

However, I encounter TypeError: Model is not immutable.

My questions:

  • Is there a better way to deal with multiple inequality filters on different properties?
  • If not, how can I resolve the TypeError above?

Thank you for the assistance.

Community
  • 1
  • 1
hyang123
  • 1,208
  • 1
  • 13
  • 32
  • Possible duplicate of [App Engine NDB query with multiple inequalities?](http://stackoverflow.com/questions/16114762/app-engine-ndb-query-with-multiple-inequalities) – Brent Washburne Nov 05 '15 at 23:17
  • It is somewhat. I had read that answer, and was curious about how might it be possible to combine multiple queries, which is what I am partially asking in this question. – hyang123 Nov 06 '15 at 02:18
  • 1
    It would be more efficient to do a keys only query, you can then create the set based on keys, and then only fetch the unique set. (though ndb caching would also save on efficiency) but as you found you can't create a set of models. So back to keys. – Tim Hoffman Nov 06 '15 at 10:53

3 Answers3

2

If you can restructure your User model, you could put in some more properties to make the queries simpler. For example, if you query on the same age ranges, then make a property that encodes the ranges:

age_range = ndb.IntegerProperty()    # 0 = 0-17, 1 = 18-29, 2 = 30-39, etc.

Then you can have:

q1 = User.query(User.age_range == 0).query(User.city != 'New York City').fetch()

If your data set is small enough, you can use @TimHoffman's approach:

q1 = User.query(User.age < 18).fetch(keys_only=True)
q2 = User.query(User.city != 'New York City').fetch(keys_only=True)
results = ndb.get_multi(set(q1).intersection(q2))

A more heavyweight approach, that will scale up to big data sets, is the MapReduce library. You can put in multiple filters to reduce your data set.

Brent Washburne
  • 12,904
  • 4
  • 60
  • 82
0

I had a similar problem. My query was:

@classmethod:
def getUnReadMessages(cls, user, date)
    return cls.query(ndb.AND(cls.created <= date,
                             cls.receiver_key == user.key,
                             cls.status != READ))

But appengine didn't let me do it. So I solved changing one inequality to:

@classmethod:
def getUnReadMessages(cls, user, date)
    return cls.query(ndb.AND(cls.created <= date,
                             cls.receiver_key == user.key,
                             ndb.OR(cls.status == SEND,
                                    cls.status == RECEIVED)))

Problem solved! I hope this helps you.

rcmstark
  • 1,031
  • 1
  • 14
  • 18
0

There are two options:

Change your data models

Add more properties or adapt the current ones so you can query the object according to the restrictions of the Datastore. This may imply categorizing continuous variables.

Find a workaround

You can make the most significant query at first and then manually filter the result. Have in mind the following considerations:

  • You can use projection to make the query more efficient.
  • Make the results iterable (iter).
  • Use get_multi for a list of keys.

Your code may look like this:

query_iter = User.query(User.age < 18).iter(projection=[User.city])
query_keys = [u.key() for u in query_iter if u.city != 'New York City']
query = ndb.get_multi(query_keys)

Or

query = [u for u in User.query(User.age < 18).fetch() if u.city != 'New York City']