0

According to App Engine restriction on inequality filters, there are some suggestions to implement something like advanced searches do (filtering the results by limiting the ranges on many properties) by filtering the properties manually in RAM:

how to effectively run two inequality filters on queries in app engine

So, is it feasible to do this amount of sorting and filtering in RAM for large datasets? is there any Java sample code to demonstrates proper implementation? is it a good idea to stick with traditional RDBMS in order to avoid this drawback?

Community
  • 1
  • 1
Ali Mortazavi
  • 41
  • 1
  • 10
  • 1
    Relational databases have their own drawbacks, including performance on complex queries. So you can not generalize like that - you need to look at specific project/data model requirements. – Andrei Volgin Aug 08 '16 at 16:01
  • Thanks @AndreiVolgin. About the first question, do you know how to handle multiple inequality queries for App Engine Datastore? – Ali Mortazavi Aug 10 '16 at 08:44
  • Yes, I do it for 7 years now :) But there is no one simple answer to your question. It all depends on your data model, distribution of data, and project requirements. – Andrei Volgin Aug 10 '16 at 13:55

1 Answers1

2

As Andrei has mentioned, there isn't a general solution to your problem of needing multiple inequality filter conditions. It really depends on your data, queries and application requirements.

Here are some possible solutions you could use:

Perform some filtering in the application. If you have two inequality conditions, A and B, and know that majority (e.g. > 80%) of the entities that meet condition A will meet condition B, then you could query without condition B against Datastore, and filter the returned results in your application code. This lets you continue to use Datastore, and the efficiency hit shouldn't be too bad, since you know > 80% will match.

However, extending this solution to more inequalities, or cases where the overlap between condition A and condition B is not great, will result in very inefficient data retrieval.

Secondary Search Index. It's possible that if you have very complicated filtering / sorting logic, you have something more akin to a search problem, for which Google App Engine Search might be more suitable. Search allows you to run very flexible queries over documents in a search index, including multiple inequality queries.

I will point out that search only offers eventual consistency, and indexes are limited to 10GB (but can be extended to 200GB on request).

Daniel Chia
  • 21
  • 1