NeedIndexError in Google App Engine, despite indexes being served

Question

I have uploaded my index.yaml file using the command line SDK:

Unfortunately I now have some of my entities indexed twice (but they are all serving):

But I am still getting a "Need Index Error" on running the page:

NeedIndexError: no matching index found. recommended index is:
- kind: RouteDetails
  ancestor: yes
  properties:
  - name: RouteName
    direction: desc

The suggested index for this query is:
- kind: RouteDetails
  ancestor: yes
  properties:
  - name: RouteName
    direction: desc

How can I get Google App Engine to recognise my entity's index?

And how do I delete the duplicates? (Do I need to?)

by the way, I also looked at this question: http://stackoverflow.com/questions/29807215/google-app-engine-error-needindexerror-no-matching-index-found and this one: http://stackoverflow.com/questions/33388390/needindexerror-at-google-app-engine-forever which got me part-way but still getting this error — Yvonne Aburrow, Mar 24 '17 at 16:18
If you look carefully you will see that the two indexes that look the same are not. Not the case for reason/Reason. These are two different indexes on two different property names "reason" and "Reason" — Tim Hoffman, Mar 25 '17 at 07:30
Yes, I added the one with lower-case 'reason' by accident, and then added the one with upper-case Reason as an attempt to correct it, and ended up with two. — Yvonne Aburrow, Mar 27 '17 at 08:26

score 1 · Accepted Answer · answered Mar 24 '17 at 20:36

Datastore requires explicit indexes to be created for each query type when the query scans over more than one property or is an ancestor query. And kinds will certainly be indexed more than once if you have different query types.

For example:

SELECT * FROM RouteDetails
WHERE __key__ HAS ANCESTOR KEY(ParentKind, 'foo')
ORDER BY RouteName ASC

requires an ascending index.

- kind: RouteDetails
  ancestor: yes
  properties:
  - name: RouteName
    direction: asc

And

SELECT * FROM RouteDetails
WHERE __key__ HAS ANCESTOR KEY(ParentKind, 'foo')
ORDER BY RouteName DESC

requires a separate descending index.

- kind: RouteDetails
  ancestor: yes
  properties:
  - name: RouteName
    direction: desc

https://cloud.google.com/datastore/docs/concepts/indexes

In your case, it appears you are performing an ancestor query with a descending ORDER BY of the RouteName property and adding the suggested index to your index.yaml file should solve the problem for you.

As for the suspected "duplicates", which indexes need to exist depend on the specific queries your application performs.

But if you determine that you have extra unused indexes, the instructions for vacuuming indexes can be found here: https://cloud.google.com/datastore/docs/tools/indexconfig#Datastore_Deleting_unused_indexes

Thanks, I've now successfully updated my indexes, but I am still getting the same error. I'm not really clear on how the ancestors thing actually works, and am wondering if that's what is causing the error — Yvonne Aburrow, Mar 27 '17 at 10:28
So it turned out I was using the wrong query, as it shouldn't have had an ancestor key in it at all. But I have got it working and serving a piece of data from my datastore, so Yay, and thank you. — Yvonne Aburrow, Mar 27 '17 at 12:36

score 1 · Answer 2 · answered Mar 26 '17 at 02:35

1

The index includes the order by direction - you can see the up arrows in the console view indicating all fields ascending.

The suggested index is a descending index on one of the properties.

Your 'duplicate' indexes have been introduced by the reason field, which you indexed as both capital r and lower case r, which are different named fields

answered Mar 26 '17 at 02:35

Nick

1,822
10
9

Thanks. Yes, I added the one with lower-case 'reason' by accident, and then added the one with upper-case Reason as an attempt to correct it, and ended up with two, which was why I needed to know how to get rid of them. – Yvonne Aburrow Mar 27 '17 at 10:27

score 0 · Answer 3 · answered Mar 27 '17 at 12:40

Just in case anyone else stumbles across this question and is having a similar problem.

The reason it was looking for an index with ancestor: yes is that I was using the wrong query, and it should not have had an ancestor key in it at all.

Here is my new query:

class RouteDetails(ndb.Model):
    """Get list of routes from Datastore """
    RouteName = ndb.StringProperty()

    @classmethod
    def query_routes(cls):
        return cls.query().order(-cls.RouteName)


class RoutesPage(webapp2.RequestHandler):
    def get(self):
        adminLink = authenticate.get_adminlink()
        authMessage = authenticate.get_authmessage()
        self.output_routes(authMessage,adminLink)

    def output_routes(self,authMessage,adminLink):
        self.response.headers['Content-Type'] = 'text/html'
        html = templates.base
        html = html.replace('#title#', templates.routes_title)
        html = html.replace('#authmessage#', authMessage)
        html = html.replace('#adminlink#', adminLink)
        self.response.out.write(html + '<ul>')
        list_name = self.request.get('list_name')
        #version_key = ndb.Key("List of routes", list_name or "*notitle*")
        routes = RouteDetails.query_routes().fetch(20)

        for route in routes:
            self.response.out.write('<li>%s</li>' % route)
        self.response.out.write('</ul>' + templates.footer)

I was using this page of the documentation, which doesn't tell you how to construct a query for a kind with no ancestor.

https://cloud.google.com/appengine/docs/standard/python/datastore/queries

I posted this as an answer so I could put the code in. Note that the query above returns the route with all its attributes, whereas you probably want to be more specific. — Yvonne Aburrow, Mar 27 '17 at 12:41

NeedIndexError in Google App Engine, despite indexes being served

3 Answers3