2

I have been searching away at this but cannot seem to find a simple example of how to access the ndb datastore from a python microservice which lives in the same application as other non service modules.

What I want to do is access the actual Model classes of the datastore, i.e. Users...and then query on that class from the microservice.

I know you cannot use the google cloud datastore api in app engine standard, but there must be another way, surely?

The same applies to the shared memcache, if I make an API call to an endpoint in a module (no service) from the microservice and set something in memcache at the endpoint, I cannot see that in the microservice. So when Google talk about shared datastore and memcache across everything in the same application including microservices, how do they propose you access it?

I'm sure I'm missing something, just can't find it.

Chez
  • 961
  • 1
  • 9
  • 20
  • 1
    Your question is a bit unclear. What do you mean by "non service modules"? Have you had a look at this doc: https://cloud.google.com/appengine/docs/standard/python/ndb/? – LundinCast Oct 13 '18 at 11:11
  • Yes I have looked at that. It just describes how to use the ndb datastore client library in app engine. What I mean by no service modules is normal app-engine modules that are not microservices. Accessing ndb in those is simple via the ndb client import and the defined Models, however, there is no clear documentation on how I could do something like ndb.query(Users).fetch() from a microservice in the same app engine project if I do not have access to the class Users from a model. – Chez Oct 13 '18 at 12:09

1 Answers1

2

To clarify: the microservice term doesn't mean anything particularly special in GAE with respect to Datastore interaction (or otherwise) - all GAE services/modules are equals from this perspective. What makes a GAE service/module a microservice is simply the functionality it performs, not its implementation or how it uses the infrastructure, see Microservices Architecture on Google App Engine.

All services of the same GAE project/application that access the datastore using the ndb library can do that by default, without any restriction or additional service configurations.

The only trick is that all services referencing a particular entity type must have a consistent view of that entity's model definition. This is a requirement coming from the particular implementation of the ndb client library, not from the datastore itself.

The simplest way to obtain such consistent view is IMHO by sharing the same ndb model definition source file(s), which can be achieved by symlinking the same actual source file(s) (or the directories holding them) across the multiple service/module directories, as described in Sharing entities between App Engine modules.

In other words all your services/modules needing to query/access/reference Users entities one way or the other would actually have the same Users model definition available to do so.

Care must be exercised when deploying changes to the model definitions (be it in different services or even between different versions of the same service) either by:

  • ensuring backward-compatibility with migration strategies
  • proper deployment orchestration - i.e. ensuring incompatible versions/services never run simultaneously

The same technique can be used for memcache, in a similar manner: a shared source code file would export the definitions of memcache keys for the memcached values that need to be shared across services. Or, better yet, provide actual read/write functions for the corresponding data, to ensure not only that the data is stored under the right keys, but it also has the matching format.

With such shared code in place the memcached data representing certain pieces of information becomes shared across the same app's services/modules (again, regardless of them being microservices or not). Almost as good as the info shared via the datastore, except without transaction support to guarantee consistency.

Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97
  • Hi, Dan, thanks for this. Could you provide a brief example of the models.yaml I'm not clear if yo mean the complete definition or just the filename. – Chez Oct 13 '18 at 15:54
  • Did the update address it? If not - please ask it again as I didn't get it :) – Dan Cornilescu Oct 13 '18 at 16:14
  • Yeah, that got it, thanks, you really have saved me a bunch of time. – Chez Oct 13 '18 at 16:18
  • To see the same datastore (and memcache) content from all services in development you need to run all services through the same development server, see https://stackoverflow.com/a/40679959/4495081. Or use the same datastore emulator (doesn't work for memcache): https://stackoverflow.com/a/50860147/4495081 – Dan Cornilescu Oct 14 '18 at 15:51