4

We are currently running a combined AppEngine / GCE app and thus far have kept all of our datastore access on the AppEngine side of things. Now we are exploring also allowing our GCE instance to make some queries into the (shared) datastore. To start, I'm trying to figure out how to run things locally. What we have so far:

  • A Go devappserver running
  • A Go standalone binary that wants to issues queries to the devappserver datastore.
  • We installed ('go get') google-api-go-client/datastore/v1beta2 so that we can use an API instead of issuing direct HTTP calls. However we are definitely willing to issue direct HTTP calls if this API library won't work in development.
  • We have service accounts set up (we already access GCS from GCE) but I doubt that's relevant for running locally...

I've seen some docs but they (a) only talk about Python & Java, and (b) discuss connecting to the (standalone) development datastore server, as opposed to the datastore embedded in AppEngine's devappserver (if those are even different?). There is also the following answer here on StackOverflow, but again it discusses connecting to the standalone development datastore server:

How to connect to the local google cloud Datastore db?

Any pointers would be much appreciated!

  • Ian
Grokify
  • 15,092
  • 6
  • 60
  • 81
Ian Rose
  • 667
  • 1
  • 6
  • 18

2 Answers2

4

Currently this is not possible in the development environment for several reasons. The Google Cloud Datastore tool (gcd.sh) uses the java development server. However when developing go for App Engine you use the python development server, which has different underlying storage. There is a bug to track this issue on the github page.

You can still develop a Google Cloud Datastore application in go however there are many bugs in the current go client library. Unfortunately, the development server does not currently support the JSON API, which the go library uses (see the note at the top of the page).

Update: I wanted to make sure proppy's comment was seen as part of the answer. His suggestion does provide a way to use the protocol version of the API, which is probably more stable than the go client library above. It could also let you use the gcd.sh tool to test this in the development server. You will have to craft the HTTP requests yourself though, and you won't be able to share the data in the datastore between your application and the Cloud Datastore in development. However it is definitely a good workaround and lets you use the Cloud Datastore API, which as it develops will be easier to work with than other workarounds. From proppy:

Note that you can still use Cloud Datastore Protobuf HTTP API with Go. The protobuf definition is available on GitHub, you can compile it to Go code using the Go protobuf compiler plugin and then send POST HTTP requests to /datastore/{version}/datasets/{datasetId}/{method}.

Patrick Costello
  • 3,616
  • 17
  • 22
  • 2
    Note that you can still use Cloud Datastore Protobuf HTTP API with Go. The [protobuf definition](https://github.com/GoogleCloudPlatform/google-cloud-datastore/blob/v1beta2-rev1-2.1.0/proto/datastore_v1.proto) is available on GitHub, you can compile it to Go code using the [Go protobuf compiler plugin](https://code.google.com/p/goprotobuf/) and then send POST HTTP requests to `/datastore/{version}/datasets/{datasetId}/{method}`. – proppy Feb 19 '14 at 18:18
  • 1
    It's 2018 and apparently, if you use Go on GAE Standard using Datastore, you still CANNOT run it locally with same code as production.Every time I see a ctx := context.Background(), rage takes over. – Feu Mar 19 '18 at 16:13
0

If the use case from your "GO" app server is straight forward enough, you may want to implement access by using an API call to your GAE service (perhaps extending the service to receive the API calls).

This has the added benefit of only having to make changes in one place if your datastore definitions or functions change.

ErstwhileIII
  • 4,829
  • 2
  • 23
  • 37
  • This workaround also has the benefit that, unlike using the Cloud Datastore API, you will be using the same development server (and thus can access the same underlying data). – Patrick Costello Feb 18 '14 at 22:50
  • I had considered that too but it also means that we need to build in some kind of auth (so that only our GCE instance will be able to access the GAE handler). Kind of a pain. Given Patrick's answer below I think we'll go with a hybrid solution where we call a handler like you are describing in devmode, but we use redesign our app so that it doesn't need datastore access (just GCS) in production. – Ian Rose Feb 18 '14 at 23:11