0

I am working with an older AppEngine GCP application and I am trying to setup a local environment with an emulated representation of the user profiles stored in datastore. For this, I am trying to export the gmail user-profiles from my GCP workspace and import these into my datastore emulator.

Is the local ndb datastore and the datastore emulator completely disconnected from eachother? Because I cant seem to interact with the data imported to the emulator. Is it possible to import it to my local datastore.db directly?

So far, this is what I am doing: I went into GCP and exported the datastore contents as a folder with gzip files. I unpacked them and stored them on my computer.

I run my dev_appserver.py with "--support_datastore_emulator true" so that I get a datastore emulator port, then I use this in postman:

POST request - localhost:[my emulator port]/v1/projects/[my project]:import

body - {"input_url":"path/to/exported_data.overall_export_metadata"}

The request is successful, but I cant seem to find any update to my local ndb database. I can add new and query for ndb elements in the admin interactive console on http://localhost:8000/console, but none of the imported elements can be queried for.

I run the app with:

dev_appserver.py . --storage_path .tmp/ --support_datastore_emulator true --enable_console --dev_appserver_log_level=debug

And the emulator with

datastore start --data-dir=.tmp/ --project=[MY-PROJECT-ID]

I get this warning when starting the emulator:

WARNING: Reusing existing data in [.tmp/].

Which is kinda what I want, no? Since I want my local datastore and the emulator data to both be accessible through ndb calls.

I have added the following environment variables:

DATASTORE_DATASET=[MY-PROJECT-ID]
DATASTORE_EMULATOR_HOST=localhost:8081
DATASTORE_EMULATOR_HOST_PATH=http://localhost:8081/datastore
DATASTORE_HOST=http://localhost:8081;DATASTORE_PROJECT_ID=[MY-PROJECT-ID]
GOOGLE_APPLICATION_CREDENTIALS=C:\path\to\service_account_key.json
paxel
  • 13
  • 4
  • As you mentioned `I unpacked them and stored them on my computer` . But as per this [Documentation](https://cloud.google.com/datastore/docs/tools/emulator-export-import#load_emulator_data_into_your_database) Before you can load entities exported from the emulator into your database, you must upload your entity export files to a Cloud Storage bucket. The managed import feature reads only from Cloud Storage buckets. – Sandeep Vokkareni Apr 21 '23 at 08:10
  • @SandeepVokkareni, that's strange, since I have been able to see that the elements are in my emulated datastore; this by importing them from my local destination. My issue is rather that I cannot seem to access the emulated values through ndb query-commands. I can query for the models that have been added to ndb by using .put() though.. – paxel Apr 24 '23 at 09:07
  • Can you check the output by running the command [`--dev_appserver_log_level=debug`](https://cloud.google.com/appengine/docs/legacy/standard/python/tools/migrate-cloud-datastore-emulator#getting_emulator_output) to see the output of Datastore Emulator. – Sandeep Vokkareni Apr 27 '23 at 07:34
  • When starting dev_appserver, I get the following output regarding my datastore: "WARNING: Detected environment variable DATASTORE_EMULATOR_HOST=localhost:8081, dev_appserver will speak to the Cloud Datastore emulator running on this address. The datastore_path .tmp/datastore.db will be neglected. If you want datastore to store on .tmp/datastore.db, remove DATASTORE_EMULATOR_HOST from environment variables and restart dev_appserver" – paxel Apr 28 '23 at 08:45
  • Prior to running dev_appserver, I start my datastore emulator host at 8081 pointing towards my local directory ".tmp". I detailed it a bit more in my comment answering NoCommandLine :) – paxel Apr 28 '23 at 08:50

1 Answers1

0

Not sure of the order in which you executed the steps but try this

  1. Start datastore emulator by running

    gcloud beta emulators datastore start --data-dir=[DATA_DIR]

  2. The above command should now give you a port number. You should note that the port number can change each time you start the emulator.

  3. Import the data you exported from the cloud using the port number you got when you started the emulator (bullet 1).

  4. Before you can run dev_appserver.py, you need to set the environment variables using the values from when you started the emulator (bullet 1). You can either set the values automatically or manually (see documentation)

  5. Now that the environment variables have been set, you can now run dev_appserver.py using

    dev_appserver.py --support_datastore_emulator=true app.yaml

NoCommandLine
  • 5,044
  • 2
  • 4
  • 15
  • Hi! I am pretty much already doing all of that. To start the emulator, I use "gcloud beta emulators datastore start --data-dir=.tmp/ --project=[MY-PROJECT-ID]". I set all environment variables prior to running by setting them in my IDE run config, and I use "DATASTORE_EMULATOR_HOST=localhost:8081" for the emulator, so it is always directed towards localhost:8081. – paxel Apr 20 '23 at 12:31
  • Okay. If something else occurs to me, I'll update. I achieve the same thing a different way (our App allows us to export the data from Prod as JSON and then import it into Dev) – NoCommandLine Apr 20 '23 at 15:17