AppEngine datastore - backup programmatically

Question

I would like to backup my app's datastore programmatically, on a regular basis.

It seems possible to create a cron that backs up the datastore, according to https://developers.google.com/appengine/articles/scheduled_backups

However, I require a more fine-grained solution: Create different backup files for dynamically changing namespaces.

Is it possible to simply call the /_ah/datastore_admin/backup.create url with GET/POST?

score 3 · Answer 1 · answered Apr 17 '13 at 19:03

3

Yes; I'm doing exactly that in order to implement some logic that couldn't be done with cron.

Use the taskqueue API to add the URL request, like this:

from google.appengine.api import taskqueue
taskqueue.add(url='/_ah/datastore_admin/backup.create',
              method='GET',
              target='ah-builtin-python-bundle',
              params={'kind': ('MyKind1', 'MyKind2')})

If you want to use more parameters that would otherwise go into the cron url, like 'filesystem', put those in the params dict alongside 'kind'.

answered Apr 17 '13 at 19:03

Jamie Niemasik

755
6
16

Trying to do so, getting 404, any ideas?? – Gilad Apr 21 '13 at 13:17
Have you enabled datastore_admin for your app? See https://developers.google.com/appengine/docs/adminconsole/datastoreadmin#Enable_datastore_admin – Jamie Niemasik Apr 23 '13 at 00:41
@JamieNiemasik how do you specify the namespace in this case? – husayt Sep 13 '13 at 04:28
I didn't have to do that, but it seems like it should work if you specify that in the `params` dict. – Jamie Niemasik Sep 13 '13 at 19:20
Hi Jamie, are you sure you're not missing a step here? The API says you can't call url='/_ah/datastore_admin/backup.create' programmaticaly, only from cron. Like user Gilad, I'm getting 404 when I try to do this programmatically..do we need to prefix with our application hostname? – doright Jun 27 '14 at 12:42
Interesting; I see that comment, yet this worked for me. Are you using taskqueue rather than urlfetch? Have you enabled datastore admin? – Jamie Niemasik Jun 28 '14 at 14:33

score 2 · Answer 2 · answered Jun 09 '15 at 14:35

Programmatically backup datastore based on environment

This comes in addition to Jamie's answer. I needed to backup the datastore to Cloud Storage, based on the environment (staging/production). Unfortunately, this can no longer be achieved via a cronjob so I needed to do it programmatically and create a cron to my script. I can confirm that what's below is working, as I saw there were some people complaining that they get a 404. However, it's only working on a live environment, not on the local development server.

from datetime import datetime

from flask.views import MethodView

from google.appengine.api import taskqueue
from google.appengine.api.app_identity import app_identity


class BackupDatastoreView(MethodView):

    BUCKETS = {
        'app-id-staging': 'datastore-backup-staging',
        'app-id-production': 'datastore-backup-production'
    }

    def get(self):

        environment = app_identity.get_application_id()

        task = taskqueue.add(
            url='/_ah/datastore_admin/backup.create',
            method='GET',
            target='ah-builtin-python-bundle',
            queue_name='backup',
            params={
                'filesystem': 'gs',
                'gs_bucket_name': self.get_bucket_name(environment),
                'kind': (
                    'Kind1',
                    'Kind2',
                    'Kind3'
                )
            }
        )

        if task:
            return 'Started backing up %s' % environment

    def get_bucket_name(self, environment):

        return "{bucket}/{date}".format(
            bucket=self.BUCKETS.get(environment, 'datastore-backup'),
            date=datetime.now().strftime("%d-%m-%Y %H:%M")
        )

I'm assuming you could poll the taskqueue once every X minutes to see if the task is still processing — kioleanu, Aug 12 '16 at 06:44

score 0 · Answer 3 · answered Mar 02 '18 at 00:51

0

You can now use the managed export and import feature, which can be accessed through gcloud or the Datastore Admin API:

Exporting and Importing Entities

Scheduling an Export

answered Mar 02 '18 at 00:51

Juan Lara

6,454
1
22
31

AppEngine datastore - backup programmatically

3 Answers3

Linked