How can one load an AppEngine cloud storage backup to a local development server?

Question

I'm experimenting with the Google cloud storage backup feature for an application.

After downloading the backup files using gsutil, how can they be loaded into a local development server?

Is there a parser available for these formats (eg, protocol buffers)?

Are you using python or java? – proppy Sep 10 '12 at 00:41 — proppy, Sep 10 '12 at 00:41

proppy · Answer 1 · 2012-09-18T15:50:28.997

7

Backups are stored in leveldb record format, you should be able to read then using:

edited Sep 18 '12 at 15:50

answered Sep 18 '12 at 15:21

proppy

10,495
5
37
66

Thanks for the links. I was looking for exactly this. – Bryce Cutt Sep 18 '12 at 21:13
For a complete set of instructions for this approach, see here: https://github.com/Venryx/firestore-leveldb-tools – Venryx Feb 22 '20 at 10:01

score 7 · Accepted Answer · edited Nov 27 '15 at 12:01

7

Greg Bayer wrote some Python code showing how to do this in a blog post:

# Make sure App Engine SDK is available
import sys
sys.path.append('/usr/local/google_appengine')
from google.appengine.api.files import records
from google.appengine.datastore import entity_pb
from google.appengine.api import datastore

raw = open('path_to_datastore_export_file', 'r')
reader = records.RecordsReader(raw)
for record in reader:
  entity_proto = entity_pb.EntityProto(contents=record)
  entity = datastore.Entity.FromPb(entity_proto)
  #Entity is available as a dictionary!

edited Nov 27 '15 at 12:01

matt burns

24,742
13
105
107

answered Nov 12 '12 at 18:48

John Lowry

249
3
4

*RecordsReader instead of RecordReader – orcaman Sep 19 '14 at 15:58

score 0 · Answer 3 · answered Jun 12 '16 at 23:42

0

For those using windows change the open line to: raw = open('path_to_datastore_export_file', 'rb')

The file must be opened in binary mode!

answered Jun 12 '16 at 23:42

Dabble

39
2

How can one load an AppEngine cloud storage backup to a local development server?

3 Answers3

Linked