15

Are there any tools to aid in data migration from dev to staging to prod? If not, are there plans to build them?

I know you can Export JSON and Import JSON from Forge, but that doesn't include authorization and security settings.

sprugman
  • 19,351
  • 35
  • 110
  • 163

5 Answers5

16

All of our data is available through a REST API, so you could easily write a script to do this yourself. You can export the data by setting format=export (this includes all of the priority data in the response):

curl https://myapp.firebaseIO.com/.json?format=export&auth=YOUR_FIREBASE_SECRET

As for exporting the security rules, you can access them here:

curl https://myapp.firebaseIO.com/.settings/rules/.json?auth=YOUR_FIREBASE_SECRET

You can then write them back to the new Firebase using PUT.

The various Auth settings can't easily be automatically transferred (such as the Authorized Origins), but they probably shouldn't be as they'll differ between staging and production.

Andrew Lee
  • 10,127
  • 3
  • 46
  • 40
  • 1
    how do you do the curl -X put filename filedestination? what would the syntax be? – ingrid Jun 10 '16 at 16:09
  • here is the documentation about the @Andrew's answer https://firebase.google.com/docs/database/rest/app-management – epool Sep 28 '16 at 15:14
5

What Andrew said above is mostly correct, however this can be a pain with large firebases. There is an import project at https://github.com/firebase/firebase-import that will help import large firebases by breaking up the put requests. Also something to note, you will need to use quotes around the curl url, otherwise the & will background the process. So what Andrew gave above will work instead as

curl -o outputfile.json "https://myapp.firebaseIO.com/.json?format=export&auth=YOUR_FIREBASE_SECRET"

Then you can use the import module I linked with that json file.

Good Luck!

Seth Caldwell
  • 179
  • 2
  • 5
2

If you want an option that doesn't require cURL, and you have the firebase-tools project installed, you can run this:

firebase database:get --export -o backup.json /

Note that this should be run from a working directory configured as a Firebase project. The advantage of this option is it will use the Auth you've set up for that project, so you don't need to hard-code auth keys into command lines (for the security-conscious) and it doesn't rely on the deprecated auth-key pattern.

Command-line Fu: Another cool technique if you want separate files for each top-level key is calling:

for i in `firebase database:get --shallow / | jq -r 'keys[]'`; do
    echo "Downloading $i..."
    firebase database:get --export -o $i.json /$i
done

You will need the "jq" tool installed for this to work. Exporting each collection separately can be really useful if you later want to restore or work with just a portion of your data.

Chad Robinson
  • 4,575
  • 22
  • 27
0

Firebase is working on a new service "S3 Customer Backups" that will copy a .gz compressed backup of your entire firebase nightly into an s3 bucket you give them. I'm evaluating the beta of this service right now, but if it is something you need, I recommend asking support about it.

Our firebase got too large for the curl operation to complete, and this new solution will enable us to manage our dev environments. So if you have a large firebase, setup the S3 Customer Backups then use firebase-import to shove the data into your dev/staging firebases. Victory!

Seth Caldwell
  • 179
  • 2
  • 5
0

I just created this ruby gem for cloning a firebase remote config data from an existing project o a new one project.

epool
  • 6,710
  • 7
  • 38
  • 43