Export json from Firestore

Question

As we can download json file at Firebase RTDB console, are there any way to export json file of Firestore collection/document data?

One of my main objectives is to compare data before/after updating document.

Answer 1

Google made it harder than it needed to be, so the community found a workaround. If you have npm installed, you can do this:

Export

npx -p node-firestore-import-export firestore-export -a credentials.json -b backup.json

Import

npx -p node-firestore-import-export firestore-import -a credentials.json -b backup.json

Source

Answer 2

I just wrote a backup and restore for Firestore. You can have a try on my GitHub.

https://github.com/dalenguyen/firestore-backup-restore

Thanks,

Answer 3

There is not, you'd need to come up with your own process such as querying a collection and looping over everything.

Update

As of August 7th, 2018, we do have a managed export system that allows you to dump your data into a GCS bucket. While this isn't JSON, it is a format that is the same as Cloud Datastore uses, so BigQuery understands it. This means you can then import it into BigQuery.

Answer 4

I've written a tool that traverses the collections/documents of the database and exports everything into a single json file. Plus, it will import the same structure as well (helpful for cloning/moving Firestore databases). Since I've had a few colleagues use the code, I figured I would publish it as an NPM package. Feel free to try it and give some feedback.

https://www.npmjs.com/package/node-firestore-import-export

Answer 5

If someone wants a solution using Python 2 or 3.

Edit: note that this does not backup the rules

Fork it on https://github.com/RobinManoli/python-firebase-admin-firestore-backup

First install and setup Firebase Admin Python SDK: https://firebase.google.com/docs/admin/setup

Then install it in your python environment:

pip install firebase-admin

Install the Firestore module:

pip install google-cloud-core
pip install google-cloud-firestore

(from ImportError: Failed to import the Cloud Firestore library for Python)

Python Code

# -*- coding: UTF-8 -*-

import firebase_admin
from firebase_admin import credentials, firestore
import json

cred = credentials.Certificate('xxxxx-adminsdk-xxxxx-xxxxxxx.json') # from firebase project settings
default_app = firebase_admin.initialize_app(cred, {
    'databaseURL' : 'https://xxxxx.firebaseio.com'
})

db = firebase_admin.firestore.client()

# add your collections manually
collection_names = ['myFirstCollection', 'mySecondCollection']
collections = dict()
dict4json = dict()
n_documents = 0

for collection in collection_names:
    collections[collection] = db.collection(collection).get()
    dict4json[collection] = {}
    for document in collections[collection]:
        docdict = document.to_dict()
        dict4json[collection][document.id] = docdict
        n_documents += 1

jsonfromdict = json.dumps(dict4json)

path_filename = "/mypath/databases/firestore.json"
print "Downloaded %d collections, %d documents and now writing %d json characters to %s" % ( len(collection_names), n_documents, len(jsonfromdict), path_filename )
with open(path_filename, 'w') as the_file:
    the_file.write(jsonfromdict)

Answer 6

There is an npm for firestore export / import

Project to export Goto -> project settings -> Service account -> Generate new private key -> save it as exportedDB.json

Project to import Goto -> project settings -> Service account -> Generate new private key -> save it as importedDB.json

run these 2 commands from the folder where u saved the files

Export: npx -p node-firestore-import-export firestore-export -a exportedDB.json -b backup.json

Import: npx -p node-firestore-import-export firestore-import -a importedDB.json -b backup.json

Answer 7

Firestore is still early in its development so please check the docs on backups for any information pertaining to Firestore.

I found this npm package, node-firestore-backup, to be easy and useful.

Note that the --accountCredentials path/to/credentials/file.json is referring to a service account key json file that you can get by following instructions from https://developers.google.com/identity/protocols/application-default-credentials.

1. Go to the API Console Credentials page.
2. From the project drop-down, select your project.
3. On the Credentials page, select the Create credentials drop-down, then select Service account key.
4. From the Service account drop-down, select an existing service account or create a new one.
5. For Key type, select the JSON key option, then select Create. The file automatically downloads to your computer.
6. Put the *.json file you just downloaded in a directory of your choosing. This directory must be private (you can't let anyone get access to this), but accessible to your web server code.

Answer 8

It works for me.

I used Cloud Functions to export all data in Firestore to JSON format. The function that I was used:

exports.exportFirestore2Json = functions.https.onRequest((request, response) => {
    db.collection("data").get().then(function(querySnapshot) {
        const orders = [];
        var order = null

         querySnapshot.forEach(doc => {
             order = doc.data();
             orders.push(order);
         });

         response.send(JSON.stringify(orders))

         return true
    })
    .catch(function(error) {
        console.error("Error adding document: ", error);
        return false
    });
})

Then, go to https://your-project-id.cloudfunctions.net/exportFirestore2Json you will see something like this

Answer 9

Yes you can, you did not need to start billing in your firebase console. There is a great npm package https://www.npmjs.com/package/firestore-export-import with this you can export and import firestore collection and documents easily. Just follow some steps:

-Get your service account key Open Firebase console > Project settings > Service accounts > generate new private key

rename the downloaded file with serviceAccountKey.json

-Now create a new folder and index.js file.

-Paste you servicekey.json in this folder

-Now install this package

npm install firestore-export-import

OR

yarn add firestore-export-import

Exporting data from firebase

const { initializeApp} =  require('firestore-export-import')

const  serviceAccount  =  require('./serviceAccountKey.json')

const  appName  =  '[DEFAULT]'

initializeApp(serviceAccount, appName)

const  fs  =  require('fs');

const { backup } =  require('firestore-export-import')
//backup('collection name')

backup('users').then((data) =>
{
    const  json  =  JSON.stringify(data);

    //where collection.json is your output file name.
    fs.writeFile('collection.json', json, 'utf8',()=>{

    console.log('done');

})

});

Execute node index.js and you should see a new collection.json file with your collection and documents in it. If it looks a little messy pretty format it online with https://codebeautify.org/jsonviewer

This index.js was just a very basic configuration which exports the whole collection with everything in it, read their documentation you could do queries and much more!

Importing data to firebase

const { initializeApp,restore } =  require('firestore-export-import')

const  serviceAccount  =  require('./serviceAccountKey.json')
const  appName  =  '[DEFAULT]'

initializeApp(serviceAccount, appName)
restore('collection.json', {
//where refs is an array of key items
    refs: ['users'],
    //autoParseDates to parse dates if documents have timestamps
    autoParseDates: true,

    },()=>{

console.log('done');
})

After execution you should see your firestore populated with collection users!

Answer 10

for dumping json from your local to firestoreDB:

npx -p node-firestore-import-export firestore-import -a credentials.json -b backup.json

for downloading data from firestoreDB to your local:

npx -p node-firestore-import-export firestore-export -a credentials.json -b backup.json

to generate credentials.json, go to project settings -> service accounts -> generate a private key.

Answer 11

1. Create a blank folder (call it firebaseImportExport ) and run npm init
2. Go to the source Firebase project -> Settings -> Service Accounts
3. Click on the Generate new private key button and rename the file as source.json and put it in the firebaseImportExport folder
4. Do the same (step 2 & 3) for the destination project and rename the file as destination.json
5. Install the npm i firebase-admin npm package.
6. Write the following code in the index.js

const firebase = require('firebase-admin');

var serviceAccountSource = require("./source.json");
var serviceAccountDestination = require("./destination.json");

const sourceAdmin = firebase.initializeApp({
    credential: firebase.credential.cert(serviceAccountSource),
    databaseURL: "https://**********.firebaseio.com" // replace with source
});

const destinationAdmin = firebase.initializeApp({
    credential: firebase.credential.cert(serviceAccountDestination),
    databaseURL: "https://$$$$$.firebaseio.com"
  }, "destination");

const collections = [ "books", "authors",   ...]; // replace with your collections

    var source = sourceAdmin.firestore();
    var destination = destinationAdmin.firestore();
   collections.forEach(colName => {
    source.collection(colName).get().then(function(querySnapshot) {
        querySnapshot.forEach(function(doc) {
            destination.collection(colName).doc(doc.id).set({...doc.data()});
        });
    });
   });

Answer 12

Documents can also be downloaded as JSON via the REST API.

This is an example using curl in conjunction with the Cloud SDK to obtain an access token:

curl -H "Authorization: Bearer "$(gcloud auth print-access-token) \
  "https://firestore.googleapis.com/v1/projects/$PROJECT/databases/(default)/documents/$COLLECTION/$DOCUMENT"

Answer 13

Open any of your clientside firebase apps (React, Angular, etc.). Use this code anywhere to log console and copy

const products = await db
  .collection("collectionName")
  .where("time", ">", new Date("2020-09-01"))
  .get()


const json = JSON.stringify(products.docs.map((doc) => ({ ...doc.data() })))
console.log(json)

Answer 14

I found an easier solution. There is a tool called Firefoo. It lists all the collection documents along with created users with multiple providers(email & password, phone number, google, facebook etc). You can export data in JSON & CSV along with that you can view the data in simplified format like Table, Tree, JSON.

Note:- You don't have to go through all the process for importing or exporting data from your firebase console.

Answer 15

Create a blank folder (Ex: firebaseImportExport ) and run npm init

Go to the source Firebase project -> Settings -> Service Accounts

Click on the Generate new private key button and rename the file as appConfig.json and put it in the firebaseImportExport folder

add packages

npm install firestore-export-import

create an index.js file.

paste the below code

const { initializeFirebaseApp } = require('firestore-export-import')

const  serviceAccount  =  require('./appConfig.json')

const  appName  =  '[DEFAULT]'

const firestore = initializeFirebaseApp(serviceAccount, appName)

const  fs  =  require('fs');

const { backup } = require('firestore-export-import')
//backup('collection name')

backup(firestore, 'collection-name').then((data) =>
{
    const  json  =  JSON.stringify(data);

    //where collection.json is your output file name.
    fs.writeFile('collection.json', json, 'utf8',()=>{

    console.log('done');

})

});