How to import data from cloud firestore to the local emulator?

Question

I want to be able to run cloud functions locally and debug against a copy from the production data. Is there a way to copy the data that is online to the local firestore emulator?

Answer 1

This can be accomplished through a set of commands in terminal on the existing project:

1. Login to firebase and Gcloud:

firebase login
gcloud auth login

2. See a list of your projects and connect to one:

firebase projects:list
firebase use your-project-name

gcloud projects list
gcloud config set project your-project-name

3. Export your production data to gcloud bucket with chosen name:

gcloud firestore export gs://your-project-name.appspot.com/your-choosen-folder-name

4. Now copy this folder to your local machine, I do that in functions folder directly:

Note : Don't miss the dot ( . ) at the end of below command

cd functions
gsutil -m cp -r gs://your-project-name.appspot.com/your-choosen-folder-name .

5. Now we just want to import this folder. This should work with the basic command, thanks to latest update from Firebase team https://github.com/firebase/firebase-tools/pull/2519.

firebase emulators:start --import ./your-choosen-folder-name

Check out my article on Medium about it and a shorthanded script to do the job for you https://medium.com/firebase-developers/how-to-import-production-data-from-cloud-firestore-to-the-local-emulator-e82ae1c6ed8

Note: Its better to use a different bucket for it, as copying into your project bucket will result in the folder created in your firebase storage.
If you are interested in gsutil arguments like -m, you can see them described by executing gsutil --help.

Answer 2

My method is somewhat manual but it does the trick. I've shared it in this useful Github thread but I'll list the steps I did here if you find them useful:

1. Go to my local Firebase project path.
2. Start the emulators using: firebase emulators:start
3. Create manually some mockup data using the GUI at http://localhost:4000/firestore using the buttons provided: + Start Collection and + Add Document.
4. Export this data locally using: emulators:export ./mydirectory
5. About the project data located at Firebase Database / Cloud Firestore, I exported a single collection like this: gcloud firestore export gs://my-project-bucket-id.appspot.com --collection-ids=myCollection The export is now located under Firebase Storage in a folder with a timestamp as name (I didn't use a prefix for my test)
6. Download this folder to local drive with: gsutil cp -r gs://my-project-bucket-id.appspot.com/myCollection ./production_data_export NOTE: I did this in a Windows environment... gsutil will throw this error: "OSError: The filename, directory name, or volume label syntax is incorrect" if the folder has invalid characters for a folder name in Windows (i.e. colons) or this error: "OSError: Invalid argument.9.0 B]" if an inner file in the folder has invalid characters too. To be able to download the export locally, rename these with a valid Windows name (i.e. removing the colons) like this: gsutil mv gs://my-project-bucket-id.appspot.com/2020-05-22T02:01:06_86152 gs://my-project-bucket-id.appspot.com/myCollection
7. Once downloaded, imitate the local export structure renaming the folder to firestore_export and copying the firebase-export-metadata.json file from the local export folder. Just to be visual, here's the structure I got:

$ tree .
.
├── local_data_export
│   ├── firebase-export-metadata.json
│   └── firestore_export
│       ├── all_namespaces
│       │   └── all_kinds
│       │       ├── all_namespaces_all_kinds.export_metadata
│       │       └── output-0
│       └── firestore_export.overall_export_metadata
└── production_data_export
    ├── firebase-export-metadata.json
    └── firestore_export
        ├── all_namespaces
        │   └── kind_myCollection
        │       ├── all_namespaces_kind_myCollection.export_metadata
        │       ├── output-0
        │       └── output-1
        └── firestore_export.overall_export_metadata

8 directories, 9 files

8. Finally, start the local emulator pointing to this production data to be imported: firebase emulators:start --import=./mock_up_data/production_data_export/
9. You should see the imported data at: http://localhost:4000/firestore/

This should assist readers for now, while we await a more robust solution from the Firebase folks.

Answer 3

You can use the firestore-backup-restore to export and import your production data as JSON files.

I wrote a quick hack to allow for importing these JSON in the Firebase Simulator Firestore instance.

I proposed a pull request and made this npm module in the meantime.

You can use it this way:

const firestoreService = require('@crapougnax/firestore-export-import')
const path = require('path')

// list of JSON files generated with the export service
// Must be in the same folder as this script
const collections = ['languages', 'roles']

// Start your firestore emulator for (at least) firestore
// firebase emulators:start --only firestore

// Initiate Firebase Test App
const db = firestoreService.initializeTestApp('test', {
   uid: 'john',
   email: 'john@doe.com',
})

// Start importing your data
let promises = []
try {
   collections.map(collection =>
      promises.push(
         firestoreService.fixtures(
            path.resolve(__dirname, `./${collection}.json`),
            [],
            [],
            db,
         ),
      ),
   )
   Promise.all(promises).then(process.exit)
} catch (err) {
   console.error(err)
}

Obviously, since this data won't persist in the emulator, you'll typically inject them in the before() function of your test suite or even before every test.

Answer 4

I was able to make some npm scripts to import from remote to local emulator and vice-versa.

"serve": "yarn build && firebase emulators:start --only functions,firestore --import=./firestore_export",
"db:update-local-from-remote": "yarn db:backup-remote && gsutil -m cp -r gs://my-firebase-bucket.appspot.com/firestore_export .",
"db:update-remote-from-local": "yarn db:backup-local && yarn db:backup-remote && gsutil -m cp -r ./firestore_export gs://my-firebase-bucket.appspot.com && yarn run db:import-remote",
"db:import-remote": "gcloud firestore import gs://my-firebase-bucket.appspot.com/firestore_export",
"db:backup-local": "firebase emulators:export --force .",
"db:rename-remote-backup-folder": "gsutil mv gs://my-firebase-bucket.appspot.com/firestore_export gs://my-firebase-bucket.appspot.com/firestore_export_$(date +%d-%m-%Y-%H-%M)",
"db:backup-remote": "yarn db:rename-remote-backup-folder && gcloud firestore export gs://my-firebase-bucket.appspot.com/firestore_export"

So you can export the local Firestore data to remote with:

npm db:update-remote-from-local

Or to update your local Firestore data with remote one, do:

npm db:update-local-from-remote

These operations will backup the remote Firestore data, making a copy of it and storing it on Firebase Storage.

Answer 5

you can use fire-import npm package. for importing both firestore and firebase storage

Answer 6

There is no built-in way to copy data from a cloud project to the local emulator. Since the emulator doesn't persist any data, you will have to re-generate the initial data set on every run.

Answer 7

I was about to go add a cli option to firebase-tools but pretty happy with the node-firestore-import-export package.

yarn add -D node-firestore-import-export

  "scripts": {
    "db:export": "firestore-export -a ./serviceAccountKey.json -b ./data/firestore.json",
    "db:import": "firestore-import -a ./serviceAccountKey.json -b ./data/firestore.json",
    "db:emulator:export": "export FIRESTORE_EMULATOR_HOST=localhost:8080 && yarn db:export",
    "db:emulator:import": "export FIRESTORE_EMULATOR_HOST=localhost:8080 && yarn db:import",
    "db:backup": "cp ./data/firestore.json ./data/firestore-$(date +%d-%m-%Y-%H-%M).json",
    "dev": "firebase emulators:start --import=./data --export-on-exit=./data",
  },

You will need to create a service account in the firebase console.

You can replace the GCLOUD_PROJECT environment variable with hard coded values.

open https://console.firebase.google.com/project/$GCLOUD_PROJECT/settings/serviceaccounts/adminsdk
mv ~/Downloads/myProjectHecticKeyName.json ./serviceAccountKey.json

That being said the gcloud tools are definitely the way to go in production, as you will need s3 backups anyway.

Answer 8

There is also a way to import data to local storage from Google Cloud Storage without any commands:

• export Firestore to Google cloud storage bucket by clicking More in google cloud
• choose your desired file in google cloud storage bucket
• open terminal (Google terminal shell near the search bar) in terminal click Open editor right click on desired file in online VSCode and click download.

You shoud start downloading of .tar file which is in fact your exported data from firestore.

• Create a folder in your root (as example you may call it 'firestore-local-data')
• Copy paste (or unarchive data) to this folder from archive file .tar
• run firebase emulators:start --import ./firestore-local-data This should do the trick

Answer 9

I wrote a little script to able to do that:

const db = admin.firestore();
const collections = ['albums', 'artists'];
let rawData: any;
    
for (const i in collections) {
    rawData = fs.readFileSync(`./${collections[i]}.json`);
    const arr = JSON.parse(rawData);

    for (const j in arr) {
        db.collection(collections[i]).add(arr[j])
        .then(val => console.log(val))
        .catch(err => console.log('ERRO: ', err))
    }
        
}