1

I have a Firebase child with nearly 500k records. I want to duplicate this child in order to create a backup of the data (within the same Firebase). So, if my child is called lines, I want to duplicate it but with a new name such as lines_backup_02_02_2016. Once this is done I want the original lines left as is.

Is there some way to do this without grabbing a dataSnapshot via Javascript (downloading the data) and then using set() to create the copy? For example:

fb.child("lines").once('value', function(snapshot) {
    fb.child("lines_backup_02_02_2016").set(snapshot.val());
  });

With 500k records I'm finding this takes longer than I've been able to wait so far (for example, I've typed this question while still waiting for it to finish).

mix
  • 6,943
  • 15
  • 61
  • 90
  • How big is the data at that location? Can you export it from the Firebase dashboard? – Frank van Puffelen Feb 02 '16 at 23:34
  • Total data size is 223MB. The export option says it's too big to use that function. But it looks I can try the REST option they list there since it's under 256MB. Will try (though not sure how to upload it back then). – mix Feb 02 '16 at 23:39
  • So I was able to grab the data via REST. Interestingly, it comes through as only 128MB of json, though the number of records seems correct. – mix Feb 02 '16 at 23:45
  • Yup. A REST call is indeed step one. If you've already downloaded the JSON, you can also add it with https://github.com/firebase/firebase-streaming-import. – Frank van Puffelen Feb 02 '16 at 23:45

1 Answers1

3

Trying to access/query a node with 500k child records is a bad idea. It's also not very likely that you need to synchronize those 500k lines to users. So please work hard on separating the active from the historical data. You current approach of creating a backup copy is a great start for that. Just also delete nodes while you're writing them to their new location.

To get this data is going to be "a bit involved". It indeed involves first getting a list of the child keys with a shallow REST call:

https://yours.firebaseio.com/path-to-list.json?shallow=true

Then you loop through the keys and access each child in turn, with a regular JavaScript client:

ref.child('path-to-list').child(key).once('value', function(snapshot) {
  ref.child('path-to-backup').child(key).set(snapshot.val(), function(error) {
    if (!error) {
      // Delete the original item, so that the list stays small
      snapshot.ref().remove();
    }
  });

});

This loop doesn't create problems, since you're not querying the entire list. Instead you are accessing specific child nodes directly, which is fine.

Be sure to throttle the loop, because you might otherwise swamp you database with small writes.

Frank van Puffelen
  • 565,676
  • 79
  • 828
  • 807