Add new field or change the structure on all Firestore documents

Question

Consider a collection of users. Each document in the collection has name and email as fields.

{
  "users": {
    "uid1": {
      "name": "Alex Saveau",
      "email": "saveau.alexandre@gmail.com"
    },
    "uid2": { ... },
    "uid3": { ... }
  }
}

Consider now that with this working Cloud Firestore database structure I launch my first version of a mobile application. Then, at some point I realize I want to include another field such as last_login.

In the code, reading all the users documents from the Firestore DB using Java would be done as

FirebaseFirestore.getInstance().collection("users").get()
        .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
            @Override
            public void onComplete(@NonNull Task<QuerySnapshot> task) {
                if (task.isSuccessful()) {
                    for (DocumentSnapshot document : task.getResult()) {
                        mUsers.add(document.toObject(User.class));
                    }
                }
            }
        });

where the class User contains now name, email and last_login.

Since the new User field (last_login) is not included in the old users stored in the DB, the application is crashing because the new User class is expecting a last_login field which is returned as null by the get() method.

What would be the best practice to include last_login in all the existing User documents of the DB without losing their data on a new version of the app? Should I run an snippet just once to do this task or are there any better approaches to the problem?

The Firebase clients typically don't crash on properties missing from the JSON. Can you share the *minimal* `User` class that is needed to reproduce the crash, as well as the full stack trace of the crash? — Frank van Puffelen, Mar 12 '18 at 14:16
The crash is not actually caused by the Firebase method, it is caused because I get a `null` value. So far I have created a function to rename the key (field) name of all the documents in a collection. I will post it later. — b-fg, Mar 12 '18 at 22:16
I have added an answer. It comprises the functions I wrote to solve this issue. It's not elegant, but it works. — b-fg, Apr 19 '20 at 09:00
I have the same problem but i expect that the solution should be dealt from the server side not the client side what i think would be the best approach is to execute a script in the server side before publishing the next version and if you want to prevent exceeding the quotas i guess it's better to apply the script using paging for (n) days and when it'll be done publish the next update and so on — Farido mastr, Feb 13 '21 at 12:54

score 10 · Answer 1 · edited Sep 11 '21 at 16:49

10

You fell into a gap of NOSQL databases: Document oriented databases do not guarantee structural integrity of the data (as RDBMS do)

The deal is:

in an RDBMS all stored data have the same structure at any given time (within the same instance or cluster). When changing the structure (ER-diagram) you have to migrate the data for all existing records which costs time and effort.

As a result, your application can be optimized for the current version of the data structure.
in a Document oriented database each record is an independent "Page" with its own independent structure. If you change the structure it only applies to new documents. So you don't need to migrate the existing data.

As a result, your application must be able to deal with all versions of the data structure you've ever used in your current database.

I don't know about firebase in detail but in general you never update a document in a NOSQL database. You only create a new version of the document. So even if you update all documents your application must be prepared to deal with the "old" data structure...

edited Sep 11 '21 at 16:49

bmurf17

83
7

answered Mar 12 '18 at 13:33

Timothy Truckle

15,071
2
27
51

1

Thanks for the information, this is useful. I have coded a small function which updates the field name of all documents in a collection. This is done in a hard-way (delete old field, write a new one) and I was expecting something more integrated with Firestore. I have not been able to find it so far. – b-fg Mar 12 '18 at 22:20
see this https://stackoverflow.com/questions/52109026/firestore-schema-versioning-and-backward-compatibility-with-android-app-to-preve but trying to see if we can do something like what protobufs have. Can we send a version info from client to firestore or something and that way Firestore can do something intelligent in terms of deciding which version to use? – Sudhakar R Sep 01 '18 at 02:27
1

@Timothy Truckle: "your application must be prepaed to deal with the "old" data structure" - do you have any suggestions/references for how you handle this for NOSQL databases? From this it sounds like each document should come with a version number that the application checks, but that seems excessive... Thanks – smörkex Oct 22 '18 at 06:21
@smörkex Yes, a *structure version number* maintained by your application is the only way I see to come around this efficiently. Another approach would be to "try" structure versions until you find a matching one. This does not sound batter anyhow... – Timothy Truckle Oct 22 '18 at 10:47
@b-fg please how did you go about this?? Codes?? or can it be done directly in firebase console.? or can you give code you used in adding a field to each documents in a collection. Tnks – Ibramazin Apr 15 '20 at 16:49
@Ibramazin I have added an answer. It comprises the functions I wrote to solve this issue. It's not elegant, but it works. – b-fg Apr 19 '20 at 09:00

b-fg · Answer 2 · 2020-05-23T08:55:40.817

I wrote some routines to help automate this process back when I posted the question. I did not post them since these are a bit rudimentary and I was hoping for an elegant Firestore-based solution. Because such solution is not still available, here are the functions I wrote.

In short, we have functions for renaming a field, adding a field, or deleting a field. To rename a field, different functions are used depending on the data type. Maybe someone could generalise this better? The functions below are:

add_field: Adds a field in all documents of a collection.
delete_field: Deletes a field in all documents of a collection.
rename_*_field: Renames a field containing a certain data type (*) in all documents of a collection. Here I include examples for String, Integer, and Date.

Add field:

public void add_field (final String key, final Object value, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {
                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            Map<String, Object> new_map = new HashMap<>();
                            new_map.put(key, value);
                            batch.update(docRef, new_map);
                        }
                        batch.commit();
                    } else {
                        // ... "Error adding field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting documents -> " + e
                }
            });
}

Delete field:

public void delete_field (final String key, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {

                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            Map<String, Object> delete_field = new HashMap<>();
                            delete_field.put(key, FieldValue.delete());
                            batch.update(docRef, delete_field);
                        }
                        // Commit the batch
                        batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
                            @Override
                            public void onComplete(@NonNull Task<Void> task) {
                                // ...
                            }
                        });

                    } else {
                        // ... "Error updating field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting notices -> " + e
                }
            });
}

Rename field:

public void rename_string_field (final String old_key, final String new_key, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {

                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            String old_value = document.getString(old_key);

                            if (old_value != null) {
                                Map<String, Object> new_map = new HashMap<>();
                                new_map.put(new_key, old_value);

                                Map<String, Object> delete_old = new HashMap<>();
                                delete_old.put(old_key, FieldValue.delete());

                                batch.update(docRef, new_map);
                                batch.update(docRef, delete_old);
                            }
                        }
                        // Commit the batch
                        batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
                            @Override
                            public void onComplete(@NonNull Task<Void> task) {
                                // ...
                            }
                        });

                    } else {
                        // ... "Error updating field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting notices ->" + e
                }
            });
}

public void rename_integer_field (final String old_key, final String new_key, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {

                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            int old_value = document.getDouble(old_key).intValue();
                            Integer ov = old_value;
                            if (ov != null) {
                                Map<String, Object> new_map = new HashMap<>();
                                new_map.put(new_key, old_value);

                                Map<String, Object> delete_old = new HashMap<>();
                                delete_old.put(old_key, FieldValue.delete());

                                batch.update(docRef, new_map);
                                batch.update(docRef, delete_old);
                            }
                        }
                        // Commit the batch
                        batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
                            @Override
                            public void onComplete(@NonNull Task<Void> task) {
                                // ...
                            }
                        });

                    } else {
                        // ... "Error updating field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting notices -> " + e
                }
            });
}

public void rename_date_field (final String old_key, final String new_key, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {

                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            Date old_value = document.getDate(old_key);
                            if (old_value != null) {
                                Map<String, Object> new_map = new HashMap<>();
                                new_map.put(new_key, old_value);

                                Map<String, Object> delete_old = new HashMap<>();
                                delete_old.put(old_key, FieldValue.delete());

                                batch.update(docRef, new_map);
                                batch.update(docRef, delete_old);
                            }
                        }
                        // Commit the batch
                        batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
                            @Override
                            public void onComplete(@NonNull Task<Void> task) {
                                // ...
                            }
                        });

                    } else {
                        // ... "Error updating field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting notices -> " + e
                }
            });
}

score 7 · Answer 3 · answered Sep 29 '20 at 04:57

7

Just wanted to share because I read that you were hoping for a Firestore based solution.

This worked for me. forEach will query each document in the collection and you can manipulate as you like.

    db.collection("collectionName").get().then(function(querySnapshot) {
        querySnapshot.forEach(async function(doc) {
            await db.collection("collectionName").doc(doc.id).set({newField: value}, {merge: true});
            // doc.data() is never undefined for query doc snapshots
            console.log(doc.id, " => ", doc.data());
        });
    });

answered Sep 29 '20 at 04:57

Gabby Villoria

71
1
1

1

Gold `.doc(doc.id).set({newField: value}, {merge: true});` – Petro Mar 02 '22 at 15:09
This answer helped me so much. Thank you. – SatelBill Mar 14 '22 at 17:54

score 4 · Answer 4 · edited May 08 '19 at 09:18

4

To solve this, you need to update each user to have the new property and for that I recommend you to use a Map. If you are using a model class when you are creating the users as explained in my answer from this post, to update all users, just iterate over the users collection amd use the following code:

Map<String, Object> map = new HashMap<>();
map.put("timestamp", FieldValue.serverTimestamp());
userDocumentReference.set(map, SetOptions.merge());

edited May 08 '19 at 09:18

Ryuga Law

160
7

answered Mar 12 '18 at 13:42

Alex Mamo

130,605
17
163
193

@KumarSantanu You're very welcome, Kumar. – Alex Mamo Jul 16 '21 at 08:44

score 1 · Answer 5 · answered Feb 15 '19 at 12:00

I am guessing that last_login is a primitive data type, maybe a long to hold a timestamp. An auto-generated setter would look like this:

private long last_login;

public void setLast_login(long last_login) {
    this.last_login = last_login;
}

This leads to a crash when old documents that lack the field are fetched due to null assignment to a variable of a primitive data type. One way around it is to modify your setter to pass in a variable of the equivalent wrapper class - Long instead of long in this case, and put a null check in the setter.

private long last_login;

public void setLast_login(Long last_login) {
    if(last_login != null) {
        this.last_login = last_login;
    }
}

The cost of avoiding the null pointer exception is the boxing-unboxing overhead.

Add new field or change the structure on all Firestore documents

5 Answers5

Linked