Slow fetching data when querying in large data in firebase realtime

Question

i have Big Data has nearly 1 million records and more , the query takes a long time in fisrt query And most of the time.

my Rules:

{
  "rules": {
    "iBkH2321":{
      "$uid":{
        ".read": "false",
          ".write": "true"    
      }
    },
      "Contacts":{
            ".indexOn": ["mobile","name","android_id"],
           ".read":"root.child('iBkH2321').child(auth.uid).val() == true",
          ".write":"root.child('iBkH2321').child(auth.uid).val() == true",
      },
    "AppVersion":{
      ".read": "true",
          ".write": "false",
            ".indexOn":["VersionNumber"]
    }
  }
}

an data like this:

 

{
  "contacts" : {
  
         "-Moj9LK5rgNYVcD1PJ2S" : {
          "android_id" : "312e62167b076ce1",
          "mobile" : "770966855",
          "name" : "عبدالله سيلان / فرع اب الشركة"
          },
         "-Moj9LKEZgZqu8tyW-4c" : {
          "android_id" : "312e62167b076ce1",
          "mobile" : "711237507",
          "name" : "عبد الملك عمر / الهلال للصرافة تعز"
         },
         "-Moj9LKFmA0E6jJMuWfD" : {
          "android_id" : "312e62167b076ce1",
          "mobile" : "711290007",
          "name" : "احمد هاشم /ايتي عدن للصرافة"
         },
         .
         .
         .
         .
         .
      } 
  "iBkHop2321" : {
    "026Z0V8ad8MnJtnfn9qt6atgMWS2" : true,
    "0CDZ0V8aAxMnJtnre9qt6atgZiw2" : true,
         .
         .
         .
         . 
     }

and this my code:

private  DatabaseReference myRef ;
.
.
.
final FirebaseDatabase database = FirebaseDatabase.getInstance();
        myRef = database.getReference("Contacts");
auth = FirebaseAuth.getInstance();
.
.
.

   private  void TestShearch(String number){
        AtomicBoolean getResult = new AtomicBoolean(false);
        ArrayList<Contacts> arrayList = new ArrayList<>();
        ArrayContacts arrayContacts = new ArrayContacts(context, arrayList);
        listView.setAdapter(arrayContacts);
        Query query = myRef.orderByChild("mobile").equalTo(number).limitToLast(30);
        ValueEventListener valueEventListener = new ValueEventListener() {
           @Override
            public void onDataChange(@NonNull DataSnapshot snapshot) {
                if(isAdded() && activity != null) {
                    arrayList.clear();
                        getResult.set(true);
                        for (DataSnapshot ds : snapshot.getChildren()) {
                            Contacts con = ds.getValue(Contacts.class);
                            arrayList.add(con);
                            assert con != null;
                        }
                        arrayContacts.notifyDataSetChanged();
                        listView.setAdapter(arrayContacts);
                    } else {
                        getResult.set(false);
                        Toast.makeText(context, "لا توجد بيانات لهذا الرقم حالياً", Toast.LENGTH_LONG).show();
                    }
                    progressBar.setVisibility(View.GONE);
                    btn_search.setVisibility(View.VISIBLE);
                    myRef.removeEventListener(this);
                }
            }
            @Override
            public void onCancelled(@NonNull DatabaseError error) {
                //do somthing here
            }
        };
 query.addListenerForSingleValueEvent(valueEventListener);
}

note: am using authentication in onStart():

@Override
    protected void onStart() {
        super.onStart();
        FirebaseUser currentUser = mAuth.getCurrentUser();
        updateUI(currentUser);
    }

private void updateUI(FirebaseUser currentUser) {
        if(currentUser == null){
            mAuth.signInAnonymously().addOnCompleteListener(this, new         OnCompleteListener<AuthResult>() {
                @Override
                public void onComplete(@NonNull Task<AuthResult> task) {
                    if(task.isSuccessful()){

                        FirebaseUser user = mAuth.getCurrentUser();
                        updateUI(user);
                        myRef2.child(Objects.requireNonNull(mAuth.getUid())).setValue(true);
                        Log.d("userAuth11", "signInAnonymously:parent success");

                    } else{
                        Log.d("userAuth11", "signInAnonymously:parent failed");
                        updateUI(null);
                    }
                }
            });

        }

when i launch the app the query takes longer than 30-60 sec after that becomes faster than old query!

can you help me? thanks.

I think that the problem is in the size of the data that I inquired from, as the size of the data is approximately 2 gigabytes?

Frank van Puffelen · Accepted Answer · 2023-08-11T13:25:51.897

0

i have Big Data has nearly 1 million records and more

That's probably the problem. Firebase Realtime Database keeps its indexes in memory for fast access, but that does mean it has limits to the number of items in can hold in an index before performance deteriorates.

You'll need to consider changing your data model to better fit your needs. Re-modeling the data to fit your use-cases is quite common when using NoSQL databases.

For example, if you want to access the contacts by their phone number, consider creating an extra data structure that stores precisely that data:

contacts_by_mobile: {
  "770966855": {
    "-Moj9LK306LUDCXIbypG": true
  },
  "711237507": {
    "-Moj9LK306LUDCXIbypG": true
  },
  "711290007": {
    "-Moj9LK306LUDCXIbypG": true
  }
  ...
}

You can use this to look up the contact keys without needing to query the whole contacts list, and then look up the individual contacts one by one without a query. There are many ways to do this, pretty much whatever works to meet your use-case.

Also see:

edited Aug 11 '23 at 13:25

answered Aug 10 '23 at 23:54

Frank van Puffelen

565,676
79
828
807

But i need other data like name and android id when i orderByChild("number").equalTo(number) i need them, and also i have another function i used it to find data by name like ths orderByChild("name"). startAt(name) – antiko11 Aug 11 '23 at 06:53
You can either *also* store the addition data instead of the `true` value I used above, or you can look up each individual contact. The performance of the latter will be better than you might expect, as [Firebase pipelines such requests](https://stackoverflow.com/questions/35931526/speed-up-fetching-posts-for-my-social-network-app-by-using-query-instead-of-obse/35932786#35932786). --- If you also need to look up by name, you can create yet another similar data structure `contacts_by_name` to use for *that* look up. – Frank van Puffelen Aug 11 '23 at 13:25
Ok thank you, but how to restructuring large data 2gb? – antiko11 Aug 11 '23 at 13:38
Nothing I said suggested restructuring the existing data, but rather to add **additional** structure to allow the new use-cases. To get the existing data into these new structures too, you can pull only the keys from `contacts` through the rest API, and then request and process them one by one – Frank van Puffelen Aug 11 '23 at 22:01
Thank you for helping, but I recently noticed that I changed the data structure as follows, excuse me look my questions again I have modified it – antiko11 Aug 12 '23 at 22:55
The logic remains the same as I showed in my answer: there is no way to query a list of that size efficiently in Firebase Realtime Database, so you'll need to create a separate data structure that allows you to perform the lookup. – Frank van Puffelen Aug 13 '23 at 03:26

Slow fetching data when querying in large data in firebase realtime

1 Answers1