0

I tried to add a large array of objects in firebase collection but it is taking too much time. The array has around 60 000 rows and I need to provide every single object. To do this I need to loop thru all array elements:

for (let i = 0; i < array.length; i++ {
  this.someFunName(array[i])
}

The firebase function is:

  someFunName(element) {
    return this._firestore
    .collection('collName').add(element);
  }

For around 5 min only 1500 elements where recorded. Do you have any suggestions how to handle it or some alternative? I know that the array is too large.

Regards

Atanas Ivanov
  • 69
  • 7
  • Do you need to create a new document for each element in your array? – Alex Mamo Aug 16 '21 at 09:12
  • Yes, this is the function. In this way a separated document with id is created in the collection. – Atanas Ivanov Aug 16 '21 at 09:26
  • Is it a one time operation? E.g. an initialization of the database. – Renaud Tarnec Aug 16 '21 at 09:57
  • No it is not a one time operation and from here the issue comes because time to time a such file will be needed to be uploaded and this will take huge amount of time. – Atanas Ivanov Aug 16 '21 at 10:27
  • I think you should either use ‘map’ or ‘forEach’ to reduce the time instead of ‘for’ loop. For reference you can go through these documents: [1] https://rxjs-dev.firebaseapp.com/api/operators/map [2] https://docs.angularjs.org/api/ng/function/angular.forEach – Zeenath S N Aug 17 '21 at 08:47
  • Regarding map and forEach you are definitely right, they are faster but the problem is not to loop thru the array, the problem is these 60 k request to firebase. Firebase has some limits and I'm searching a proper way to write this large information faster. For example if I load these 60k rows from a csv file with forEach the file is rendered for 10-15 seconds but the requests to firebase for these 60k rows will took around 40 min (around 1500 records for 5 min). – Atanas Ivanov Aug 17 '21 at 10:48
  • Sorry, they are 40 requests (not 40 min) x 5 min each = 200 min (over 3hours). That is the reason to search more optimal way to do it. – Atanas Ivanov Aug 17 '21 at 12:16
  • According to this documentation[1], you can use the server client library for bulk writes. And you can refer to this stackoverflow answer[2] in which a Firebase engineer has explained in detail about the time taken to write while using the server client library. I think this should help you. [1] https://cloud.google.com/firestore/docs/manage-data/transactions#batched-writes [2] https://stackoverflow.com/a/58897275/15774177 – Zeenath S N Aug 19 '21 at 05:23

0 Answers0