0

I am using this

exports.countposts = functions.database.ref('/rooms/{roomId}/posts').onWrite((change, context) => {

It works! As soon as data is posted (in 99% of cases POSTed) onto the "array" posts (it is an array, I guess in the firebase database world no such exist), this function is called and all is well.

However when the array posts get big, somewhere above 10000 (ten thousand), the function specified silently does not get called at all. I get no error logs. I am assuming/guessing this is a performance issue - when the number of children inside posts gets higher, the execution time of each function gets longer - more or less linearly.

What is the problem here, how can I mitigate this?

ReyAnthonyRenacia
  • 17,219
  • 5
  • 37
  • 56
Jonny
  • 15,955
  • 18
  • 111
  • 232
  • I'm not sure which one you are suggesting, maybe "Size of a single event triggered by a write"? Would an event trigged by a "small post" somehow get big because the array affected, and maybe take up memory as parameter to the function, ? – Jonny Apr 13 '18 at 09:41
  • How big is the JSON for 10K posts? Because the [maximum size of a database trigger event is 10MB](https://cloud.google.com/functions/quotas#resource_limits). – Frank van Puffelen Apr 13 '18 at 13:52
  • The whole thing? I don't know, each new post is at most a few characters long and with that two or three bookkeeping fields so in all around 100 bytes(?) per new post. I'm not sure of the whole array of posts because that would depend on the contents. I was thinking how I could measure this. Maybe I can just make a big JSON of the changed parameter... it's strange to me however (if it's true) that the whole contents of the array gets somehow copied into the changed parameter of the onWrite clause – Jonny Apr 13 '18 at 20:11
  • At 10K posts, the numbers add up. The first parameter gets all data that triggered your function. That why it's usually called `data`. – Frank van Puffelen Apr 13 '18 at 20:33
  • OK. I'm doing this to calculate the number of elements in the array (like everyone else), and put that number somewhere else where my clients can pick it up. I just wonder why this thing needs to have all that data in the changed parameter - I imagine it could be useful if that data could be loaded lazily etc when needed. Or if there was a shallow parameter like with that GET function. I'm certainly not interested in all of the other existing content, or even the new content in this case, I just need the "on", not "what"... – Jonny Apr 13 '18 at 21:21
  • There is no way to get shallow data with Firebase Realtime Database. There is no way to get shallow data with Firebase Realtime Database As said before: the parameter is called `data` normally, because it's the data that you're asking Cloud Functions to trigger on. – Frank van Puffelen Apr 13 '18 at 21:54
  • I guess I could do fine, of course, with getting only the newly added post, so maybe triggering on the path 'rooms/{roomId}/posts/{postId}' would be a better way to achieve that..? – Jonny Apr 15 '18 at 02:30
  • ^ I was on the right track there - avoiding to pull all of the data in the array/collection - but I still needed to get the whole array in order to get the size and I would likely end up with the same/similar problem of handling too much data at once, just to get the child count (though the method really worked). Then I found a better method which you mentioned in this comment: https://stackoverflow.com/a/44761902/129202 thanks, this will fit my needs and was a good lesson learned. I might write a decent answer to this question later. – Jonny Apr 15 '18 at 07:50
  • @Jonny may I ask you to post your solution as an answer? It may help the community in the future. – gr7 May 02 '18 at 14:02

0 Answers0