38

I've just noticed that my app's storage started to increase significantly.

After having a closer look, it appeared that this was caused by the "artifacts" bucket.

I can see that the "artifacts" storage keeps increasing by about ~800Mb every week which worries me to say the least.

I assume this is related to firestore functions deploys (or not?), but is this really expected? Can I cleanup this artifacts safely?

Appreciate any suggestions on how to safely handle storage size in this case and to keep its consumption at minimum.

vir us
  • 9,920
  • 6
  • 57
  • 66

3 Answers3

62

Figured out a solution - it appeared there is a way to setup an auto deletion rule in google cloud console for those images that clutter the storage.

  1. go to the google cloud console, select your project -> storage -> browser https://console.cloud.google.com/storage/browser

  2. Select the "artifacts" bucket

  3. Under the "lifecycle" tab add a rule to auto delete old images (in my case I put "delete after 1 day since update" which works fine for me)

enter image description here

Storage is safe now!

NOTE: if you face any deployment issues later, like if you deploy several days in a row and if it gives you an error on deploy, just delete the whole "container" folder manually in the artifacts which should solve it and then redeploy again. (make sure not to delete the artifacts bucket itself!)

Hope the firebase team will improve that - the current behavior looks confusing as it easily leads to an unexpected bill unless you take extra steps to prevent that. But you'll never know that it will happen until it does.

vir us
  • 9,920
  • 6
  • 57
  • 66
  • I also surprised, I thought someone has hacked my database :D since I got about 82 objects and total storage around 16 Mb. But my cloud storage shows me 443Mb has stored. I only use the cloud function to store ratings and count the comments. Can you explain what is happening here, I checked my bucket details as you did above. When I checked my us.artifacts, there are many files with images and some files contain more than 60 MB. I even did not use funtions to change the image quality. Where did you find information related to this? or can you explain what is behind the scene ? – Isuru Bandara Sep 30 '20 at 18:03
  • 1
    @IsuruBandara - it all looked quite unexpected to me so I started to google and found a similar question about "what is it?" in the other thread on SO - I've included that link in my question. As Frank mentions there, this is how firebase handles functions deploys now if you use Node10+. And as Doug confirms here, this is now an expected behavior, which I hope will change at some point. – vir us Oct 01 '20 at 10:28
  • More info here: https://stackoverflow.com/a/64144324/2228559 – aleclarson Feb 02 '21 at 18:39
  • 1
    I'm 90% sure this setting causes deployment failures with cloud functions. I think this is a recent issue: https://stackoverflow.com/questions/67338215/cloud-function-build-error-failed-to-get-os-from-config-file-for-image. Is anyone else seeing this? – Sameer Madan May 04 '21 at 00:07
  • @SameerMadan if you face deployment issues you can try to delete the whole "container" folder manually in the artifacts and try to redeploy again. (just make sure not to delete the artifacts bucket itself) – vir us May 04 '21 at 08:19
  • That means the problem is somewhere else – vir us May 05 '21 at 06:24
  • Yep, I can confirm that this solution is now causing issues with deployments. It didn't use to. But now it does. https://groups.google.com/g/google-cloud-dev/c/xHqqTbRMXrk – koullislp May 30 '21 at 13:46
  • @koullislp you will face a deployment issue if you deploy several days in a row. This will lead to deleting some files of the images while keeping others and you end up with inconsistent images leading to deploy errors. In this case you can either wait for another day and the issue will resolve itself or delete the whole container as mentioned in the note from the answer. Again, if that doesn't solve your problem, your issue is somewhere else. – vir us May 30 '21 at 15:15
  • Maybe setting the lifecycle rule to a few days (week) instead of one already solves the deployment issue? – Fr4nc3sc0NL Jan 07 '22 at 19:56
4

I assume this is related to firestore functions deploys (or not?), but is this really expected?

Yes, it's expected. Every time you deploy functions, Cloud Build will use a dedicated Cloud Storage space for the built docker image, and retain it until you delete it.

Can I cleanup this artifacts safely?

Yes, but then you won't be able to easily revert to a prior image. You would have to deploy again from your own source code.

Doug Stevenson
  • 297,357
  • 32
  • 422
  • 441
  • Thanks Dough. Is there a way to manage that process so it doesn't consume such a huge amount of space? And is there any upper limit or will it keep increasing indefinitely with future deploys? I guess it never behave like that previously. – vir us Sep 11 '20 at 15:46
  • It's entirely up to you to devise and execute a plan to keep costs under control in a way that suits your preferences. Every deploy will require more space, so you should manage the space as needed. – Doug Stevenson Sep 11 '20 at 15:47
  • 1
    any docs/references on how to keep this process under control? Currently from your answer it looks like by default it will keep increasing through the roofs unless we somehow explicitly control that on our end. Would appreciate if you could elaborate on that. – vir us Sep 11 '20 at 15:52
  • I don't know. A web search might be helpful. – Doug Stevenson Sep 11 '20 at 15:53
  • ok, I though it's a firebase related process which is documented somewhere. Thanks anyway. – vir us Sep 11 '20 at 18:38
  • 1
    hey @DougStevenson when you say deploy a function what do you mean. Do you mean each time a function is run, for example in my app if i make it in such a way that every time a button is clicked a cloud function would run to perform a server function, would my storage increase each time, or my storage taken by functions would only increase when I make a new function, which means if i don't edit or make a new function for a year the storage taken by functions would also not increase for a year ? Please answer thanks – Jesus Loves You Sep 12 '20 at 07:34
  • when i say run a function i mean per every invocation.. – Jesus Loves You Sep 12 '20 at 07:36
1

On top of the GCP's Life Cycle settings for artifacts images, you can also consider the following for further optimization and costs reduction of your Firebase Functions deployment:

  1. Clean up your functions folder, don't put unnecessary files in it, as we do not know if Google will only upload files by dependencies or by the whole functions folder. Feel free to refine this item if anyone of you can confirm this.
  2. Remove unnecessary dependencies from functions/package.json, functions/node_modules and require statements from your JS files, e.g. functions/index.js.
  3. Compact and compress your function's JS files by removing unnecessary comments, console loggings etc. You can achieve this with the help of grunt and uglify NPM packages. Again, we're not sure if the Cloud Build (or any of the Google functions deployment system) will auto-compress the function's images for us before storing them into the Container Registry or Cloud Storage (please refine this item if you have better answer).
  4. Organize your functions properly by creating relevant function groups so that you can deploy only certain group(s) of function rather than simply firebase deploy --only functions.
  5. If necessary, write codes that automatically detect and resolve environmental differences, e.g. environment variables from local emulators to production/staging, because the Firebase emulators and production environments may not be 100% consistent. If you don't do that, you may end up needing to deploy several times per day due to certain negligence -- this will spike up your deployment cost.
  6. If necessary, change your deployment plan: from daily to weekly, or even from weekly to monthly, depending on your monthly budgets, criticality, and urgency.

Lastly, I hope the community can also help to add more recommended costs reduction plans and strategies on this post in order to help some small businesses and individuals to survive better on Firebase and Google Cloud Platform as a whole. Even just some links to good articles would help. Thanks!

Antonio Ooi
  • 1,601
  • 1
  • 18
  • 32
  • Yo, if I delete the entire folder, does my env files that I set with firebase also be gone? – chichi Dec 12 '21 at 13:51