64

I have a Google App Engine app, which connects to Google Cloud Storage.

I noticed that the amount of data stored was unreasonably high (4.01 GB, when it should be 100MB or so).

So, I looked at how much each bucket was storing, and I found that there was an automatically created bucket called us.artificats. that was taking up most of the space.

I looked inside, and all it has is one folder: containers/images/.

From what I've Googled, it seems like these images come from Google Cloud Build.

My question is, can I delete them without compromising my entire application?

Caleb H.
  • 1,657
  • 1
  • 10
  • 31
  • 1
    Only delete container images using the Console or the CLI `gcloud container images delete`. A container consists of layers. You are seeing these layers as objects in Cloud Storage. Do not directly delete these objects. In simple terms, these objects are cached layers that are used to `build` a container image. For some tools if a layer is not cached it will first be pulled (downloaded), but I have not tried to deliberately delete container registry objects to see what happens. – John Hanley Jan 27 '20 at 20:10
  • Thank you for the clarification! What do I use as my image name? – Caleb H. Jan 27 '20 at 20:36
  • `gcloud container images list` says there are no images... – Caleb H. Jan 27 '20 at 20:37
  • 1
    Do you have more than one repository? Use the `--repository=[HOSTNAME]/[PROJECT-ID]` Repository locations are gcr.io, us.gcr.io, eu.gcr.io, and asia.gcr.io. Also review this document: https://cloud.google.com/container-registry/docs/managing – John Hanley Jan 27 '20 at 20:54
  • I found images in the `us.gcr.io` repository. I deleted images from the UI, but they still exist in Cloud Storage. It looks like they were only deleted from the Cloud Registry. – Caleb H. Jan 27 '20 at 20:56
  • I don't have an exact answer for you if these objects are required or just cached (saved) for future use. – John Hanley Jan 27 '20 at 21:51
  • In your post, you mention that it should have only stored around 100 MB. Where did you get this number from? – Evan Baldonado Sep 09 '20 at 12:07
  • 1
    @EvanBaldonado I got that from the number and size of the files I was intentionally uploading. – Caleb H. Sep 09 '20 at 18:54

7 Answers7

62

I have solved this problem by applying a deletion rule. Here's how to do it:

  1. Open the project in Google Cloud console
  2. Open the storage management (search for "Storage" for example).
  3. In the Browser tab, select the container us.artifacts....
  4. Now, open the Lifecycle section. You should see something like:

enter image description here

  1. Click on Add a rule and provide the following conditions:
    1. In the action, select Delete object
    2. In the conditions, select Age and enter for example 3 days
  2. Click on create to confirm the creation

Now all objects older than 3 days will be automatically deleted. It might take a few minutes for this new rule to be applied by Google Cloud.

Frenchcooc
  • 910
  • 6
  • 20
HarshitG
  • 2,677
  • 3
  • 16
  • 13
  • Could you clarify which rules you added? – Evan Baldonado Sep 09 '20 at 12:07
  • 3
    Use a rule ... like old than 3 days... delete automatically... then it will only have latest artifacts ... and saves your space – HarshitG Sep 10 '20 at 14:23
  • 2
    @CalebH This should be the correct answer. Relying on a third party tool is specific to your use case. – D G Oct 06 '20 at 02:38
  • Has anyone had issues with their CFs with the above setup? From my reading, it looks like these files that we'd be cleaning up with the "3 day old" rule are just interim build files, and not actually used in production? Does anyone know of a way to configure such a rule by command line or config file or something? I have several instance of my project so I'd rather configure by script than by hand. – Greg Fenton Jan 17 '21 at 19:00
  • 1
    Help! I set this up a few days ago (3 day lifecycle) and now it's failing with NoSuchKey when I try to deploy via gcloud command line. It says it's looking for a specific sha256 image file which now apparently doesn't exist. Should I delete the whole container image folder? – policenauts Jan 29 '21 at 18:01
  • 1
    @policenauts , u can delete whole container registry manually , it will be get automatically created with latest artifacts once u deploy new version. – HarshitG Mar 16 '21 at 19:56
  • 6
    Weird, why is this not done automatically? Now I'm being charged for garbage google decided had to be stored for me... – Fr4nc3sc0NL Jan 07 '22 at 19:11
29

For those of you seeing this later on, I ended up deleting the folder, and everything was fine.

When I ran Google Cloud Build again, it added items back into the bucket, which I had to delete later on.

As @HarshitG mentioned, this can be set up to happen automatically via deletion rules in cloud storage. As for myself, I added a deletion step to my deployment GitHub action.

Caleb H.
  • 1,657
  • 1
  • 10
  • 31
  • 1
    If there's someone else like me, with no idea about where to delete these files, you can go to https://console.cloud.google.com/ – Mr. DMX Aug 31 '20 at 16:09
  • I already deleted 2 times but he always comeback and always comeback with more increase than before... – luke cross Sep 09 '20 at 18:48
  • @lukecross I ended up adding the deletion as a step in a Github Action so it automatically deletes after deployment. But, you could also look at the answer by HarshitG, where he uses deletion rules. – Caleb H. Sep 09 '20 at 18:52
  • He says "I using exclude rules" but not says which "Rules" he used... lol – luke cross Sep 09 '20 at 19:42
  • 4
    The Firebase until today not said or posted about this auto bucket... The only thing that her says is "Node JS 10 brought this creation as required to run functions..." i think that's a big irresponsability with us. – luke cross Sep 09 '20 at 19:44
  • @lukecross that may be, but until they do anything about it, we'll just have to live with it. As for deletion rules, I would try setting it to delete objects after a few hours or so - depending on how long your functions take to run. As long as the files are available when they are needed, you should be fine to have them automatically delete after that. – Caleb H. Sep 11 '20 at 16:11
  • I've an cloud function with pub/sub that is happened every day at 23hours, i think that i can use this function to delete the _unnecessary_ bucket... But, Storage with CF is not good idea – luke cross Sep 12 '20 at 12:47
  • 2
    I also deleted the whole artifact bucket 3 days ago. The storage fell down from 500 Mb to 8Mb, but the bill did not decrease. I am still been charged in a few cents. – Aliton Oliveira Oct 18 '20 at 18:58
  • Help! I set up a deletion rule (3 day lifecycle) and now it's failing with NoSuchKey when I try to deploy via gcloud command line. It says it's looking for a specific sha256 image file which now apparently doesn't exist. Should I delete the whole container image folder? – policenauts Jan 29 '21 at 18:13
19

Here is the reference to the documentation: link

Built container images are stored in the app-engine folder in Container Registry. You can download these images to keep or run elsewhere. Once deployment is complete, App Engine no longer needs the container images. Note that they are not automatically deleted, so to avoid reaching your storage quota, you can safely delete any images you don't need. For more information about managing images in Container Registry, see the Container Registry documentation.

This can be automated by adding a Lifecycle rules like @HarshitG mentioned. enter image description here

Shadorian
  • 182
  • 1
  • 9
samchuang
  • 411
  • 5
  • 15
6

Same issue. Thanks for the update, Caleb.

I'm having the same issue, but I don't have an App running; I just have:

  • Firebase Auth
  • Firestore
  • Firebase Functions
  • Cloud Storage

Not sure why I have 4GB stored in those containers, and I'm not sure if I should delete them or if that would break my functions.

UPDATE: I deleted the container folder and all still works. Not sure if those are backups or whatnot, but I can't find anything online or in the docs. I will post here if something happens. As soon as a cloud function ran, the folder had 33 files again.

Eric Crescioni
  • 299
  • 3
  • 10
  • After the cloud function ran, how much space did the 33 files take up? Was it still 4GB, or was it significantly less? – Evan Baldonado Sep 09 '20 at 12:06
  • I ended up deleting the whole folder (artifact bucket) and I'm afraid that my cloud functions stop working. Is there any chance of that happening? – Aliton Oliveira Oct 15 '20 at 21:31
6

You can add a trigger to your lifecycle rules on console.cloud.google.com.

  1. Access the bucket with artifacts (default is "us.artifacts.yourAppName.appspot.com")
  2. Go to "Life cycle".
  3. Click on "Add a rule".
  4. Check "Object delete" and press "Continue".
  5. Check the filter to delete the bucket, I chose "age" and selected three days as the number of auto delete old elements (after element has 3 days of life it's auto deleted).
  6. Click on "Create" and the rule is working now, then you do not need to visit every day to clean the bucket.
DjongaNelson Lontrowski
  • 163
  • 2
  • 8
3

I recommend against setting up a lifecycle rules on your Storage buckets. It will likely lead to breaking subsequent updates to the function (as described in Cloud Function build error - failed to get OS from config file for image)

If you are interested in cleaning up container images, you should instead delete container images stored in Google Cloud Registry https://console.cloud.google.com/gcr. Deleting container images on GCR repos will automatically cleanup objects stored in your Cloud Storage.

https://issuetracker.google.com/issues/186832976 has relevant information from a Google Cloud Functions engineer.

Daniel L
  • 450
  • 3
  • 6
  • 1
    "Deleting container images on GCR repos will automatically cleanup objects stored in your Cloud Storage." does not seem to work for me. – Nakilon Mar 20 '22 at 19:34
0

To optimize cloud storage cost it is always recommended practice to add lifecycle rules on cloud storage buckets.

To add lifecycle rules in google cloud console search for storage buckets and navigate to your bucket.

Cloud Storage Life Cycle management

In life cycle rules create a Delete action and then set the condition for delete.

Delete Rule for object Life Cycle management

When any objects inside this storage buckets reaches age greater than specified in condition it is deleted. This rule is applicable to all objects inside storage bucket.

Learn more about Google Cloud storage object life cycle management here https://googlecloudtutorials.com/google-cloud-storage-life-cycle-management-and-optimize-storage-cost/

prasad rane
  • 1
  • 2