How to unzip folder in google cloud platform?

Question

My dataset is in the following form with 3 folders labelled 0, 5, 10 and each has about 200,000 images :

frames_zip :
           ->0
           ->5
           ->10

I have been trying to unzip my folder with the Dataflow API using Bulk Decompress Cloud Storage Files but the folders get unzipped into a single file. I have tried everything in this question as well but nothing is working.

What is the total size of an unzipped file? I might have a workaround — guillaume blaquiere, Aug 24 '20 at 20:37
23GB I'm thinking I might just zip each folder up individually then run it through the Bulk Decompress API — yudhiesh, Aug 25 '20 at 03:25
Do you need to perform this unzip only 1 time? or at least asynchronously? — guillaume blaquiere, Aug 25 '20 at 10:04
Just one time and I tried the Bulk Decompress on each folder individually and they are converted to text. — yudhiesh, Aug 25 '20 at 12:49

score 1 · Accepted Answer · answered Aug 25 '20 at 13:34

There is several way to achieve this.

If it's one time, simply
- create a compute engine,
- install zip on it
- Download your file from Storage
- unzip the file locally
- upload the uncompressed file and folder architecture to Cloud Storage gsutil -m cp -r ./local-dir gs://myBucket
- Delete the VM
If it's a periodic task to uncompress the file (for example every week)
- Create a Cloud Build pipeline, with 1 step which perform exactly the same things as before with the VM (install zip, download the zip files, uncompress and send back the uncompressed file).
- Schedule periodically Cloud Build pipeline with Cloud Scheduler
- Cloud Build is serverless and you can have up to 1000Gb of local storage

diskSizeGb: Use the diskSizeGb option to request a custom disk size for your build. The maximum size you can request is 1000 GB.

Worked perfectly I already had the unzipped folder locally I just had to set up gcloud on my laptop and upload it. — yudhiesh, Aug 26 '20 at 01:59

How to unzip folder in google cloud platform?

1 Answers1