How do I unzip a .zip file in google cloud storage?

Question

How do I unzip a .zip file in Goolge Cloud Storage Bucket? (If we have some other tool like 'CloudBerry Explorer' for AWS, that will be great.)

Answer 1

You can use Python, e.g. from a Cloud Function:

    from google.cloud import storage
    from zipfile import ZipFile
    from zipfile import is_zipfile
    import io

    def zipextract(bucketname, zipfilename_with_path):

        storage_client = storage.Client()
        bucket = storage_client.get_bucket(bucketname)

        destination_blob_pathname = zipfilename_with_path
        
        blob = bucket.blob(destination_blob_pathname)
        zipbytes = io.BytesIO(blob.download_as_string())

        if is_zipfile(zipbytes):
            with ZipFile(zipbytes, 'r') as myzip:
                for contentfilename in myzip.namelist():
                    contentfile = myzip.read(contentfilename)
                    blob = bucket.blob(zipfilename_with_path + "/" + contentfilename)
                    blob.upload_from_string(contentfile)

    zipextract("mybucket", "path/file.zip") # if the file is gs://mybucket/path/file.zip

Answer 2

Here is some code I created to run as a Firebase Cloud Function. It is designed to listen to files loaded into a bucket with the content-type 'application/zip' and extract them in place.

    const functions = require('firebase-functions');
    const admin = require("firebase-admin");
    const path = require('path');
    const fs = require('fs');
    const os = require('os');
    const unzip = require('unzipper')

    admin.initializeApp();

    const storage = admin.storage();


    const runtimeOpts = {
      timeoutSeconds: 540,
      memory: '2GB'
    }

    exports.unzip = functions.runWith(runtimeOpts).storage.object().onFinalize((object) => {

        return new Promise((resolve, reject) => {
            //console.log(object)
            if (object.contentType !== 'application/zip') {
              reject();
            } else {
              const bucket = firebase.storage.bucket(object.bucket)
              const remoteFile = bucket.file(object.name)
              const remoteDir = object.name.replace('.zip', '')

              console.log(`Downloading ${remoteFile}`)

              remoteFile.createReadStream()
                .on('error', err => {
                  console.error(err)
                  reject(err);
                })
                .on('response', response => {
                  // Server connected and responded with the specified status and headers.
                  //console.log(response)
                })
                .on('end', () => {
                  // The file is fully downloaded.
                  console.log("Finished downloading.")
                  resolve();
                })
                .pipe(unzip.Parse())
                .on('entry', entry => {
                  const file = bucket.file(`${remoteDir}/${entry.path}`)

                  entry.pipe(file.createWriteStream())
                  .on('error', err => {
                    console.log(err)
                    reject(err);
                  })
                  .on('finish', () => {
                    console.log(`Finsihed extracting ${remoteDir}/${entry.path}`)
                  });

                  entry.autodrain();

                });
            }
        })

    });

Answer 3

If you ended up having a zip file on your Google Cloud Storage bucket because you had to move large files from another server with the gsutil cp command, you could instead gzip when copying and it will be transferred in compressed format and unzippet when arriving to the bucket.

It is built in gsutil cp by using the -Z argument.

E.g.

gsutil cp -Z largefile.txt gs://bucket/largefile.txt

Answer 4

In shell, you can use the below command to unzip a compressed file

gsutil cat gs://bucket/obj.csv.gz | zcat |  gsutil cp - gs://bucket/obj.csv

Answer 5

There is no mechanism in the GCS to unzip the files. A feature request regarding the same has already been forwarded to the Google development team.

As an alternative, you can upload the ZIP files to the GCS bucket and then download them to a persistent disk attached to a VM instance, unzip them there, and upload the unzipped files using the gsutil tool.

Answer 6

There are Data flow templates in google Cloud data flow which helps to Zip/unzip the files in cloud storage.Refer below screenshots.

This template stages a batch pipeline that decompresses files on Cloud Storage to a specified location. This functionality is useful when you want to use compressed data to minimize network bandwidth costs. The pipeline automatically handles multiple compression modes during a single execution and determines the decompression mode to use based on the file extension (.bzip2, .deflate, .gz, .zip).

Pipeline requirements

The files to decompress must be in one of the following formats: Bzip2, Deflate, Gzip, Zip.

The output directory must exist prior to pipeline execution.

Answer 7

1. Enable Dataflow API in your gcloud console
2. Create a temp dir in your bucket (cant use root).
3. Replace YOUR_REGION (e.g. europe-west6) and YOUR_BUCKET in the below command and run it with gcloud cli (presumption is gz file is at root - change if not):

gcloud dataflow jobs run unzip \
--gcs-location gs://dataflow-templates-YOUR_REGION/latest/Bulk_Decompress_GCS_Files \
--region YOUR_REGION \
--num-workers 1 \
--staging-location gs://YOUR_BUCKET/temp \
--parameters inputFilePattern=gs://YOUR_BUCKET/*.gz,outputDirectory=gs://YOUR_BUCKET/,outputFailureFile=gs://YOUR_BUCKET/decomperror.txt

Answer 8

I'm afraid that by default in Google Cloud no program could do this..., but you can have this functionality, for example, using Python.

Universal method available on any machine where Python is installed (so also on Google Cloud):

You need to enter the following commands:

python

or if you need administrator rights:

sudo python

and then in the Python Interpreter:

>>> from zipfile import ZipFile
>>> zip_file = ZipFile('path_to_file/t.zip', 'r')
>>> zip_file.extractall('path_to_extract_folder')

and finally, press Ctrl+D to exit the Python Interpreter.

The unpacked files will be located in the location you specify (of course, if you had the appropriate permissions for these locations).

The above method works identically for Python 2 and Python 3.

Enjoy it to the fullest! :)

Answer 9

Another fast way to do it using Python in version 3.2 or higher:

import shutil
shutil.unpack_archive('filename')

The method also allows you to indicate the destination folder:

shutil.unpack_archive('filename', 'extract_dir')

The above method works not only for zip archives, but also for tar, gztar, bztar, or xztar archives.

If you need more options look into documentation of shutil module: shutil.unpack_archive

Answer 10

You can create a Google Cloud Function with a Cloud Storage trigger.

When a new object is created, the function will be triggered.

const functions = require("@google-cloud/functions-framework");
const {Storage} = require("@google-cloud/storage");
const unzip = require("unzip-stream");

functions.cloudEvent("gcs-unzip", async cloudEvent => {

    //console.log(JSON.stringify(cloudEvent, null, 4));

    const zipFile = cloudEvent.data;

    //console.log(JSON.stringify(file, null, 4));

    if (zipFile.contentType === "application/zip") {
        const storage = new Storage();

        async function unzipAndUploadContainedFiles() {
            await storage
                .bucket(zipFile.bucket)
                .file(zipFile.name)
                .createReadStream()
                .pipe(unzip.Parse())
                .on("entry", async function (entry) { //there could be multiple files and even a directory structure in the zip file
                    //console.log(JSON.stringify(entry, null, 4));

                    const gcsTargetFileName = zipFile.name.replace(".zip", "") + "/" + entry.path;
                    if (entry.type === "File") {
                        entry.pipe(storage.bucket(zipFile.bucket).file(gcsTargetFileName).createWriteStream());
                    }
                });
        }

        await unzipAndUploadContainedFiles().catch(err => {
            console.error(err);
        });

    } else {
        console.log("Non-zip file ignored.");
    }

});

Answer 11

To unzip all files within a zip I used this one-liner on the cloud shell terminal:

gsutil cat gs://{bucket_name}/data.zip | for i in $(jar --list); do gsutil cat gs://{bucket_name}/data.zip | jar x $i && cat $i | gsutil cp - gs://{bucket_name}/unzipped/$i && rm ./$i; done;