Extract Google Drive zip from Google colab notebook

Question

I already have a zip of (2K images) dataset on a google drive. I have to use it in a ML training algorithm. Below Code extracts the content in a string format:

from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
import io
import zipfile
# Authenticate and create the PyDrive client.
# This only needs to be done once per notebook.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

# Download a file based on its file ID.
#
# A file ID looks like: laggVyWshwcyP6kEI-y_W3P8D26sz
file_id = '1T80o3Jh3tHPO7hI5FBxcX-jFnxEuUE9K' #-- Updated File ID for my zip
downloaded = drive.CreateFile({'id': file_id})
#print('Downloaded content "{}"'.format(downloaded.GetContentString(encoding='cp862')))

But I have to extract and store it in a separate directory as it would be easier for processing (as well as for understanding) of the dataset.

I tried to extract it further, but getting "Not a zipfile error"

dataset = io.BytesIO(downloaded.encode('cp862'))
zip_ref = zipfile.ZipFile(dataset, "r")
zip_ref.extractall()
zip_ref.close()

Google Drive Dataset

Note: Dataset is just for reference, I have already downloaded this zip to my google drive, and I'm referring to file in my drive only.

score 114 · Answer 1 · edited Apr 16 '20 at 11:26

114

You can simply use this

!unzip file_location

edited Apr 16 '20 at 11:26

Zeeshan Ali

137
1
10

answered Apr 14 '18 at 19:06

Harsh Gupta

1,433
1
10
7

1

@zeeshan how can i specify output directory ? – Asghar Nazir Nov 12 '21 at 12:39
1

Specify it next to the `file_location`. – Zeeshan Ali Nov 15 '21 at 12:02

score 90 · Answer 2 · answered Oct 23 '19 at 04:51

90

TO unzip a file to a directory:

!unzip path_to_file.zip -d path_to_directory

answered Oct 23 '19 at 04:51

giapnh

2,950
24
20

Works great. Fast. Example !unzip drive/MyDrive/feedback-prize-2021/train.zip -d drive/MyDrive/feedback-prize-2021/train/ – Sahar Millis Feb 12 '23 at 10:39

score 28 · Answer 3 · answered Apr 30 '19 at 15:21

28

To extract Google Drive zip from a Google colab notebook:

import zipfile
from google.colab import drive

drive.mount('/content/drive/')

zip_ref = zipfile.ZipFile("/content/drive/My Drive/ML/DataSet.zip", 'r')
zip_ref.extractall("/tmp")
zip_ref.close()

answered Apr 30 '19 at 15:21

Alon Lavian

1,149
13
14

How much time does it take for the files to reflect in the destination directory? My zip file size is around 10 gb – Abhishek Boorugu May 27 '22 at 13:01
This code snippet is not generating output at all. Am I missing something? – Abhishek Boorugu May 27 '22 at 13:21
1

I was trying to unzip in a shared folder location. Issue was resolved when I used the absolute path. – Abhishek Boorugu May 27 '22 at 15:02

Suvo · Answer 4 · 2020-07-30T10:56:27.817

Colab research team has a notebook for helping you out.

Still, in short, if you are dealing with a zip file, like for me it is mostly thousands of images and I want to store them in a folder within drive then do this --

!unzip -u "/content/drive/My Drive/folder/example.zip" -d "/content/drive/My Drive/folder/NewFolder"

-u part controls extraction only if new/necessary. It is important if suddenly you lose connection or hardware switches off.

-d creates the directory and extracted files are stored there.

Of course before doing this you need to mount your drive

from google.colab import drive 
drive.mount('/content/drive')

I hope this helps! Cheers!!

score 9 · Answer 5 · answered Aug 15 '19 at 09:58

9

First, install unzip on colab:

!apt install unzip

then use unzip to extract your files:

!unzip  source.zip -d destination.zip

answered Aug 15 '19 at 09:58

Omid Ghahroodi

91
1
3

what if the file is protected with a password? I tried to add -pPassword but it does not work, I found this error "caution: filename not matched: MyPassword" – Naham Al-Zuhairi Apr 17 '23 at 17:33

score 6 · Answer 6 · edited Sep 08 '19 at 17:03

6

Mount GDrive:

from google.colab import drive
drive.mount('/content/gdrive')

Open the link -> copy authorization code -> paste that into the prompt and press "Enter"

Check GDrive access:

!ls "/content/gdrive/My Drive"

Unzip (q stands for "quiet") file from GDrive:

!unzip -q "/content/gdrive/My Drive/dataset.zip"

edited Sep 08 '19 at 17:03

Community

1
1

answered May 20 '19 at 21:28

Plo_Koon

2,953
3
34
41

1

for unzip isn't it I/O-wise better to first copy the zip file to local Colab storage then do the unzip operation? – Farzan Feb 22 '20 at 18:51

score 6 · Answer 7 · answered Aug 22 '19 at 22:51

6

First create a new directory:

!mkdir file_destination

Now, it's the time to inflate the directory with the unzipped files with this:

!unzip file_location -d file_destination

answered Aug 22 '19 at 22:51

sargupta

953
13
25

score 5 · Answer 8 · edited May 27 '20 at 14:16

5

For Python

Connect to drive,

from google.colab import drive
drive.mount('/content/drive')

Check for directory

!ls and !pwd

For unzip

!unzip drive/"My Drive"/images.zip

edited May 27 '20 at 14:16

Krunal Kapadiya

2,853
3
16
37

answered Nov 27 '19 at 14:16

abdul

526
1
5
10

1

For the code part, please use the language-specific format ( it's Python in this context). – Vincent Jia Nov 27 '19 at 14:25
Have added, Thanks – abdul Nov 27 '19 at 17:22

score 5 · Answer 9 · edited Sep 06 '21 at 13:28

5

Please use this command in google colab

Unzip the file you want to extract and then the location

!unzip "drive/My Drive/Project/yourfilename.zip" -d "drive/My Drive/Project/yourfolder"

edited Sep 06 '21 at 13:28

MD Mushfirat Mohaimin

1,966
3
10
22

answered Sep 06 '21 at 04:24

Floyd Fernandes

147
1
4

Md Hishamur Rahman · Answer 10 · 2020-05-21T11:07:57.003

4

After mounting on drive, use shutil.unpack_archive. It works with almost all archive formats (e.g., “zip”, “tar”, “gztar”, “bztar”, “xztar”) and it's simple:

import shutil
shutil.unpack_archive("filename", "path_to_extract")

edited May 21 '20 at 11:07

answered Mar 22 '20 at 09:30

Md Hishamur Rahman

334
4
11

It doesn't work for large datasets. I have tried it on a 60GB zip file, but it didn't work out! – Rajdeep Borgohain Feb 14 '22 at 06:26
It's not a problem with shutil; Colab RAM does not have the capacity to hold your 60 GB zip file in the first place. – Md Hishamur Rahman Feb 14 '22 at 08:03
I got 35GB ram, and I think that is sufficient. I think it doesn't load the whole file to the memory. I don't know but in my local system with 8GB I have extracted more than 50GB without any issue. – Rajdeep Borgohain Feb 14 '22 at 08:07

score 2 · Answer 11 · answered Jun 13 '22 at 20:24

2

This is what worked for me.

!apt install unzip

Then I used this code to unzip the file

!unzip /content/file.zip -d /content/

Without installing unzip on Colab first you'll always receive error messages.

answered Jun 13 '22 at 20:24

Taeef Najib

57
4

score 1 · Answer 12 · answered Apr 07 '18 at 07:19

1

Instead of GetContentString(), use GetContentFile() instead. It will save the file instead of returning the string.

downloaded.GetContentFile('images.zip')

Then you can unzip it later with unzip.

answered Apr 07 '18 at 07:19

korakot

37,818
16
123
144

It seemed to work, but was too fast perhaps. But then unzip didn't work. Archive: flowers.zip End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive. unzip: cannot find zipfile directory in one of flowers or flowers.zip, and cannot find flowers.ZIP, period. – Rafael_Espericueta Jul 22 '19 at 22:27

score 1 · Answer 13 · answered Apr 02 '19 at 19:46

SIMPLE WAY TO CONNECT

1) You'll have to verify authentication

from google.colab import auth
auth.authenticate_user()
from oauth2client.client import GoogleCredentials
creds = GoogleCredentials.get_application_default()

2)To fuse google drive

!apt-get install -y -qq software-properties-common python-software-properties module-init-tools
!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null
!apt-get update -qq 2>&1 > /dev/null
!apt-get -y install -qq google-drive-ocamlfuse fuse

3)To verify credentials

import getpass
!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL
vcode = getpass.getpass()
!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}

4)Create a drive name to use it in colab ('gdrive') and check if it's working

!mkdir gdrive
!google-drive-ocamlfuse gdrive
!ls gdrive
!cd gdrive

score 0 · Answer 14 · answered Jul 28 '20 at 20:37

Try this:

!unpack file.zip

If its now working or file is 7z try below

!apt-get install p7zip-full
!p7zip -d file_name.tar.7z
!tar -xvf file_name.tar

Or

!pip install pyunpack
!pip install patool

from pyunpack import Archive
Archive(‘file_name.tar.7z’).extractall(‘path/to/’)
!tar -xvf file_name.tar

score 0 · Answer 15 · answered Aug 29 '21 at 08:36

0

We assumed that you're already mounting your googleDrive to googleColab. in case if you just want to extract zip file thats contains .csv extention. just call pandas attribute read_csv

pd.read_csv('/content/drive/My Drive/folder/example.zip')

answered Aug 29 '21 at 08:36

Kanade - san

21
1

score -2 · Answer 16 · answered Feb 22 '21 at 20:28

-2

in my idea, you must go to a certain path for example:

from google.colab import drive drive.mount('/content/drive/') cd drive/MyDrive/f/

then :

!apt install unzip !unzip zip_folder.zip -d unzip_folder enter image description here

answered Feb 22 '21 at 20:28

hamed aghapanah

11
1

Extract Google Drive zip from Google colab notebook

16 Answers16

Linked