How to read (stream) files from Google Drive in memory in python?

Question

I'm trying to read large CSV files that are dropped on Google Drive using the google-api-python-client https://google.github.io/google-api-python-client/docs/epy/googleapiclient.http.MediaIoBaseDownload-class.html

I was able to download the file on the hard drive doing this:

request = drive_service.files().get_media(fileId=file_id)
fh = io.FileIO('test.csv', mode='w')
downloader = MediaIoBaseDownload(fh, request)
done = False
while done is False:
    status, done = downloader.next_chunk()

But I was wondering if there's a simple way to read it in chunks in memory.

Possible duplicate of [Python - Download File Using Requests, Directly to Memory](https://stackoverflow.com/questions/22340265/python-download-file-using-requests-directly-to-memory) — Linda Lawton - DaImTo, Aug 05 '17 at 13:25

score 1 · Accepted Answer · answered Aug 18 '17 at 23:42

    api_service_object = self.service
    request = api_service_object.files().get_media(fileId=file_id)
    stream = io.BytesIO()
    downloader = MediaIoBaseDownload(stream, request)
    done = False
    # Retry if we received HttpError
    for retry in range(0, 5):
        try:
            while done is False:
                status, done = downloader.next_chunk()
                print "Download %d%%." % int(status.progress() * 100)
            return stream.getvalue()
        except HTTPError as error:
            print 'There was an API error: {}. Try # {} failed.'.format(
                error.response,
                retry,
            )

How to read (stream) files from Google Drive in memory in python?

1 Answers1