1

I can move data in google storage to buckets using the following:

gsutil cp afile.txt gs://my-bucket

How to do the same using the python api library:

from google.cloud import storage
storage_client = storage.Client()

# Make an authenticated API request
buckets = list(storage_client.list_buckets())
print(buckets)

Cant find anything more than the above.

Mpizos Dimitris
  • 4,819
  • 12
  • 58
  • 100
  • Hi Mpizos, do you mean upload data to a cloud storage bucket (e.g. from your local machine)? – Paul Jun 18 '18 at 13:13
  • @Paul yes. Not necessarily from local machine. But generally how to move data to a cloud storage bucket with the python api – Mpizos Dimitris Jun 18 '18 at 15:59

1 Answers1

4

There is an API Client Library code sample code here. My code typically looks like below which is a slight variant on the code they provide:

from google.cloud import storage

client = storage.Client(project='<myprojectname>')
mybucket = storage.bucket.Bucket(client=client, name='mybucket')
mydatapath = 'C:\whatever\something' + '\\'   #etc
blob = mybucket.blob('afile.txt')
blob.upload_from_filename(mydatapath + 'afile.txt')

In case it is of interest, another method is to run the "gsutil" command line how you have typed in your Original Post using the subprocess command, e.g.:

import subprocess
subprocess.call("gsutil cp afile.txt gs://mybucket/", shell=True)

In my view, there are pros and cons of both methods depending on what you are trying to achieve - the latter method allows multi-threading if you have many files to upload whereas the former method perhaps allows better control, specification of metadata for each file, etc.

Paul
  • 528
  • 5
  • 17
  • I would like to add to Paul's answer two links that may be of help. [Cloud Storage Libraries](https://cloud.google.com/storage/docs/reference/libraries) and [GIT: Python GCS samples](https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/storage) – Iñigo Jul 11 '18 at 11:25
  • `shell=True` may have security vulnerabilities: https://stackoverflow.com/questions/3172470/actual-meaning-of-shell-true-in-subprocess – Dave Liu Feb 10 '21 at 21:52