3

I have a URL (https://example.com/myfile.txt) of a file and I want to upload it to my bucket (gs://my-sample-bucket) on Google Cloud Storage.

What I am currently doing is:

  1. Downloading the file to my system using the requests library.

  2. Uploading that file to my bucket using python function.

Is there any way I can upload the file directly using the URL.

Cool Breeze
  • 738
  • 2
  • 10
  • 26

2 Answers2

0

You can use urllib2 or requests library to get the file from HTTP, then your existing python code to upload to Cloud Storage. Something like this should work:

import urllib2
from google.cloud import storage

client = storage.Client()

filedata = urllib2.urlopen('http://example.com/myfile.txt')
datatoupload = filedata.read()

bucket = client.get_bucket('bucket-id-here')
blob = Blob("myfile.txt", bucket)
blob.upload_from_string(datatoupload)

It still downloads the file into memory on your system, but I don't think there's a way to tell Cloud Storage to do that for you.

Michael
  • 1,428
  • 3
  • 15
  • 34
0

There is a way to do this, using a Cloud Storage Transfer job, but depending on your use case, it may be worth doing or not. You would need to create a transfer job to transfer a URL list.

I marked this question as duplicated from this.

Pablo Almécija Rodríguez
  • 1,470
  • 8
  • 14