0

I wonder if one can upload a remote file into Google Storage without downloading it first into local storage. Basically what I would like to do is to pass a URL to the storage client and wait until it pulls the file. Thanks!

overbet13
  • 1,654
  • 1
  • 20
  • 36
  • I found this similar [question](https://stackoverflow.com/questions/54235721/transfer-file-from-url-to-cloud-storage) to yours, basically the answer mentions that transferring URLs into GCS is possible through the [Cloud Transfer](https://cloud.google.com/products/data-transfer/).This solution is targeted towards a situation with millions of URLs that need to become GCS objects.Another option is writing a job that pumps an incoming stream from reading a URL into a write stream to GCS and running that somewhere in the Google Cloud close to the bucket. – Andie Vanille Jun 19 '20 at 22:54
  • Can you detail more your use case? Are you on a website? What mean "local storage" for you? A server? your local environment? – guillaume blaquiere Jun 20 '20 at 12:37
  • Basically I'm running an app on a vm and I would like to upload some files from a third party directly to GS without downloading it first. Hence local storage would mean storage of the VM. Sorry if this was not clear enough. – overbet13 Jun 21 '20 at 19:47
  • I really think that is not possible upload the files directly to Google Cloud Storage without downloading it first :( maybe using code you can achieve this. – Andie Vanille Jun 25 '20 at 17:45
  • Did you ever find a solution to this? I am trying to do the same thing. Closest I have been able to come is downloading it locally into a folder that uses GSFUSE. https://cloud.google.com/storage/docs/gcs-fuse It's still downloading locally but that local server is actually a Google Computer Engine (Google VM). So the compute engine and GSFUSE directory (which is a Google Cloud Storage bucket) are on the same private google network. – fred Aug 30 '20 at 22:28
  • 1
    hey @fred I came to use GSFUSE which is quite impressive in my opinion. If set up correctly, it works like a charm. I mounted a bucket and used it as local file system. You might want to keep an eye on the key differences between GSFUSE and a POSIX file system (concurrency was a bit of an issue for me) and on the pricing as well. So I recommend this, it worked almost perfectly for me. – overbet13 Aug 31 '20 at 12:18

1 Answers1

2

You should be able to use a google colaboratory instance to do this. You can download a file to the instance and then save it to your drive (though you have to "mount" the drive first). Then, you can save file from the instance to your drive. While this is still a 2-step process, it at least avoids having to transfer files to your local machine

DerekG
  • 3,555
  • 1
  • 11
  • 21