2

My problem is the following:

I have two .p (pickle files) in my bucket in the google cloud storage and I would like to load them on my jupyter notebook (where I run my code on the VM instance).

I tried the following 

url = "http:\\<localhost>\tree\storage.googleapis.com\<bucket-name>"
data_x = pickle.load(open(url + "\<filename>.p", "rb" ))

but it said that there is no such file or directory. I have also tried to create a public link on the google storage with no success. I managed to load .mat files and .pts files that way but I can't seem to be able to load the .p file.

There was a similar question posted here: pickling python objects to google cloud storage

but it is depreciated and I did not manage to use the answer.

This is also a resource that could be useful: https://pypi.python.org/pypi/google-cloud-storage

Thanks a lot for you help!!

Doug Stevenson
  • 297,357
  • 32
  • 422
  • 441
Tom
  • 275
  • 2
  • 16
  • Please note that there are different client libraries for accessing the cloud storage depending on where you're running your app: GAE standard/flex, GCE, outside Google's cloud, etc. See https://cloud.google.com/apis/docs/client-libraries-explained – Dan Cornilescu Jan 17 '18 at 15:16
  • Thaks for your answer! I am not using the app engine but simply the compute engine on the jupyter notebook. I have the .p file on both my storage bucket and my VM instance. – Tom Jan 17 '18 at 17:19

1 Answers1

3

For Python >= 3.5 users :

One thing you can do is to use gcsfs library from Dask creator that works really similar to open but almost just replacing with fs.open

Example reading :

import gcsfs, pickle

fs = gcsfs.GCSFileSystem(project = 'my-google-project')
fs.ls('my-bucket')
>>> ['my-file.txt']
with fs.open('my-bucket/my-file.txt', 'rb') as file:
    print(pickle.load(file))

To write, this is similar. You just need to replace rb by wb and load with dump

LaSul
  • 2,231
  • 1
  • 20
  • 36