6

I've been pickling the objects to filesystem and reading them back when needed to work with those objects. Currently I've this code for that purpose.

def pickle(self, directory, filename):
    if not os.path.exists(directory):
        os.makedirs(directory)
    with open(directory + '/' + filename, 'wb') as handle:
        pickle.dump(self, handle)

@staticmethod
def load(filename):
    with open(filename, 'rb') as handle:
        element = pickle.load(handle)
    return element

Now I'm moving my applictation(django) to Google app engine and figured that app engine does not allow me to write to file system. Google cloud storage seemed my only choice but I could not understand how could I pickle my objects as cloud storage objects and read them back to create the original python object.

Jo Kachikaran
  • 562
  • 1
  • 6
  • 12

3 Answers3

6

You can use the Cloud Storage client library.

Instead of open() use cloudstorage.open() (or gcs.open() if importing cloudstorage as gcs, as in the above-mentioned doc) and note that the full filepath starts with the GCS bucket name (as a dir).

More details in the cloudstorage.open() documentation.

Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97
  • Thanks for your time @Dan. I'll give it a try and let you know if I face any issues. – Jo Kachikaran Jun 12 '16 at 20:51
  • That worked! I had to run my application through app engine dev server. Although I cannot see the files in the bucket through the browser, I'm able to read them through program. What am I supposed to do to view those files in browser? – Jo Kachikaran Jun 13 '16 at 02:53
  • 1
    I didn't actually use GCS yet, so I can't test, but this might help: http://stackoverflow.com/questions/25110110/local-storage-browser-for-google-cloud-storage-and-dev-appserver-py – Dan Cornilescu Jun 13 '16 at 12:44
  • I figured that myself though when I tried to deploy my application while my dev server is running. It brought up local dev server and I could see all the files I created using cloudstorage. It seems, they are simulating the cloudstorage behavior in my local rather than creating the files actually on the cloud. I'm gonna test my application on app engine today and I'll update if I can see the objects through browser. – Jo Kachikaran Jun 13 '16 at 19:18
6

For Python 3 users, you can use gcsfs library from Dask creator to solve your issue.

Example reading:

import gcsfs

fs = gcsfs.GCSFileSystem(project='my-google-project')
fs.ls('my-bucket')
>>> ['my-file.txt']
with fs.open('my-bucket/my-file.txt', 'rb') as f:
    print(f.read())

It basically is identical with pickle, though:

with fs.open(directory + '/' + filename, 'wb') as handle:
        pickle.dump(shandle)

To read, this is similar, but replace wb by rb and dump with load:

with fs.open(directory + '/' + filename, 'rb') as handle:
        pickle.load(handle)
tripleee
  • 175,061
  • 34
  • 275
  • 318
LaSul
  • 2,231
  • 1
  • 20
  • 36
3

One other option (I tested it with Tensorflow 2.2.0) which also works with Python 3:

from tensorflow.python.lib.io import file_io

with file_io.FileIO('gs://....', mode='rb') as f:
    pickle.load(f)

This is very useful if you already use Tensorflow for example.

Dr. Fabien Tarrade
  • 1,556
  • 6
  • 23
  • 49