0

I am currently trying to zip a large file (> 1GB) using python on google app engine and I have used the following solution due to the limitations google app engine places on the memory cache for a process. Create a zip file from a generator in Python?

When I run the code on the app engine, I get the following error

Traceback (most recent call last):
  File "/base/data/home/apps/s~whohasfiles/frontend.379535120592235032/gluon/restricted.py", line 212, in restricted
    exec ccode in environment
  File "/base/data/home/apps/s~whohasfiles/frontend.379535120592235032/applications/onefile/controllers/page.py", line 742, in <module>
  File "/base/data/home/apps/s~whohasfiles/frontend.379535120592235032/gluon/globals.py", line 194, in <lambda>
    self._caller = lambda f: f()
  File "/base/data/home/apps/s~whohasfiles/frontend.379535120592235032/applications/onefile/controllers/page.py", line 673, in download
    zip_response = page_store.gcs_zip_page(page, visitor)
  File "applications/onefile/modules/page_store.py", line 339, in gcs_zip_page
    w = z.start_entry(ZipInfo('%s-%s' %(file.created_on, file.name) ))
  File "applications/onefile/modules/page_store.py", line 481, in start_entry
    r, w = os.pipe()
OSError: [Errno 38] Function not implemented

Does the google app engine not support the OS.pipe() function? How can I get a work around please?

Community
  • 1
  • 1
Arnold Ewin
  • 1,113
  • 1
  • 13
  • 26

1 Answers1

0

The 'os' module is available but with unsupported features disabled such as pipe() as it operates on file objects [1]. You would need to use a Google Cloud Storage bucket as a temporary object as there is no concept of file objects you can use for storage local to the App Engine runtime. The GCS Client Library will give you file-like access to a bucket which you can use for this purpose [2]. Every app has access to a default storage bucket which you may need to first activate [3].

[1] https://cloud.google.com/appengine/docs/python/#Python_Pure_Python
[2] https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/
[3] https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/activate

Adam
  • 5,697
  • 1
  • 20
  • 52