0

I'm having trouble sifting through all the threads about uploading from a form and can't seem to find anything related to downloading and send from a URL.

I have a large gzipped json file that I need to download from an external server and process its results on App Engine. I have it working right now so that it is downloads the file into memory and unzips it, and then processes it into a task queue in small tasks. However, before making small code memory optimizations, I was hitting my 128MB limit on App Engine. I'm worried this is eventually going to happen again.

Here is my code just in case it is helpful to anyone else:

READ_BLOCK_SIZE = 1024*8
request = urllib2.Request(url)
response = urllib2.urlopen(request)
d = zlib.decompressobj(16+zlib.MAX_WBITS)

str = ""
while True:
    data = response.read(READ_BLOCK_SIZE)
    if not data:
        break
    data = d.decompress(data)
    str += data
return str

Does anyone have any thoughts? Is there a good way to handle large files on App Engine and send them right to GCS so I don't have to hold it all in memory? Stream the download right to GCS somehow?

normmcgarry
  • 661
  • 2
  • 6
  • 22
  • 1
    As a band-aid solution you could also (maybe just temporarily) bump your instance class to one with more memory. – Dan Cornilescu Jun 04 '18 at 04:37
  • While bumping the instance level up is a good suggestion, is there a reason you marked this as a duplicate? I clicked the answer you submitted as a duplicate, which has to do with uploading from a URL, and that's exactly what I pointed out in the first sentence of my post. I'm downloading from a URL on an external server, not uploading to a GCS upload url. – normmcgarry Jun 04 '18 at 08:06
  • I cover that as well in the answer. My intention was to just suggest the duplicate, usually others vote as well, but since I got the gold badge the voting process is bypassed. I'll revert that. – Dan Cornilescu Jun 04 '18 at 13:52
  • Possible duplicate: https://stackoverflow.com/questions/32972477/is-it-possible-to-upload-a-file-by-url-to-cloud-storage-on-app-engine-without-wr – Dan Cornilescu Jun 04 '18 at 13:53

0 Answers0