0

Hi i am being supplied a zip file as a number of chunks, . If i join the chunks together i run out of memory (for large files) so I need to stream .i can call a API to get each chunk, but i do not want to join them all together in the program. how do i stream ? I'm trying to use the stream_unzip library. eg.

Note, the below example won't work for me i'm not streaming a zip from a web site, so the below code is not relevant - but I would like to call api multiple times instead.

   from stream_unzip import stream_unzip
   import httpx
    
    def zipped_chunks():  # how to call a function repeated times?
        with httpx.stream('GET', 'https://www.example.com/my.zip') as r:
            yield from r.iter_bytes()
    
    unzipped_chunks = iterable_subprocess(['funzip'], zipped_chunks())
    
    for chunk in unzipped_chunks:
        print(chunk)

Something like this example

from iterable_subprocess import iterable_subprocess

def yield_input():
    # In a real case could read from the filesystem or the network - how?
    yield b'first\n'
    yield b'second\n'
    yield b'third\n'

output = iterable_subprocess(['cat'], yield_input())

for chunk in output:
    print(chunk)

The above works - but harded coded - so not suitable - ie how can i an api to get the next chunk. eg. This example (below) calls a function. But how to call multiple times an indicate end of input?

from iterable_subprocess import iterable_subprocess
def nextchunkcallapi():
    return b'call api to get next n chunk\n'  # something like this
def yield_input():
    # In a real case could read from the filesystem or the network - how !?
    yield nextchunkcallapi() # how to do this?
    
output = iterable_subprocess(['cat'], yield_input())

for chunk in output:
    print(chunk)
ozmike
  • 2,738
  • 1
  • 33
  • 40
  • I think this will answer your question - https://stackoverflow.com/questions/72523108/decompressing-gzip-chunks-of-response-from-http-client-call – Rashid88 Jan 17 '23 at 05:28

0 Answers0