Hi i am being supplied a zip file as a number of chunks, . If i join the chunks together i run out of memory (for large files) so I need to stream .i can call a API to get each chunk, but i do not want to join them all together in the program. how do i stream ? I'm trying to use the stream_unzip library. eg.
Note, the below example won't work for me i'm not streaming a zip from a web site, so the below code is not relevant - but I would like to call api multiple times instead.
from stream_unzip import stream_unzip
import httpx
def zipped_chunks(): # how to call a function repeated times?
with httpx.stream('GET', 'https://www.example.com/my.zip') as r:
yield from r.iter_bytes()
unzipped_chunks = iterable_subprocess(['funzip'], zipped_chunks())
for chunk in unzipped_chunks:
print(chunk)
Something like this example
from iterable_subprocess import iterable_subprocess
def yield_input():
# In a real case could read from the filesystem or the network - how?
yield b'first\n'
yield b'second\n'
yield b'third\n'
output = iterable_subprocess(['cat'], yield_input())
for chunk in output:
print(chunk)
The above works - but harded coded - so not suitable - ie how can i an api to get the next chunk. eg. This example (below) calls a function. But how to call multiple times an indicate end of input?
from iterable_subprocess import iterable_subprocess
def nextchunkcallapi():
return b'call api to get next n chunk\n' # something like this
def yield_input():
# In a real case could read from the filesystem or the network - how !?
yield nextchunkcallapi() # how to do this?
output = iterable_subprocess(['cat'], yield_input())
for chunk in output:
print(chunk)