1

I query multiple services with asynchronously using httpx. These services return csv data that could be very large so I'm using streams.

So far so good.

The problem I'm having is that the Python standard csv.reader doesn't work with AsyncIterator's and the async stream won't let you call the sync functions like iter_lines(). Doing that gets you the following error:

RuntimeError: Attempted to call a sync iterator on an async stream.

So you're stuck with the async functions like aiter_lines() which return an AsyncIterator.

I've seen aiocsv but that can't be used with AsyncIterator's since it expects an object with a read(size: int).

Example code below:

from httpx import AsyncClient
from csv import reader

async with AsyncClient() as client:
    async with client.stream("POST", url, content=query, headers=HEADER, timeout=25) as resp:
        lines = resp.aiter_lines()
        await anext(lines) # skip headers

        for line in reader(lines):
             yield QueryResult(*line) # create namedtuple for ease of use

TypeError: 'async_generator' object is not iterable

mkrieger1
  • 19,194
  • 5
  • 54
  • 65
fanta fles
  • 149
  • 8
  • 1
    Does this answer your question? [How to wrap asyncio with iterator](https://stackoverflow.com/questions/55132673/how-to-wrap-asyncio-with-iterator) – mkrieger1 Apr 19 '23 at 09:55

0 Answers0