Associating aiohttp requests with their responses

Question

Very simply, I would simply like to associate responses from aiohttp asynchronous HTTP requests with an identifier, such as a dictonary key, so that I know which response corresponds to which request.

For example, the function below calls URIs suffixed with the dict values 1, 2 and 3. How could it be modified to return the the keys associated with each result? I just need to be able to keep track of which request is which… Doubtless trivial for someone familiar with asyncio

import asyncio
import aiohttp

items = {'a': '1', 'b': '2', 'c': '3'}

def async_requests(items):
    async def fetch(item):
        url = 'http://jsonplaceholder.typicode.com/posts/'
        async with aiohttp.ClientSession() as session:
            async with session.get(url + item) as response:
                return await response.json()

    async def run(loop):
        tasks = []
        for k, v in items.items():
            task = asyncio.ensure_future(fetch(v))
            tasks.append(task)
        responses = await asyncio.gather(*tasks)
        print(responses)

    loop = asyncio.get_event_loop()
    future = asyncio.ensure_future(run(loop))
    loop.run_until_complete(future)

async_requests(items)

Output (abbreviated):

[{'id': 2, ...}, {'id': 3, ...}, {'id': 1...}]

Desired output (for example):

{'b': {'id': 2, ...}, 'c': {'id': 3, ...}, 'a': {'id': 1, ...}}

score 5 · Accepted Answer · answered Apr 27 '16 at 16:49

5

Pass keys to fetch(), to return them with the corresponding response:

#!/usr/bin/env python
import asyncio
import aiohttp  # $ pip install aiohttp

async def fetch(session, key, item, base_url='http://example.com/posts/'):
    async with session.get(base_url + item) as response:
        return key, await response.json()

async def main():
    d = {'a': '1', 'b': '2', 'c': '3'}
    with aiohttp.ClientSession() as session:
        ####tasks = map(functools.partial(fetch, session), *zip(*d.items()))
        tasks = [fetch(session, *item) for item in d.items()]
        responses = await asyncio.gather(*tasks)
    print(dict(responses))

asyncio.get_event_loop().run_until_complete(main())

answered Apr 27 '16 at 16:49

jfs

399,953
195
994
1,670

Thank you, this is precisely what I was after. I notice you open the `ClientSession()` inside `main()` rather than the `fetch()` – is this merely a matter of preference? – Bede Constantinides Apr 28 '16 at 14:26
1

@bedeabc a better question why do you need multiple sessions here? – jfs Apr 28 '16 at 14:31
@j-f-sebastian Thanks, I see your point. I was using patterns from the following article: http://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html – Bede Constantinides Apr 28 '16 at 16:53
@j-f-sebastian Hi, would you mind taking a look at this? http://stackoverflow.com/questions/37901292/asyncio-aiohttp-progress-bar-with-tqdm – Bede Constantinides Jun 18 '16 at 21:17

Associating aiohttp requests with their responses

1 Answers1