Fetching multiple urls with aiohttp in python

Question

In a previous question, a user suggested the following approach for fetching multiple urls (API calls) with aiohttp:

import asyncio
import aiohttp


url_list = ['https://api.pushshift.io/reddit/search/comment/?q=Nestle&size=30&after=1530396000&before=1530436000', 'https://api.pushshift.io/reddit/search/comment/?q=Nestle&size=30&after=1530436000&before=1530476000']

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.json()['data']


async def fetch_all(session, urls, loop):
    results = await asyncio.gather(*[loop.create_task(fetch(session, url)) for url in urls], return_exceptions= True)
    return results

if __name__=='__main__':
    loop = asyncio.get_event_loop()
    urls = url_list
    with aiohttp.ClientSession(loop=loop) as session:
        htmls = loop.run_until_complete(fetch_all(session, urls, loop))
    print(htmls)

However, this results in only returning Attribute errors:

[AttributeError('__aexit__',), AttributeError('__aexit__',)]

(which I enabled, otherwhise it would just break). I really hope there is somebody here, who can help with this, it is still kind of hard to find resources for asyncio etc. The returned data is in json format. In the end I would like to put all json dicts in a list.

Yurii Kramarenko · Accepted Answer · 2018-08-07T13:35:52.303

19

Working example:

import asyncio
import aiohttp
import ssl

url_list = ['https://api.pushshift.io/reddit/search/comment/?q=Nestle&size=30&after=1530396000&before=1530436000',
            'https://api.pushshift.io/reddit/search/comment/?q=Nestle&size=30&after=1530436000&before=1530476000']


async def fetch(session, url):
    async with session.get(url, ssl=ssl.SSLContext()) as response:
        return await response.json()


async def fetch_all(urls, loop):
    async with aiohttp.ClientSession(loop=loop) as session:
        results = await asyncio.gather(*[fetch(session, url) for url in urls], return_exceptions=True)
        return results


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    urls = url_list
    htmls = loop.run_until_complete(fetch_all(urls, loop))
    print(htmls)

edited Aug 07 '18 at 13:35

answered Aug 07 '18 at 13:29

Yurii Kramarenko

1,025
6
16

2

```async with aiohhtp.ClientSession(connector=aiohttp.TCPConnector(verify_ssl=False))``` no more tears to cry – py_dude Aug 07 '18 at 13:32
@Jannik looks strange, which version of aiohttp\python you are using? – Yurii Kramarenko Aug 08 '18 at 08:20
@YuriiKramarenko i am using aiohttp==0.7.2, asyncio 3.4.3 and python 3.6 – Jannik Aug 08 '18 at 08:37
@YuriiKramarenko you were so right. I updated it, now it works! thank you so much. I have no idea, why my version was so outdated – Jannik Aug 08 '18 at 08:51
2

Not working for me got below error : RuntimeError: This event loop is already running – Krissh Dec 29 '19 at 06:44
@Krissh the problem is here `loop.run_until_complete(fetch_all(urls, loop))`, your loop has been already ran. Just call `await fetch_all(urls, loop)` – Yurii Kramarenko Dec 30 '19 at 12:16
asyncio.get_event_loop is deprecated in Python 3.10. https://docs.python.org/3.10/library/asyncio-eventloop.html#asyncio.get_event_loop – Bufke Aug 03 '22 at 13:57

Fetching multiple urls with aiohttp in python

1 Answers1

Linked