0

Today I was trying to speed up my script and found great example code from another stackoverflow post. Basically I found a way to make async requests using aiohttp to web instead of using requests. Here is the link to that post (I copied code from DragonBobZ's answer).

Link to other stackoverflow post from which I copied code

The issue is that I am trying to get it to return 2 values (url, response) instead of just the response from the request made. Here is the code I took.

def async_aiohttp_get_all(urls, cookies):
    async def get_all(urls):
        async with aiohttp.ClientSession(cookies=cookies) as session:
            async def fetch(url):
                async with session.get(url) as response:
                    return await response.json()
            return await asyncio.gather(*[
                fetch(url) for url in urls
            ])

    return sync.async_to_sync(get_all)(urls)

for x in async_aiohttp_get_all(urls_list, s.cookies.get_dict()):
    print(x)

Now I am successfully able to get responses from all urls within fraction of time it was taking with requests but I want the function to also return the url with:

return await response.json()

I tried this but nothing works and this is my first day to ever use async practices in python so I am not even able to search for a solution as nothing makes sense.

return await url, response.json()
return await (url, response.json())
Mansoor Akram
  • 1,997
  • 4
  • 24
  • 40
  • So it turns out the error was just because I was trying to await url which of course was wrong. When I changed to return await response.json(), url from return await url, response.json(), then it all worked correctly. Now I get it that function needs to wait for response since url is just a static variable already present. – Mansoor Akram Apr 19 '22 at 06:44
  • 1
    However, I accepted the answer as it solves my issue and that too without using sync (asgiref) library. – Mansoor Akram Apr 19 '22 at 06:45

1 Answers1

1

I could not run the code exactly how you do, but I returned a tuple with no problem. Also removed the sync call, since asyncio gives you enough flexibility.

import asyncio
import aiohttp

urls_list = [
    "https://www.google.com",
    "https://www.facebook.com",
    "https://www.twitter.com",
]

async def async_aiohttp_get_all(urls, cookies):
    async with aiohttp.ClientSession(cookies=cookies) as session:
        async def fetch(url):
            async with session.get(url) as response:
                return await response.text(), url
        return await asyncio.gather(*[
            fetch(url) for url in urls
        ])

results = asyncio.run(async_aiohttp_get_all(urls_list, None))
for res in results:
    print(res[0][:10], res[1])

Output:

<!doctype  https://www.google.com
<!DOCTYPE  https://www.facebook.com
<!DOCTYPE  https://www.twitter.com

So, in your case, return await response.json(), url should work.

svex99
  • 518
  • 4
  • 10
  • Thank you. Your code suggestion works exactly as expected. Now sure why the code in question did not work for me. I kept getting this error when I tried to return both response and url. TypeError: object tuple can't be used in 'await' expression – Mansoor Akram Apr 19 '22 at 06:21
  • That error ocurrs if you call `await (response.json(), url)`. `( )` in python is a constructor for tuple type too and await expects a coroutine not a tuple, a coroutine is obtained invoking an async def like in this case `response.json()`. – svex99 Apr 19 '22 at 15:15