1

I am learning web scraping with asyncio and aiohttp. The scraper seems to work for few times and then shows some errors for multiple tries and then works again. The same URL opens normally in browser every time. Tried few fixes found from web but nothing seems to fix this problem and also not many fixes about it is available. Here is my code;

import asyncio
import time
import aiohttp

class TestScraper:
    def __init__(self, query):
        self.query = query

    async def main(self):
        url = f"https://something.com/query-{self.query}.html"
        headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36",
            "Connection": "keep-alive",
            "Accept": "*/*",
        }
        async with aiohttp.ClientSession(headers=headers) as session:
            async with session.get(url) as r:
                print(await r.json())

if __name__ == "__main__":
    time_start = time.time()
    print("Starting..........\n\n")
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
    asyncio.run(TestScraper("fruits").main())
    print("\n\n..........Finished")
    time_end = time.time()
    print(f"Time taken: {time_end - time_start}")

And the errors showing are;

ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

The above exception was the direct cause of the following exception:
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host something.com:443 ssl:default [An existing connection was forcibly closed by the remote host]

What's causing the problem and how to fix it?

Thanks and have a nice day.

micBighne.98
  • 31
  • 1
  • 7

0 Answers0