2

I've a scraping engine which uses any proxy list, and retries in case the proxy doesn't work. So there are plenty of proxies that timeout, have connection refused, bad certificates etc. After I switched to httpx from aiohttp I have plenty of internal exceptions that don't seem to hinder, just spam the log.

16:47:37: Future exception was never retrieved
future: <Future finished exception=BrokenResourceError()>
Traceback (most recent call last):
  File "/usr/lib/python3.9/asyncio/selector_events.py", line 856, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/pooh/venv39/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 60, in read
    return await self.stream.receive(n)
  File "/home/pooh/venv39/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1095, in receive
    raise self._protocol.exception
anyio.BrokenResourceError

Maybe someone from the developers can shed a light at what it is?

oguz ismail
  • 1
  • 16
  • 47
  • 69
aikipooh
  • 137
  • 1
  • 19
  • This may be due to a huge number of concurrent connections also know as the [C10k problem](https://en.wikipedia.org/wiki/C10k_problem) You should try to add [Semaphore](https://docs.python.org/3/library/asyncio-sync.html#asyncio.Semaphore) into your code to limit the number of concurrent requests – Alain Berrier Aug 03 '21 at 12:33
  • @AlainBerrier, thank you! I've semaphores, so I can limit a number of concurrent tasks. Usually it's 100 and very rarely up to 1000, which should be a complete nothing. I'll try to play with it though, setting it to 10 for example. The problem never happens with direct connection, will try to play with higher number of concurrent tasks (if the target site permits:). – aikipooh Aug 04 '21 at 07:48
  • @AlainBerrier, no, with lesser number of connections it's the same. It must be something about connection error handling within the module, so I surmise. – aikipooh Aug 28 '21 at 07:21

0 Answers0