0

Very similar problem described in another question and have good answer, but didn't help to solve the problem.

I wrote Scraper class which support async context manager. COUNT_OF_TESTS is defining number of methods which will be added to TestScraper for simulating a lot of tests.

mre.py:

import unittest

from aiohttp import ClientSession, ClientResponse


COUNT_OF_TESTS = 20


class Scraper:
    async def do_get_request(self, url: str) -> ClientResponse:
        return await self.session.get(url)

    async def __aenter__(self) -> 'Scraper':
        self.session = await ClientSession().__aenter__()
        return self

    async def __aexit__(self, exc_type, exc_val, exc_tb) -> None:
        await self.session.close()


class TestScraper(unittest.IsolatedAsyncioTestCase):
    async def _some_test(self) -> None:
        async with Scraper() as scraper:
            resp = await scraper.do_get_request('https://icanhazip.com')
        self.assertEqual(resp.status, 200)

for i in range(1, COUNT_OF_TESTS+1):
    setattr(TestScraper, f'test_{i}', TestScraper._some_test)

python -m unittest mre.py:

...../usr/local/lib/python3.11/traceback.py:428: ResourceWarning: unclosed <socket.socket fd=6, family=2, type=1, proto=6, laddr=('192.168.0.104', 35912), raddr=('104.18.114.97', 443)>
  result.append(FrameSummary(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.11/asyncio/selector_events.py:843: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=6>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.11/traceback.py:428: ResourceWarning: unclosed <socket.socket fd=7, family=2, type=1, proto=6, laddr=('192.168.0.104', 32772), raddr=('104.18.115.97', 443)>
  result.append(FrameSummary(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.11/asyncio/selector_events.py:843: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=7>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
...../usr/local/lib/python3.11/traceback.py:428: ResourceWarning: unclosed <socket.socket fd=6, family=2, type=1, proto=6, laddr=('192.168.0.104', 32784), raddr=('104.18.115.97', 443)>
  result.append(FrameSummary(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.11/traceback.py:428: ResourceWarning: unclosed <socket.socket fd=7, family=2, type=1, proto=6, laddr=('192.168.0.104', 32792), raddr=('104.18.115.97', 443)>
  result.append(FrameSummary(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.11/traceback.py:428: ResourceWarning: unclosed <socket.socket fd=9, family=2, type=1, proto=6, laddr=('192.168.0.104', 35942), raddr=('104.18.114.97', 443)>
  result.append(FrameSummary(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.11/asyncio/selector_events.py:843: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=9>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
...../usr/local/lib/python3.11/traceback.py:428: ResourceWarning: unclosed <socket.socket fd=6, family=2, type=1, proto=6, laddr=('192.168.0.104', 32810), raddr=('104.18.115.97', 443)>
  result.append(FrameSummary(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.11/traceback.py:428: ResourceWarning: unclosed <socket.socket fd=7, family=2, type=1, proto=6, laddr=('192.168.0.104', 35948), raddr=('104.18.114.97', 443)>
  result.append(FrameSummary(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.11/traceback.py:428: ResourceWarning: unclosed <socket.socket fd=9, family=2, type=1, proto=6, laddr=('192.168.0.104', 35950), raddr=('104.18.114.97', 443)>
  result.append(FrameSummary(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.11/traceback.py:428: ResourceWarning: unclosed <socket.socket fd=11, family=2, type=1, proto=6, laddr=('192.168.0.104', 35964), raddr=('104.18.114.97', 443)>
  result.append(FrameSummary(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.11/asyncio/selector_events.py:843: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=11>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
.....
----------------------------------------------------------------------
Ran 20 tests in 1.330s

OK

All tests passed, but these warnings... I tried manually close ClientSession:

async def _some_test(self) -> None:
    async with Scraper() as scraper:
        resp = await scraper.do_get_request('https://icanhazip.com')
        await scraper.session.close()  # <-- Closing session
    self.assertEqual(resp.status, 200)

Also tried creating session in asyncSetUp() and close it in asyncTearDown():

class Scraper:
    def __init__(self, session: ClientSession | None = None):
        self.session = session

    async def do_get_request(self, url: str) -> ClientResponse:
        return await self.session.get(url)

    async def __aenter__(self) -> 'Scraper':
        if self.session is None:
            self.session = await ClientSession().__aenter__()
        return self

    async def __aexit__(self, exc_type, exc_val, exc_tb) -> None:
        await self.session.close()


class TestScraper(unittest.IsolatedAsyncioTestCase):
    async def asyncSetUp(self) -> None:
        self.session = await ClientSession().__aenter__()

    async def asyncTearDown(self) -> None: 
        await self.session.close()

    async def _some_test(self) -> None:
        # pass session from asyncSetUp
        async with Scraper(self.session) as scraper:
            resp = await scraper.do_get_request('https://icanhazip.com')
        self.assertEqual(resp.status, 200)

Anyway getting these warnings. Any advice would be helpful, thanks!

P.S. I can't reproduce same issues if COUNT_OF_TEST <= 3. If it's > 5 getting error from time to time, but of course in production i have more than 20 tests

555Russich
  • 354
  • 2
  • 10
  • According to https://docs.aiohttp.org/en/stable/client_advanced.html#graceful-shutdown, you need to add arbitrary sleep after closing session, and here I can "fix" your problem by adding `await asyncio.sleep(0.250)`. – bennylp Jun 06 '23 at 06:10

0 Answers0