0

Stuck on trying to get Playwright to work in Jupyter labs on Windows ... Jupyter Labs 3.5.2, Python 3.9.15, Playwright 1.29.0. Code is ...

import asyncio

from playwright.async_api import async_playwright

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto("http://playwright.dev")
        print(await page.title())
        await browser.close()
        
await main()

Error is:

Task exception was never retrieved
future: <Task finished name='Task-5' coro=<Connection.run() done, defined at C:\Users\Larry\anaconda3\envs\TDA_virtual\lib\site-packages\playwright\_impl\_connection.py:240> exception=NotImplementedError()>
Traceback (most recent call last):
  File "C:\Users\Larry\anaconda3\envs\TDA_virtual\lib\site-packages\playwright\_impl\_connection.py", line 247, in run
    await self._transport.connect()
  File "C:\Users\Larry\anaconda3\envs\TDA_virtual\lib\site-packages\playwright\_impl\_transport.py", line 132, in connect
    raise exc
  File "C:\Users\Larry\anaconda3\envs\TDA_virtual\lib\site-packages\playwright\_impl\_transport.py", line 120, in connect
    self._proc = await asyncio.create_subprocess_exec(
  File "C:\Users\Larry\anaconda3\envs\TDA_virtual\lib\asyncio\subprocess.py", line 236, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "C:\Users\Larry\anaconda3\envs\TDA_virtual\lib\asyncio\base_events.py", line 1676, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "C:\Users\Larry\anaconda3\envs\TDA_virtual\lib\asyncio\base_events.py", line 498, in _make_subprocess_transport
    raise NotImplementedError
NotImplementedError
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
Cell In[1], line 16
     13         print(await page.title())
     14         await browser.close()
---> 16 await main()

Cell In[1], line 9, in main()
      8 async def main():
----> 9     async with async_playwright() as p:
     10         browser = await p.chromium.launch()
     11         page = await browser.new_page()

File ~\anaconda3\envs\TDA_virtual\lib\site-packages\playwright\async_api\_context_manager.py:46, in PlaywrightContextManager.__aenter__(self)
     44 if not playwright_future.done():
     45     playwright_future.cancel()
---> 46 playwright = AsyncPlaywright(next(iter(done)).result())
     47 playwright.stop = self.__aexit__  # type: ignore
     48 return playwright

File ~\anaconda3\envs\TDA_virtual\lib\site-packages\playwright\_impl\_connection.py:247, in Connection.run(self)
    244 async def init() -> None:
    245     self.playwright_future.set_result(await self._root_object.initialize())
--> 247 await self._transport.connect()
    248 self._init_task = self._loop.create_task(init())
    249 await self._transport.run()

File ~\anaconda3\envs\TDA_virtual\lib\site-packages\playwright\_impl\_transport.py:132, in PipeTransport.connect(self)
    130 except Exception as exc:
    131     self.on_error_future.set_exception(exc)
--> 132     raise exc
    134 self._output = self._proc.stdin

File ~\anaconda3\envs\TDA_virtual\lib\site-packages\playwright\_impl\_transport.py:120, in PipeTransport.connect(self)
    117     if getattr(sys, "frozen", False):
    118         env.setdefault("PLAYWRIGHT_BROWSERS_PATH", "0")
--> 120     self._proc = await asyncio.create_subprocess_exec(
    121         str(self._driver_executable),
    122         "run-driver",
    123         stdin=asyncio.subprocess.PIPE,
    124         stdout=asyncio.subprocess.PIPE,
    125         stderr=_get_stderr_fileno(),
    126         limit=32768,
    127         creationflags=creationflags,
    128         env=env,
    129     )
    130 except Exception as exc:
    131     self.on_error_future.set_exception(exc)

File ~\anaconda3\envs\TDA_virtual\lib\asyncio\subprocess.py:236, in create_subprocess_exec(program, stdin, stdout, stderr, loop, limit, *args, **kwds)
    229     warnings.warn("The loop argument is deprecated since Python 3.8 "
    230                   "and scheduled for removal in Python 3.10.",
    231                   DeprecationWarning,
    232                   stacklevel=2
    233     )
    234 protocol_factory = lambda: SubprocessStreamProtocol(limit=limit,
    235                                                     loop=loop)
--> 236 transport, protocol = await loop.subprocess_exec(
    237     protocol_factory,
    238     program, *args,
    239     stdin=stdin, stdout=stdout,
    240     stderr=stderr, **kwds)
    241 return Process(transport, protocol, loop)

File ~\anaconda3\envs\TDA_virtual\lib\asyncio\base_events.py:1676, in BaseEventLoop.subprocess_exec(self, protocol_factory, program, stdin, stdout, stderr, universal_newlines, shell, bufsize, encoding, errors, text, *args, **kwargs)
   1674     debug_log = f'execute program {program!r}'
   1675     self._log_subprocess(debug_log, stdin, stdout, stderr)
-> 1676 transport = await self._make_subprocess_transport(
   1677     protocol, popen_args, False, stdin, stdout, stderr,
   1678     bufsize, **kwargs)
   1679 if self._debug and debug_log is not None:
   1680     logger.info('%s: %r', debug_log, transport)

File ~\anaconda3\envs\TDA_virtual\lib\asyncio\base_events.py:498, in BaseEventLoop._make_subprocess_transport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, extra, **kwargs)
    494 async def _make_subprocess_transport(self, protocol, args, shell,
    495                                      stdin, stdout, stderr, bufsize,
    496                                      extra=None, **kwargs):
    497     """Create subprocess transport."""
--> 498     raise NotImplementedError

NotImplementedError:

Have tried everything I have found including modifying init.py of Asyncio with each of the commented line of the following:

import asyncio

if sys.platform == 'win32':
    from .windows_events import *
#    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
#    asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())
    __all__ += windows_events.__all__
else:
    from .unix_events import *  # pragma: no cover
    __all__ += unix_events.__all__

So far no joy ... anyone help would be appreciated.

I believe the error comes from the following snippet from Playwright _transport.py routine

 try:
            # For pyinstaller
            env = get_driver_env()
            if getattr(sys, "frozen", False):
                env.setdefault("PLAYWRIGHT_BROWSERS_PATH", "0")

            self._proc = await asyncio.create_subprocess_exec(
                str(self._driver_executable),
                "run-driver",
                stdin=asyncio.subprocess.PIPE,
                stdout=asyncio.subprocess.PIPE,
                stderr=_get_stderr_fileno(),
                limit=32768,
                creationflags=creationflags,
                env=env,
            )
        except Exception as exc:
            self.on_error_future.set_exception(exc)
            raise exc

and I am thinking that the IOLoop from JupyterLabs is selectoreventlooppolicy and does not support pipes, perhaps that is what the "Not Implemented" is about, but I am still digging into this and into if it this is true and if the behaviour can be modified.

Again, all input is appreciated.

vandel
  • 41
  • 4
  • I'm seeing `asyncio.run(main())` [here](https://stackoverflow.com/a/74518471/8508004) instead of `await main()`. Tried that? Oh wait maybe you got that `await main()` from [here](https://github.com/microsoft/playwright-python/issues/178#issuecomment-1302869947), which I found [thanks to this post](https://stackoverflow.com/a/71702599/8508004). If so, that information on why you modified "init.py" should be referenced as part of your post to make it more obvious if this is the same issue as [here](https://stackoverflow.com/q/71228742/8508004) or a new one. – Wayne Jan 23 '23 at 20:01
  • Cross-posted [here](https://discourse.jupyter.org/t/running-playwright-in-jupyterlab-notebook-problem-not-implemented-error/17653?u=fomightez). Please, don't cross-post without linking **each and every post in each location**. It wastes time of those answering if they say the same thing said elsewhere and makes it hard for people following in your footsteps to find the solution. To help understand why that's important, you may want to consider the person who may be following in your footsteps could be "future you" coming across this issue again several months from now. – Wayne Jan 23 '23 at 20:07
  • Hello, thanks for getting back. I had reviewed all of those links prior to posting here. As indicated in the post, my belief is that this is specific to the type of event loop policy as Selector does not support pipes and playwright uses them. This problem is specific to Win and not Colab or Mac so posts dealing with those implementations did not yield results. The code is from Playwright examples hence the await. The asyncio.run can not be used on a running loop. I modified init.py because I thought that might prevent switch over to Selector policy. Answer will show where it happens. – vandel Jan 24 '23 at 14:09
  • 1
    With respect to Cross-posting, thank you for tagging the Post in the Jupyter Discord group. Since there was no response here for 24 hours, I added it there and was unaware of any relationship between groups and hence any etiquette rules. In the future I will leave a note about cross-posting. I am drafting a solution to the problem now and will post it in the next few minutes. – vandel Jan 24 '23 at 14:13

1 Answers1

2

It appears that the problem is specific to Windows Platform and as identified in the original question as the use of SelectorEventLoopPolicy rather than ProactorEventloopPolicy in JupyterLabs is the source of the issue.

Since SelectorEventLoopPolicy does not support Pipes the Not implemented error occurs when Playwright first tries to open a pipe. This link identifies that the EventLoopPolicy for JupyterLab is set in the routine kernelapp.py of ipykernel.

        if sys.platform.startswith("win") and sys.version_info >= (3, 8):
            import asyncio

            try:
                from asyncio import (
                    WindowsProactorEventLoopPolicy,
                    WindowsSelectorEventLoopPolicy,
                )
            except ImportError:
                pass
                # not affected
            else:
                if type(asyncio.get_event_loop_policy()) is WindowsProactorEventLoopPolicy:
                    # WindowsProactorEventLoopPolicy is not compatible with tornado 6
                    # fallback to the pre-3.8 default of Selector
#Comment out  - 637              asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())
#Add this to complete the if     print(f'do not change to Selector Event Loop')

Not sure if this will have any adverse effects on other operations, but it does cause the original code to execute in JupyterLab.

I implemented this as separate environment in case it has side effects. Perhaps the folks responsible for JupyterLab can comment on what the side effects might be.

vandel
  • 41
  • 4