I am using asyncio and pyppeteer to test scraping sites. Currently I have:
browser = await launch(
args=[f'--proxy-server={proxyUrl}'],
headless=True,
autoClose=False
)
to launch the browser. I am using autoClose=False as when I ran it as True I was left with hanging Chromium processes.
After struggling with many network errors, and hanging processes the best way I found this to be the best way:
def close_browser(browser):
browser.process.terminate()
However, sometimes the operation will result in no errors being thrown, or up to 18 errors. All the errors are:
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.')>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed.
This is the output when I looped through the code 10 times:
Completed Number 4
Completed Secondary
Completed Primary
Completed Number 5
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.')>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed.
Completed Tertiary
Done
Running all tasks took 8.363972902297974
All tasks successfully ran.
Completed Secondary
Completed Tertiary
Completed Number 4
Completed Primary
Completed Number 5
Done
Running all tasks took 9.243869304656982
All tasks successfully ran.
Completed Tertiary
Completed Primary
Completed Number 4
Completed Number 5
Completed Secondary
Done
Running all tasks took 8.441104650497437
All tasks successfully ran.
Completed Number 5
Completed Primary
Completed Secondary
Completed Number 4
Completed Tertiary
Done
Running all tasks took 8.054714918136597
All tasks successfully ran.
Completed Number 5
Completed Number 4
Completed Secondary
Completed Tertiary
Completed Primary
Done
Running all tasks took 8.639416217803955
All tasks successfully ran.
Completed Number 5
Completed Secondary
Completed Number 4
Completed Tertiary
Completed Primary
Done
Running all tasks took 8.757359027862549
All tasks successfully ran.
Completed Tertiary
Completed Number 5
Completed Primary
Completed Number 4
Completed Secondary
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.')>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed.
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.')>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed.
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.')>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed.
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.')>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed.
Done
Running all tasks took 8.711585521697998
All tasks successfully ran.
Completed Number 4
Completed Tertiary
Completed Secondary
Completed Number 5
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.')>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed.
Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.')>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed.
Completed Primary
Done
Running all tasks took 8.28357720375061
All tasks successfully ran.
Completed Tertiary
Completed Secondary
Completed Number 5
Completed Primary
Completed Number 4
Done
Running all tasks took 8.737911939620972
All tasks successfully ran.
Completed Secondary
Completed Number 5
Completed Number 4
Completed Primary
Completed Tertiary
Done
Running all tasks took 8.866266965866089
All tasks successfully ran.
