1
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get("https://twitter.com/WholesomeMeme")
html = driver.page_source
driver.close()
open("twitter.html", "w", encoding='utf8').write(html)

A Google chrome instance was successfully opened by executing the code above and I was able to see the wholesomeMeme profile page, but when I open twitter.html, only "Sorry, that page doesn't exist" shows up. enter image description here

Anonny
  • 459
  • 1
  • 3
  • 17

1 Answers1

0

I have used your code with a couple of additional options and was able to retrive the Page Source partially before writing the partial Page Source to twitter.html as follows:

  • Code Block:

    options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://twitter.com/WholesomeMeme')
html = driver.page_source
print(driver.page_source)
driver.quit()
open("twitter.html", "w", encoding='utf8').write(html)

  • Console Output:

    <html dir="ltr" lang="en" style="font-size: 15px;"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=0,viewport-fit=cover">
.
<html dir="ltr" lang="en" style="font-size: 15px;"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=0,viewport-fit=cover">
.
.
.
  <noscript>
    <form action="https://mobile.twitter.com/i/nojs_router?path=%2FWholesomeMeme" method="POST" style="background-color: #fff; position: fixed; top: 0; left: 0; right: 0; bottom: 0; z-index: 9999;">
      <div style="font-size: 18px; font-family: Helvetica,sans-serif; line-height: 24px; margin: 10%; width: 80%;">
    <p>We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?</p>
    <p style="margin: 20px 0;">
      <button type="submit" style="background-color: #1da1f2; border-radius: 100px; border: none; box-shadow: none; color: #fff; cursor: pointer; font-size: 14px; font-weight: bold; line-height: 20px; padding: 6px 16px;">Yes</button>
    </p>
      </div>
    </form>
  </noscript>
  .
  <div id="react-root" style="height:100%;display:flex;"><div class="css-1dbjc4n r-13awgt0 r-12vffkv">
  .
//# sourceMappingURL=runtime.19384dc4.js.map
</script>
<script type="text/javascript" charset="utf-8" nonce="" crossorigin="anonymous" src="https://abs.twimg.com/responsive-web/web/polyfills.604422d4.js"></script>
<script type="text/javascript" charset="utf-8" nonce="" crossorigin="anonymous" src="https://abs.twimg.com/responsive-web/web/vendors~main.31775294.js"></script>
<script type="text/javascript" charset="utf-8" nonce="" crossorigin="anonymous" src="https://abs.twimg.com/responsive-web/web/i18n-rweb/en.cb24d894.js"></script>
<script type="text/javascript" charset="utf-8" nonce="" crossorigin="anonymous" src="https://abs.twimg.com/responsive-web/web/i18n-horizon/en.d212af84.js"></script>
<script type="text/javascript" charset="utf-8" nonce="" crossorigin="anonymous" src="https://abs.twimg.com/responsive-web/web/main.a4500454.js"></script>
<script nonce="">
  if (!window.__SCRIPTS_LOADED__['main']) {
    document.getElementById('ScriptLoadFailure').style.display = 'block';
  }
</script>
<script async="" src="https://twitter.com/i/js_inst?c_name=ui_metrics"></script></body></html>
[11684:11992:0605/150054.450:ERROR:broker_win.cc(55)] Error reading broker pipe: The pipe has been ended. (0x6D)

  • Observation: The following error is observed:

    Error reading broker pipe: The pipe has been ended. (0x6D)

Error reading broker pipe: The pipe has been ended. (0x6D)

This error message...

ERROR:broker_win.cc(55)] Error reading broker pipe: The pipe has been ended. (0x6D)

...implies that the pipe is broken as if the browser side has been closed.

This error is defined in broker_win.cc within the Chromium code repository as follows:

Channel::MessagePtr WaitForBrokerMessage(PlatformHandle platform_handle,
                     BrokerMessageType expected_type) {
  char buffer[kMaxBrokerMessageSize];
  DWORD bytes_read = 0;
  BOOL result = ::ReadFile(platform_handle.handle, buffer,
               kMaxBrokerMessageSize, &bytes_read, nullptr);
  if (!result) {
    // The pipe may be broken if the browser side has been closed, e.g. during
    // browser shutdown. In that case the ReadFile call will fail and we
    // shouldn't continue waiting.
    PLOG(ERROR) << "Error reading broker pipe";
    return nullptr;
  }

Reason

The main reason you see this error is because the ChromeDriver controlled Chrome browser gets detected and the navigation gets blocked.

Solution

As a solution you may need to configure the ChromeDriver / Chrome with certain configurations so Selenium driven Chrome Browsing Context doesn't get detected.

References

You can find a couple of relevant detailed discussions in:

tl; dr

Broken pipe error selenium webdriver, when there is a gap between commands?

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352