2

I'm using:

  • Python 3.8
  • Selenium 3.141.0
  • Windows 10 (behind a proxy)
  • Chrome:84.0.4147.105
  • Chromedriver:84.0.4147.30
  • Mac 10.15.6 (does not has a proxy)

Here is my code:

from selenium import webdriver

driver = webdriver.Chrome("D:/webdriver/chromedriver.exe")
driver.get("https://github.com")
driver.quit()

When executing driver.quit(), the exception raise:

Traceback (most recent call last):
  File "C:/Users/taiping/Desktop/data_test/selenium_test.py", line 5, in <module>
    driver.quit()
  File "D:\python3.8\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 158, in quit
    self.service.stop()
  File "D:\python3.8\lib\site-packages\selenium\webdriver\common\service.py", line 151, in stop
    self.send_remote_shutdown_command()
  File "D:\python3.8\lib\site-packages\selenium\webdriver\common\service.py", line 127, in send_remote_shutdown_command
    url_request.urlopen("%s/shutdown" % self.service_url)
  File "D:\python3.8\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "D:\python3.8\lib\urllib\request.py", line 525, in open
    response = self._open(req, data)
  File "D:\python3.8\lib\urllib\request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "D:\python3.8\lib\urllib\request.py", line 502, in _call_chain
    result = func(*args)
  File "D:\python3.8\lib\urllib\request.py", line 1379, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "D:\python3.8\lib\urllib\request.py", line 1354, in do_open
    r = h.getresponse()
  File "D:\python3.8\lib\http\client.py", line 1332, in getresponse
    response.begin()
  File "D:\python3.8\lib\http\client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "D:\python3.8\lib\http\client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

But this code has no error on my Macbook. What's the problem?

Updated: 2020-08-05

I open the debugger and found that every HTTPConnection object have been set the system wide http proxy. But I did not set any options explicitly in my code. And the driver.quit method will send http://localhost:59717/shutdown to chrome to perform quit. So I guess the shutdown url is actually sent to the proxy server, not the local browser.

And I try to use fiddler to check the request informations. There is another problem that I can not decode the https requests because of some certificates configs. So I change the argument of driver.get() to an internal web url of my company. The result is : If I close fiddler, RemoteDisconnected error raise again. And if I open fiddler, all works.

What happened? I know fiddler set the proxy to 127.0.0.1:8888, so I think there could be something wrong with the proxy settings. But I can not fix it. I guess the chrome use the system proxy so the github home page could open correctly, but when send shutdown url to chrome, the request object should not use the system proxy, but it does.

Am I right? And how to fix this problem?

mrd2242
  • 97
  • 1
  • 7
  • 1
    Does this answer your question? [Python http.client.RemoteDisconnected](https://stackoverflow.com/questions/51660709/python-http-client-remotedisconnected) – Zaraki Kenpachi Aug 04 '20 at 12:27
  • Try a different url, hit google's home page just to move the goalposts a bit to see if it's the site - it might be your data stream in ongoing as it's terminated. It's a couple years old ut github talk about blocking scrapers@ https://github.com/github/site-policy/issues/56 - and your mac works because it's not been "caught" yet... all theory of course - because the code works fine for me and i got pretty much identical setup on versions – RichEdwards Aug 04 '20 at 14:15

4 Answers4

1

This error message...

    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

...implies that there was no response from the remote end and the connection to the remote end was closed.

Syntactically, there are no errors in your code block. However GitHub Home Page contains AJAX elements.

This usecase

Most possibly when the webpage attains document.readyState equals "complete" the AJAX calls are still active which implies the connection to the remote end was still not established completely. In such a scenario when you invoke driver.quit() the above error is raised.

Repro

When I executed your usecase I got a similar error on my localhost as follows:

  • Code Block:

    from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://github.com")
driver.quit()

  • Console Output:

    DevTools listening on ws://127.0.0.1:61488/devtools/browser/62c450cf-7ed5-4205-bb9c-d7e50cd13173
[2780:12796:0804/183946.157:ERROR:broker_win.cc(55)] Error reading broker pipe: The pipe has been ended. (0x6D)

Solution

For a cleaner execution of driver.quit() you need to induce WebDriverWait for some element only for the Javascript and AJAX to complete and then invoke driver.quit() as follows:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://github.com")
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//h1[text()='Built for developers']")))
driver.quit()

References

You can find a couple of detailed discussions in:

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • I simply add ` time.sleep(300)` before driver.quit(), but the error still raised. I run the code from pycharm before, this time, I run it from powershell. Some logs print before the error is raised: `DevTools listening on ws://127.0.0.1:64112/devtools/browser/81d4e2d5-1a3c-4679-af71-81903a8c92a4 [9840:8048:0805/095640.600:Error:device_event_log_imp1.cc(208)] [09:56:40.600] Bluetooth: bluetooth_adapter_winrt.cc:1074 Getting Default Adapter failed.` And my network is behind a proxy, does the proxy affect? The github home page load successfully without set any options. – mrd2242 Aug 05 '20 at 02:09
  • @mrd2242 _[9840:8048:0805/095640.600:Error:device_event_log_imp1.cc(208)] [09:56:40.600] Bluetooth: bluetooth_adapter_winrt.cc:1074 Getting Default Adapter failed._ sounds to be a different error all together. I trust your initial error _http.client.RemoteDisconnected: Remote end closed connection without response_ is resolved now. Can you raise a new question with your new requirement please? – undetected Selenium Aug 05 '20 at 13:43
  • This bluetooth_adapter_winrt.cc error print before RemoteDisconnected raised. But bluetooth_adapter_winct.cc error does not stop the program, just print to the console, and RemoteDisconnected still raised. I have updated the post, add some information and my own testing. – mrd2242 Aug 05 '20 at 14:08
  • @mrd2242 _Getting Default Adapter failed_ is a non-blocking error and you can safely ignore it for the time being. _Remote end closed connection without response_ is the error you need to address. Did you try out the answer? – undetected Selenium Aug 05 '20 at 14:42
  • Not exactly the same code in your answer. I try time.sleep(300) and the the result does not change. I think the page should be full loaded after 5 minutes. – mrd2242 Aug 05 '20 at 15:17
  • And I added some test to my original post. I guess the proxy is the point. – mrd2242 Aug 05 '20 at 15:30
1

I have opened an issue to selenium repo

My solution is to send a request which doesn't has a proxy. Here is my code:

from selenium import webdriver
import http.client

driver = webdriver.Chrome("D:/webdriver/chromedriver.exe")
driver.get("https://github.com")
# driver.quit()
# use below to quit and clean.
conn = http.client.HTTPConnection(driver.service.service_url.split("//")[1])
conn.request("GET", "/shutdown")
conn.close()
del driver

This works well. No exceptions and chromedriver.exe closed. But I think this is not the best solution.

mrd2242
  • 97
  • 1
  • 7
0

my OS is linux I resolve this issue add no_poroxy In my case add add no_proxy for localhost and intranet ips no_proxy="127.0.0.1, localhost, 10...* ...

Chanil Byun
  • 1
  • 1
    Please use proper formatting in your answer. You can read more about it here: https://stackoverflow.com/help/formatting. And welcome to Stack Overflow! – Agile_Eagle Dec 18 '20 at 04:44
0

Check to see if your os has been set to eufi and bios have been flashed to make you think that you are on your system but in all reality your proxy is run from cm and clone of your pc run in background transparent check bios version and see if they have been overwritten also check router switches and any possible server onnnetwork

Dustin taylor
  • 1