ORIGINAL POST 06-22-2021 @12:00 UTC
The error:
HTTP protocol error in client request: Server disconnected
Is being thrown by mitmproxy. I pulled the following from mitmproxy source code.
"clientconnect": "client_connected",
"clientdisconnect": "client_disconnected",
"serverconnect": "server_connect and server_connected",
"serverdisconnect": "server_disconnected",
class ServerDisconnectedHook(commands.StartHook):
"""
A server connection has been closed (either by us or the server).
"""
blocking = False
data: ServerConnectionHookData
I would recommend putting your code in a Try Except block, which will allow you to suppress the errors thrown by mitmproxy.
from mitmproxy.exceptions import MitmproxyException
from mitmproxy.exceptions import HttpReadDisconnect
try:
your driver code
except HttpReadDisconnect as e:
pass
except MitmproxyException as e:
"""
Base class for all exceptions thrown by mitmproxy.
"""
pass
finally:
driver.quit()
If the exceptions that I provided don't suppress your error than I would recommend trying some of the other Exceptions in mitmproxy.
UPDATE 06-22-2021 @15:28 UTC
In my research I noted that seleniumwire has integration code with mitmproxy. Part of this integration is capturing error message thrown by *mitmproxy."
class SendToLogger:
def log(self, entry):
"""Send a mitmproxy log message through our own logger."""
getattr(logger, entry.level.replace('warn', 'warning'), logger.info)(entry.msg)
In my testing suppressing the error in question using mitmproxy.exceptions is difficult. In testing the following exceptions, the only one that fired was HttpReadDisconnect. And that firing wasn't consistent.
- HttpException
- HttpReadDisconnect
- HttpProtocolException
- Http2ProtocolException
- MitmproxyException
- ServerException
- TlsException
I noted that if I added a standard Exception:
except Exception as error:
print('standard')
print(''.join(traceback.format_tb(error.__traceback__)))
That this line in your code consistently throws errors:
File "/Users/user_name/Python_Projects/scratch_pad/seleniumwire_test.py", line 18, in <module>
assert driver.requests[0].response.status_code == 200
When I looked at this error in more detail I found that it was related to the status_code.
<class 'AttributeError'>
'NoneType' object has no attribute 'status_code'
UPDATE 06-23-2021 @15:04 UTC
During my research I found that selenium had a service_log_path parameter that could be added to webdriver.Chrome().
class WebDriver(ChromiumDriver):
def __init__(self, executable_path="chromedriver", port=DEFAULT_PORT,
options: Options = None, service_args=None,
desired_capabilities=None, service_log_path=DEFAULT_SERVICE_LOG_PATH,
chrome_options=None, service: Service = None, keep_alive=DEFAULT_KEEP_ALIVE):
According to the documentation this parameter could be used this way: service_log_path=/dev/null
Unfortunately, the comments in class WebDriver(ChromiumDriver) indicated that this parameter is deprecated. It also failed to suppress the sys.stdout error messages.
service_log_path - Deprecated: Where to log information from the driver.
CURRENT STATUS
I reworked your code and removed the status_code lines that were throwing errors. I added some implicitly_wait() and some WebDriverWait statements to handle what you were trying to do with the status_code statement. I also added some error handling to catch specific error message types. And I added some chrome_options to suppress certain things, such a loading website images, which are not needed for scraping the target website.
Finally, I added a custom logging feature to suppress the error messages that were being sent to the sys.stdout. I tested the code many times and so far I haven't received and error message to sys.stdout. More testing might be needed it you get the messages again.
Here is a link of the code in action.
import sys
import logging
import traceback
from seleniumwire import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from mitmproxy.exceptions import HttpReadDisconnect, TcpDisconnect, TlsException
class DisableLogger():
def __enter__(self):
logging.disable(logging.WARNING)
def __exit__(self, exit_type, exit_value, exit_traceback):
logging.disable(logging.NOTSET)
options = {
"backend": "mitmproxy",
'mitm_http2': False,
'disable_capture': True,
'verify_ssl': True,
'connection_keep_alive': False,
'max_threads': 3,
'connection_timeout': None,
'proxy': {
'https': 'https://209.40.237.43:8080',
}
}
chrome_options = Options()
chrome_options.add_argument(
"Mozilla/5.0 (Macintosh; Intel Mac OS X 11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Safari/537.36")
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-logging')
chrome_options.add_argument("--disable-application-cache")
chrome_options.add_argument("--ignore-certificate-errors")
chrome_options.add_argument("--disable-blink-features=AutomationControlled")
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_experimental_option('useAutomationExtension', False)
webdriver.DesiredCapabilities.CHROME['acceptSslCerts'] = True
prefs = {
"profile.managed_default_content_settings.images": 2,
"profile.default_content_settings.images": 2
}
capabilities = webdriver.DesiredCapabilities.CHROME
chrome_options.add_experimental_option("prefs", prefs)
capabilities.update(chrome_options.to_capabilities())
driver = webdriver.Chrome(executable_path='/usr/local/bin/chromedriver',
options=chrome_options, seleniumwire_options=options)
with DisableLogger():
driver.implicitly_wait(60)
try:
driver.get('https://www.zillow.com/Houston,-TX/houses/')
wait = WebDriverWait(driver, 240)
page_title = wait.until(EC.presence_of_element_located((By.XPATH, '//*[@id="search-page-react-content"]')))
if page_title:
post_links = [i.get_attribute("href") for i in driver.find_elements_by_css_selector("article[role='presentation'] > .list-card-info > a.list-card-link")]
for individual_link in post_links:
driver.implicitly_wait(60)
driver.get(individual_link)
post_title = driver.find_element_by_css_selector("h1").text
print(post_title)
except HttpReadDisconnect as error:
print('A HttpReadDisconnect Exception has occurred')
exc_type, exc_value, exc_tb = sys.exc_info()
print(exc_type)
print(exc_value)
print(''.join(traceback.format_tb(error.__traceback__)))
driver.quit()
except TimeoutException as error:
print('A TimeOut Exception has occurred')
exc_type, exc_value, exc_tb = sys.exc_info()
print(exc_type)
print(exc_value)
print(''.join(traceback.format_tb(error.__traceback__)))
driver.quit()
except TcpDisconnect as error:
print('A TCP Disconnect Exception has occurred')
exc_type, exc_value, exc_tb = sys.exc_info()
print(exc_type)
print(exc_value)
print(''.join(traceback.format_tb(error.__traceback__)))
driver.quit()
except TlsException as error:
print('A TLS Exception has occurred')
exc_type, exc_value, exc_tb = sys.exc_info()
print(exc_type)
print(exc_value)
print(''.join(traceback.format_tb(error.__traceback__)))
driver.quit()
except Exception as error:
print('An exception has occurred')
print(''.join(traceback.format_tb(error.__traceback__)))
pass
finally:
driver.quit()
OBSERVATIONS
I noted that you are using free proxies instead of paid proxy service. The proxy in your code hxxps://136.226.33.115:80 I found was a standard HTTP proxy and it also was having latency issues, which was causing timeouts when connecting to your target website.
Another observation is that your target website has captcha, which are fired when you send too many connection requests.
I also noted that your proxy server would also have connection issues, which would cause error messages to be sent to sys.stdout. This is what you were likely encountering.
SIDE NOTE
The selenium session in your code occasionally encounters an I am human captcha from Zillow.
----------------------------------------
My system information
----------------------------------------
Platform: Mac
Python Version: 3.9
Seleniumwire: 4.3.1
Selenium: 3.141.0
mitmproxy: 6.0.2
browserVersion: 91.0.4472.114
chromedriverVersion: 90.0.4430.24
IDE: PyCharm 2021.1.2
----------------------------------------
