I have flask that is essentially using selenium to scrap a website. This app is deployed on Heroku. On my local computer, I am facing no errors although, on Heroku I am not getting this error initially, but when I reload the page, a server error page is shown and in the logs I find this.
urllib3.exceptions.MaxRetryError
This error is caused by a
NewConnectionError
Which occurs, while handling another error
ConnectionRefusedError: [Errno 111] Connection refused
The following is the code
from flask import Flask
from flask import request
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chrome_options = webdriver.ChromeOptions()
chrome_options.binary_location = os.environ.get("GOOGLE_CHROME_BIN")
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--no-sandbox")
driver = webdriver.Chrome(executable_path=os.environ.get("CHROMEDRIVER_PATH"), options=chrome_options)
app = Flask(__name__)
@app.route('/', methods=['GET', 'POST'])
def handleRequests():
url = str(request.args.get('url'))
driver.get(url)
try:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH , xpath))
)
finally:
data = element.text
driver.quit()
return str(data)
if __name__ == "__main__":
app.run()
Note CHROMEDRIVER_PATH and GOOGLE_CHROME_BIN are config vars
