Using selenium python webdriver to browse Fidelity stock screener throws error regarding 3rd party browser plugins

Question

I hope this is appropriate to ask, I've looked around for more than a month for this question.

I've been using python selenium to access Fidelity stock screener. I'm new to selenium and fairly new to Python too, I use this just to flip through the pages and download everything, since the site only lets you download 500 entries out of ~7000 entries at a time. All worked fine, but now it throws an error when I try to sign into the website.

The error mentions a technical issue due to 3rd party browser extensions and to try incognito mode. However, the error doesn't arise on my normal chrome browser, persists despite trying incognito mode and other options (below), and also only started in about March, despite working fine since last September. I can't tell if its some change with updated chrome/chomedriver (using latest ones), or the website somehow blocking me.

The basic code is below, this was cobbled together from various places but it did work:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common import by
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions as EC
import selenium.common.exceptions as exceptions
from selenium.webdriver.common.keys import Keys
import time

import os


#activate driver using executable path provided in quotes
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome('driver/chromedriver')

#go to stock screener page
driver.get('https://research2.fidelity.com/pi/stock-screener#results')


### below code requires setting username and password
username_fidelity="ABC"
password_fidelity="XYZ"


username_textbook_fidelity=driver.find_element_by_id("userId")
username_textbook_fidelity.send_keys(username_fidelity)

username_textbook_fidelity=driver.find_element_by_id("password")
username_textbook_fidelity.send_keys(password_fidelity)

login_button_fidelity=driver.find_element_by_id("Login")
login_button_fidelity.submit()

Once the username/password iis entered, it shows an error page regarding a technical error. This doesn't occer when I use my normal chrome browser. Error message says:

We are currently aware of an issue with some 3rd party browser plugins that are causing issues with access to the Fidelity website. Please try running your browser in incognito mode - or using an alternative browser as we work with the 3rd parties to resolve the issue. If you continue to receive this page please contact customer support and provide them the reference number below. Reference Error: 18.8cee2117.1653688699.1eaaf206

I'm not sure how to provide a reproducible example, since it involves Fidelity username/password but I think if you try above code with your login you'll see what I mean.

Stuff I tried (I have a limited understanding of what these mean, but I read they may be relevant, but nothing worked):

I tried setting user agent, but it seems to be identical to my normal chrome browser. (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36)

I tried a bunch of other things too including removing all selenium chromedriver options so they match the chrome command line stuff (as per url chrome://version/), but this did not work. Selenium browser has all of these command line arguments, normal chrome browser doesn't. Code below removes all of the "command line" arguments in (chrome://version/), except for "remote-debugging-port=0", not sure why. But it doesn't work.

chrome_options = webdriver.ChromeOptions()
exclude_these = ['allow-pre-commit-input',
                'disable-background-networking',
                "disable-client-side-phishing-detection",
                "disable-default-apps",
                "disable-hang-monitor",
                "disable-popup-blocking",
                "disable-prompt-on-repost",
                "disable-sync",
                "enable-automation",
                "enable-blink-features",
                "enable-logging",
                "log-level",
                "no-first-run",
                "no-service-autorun",
                "password-store",
                "remote-debugging-port",
                "test-type",
                "use-mock-keychain"]

chrome_options.add_experimental_option("excludeSwitches", exclude_these)
driver = webdriver.Chrome('driver/chromedriver', 
                          options=chrome_options)

I tried disabling extensions, automation, incgnito mode (as it mentions), none of these work either:

chrome_options.add_argument("--disable-extensions")
chrome_options.add_experimental_option('useAutomationExtension', False)
chrome_options.add_argument("--incognito")

I tried using mozilla firefox with geckodriver, same error page.

If there's anything anyoone can suggest it would be so great!! Thanks.

Answer 1

you want to use undetected_chromedriver to get past that.

https://github.com/ultrafunkamsterdam/undetected-chromedriver

just pip3 install undetected-chromedriver

and do what you're doing (some ways of specifying options/preferences to the driver may need to be changed, check out the help at github above)

also, don't go to the stock screener url to force a login

browse to www.fidelity.com (get) first, to get _abck cookie (akamai) then login by browsing to https://digital.fidelity.com/prgw/digital/login/full-page

and doing your username/pass/login thing

Answer 2

Apparently Fidelity detects when the browser was launched by webdriver and rejects login. Saving cookie does not do a thing because webdriver does it automatically anyway.
The solution is to launch the browser from command line with debug port and make webdriver to connect to it. Then use the login logic as before. Tested on Ubuntu 16.04, Chrome v. 111.0.5563.64, e.g.

command line:

chrome --remote-debugging-port=9222 --user-data-dir="~/chrome-data"

Java program:

ChromeOptions options = new ChromeOptions();
options.addArguments("−−incognito");
options.setExperimentalOption("debuggerAddress", "127.0.0.1:9222");
System.setProperty("webdriver.chrome.driver", driverPath);
webDriver = new ChromeDriver(options);

Answer 3

This answer Way to change Google Chrome user agent in Selenium? actually is what worked for me. You need to set these variables to make sure the server can't know if its a driver or not.