1

I'm trying to use AWS to automate some procedures, and need to authenticate myself with the API I'm using. The issue is, its way too slow. I'm using a selenium headless chrome browser to connect to the page, fill in the information I need and submit the form (according to other sources I found the TD Ameritrade API only accepts the browser route so there is no way to do it just with requests, if you know otherwise that would be great too). Finding the element on the page takes about 20 seconds, sending the keys maybe a full minute, and trying to submit the form it times out after 5 minutes. Is this an issue with my implementation of selenium, or is it something to do with AWS or the API I'm connecting to? My code is as shown below and the error:

        oauth = OAuth2Session(client_code, redirect_uri=redirect)
        authorization_url, state = oauth.authorization_url('https://auth.tdameritrade.com/auth')
        
        chrome_options = Options()
        chrome_options.add_argument('--headless')
        chrome_options.add_argument('--no-sandbox')
        #chrome_options.add_argument('--disable-gpu')
        chrome_options.add_argument('--window-size=1280x1696')
        chrome_options.add_argument('--user-data-dir=tmp/user-data')
        chrome_options.add_argument('--hide-scrollbars')
        chrome_options.add_argument('--enable-logging')
        chrome_options.add_argument('--log-level=0')
        chrome_options.add_argument('--v=99')
        chrome_options.add_argument('--single-process')
        chrome_options.add_argument('--data-path=tmp/data-path')
        chrome_options.add_argument('--ignore-certificate-errors')
        chrome_options.add_argument('--homedir=tmp')
        chrome_options.add_argument('--disk-cache-dir=tmp/cache-dir')
        chrome_options.add_argument('--no-proxy-server')
        chrome_options.add_argument("--proxy-server='direct://'");
        chrome_options.add_argument("--proxy-bypass-list=*");
        chrome_options.add_argument('user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36')
        chrome_options.binary_location = "/usr/bin/chromium-browser"
        
        with webdriver.Chrome(options=chrome_options) as driver:
            driver.get(authorization_url)
            
            usernamebox = driver.find_element_by_id("username0")
            print("found")
            usernamebox.send_keys(username)
            print("sent")
            passbox = driver.find_element_by_id("password")
            print("found2")
            passbox.send_keys(password)
            print("sent2")
            passbox.submit()
            print("submitted")

Error:

Traceback (most recent call last):
  File "/home/ubuntu/environment/td_ameritrade_test.py", line 229, in <module>
    td = TDClient()
  File "/home/ubuntu/environment/td_ameritrade_test.py", line 172, in __init__
    self.oauth_client = self.authenticate_user()
  File "/home/ubuntu/environment/td_ameritrade_test.py", line 90, in authenticate_user
    passbox.submit()
  File "/home/linuxbrew/.linuxbrew/Cellar/python@3.8/3.8.6/lib/python3.8/site-packages/selenium/webdriver/remote/webelement.py", line 86, in submit
    self._parent.execute_script(
  File "/home/linuxbrew/.linuxbrew/Cellar/python@3.8/3.8.6/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 634, in execute_script
    return self.execute(command, {
  File "/home/linuxbrew/.linuxbrew/Cellar/python@3.8/3.8.6/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/home/linuxbrew/.linuxbrew/Cellar/python@3.8/3.8.6/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 300.000
  (Session info: headless chrome=86.0.4240.75)

Any help would be appreciated greatly!

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
D. Name
  • 23
  • 3
  • Do you really need those many `add_argument`? – undetected Selenium Nov 11 '20 at 21:23
  • These were the options I found on the only guides I could find on using selenium in AWS, those guides all described web scrapers though and your right a few of these arguments could be useless. Do you think this could be slowing it down? – D. Name Nov 11 '20 at 21:25
  • Of coarse setting too many arguments have a negative effect on the test execution. Use the arguments as per the _Test Specs_. – undetected Selenium Nov 11 '20 at 21:27
  • Ok, I don't know how Stack Overflow works but that worked wonders, I only used the headless argument and it worked brilliantly. Thank you very much. – D. Name Nov 12 '20 at 01:17

1 Answers1

1

A bit of more details about your usecase would have helped us to analyze the reason behind the program executing too slow. Here are a few considerations:

  • A common cause for Chrome to crash during startup is running Chrome as root user (administrator) on Linux. While it is possible to work around this issue by passing --no-sandbox flag when creating your WebDriver session, such a configuration is unsupported and highly discouraged. You need to configure your environment to run Chrome as a regular user instead. So you may need to drop the --no-sandbox option.

Here is the link to the Sandbox story.

  • While using --headless option you won't be able to use --window-size=1280x1696 due to certain constraints.

You can find a couple of relevant detailed discussion in:

You can find a relevant detailed discussion in ERROR:gpu_process_transport_factory.cc(1007)-Lost UI shared context : while initializing Chrome browser through ChromeDriver in Headless mode

  • Further you haven't mentioned any specific requirement of using --hide-scrollbars, --enable-logging, --log-level=0, --v=99, --single-process, --data-path=tmp/data-path, --homedir=tmp, --disk-cache-dir=tmp/cache-dir, --no-proxy-server, --proxy-server='direct://' and --proxy-bypass-list=* arguments which you opt to drop for the time being and add them back as per your Test Specification.

References

You can find a couple of relevant detailed discussions in:

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352