0

Sorry but I'm quite new to Python. I have checked online and found that there are many videos with ways to create headless google selenium. However, upon trying those methods, I realized that they are all deprecated. Can anyone update me with the latest method to enable headless Selenium for google Chrome?

So the sequence of the program follows:

  1. Enters the login page website "https://info.bbdc.sg/members-login/"
  2. Then, it goes to submit insecure page "http://www.bbdc.sg/bbdc/bbdc_web/header2.asp" photo
  3. Lastly, it would navigate to the home page of the website which is "http://www.bbdc.sg/bbdc/b-mainframe.asp"

[Updated] I have tried adding the suggested answer to my code. This is my new code:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options

browser_options = Options()
browser_options.add_argument("--allow-running-insecure-content")
browser_options.headless = True
browser = webdriver.Chrome(executable_path=r'C:\Users\Jonathan\Projects\drivers\chromedriver_win32_V96\chromedriver.exe',options=browser_options)
browser.get("https://info.bbdc.sg/members-login/")

login_refresh = True
while login_refresh:
    try:
        if EC.url_matches('https://info.bbdc.sg/members-login/?err=Please+refresh+your+browser+and+fill+in+the+fields'
                          '+properly+to+login.+' or 'https://info.bbdc.sg/members-login/'):
            browser.refresh()
            login_id_elem = browser.find_element(By.ID, 'txtNRIC')
            login_pw_elem = browser.find_element(By.ID, 'txtPassword')
            access_btn = browser.find_element(By.ID, 'loginbtn')
            login_id_elem.send_keys("094H28022001")
            login_pw_elem.send_keys("258046")
            time.sleep(5)
            access_btn.click()
    except NoSuchElementException:
        login_refresh = False

print(browser.current_url)
WebDriverWait(browser, 10).until(EC.url_matches('http://www.bbdc.sg/bbdc/bbdc_web/header2.asp'))
send_anyway = browser.find_element(By.ID, 'proceed-button')
print(browser.current_url)
send_anyway.click()

# Home page
print(browser.current_url)
frame_element = browser.find_element(By.XPATH, '//frame[@name="leftFrame"]')
browser.switch_to.frame(frame_element)

However, I get this stacktrace error while running headless chrome. If i were to use the normal headful chrome, it works perfectly fine.

DevTools listening on ws://127.0.0.1:57679/devtools/browser/34af1be6-ebe7-4326-a4b5-072d940d9684
[1213/220648.128:INFO:CONSOLE(23)] "JQMIGRATE: Migrate is installed with logging active, version 1.4.1", source: https://info.bbdc.sg/
wp-content/plugins/enable-jquery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (23)
[1213/220648.168:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.fn.load() is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enable
-jquery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220648.168:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220648.315:INFO:CONSOLE(825)] "Uncaught TypeError: Cannot read properties of null (reading 'addEventListener')", source: https:/
/info.bbdc.sg/members-login/ (825)
[1213/220648.334:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.browser is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enable-j
query-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220648.334:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220648.354:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.attrFn is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enable-jq
uery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220648.355:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220648.370:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.fn.unload() is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enab
le-jquery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220648.370:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220649.013:INFO:CONSOLE(23)] "JQMIGRATE: Migrate is installed with logging active, version 1.4.1", source: https://info.bbdc.sg/
wp-content/plugins/enable-jquery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (23)
[1213/220649.030:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.fn.load() is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enable
-jquery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220649.030:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220649.040:INFO:CONSOLE(825)] "Uncaught TypeError: Cannot read properties of null (reading 'addEventListener')", source: https:/
/info.bbdc.sg/members-login/ (825)
[1213/220649.044:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.browser is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enable-j
query-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220649.044:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220649.062:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.attrFn is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enable-jq
uery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220649.063:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220649.086:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.fn.unload() is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enab
le-jquery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220649.087:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220657.962:INFO:CONSOLE(23)] "JQMIGRATE: Migrate is installed with logging active, version 1.4.1", source: https://info.bbdc.sg/
wp-content/plugins/enable-jquery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (23)
[1213/220657.977:INFO:CONSOLE(56)] "JQMIGRATE: jQuery.fn.load() is deprecated", source: https://info.bbdc.sg/wp-content/plugins/enable
-jquery-migrate-helper/js/jquery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (56)
[1213/220657.977:INFO:CONSOLE(58)] "console.trace", source: https://info.bbdc.sg/wp-content/plugins/enable-jquery-migrate-helper/js/jq
uery-migrate/jquery-migrate-1.4.1-wp.js?ver=1.4.1-wp (58)
[1213/220657.979:INFO:CONSOLE(385)] "Mixed Content: The page at 'https://info.bbdc.sg/members-login-redirect/' was loaded over a secur
e connection, but contains a form that targets an insecure endpoint 'http://www.bbdc.sg/bbdc/bbdc_web/header2.asp'. This endpoint shou
ld be made available over a secure connection.", source: https://info.bbdc.sg/members-login-redirect/ (385)
[1213/220657.993:INFO:CONSOLE(388)] "Mixed Content: The page at 'https://info.bbdc.sg/members-login-redirect/' was loaded over a secur
e connection, but contains a form that targets an insecure endpoint 'http://www.bbdc.sg/bbdc/bbdc_web/header2.asp'. This endpoint shou
ld be made available over a secure connection.", source: https://info.bbdc.sg/members-login-redirect/ (388)
http://www.bbdc.sg/bbdc/b-mainframe.asp
Traceback (most recent call last):
  File "C:\Users\Jonathan\PycharmProjects\BBDCBOT\BBDC_trial_headless.py", line 33, in <module>
    WebDriverWait(browser, 10).until(EC.url_matches('http://www.bbdc.sg/bbdc/bbdc_web/header2.asp'))
  File "C:\Users\Jonathan\venvs\automation\lib\site-packages\selenium\webdriver\support\wait.py", line 89, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Paper pad
  • 1
  • 5
  • Does this [answer](https://stackoverflow.com/questions/46920243/how-to-configure-chromedriver-to-initiate-chrome-browser-in-headless-mode-throug/49582462#49582462) helps you? – undetected Selenium Dec 12 '21 at 12:37

2 Answers2

1

For selenium 4 desired capabilities replaced with browser options.

from selenium.webdriver.chrome.options import Options
 
browser_options = Options()
browser_options.headless = True
driver = webdriver.Chrome(executable_path='G:\chromedriver\chromedriver.exe',options=browser_options)
Max Daroshchanka
  • 2,698
  • 2
  • 10
  • 14
  • I tried this, but it doesnt work. It have me a timeout exception for the code. Just wondering, does this code with webdriverwait and EC? Also, I have added the code and exception in the question above. Thanks! – Paper pad Dec 12 '21 at 14:26
  • In your logs there is redirection from https to http. Try to add `browser_options.add_argument("--allow-running-insecure-content")`. – Max Daroshchanka Dec 12 '21 at 15:05
  • hey @ Max Daroshchanka, I've added it but it still results in the same error – Paper pad Dec 12 '21 at 17:49
  • Right now I have no idea why it works in non-headless mode.. Also I can say that it's not related to the way, how you enable the headless mode. Headless mode is enabled in the right way. I may ask you to add some debug logs before webdriver wait.. Could you please log/print `browser.current_url` in order to understand what url was opened in the moment of error? – Max Daroshchanka Dec 12 '21 at 18:15
  • I have added the print(URL) into the code and updated it in the question above. This resulted in it printing "http://www.bbdc.sg/bbdc/b-mainframe.asp " for the first print command but realized that the second command doesn't bring anything. Also, just note that there is a picture of this "Information you're about to submit is not secure" page before i enter the main page which happens to be the 3rd url I will be assessing. I have added the photo in the question above. – Paper pad Dec 13 '21 at 14:27
  • Try to add all of this: `browser_options.set_capability('acceptInsecureCerts', true)`, `browser_options.set_capability('acceptSslCerts', true)` If not helps, also try to add: `browser_options.add_argument('--ignore-ssl-errors=yes')`, `browser_options.add_argument('--ignore-certificate-errors')`. – Max Daroshchanka Dec 13 '21 at 14:58
  • Hey I added it to my code - It still gives me the timeout exception. However, I checked it without the headless chrome and it works perfectly fine with all the addition of the codes – Paper pad Dec 13 '21 at 15:44
  • I found a lot of questions regarding ignoring sertificate errors in headless chrome. Looks like there is a bug in healess mode, so all the provided options ignored by driver. In suggesions I found that selenium 4 able to execute direct cdp commands. So, try `driver.execute_cdp_cmd("Security.setIgnoreCertificateErrors(true)")`. (Not sure it will work..) – Max Daroshchanka Dec 13 '21 at 16:20
  • @ Max Daroshchanka, could i know where i should insert it? – Paper pad Dec 16 '21 at 09:24
  • Sorry, not `driver` but `browser` in your case. Just add it one line above `browser.get(...` `browser.execute_cdp_cmd("Security.setIgnoreCertificateErrors(true)")` – Max Daroshchanka Dec 16 '21 at 09:46
  • Yup I did as follows but i got this traceback error: browser.execute_cdp_cmd("Security.setIgnoreCertificateErrors(True)") TypeError: ChromiumDriver.execute_cdp_cmd() missing 1 required positional argument: 'cmd_args' – Paper pad Dec 16 '21 at 15:08
  • Try `browser.execute_cdp_cmd("Security.setIgnoreCertificateErrors", {"ignore": true})`. – Max Daroshchanka Dec 16 '21 at 18:00
  • This allows the program to continue, however, it gives a TimeoutException again – Paper pad Dec 17 '21 at 09:46
  • Hey @Paperpad, I found, that in headless mode, chromium is used, not chrome. And I finally found the list of switches, supported by chromium: https://source.chromium.org/chromium/chromium/src/+/main:headless/app/headless_shell_switches.cc;bpv=1;bpt=1 Unfortunately, there are no switches for ignoring certificate errors or allowing running-insecure content. These switches are implemented only for chrome by the chrome team, not for headless chromium. So there is no way to workaround the mixin content error in headless mode until it will be implemented in chromium. – Max Daroshchanka Dec 17 '21 at 12:09
  • See also https://github.com/puppeteer/puppeteer/issues/3741#issuecomment-452912040 – Max Daroshchanka Dec 17 '21 at 12:09
  • @ Max Daroshchanka, Does that mean my program is unable to run in headless mode because of the insecure content? – Paper pad Dec 18 '21 at 08:52
  • Yes, you've loaded the page over `https`, but after `access_btn.click()` the page goes to send your data over `http`. That's why you see "Information you're about to submit is not secure" warning instead of the new page load. And unfortunately, headless chrome not able to ignore this warning and proceed. – Max Daroshchanka Dec 18 '21 at 11:06
  • Optionaly (if you able) you may ask someone from bbdc.sg to fix this security issue, seems it's not ok to submit login data over http. – Max Daroshchanka Dec 18 '21 at 11:15
0

I know two ways, and they works fine.

  1. Using add_argument headless

    options = webdriver.ChromeOptions()
options.add_argument("--headless")
driver = webdriver.Chrome(options = options)

  2. Using .headless = True

    options = webdriver.ChromeOptions()
options.headless = True
driver = webdriver.Chrome(options = options)
cruisepandey
  • 28,520
  • 6
  • 20
  • 38