How to click on devtools console tab with Selenium and python

Question

I have this code:

async def acess_all():                         
 def acess_localhost():                               
  option = Options()                            
  option.add_argument("--no-sandbox")       
  option.debugger_Address="127.0.0.1:8081"    
  driver = webdriver.Chrome(options=option)     
  driver.get("http://127.0.0.1:8081")        
  wait = WebDriverWait(driver, 5)            
  wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'a.item'))).click()                 
 try:                                           
  t =  threading.Thread(target=get_infos)        
  t.start()                                     
  os.system("chromium --remote-debugging-port=8081 https://google.com")                    
 except:                                        
  print("Error!")

What I need:

Guys, as you can see, this code opens the Chrome browser on the Google page, and my Selenium code opens the browser on localhost because it's accessing where the remote-debbuging address points to, but I can't access the console tab of devtools, I need to access this tab using Selenium and run a javascript command to copy the cookie in json format, but I can't, how can I access the devtools console tab?

Answer 1

I believe I've figured out a way to run a javascript command in the console and get a return value (using selenium, as in the question).

Note: I'm running this on a Windows computer, but the idea should remain the same on a different operating system. You might have to tweak a few things in this code. Also, all Chrome sessions have to be closed beforehand to get this to work.

Ok, unless I interpreted your question wrong (tell me if I did), this is the basic order of the things you want you want to happen when:

1. Open the regular chrome browser (by regular, I mean not selenium, but regular chrome.exe or Google Chrome.app) to google.com and set the debugging port (which I assume is what you're doing when you run this command: chromium --remote-debugging-port=8081 https://google.com)
2. Open Chromedriver (Selenium) and go to the locally-hosted debugger window at 127.0.0.1:8081
3. Select the "Google" window option in the list of available windows at 127.0.0.1:8081
4. Once the devtools window opens, move to the Console tab
5. Finally, run some Javascript in the Console's input box and somehow get a return value for a cookie

You already did the first few items (1-3) in your code but you needed help figuring out the rest. I think I've found a way to do it.

So assuming that you already opened the google.com window on the Chrome browser and the 127.0.0.1:8081 window on localhost, all you need to do now is access the Console. Here's what my chromedriver (selenium) browser screen looks like at this point, just for reference.

We'll start by waiting until a specific element (devtools-elements-breadcrumbs) has loaded on the page. We wait for this so we are sure that the page is loaded enough to navigate. I found this element by looking at the driver page_source. Here's how we wait for it:

wait.until(EC.presence_of_element_located((By.TAG_NAME, "devtools-elements-breadcrumbs")))

Once the breadcrumb element is located we can move to the console window by sending a shortcut to the chromedriver browser telling it to move right one tab in the Inspector window (to Console). The command is Control + ] (Windows/Linux) or Command + ] (Mac), to go to the next panel. This is how we send that shortcut to the selenium window (once again, using Windows):

click_console = ActionChains(driver)
click_console.key_down(Keys.CONTROL).send_keys(']').key_up(Keys.CONTROL).perform()

or on a Mac,

click_console = ActionChains(driver)
click_console.key_down(Keys.COMMAND).send_keys(']').key_up(Keys.COMMAND).perform()

After moving to the Console, we now have to wait for the console area to load, so we wait for all the CodeMirror class elements (once again, found using driver.page_source)

# wait for console area to open
wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME, "CodeMirror")))

Now the last thing to do is to send the javascript command to the console. In this example, I'm searching for the APISID cookie.

cookie_name = "SEARCH_SAMESITE"
# javascript find the cookie
# https://stackoverflow.com/a/59603055/11073064
js_command = '(\"; \" + document.cookie).split(\"; {}\").pop().split(\';\').shift();'.format(cookie_name)
# send the command to the selenium webbrowser
send_js_command = ActionChains(driver)
send_js_command.send_keys(js_command, Keys.ENTER).perform()

and to get the value outputted in the console after you run that command:

# wait for return value span tag to be found
element = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'span.object-value-string.source-code')))
value = element.text
driver.close()

Here's the full code I used (on Windows).

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import threading
from selenium.webdriver.chrome.options import Options
from selenium.webdriver import ActionChains
from selenium.webdriver.common.keys import Keys
import os


def start_chrome_browser_with_debug():
    # this is the path to your regular google chrome browser, not selenium chromedriver
    # note the chrome.exe, not chromedriver.exe. 
    # point the path to your regular chrome browser
    os.system('"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --remote-debugging-port=8081 https://google.com')


cookie_name = "SEARCH_SAMESITE"
js_command = '(\"; \" + document.cookie).split(\"; {}\").pop().split(\';\').shift();'.format(cookie_name)

chrome_browser = threading.Thread(target=start_chrome_browser_with_debug)
chrome_browser.start()

option = Options()
option.add_argument("--no-sandbox")
option.debugger_Address = "127.0.0.1:8081"
driver = webdriver.Chrome(options=option)
driver.get("http://127.0.0.1:8081")
wait = WebDriverWait(driver, 5)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'a.item'))).click()
wait.until(EC.presence_of_element_located((By.TAG_NAME, "devtools-elements-breadcrumbs")))
click_console = ActionChains(driver)
click_console.key_down(Keys.CONTROL).send_keys(']').key_up(Keys.CONTROL).perform()

wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME, "CodeMirror")))
send_js_command = ActionChains(driver)
send_js_command.send_keys(js_command, Keys.ENTER).perform()
element = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'span.object-value-string.source-code')))
value = element.text
driver.close()
os.system('taskkill  /F /IM chrome.exe /T')
print(value)

Answer 2

Actually, you can execute the javascript command by selenium itself :

driver.execute_script("some javascript code here");

Answer 3

You can't use Selenium to interact with the dev tools console, however you don't need it to get the cookies, just use get_cookies() from the webdriver and convert it to json using json.dumps()

import json

cookies = driver.get_cookies()
js = json.dumps(cookies)

Answer 4

I couldn't make this work on a Mac using only Python and Selenium. However, I found a solution that works if I use pyautogui in addition.

I'm working on a file, duals7.svg, saved locally. I want to be able to open it, play with it and watch what happens at the Javascript developer console in Chrome.

The script below does exactly what I want:

# chrome.py

# https://www.browserstack.com/guide/python-selenium-to-run-web-automation-test
# https://www.tutorialspoint.com/how-do-i-pass-options-to-the-selenium-chrome-driver-using-python
# https://stackoverflow.com/questions/36311191/how-to-open-chrome-developer-console-in-selenium-webdriver-using-java
# https://towardsdatascience.com/controlling-the-web-with-python-6fceb22c5f08

from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions

import pyautogui 

op     = webdriver.ChromeOptions ()
op.add_argument ('--allow-file-access-from-files')
op.add_argument ('--auto-open-devtools-for-tabs')

driver = webdriver.Chrome (options=op)
driver.get ('file:///Users/murray/projects/warp/dual7.svg')

console = (850, 175)

pyautogui.moveTo (console [0], console [1], duration = 0.5)
pyautogui.click (console [0], console [1])

(I don't think the pyautogui.moveTo command is needed. Including it allows me to watch the mouse move, which I like.)

When I run it (python chrome.py), I see my svg "app" in Chrome with the Javascript dev console open:

The coordinates of the console button were determined by trial and error. Please forgive the aesthetics of the app: it'll look nice when it's done.

When I interact with my "app" I see my console.log () debugging messages.

If you're interested, the app is being modified from

/* elliptic-mesh.js
 *
 * generates a structured mesh bounded by four splines by
 * solving the elliptic Laplace equation
 *
 * For more info see:
 * http://www.particleincell.com/2012/online-meshing-example/
 * http://www.particleincell.com/2012/bezier-splines/ 
 *
 * Lubos Brieda, Particle In Cell Consulting LLC, 2012
 * you may freely use this algorithm in your codes but whenever possible
 * please include a link/reference to the source article
 */ 

My version generalizes Dr. Brieda's code a bit, adds some debugging statements, and will add save and load options.