Returning javascript data from Python Selenium Web Driver

Question

I'm trying to run a javascript file inside a Python file using Selenium WebDriver and get the return value from the javascript function. The following attempts below should work from what I've looked up online. I've worked on this for a few hours to no avail. Any help would be appreciated.

Python Code:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
import os


driver = webdriver.Chrome(executable_path='chromedriver')
driver.get('example.com')

# Clean up the text inside the <p> tags
js_file = 'clean_text.js'
with open(js_file, 'r') as f:
    script = f.read()

# Attempt 1
text = driver.execute_script(script)

# Attempt 2
text = driver.execute_script(f'return {script}')

# Attempt 3
text = WebDriverWait(driver, 20).until(lambda driver: driver.execute_script(script))

# Attempt 4
text = WebDriverWait(driver, 20).until(lambda driver: driver.execute_script(f'return {script}'))

The attempts either return None or a selenium.common.exceptions.TimeoutException

Javascript Code:

$(document).ready(function() {
  // Collect the text within the nested <p> tags
  var text = [];

  // get all nested <p> tags
  let paragraphs = $(`div#id div[data-test-id=value] p`);

  // Replace all whitespaces with a single white space
  paragraphs.each(function(index) {
    let original_text = $(this).text();
    let cleaned_text = original_text.trim().replace(/\s\s+/g, ' ');
    $(this).text(cleaned_text);
    text.push(cleaned_text);
  });

  return text;
});

Given furas' answer below this JS code worked for Attempt 1 & 4. I needed to add "return" at the start, "();" at the end & remove the jQuery.

return function cleanText() {
  // Collect the text within the nested <p> tags
  var text = [];

  // get all nested <p> tags
  let paragraphs = $(`div#id div[data-test-id=value] p`);

  // Replace all whitespaces with a single white space
  paragraphs.each(function(index) {
    let original_text = $(this).text();
    let cleaned_text = original_text.trim().replace(/\s\s+/g, ' ');
    $(this).text(cleaned_text);
    text.push(cleaned_text);
  });

  return text;
}();

did you load `jQuery` in your code? You can't use `$(...)` without `jQuery`. — furas, May 11 '21 at 08:48
first you could use `console.log(...)` in javaScript to display information in JavaScript's console in DevTools in web browser. and this way you could check (in JavaScript) what you have in variables and which part of code is executed. — furas, May 11 '21 at 08:50
jQuery is loaded into the script already. I've added an alert(text); right above return text; I went back and added a conosle.log(...); Both of these worked. Selenium is still receiving None. — Osuynonma, May 11 '21 at 09:21
I think you use `jQuery` in wrong way - `function()` is executed inside `ready()` so `return` sends `text` to `ready()`, not to you. You should run directly `function()` to get its result. — furas, May 11 '21 at 09:35

furas · Accepted Answer · 2021-05-11T10:07:43.670

All problem can be because you run function() inside ready() and it return sends text to ready(), not to you. You have to directly run function to get its result.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
import os
import time

driver = webdriver.Chrome()# executable_path='chromedriver')
driver.get('http://quotes.toscrape.com/js/')

#script = '$(document).ready(function() { return "Hello World"; })';
script = 'function test() { return "Hello World"; })()';

time.sleep(3)

# Attempt 1
text = driver.execute_script(script)
print(text)  # None

# Attempt 2
text = driver.execute_script(f'return {script}')
print(text)  # Hello World

# Attempt 3
#text = WebDriverWait(driver, 20).until(lambda driver: driver.execute_script(script))
#print(text)  # error

# Attempt 4
text = WebDriverWait(driver, 20).until(lambda driver: driver.execute_script(f'return {script}'))
print(text)  # Hello World

EDIT:

if you really need to check if document is ready then you should do it inside your function. You could try to use while-loop with variable $.isReady

In JQuery, how do I check if the DOM is ready?

Returning javascript data from Python Selenium Web Driver

1 Answers1