Finding a string in an URL's source code with selenium webdriver

Question

I'm trying to extract a keyword/string from a website's source code using this python 2.7 script:

from selenium import webdriver

keyword = ['googleadservices']

driver = webdriver.Chrome(executable_path=r'C:\Users\Jacob\PycharmProjects\Testing\chromedriver_win32\chromedriver.exe')
driver.get('https://www.vacatures.nl/')

elem = driver.find_element_by_xpath("//*")
source_code = elem.get_attribute("outerHTML")

for searchstring in keyword:
    if searchstring.lower() in str(source_code).lower():
        print (searchstring, 'found')
    else:
        print (searchstring, 'not found')

The browser fortunately opens when the script is running, but I'm not able to extract the desired keywords from it's source code. Any help?

http://stackoverflow.com/questions/7861775/python-selenium-accessing-html-source This might help you. — Kishan Patel, Feb 28 '17 at 07:00
Are you sure `googleadservices` is present in the source code of `https://www.vacatures.nl/`? I checked, it is NOT present — Naveen Kumar R B, Feb 28 '17 at 07:18

Naveen Kumar R B · Answer 1 · 2017-02-28T10:17:55.810

I observed that googleadservices is NOT present in the web page source code.

There is NO issue with the code.

I tried with GoogleAnalyticsObject, and it is found.

from selenium import webdriver

keyword = ['googleadservices', 'GoogleAnalyticsObject']

driver = webdriver.Chrome()
driver.get('https://www.vacatures.nl/')

elem = driver.find_element_by_xpath("//*")
source_code = elem.get_attribute("outerHTML")

for searchstring in keyword:
    if searchstring.lower() in str(source_code).lower():
        print (searchstring, 'found')
    else:
        print (searchstring, 'not found')

Instead of using //* to find the source code

elem = driver.find_element_by_xpath("//*")
source_code = elem.get_attribute("outerHTML")

Use the following code:

source_code = driver.page_source

score 0 · Answer 2 · answered Feb 28 '17 at 09:17

As others have said, the issue isn't your code but simply that googleadservice isn't present in the source code.

What I want to add, is that your code is a bit over engineered, since all you seem to do is either return true or false if a certain string is present in the source code.

You can achieve that much easier with a better xpath like //script[contains(text(),'googletagmanager')] and than use find_element_by_xpath and catch the possible NoSuchElementException. That might save you time and you don't need the for loop.
There are other possiblities as well, using ExpectedConditions or find_elements_by_xpath and then check if the returned list is greater than 0.

Finding a string in an URL's source code with selenium webdriver

2 Answers2