Can't find element by through any locator using Selenium in Python

Question

I am trying to the scrape contents of https://patents.google.com/patent/US4718386 using selenium in Python with PyCharm 2019.2. In particular, I need the the classification code + title (A23L3/358 - Inorganic compounds).

Google Patents changed this element recently, so that my previous code can't capture the contents anymore.

The HTML is now:

<div class="style-scope classification-tree">
              <concept-mention class="style-scope classification-tree">

    <span id="target" tabindex="0" aria-label="Details of concept" role="link" class="style-scope concept-mention">

        <iron-icon class="inline-icon style-scope concept-mention x-scope iron-icon-0" icon="icons:label"><svg viewBox="0 0 24 24" preserveAspectRatio="xMidYMid meet" focusable="false" class="style-scope iron-icon" style="pointer-events: none; display: block; width: 100%; height: 100%;"><g class="style-scope iron-icon"><path d="M17.63 5.84C17.27 5.33 16.67 5 16 5L5 5.01C3.9 5.01 3 5.9 3 7v10c0 1.1.9 1.99 2 1.99L16 19c.67 0 1.27-.33 1.63-.84L22 12l-4.37-6.16z" class="style-scope iron-icon"></path></g></svg>

  </iron-icon>
      <template is="dom-if" class="style-scope concept-mention"></template>

                <state-modifier class="code style-scope classification-tree" act="{&quot;type&quot;: &quot;QUERY_ADD_CPC&quot;, &quot;cpc&quot;: &quot;$cpc&quot;}" first="true" data-cpc="A23L3/358"><a id="link" href="/?q=APPLE&amp;q=A23L3%2f358" class="style-scope state-modifier">A23L3/358</a></state-modifier>
                <span class="description style-scope classification-tree">Inorganic compounds</span>

      <template is="dom-if" restamp="" class="style-scope concept-mention"></template>
    </span>
  </concept-mention>
            </div>

This was my previous code I used:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By

Class_Content_year = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='style-scope classification-tree' and not(@hidden)]/state-modifier[@class='code style-scope classification-tree']/a[@id='link' and @class='style-scope state-modifier']"))).get_attribute("innerHTML")

Class_Content_title = WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='style-scope classification-tree' and not (@hidden)]/span[@class='description style-scope classification-tree']"))).get_attribute("innerHTML")

I expected it to still find at least the title, but for some reason, it can't. Can someone please help?

Thank you!

Both. ```Class_Content_year``` should find ```A23L3/358```and ```Class_Content_title```the title ```Inorganic compounds``` — Jen G, Aug 27 '19 at 07:27

score 0 · Answer 1 · edited Aug 27 '19 at 09:59

0

Use this find_element_by class name :

driver.find_elements_by_class_name("style-scope classification-tree");

By XPATH you can get id and class as well but you have to put many things manually .

edited Aug 27 '19 at 09:59

frianH

7,295
6
20
45

answered Aug 27 '19 at 07:23

uzair iqbal

1
7

undetected Selenium · Answer 2 · 2019-08-27T07:41:37.743

To extract the text A23L3/358 and Inorganic compounds you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

To extract A23L3/358:

Using CSS_SELECTOR:

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.classification-tree span.concept-mention state-modifier>a.state-modifier"))).get_attribute("innerHTML"))

Using XPATH:

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='style-scope classification-tree']//span[@class='style-scope concept-mention']//state-modifier/a[@class='style-scope state-modifier']"))).get_attribute("innerHTML"))

To extract Inorganic compounds:

Using CSS_SELECTOR:

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.classification-tree span.concept-mention span.description.classification-tree"))).get_attribute("innerHTML"))

Using XPATH:

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='style-scope classification-tree']//span[@class='style-scope concept-mention']//span[@class='description style-scope classification-tree']"))).get_attribute("innerHTML"))

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Thank you for the answer! However, it still can't find the elements. I tried to add ```and not (@hidden)``` because there are other dynamic hidden elements there, but I guess it did not help. ```WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='style-scope classification-tree' and not (@hidden)]//....``` — Jen G, Aug 27 '19 at 07:55

Can't find element by through any locator using Selenium in Python

2 Answers2