Selenium not extracting info using xpath

Question

I am trying to extract some information from the amazon website using selenium. But I am not able to scrape that information using xpath in selenium.

In the image below I want to extract the info highlighted.

This is the code I am using

try:
    path = "//div[@id='desktop_buybox']//div[@class='a-box-inner']//span[@class='a-size-small')]"
    seller_element = WebDriverWait(driver, 5).until(
        EC.visibility_of_element_located((By.XPATH, path)))
except Exception as e:
    print(e)

When I run this code, it shows that there is an error with seller_element = WebDriverWait(driver, 5).until( EC.visibility_of_element_located((By.XPATH, path))) but does not say what exception it is.

I tried looking online and found that this happens when selenium is not able to find the element in the webpage.

But I think the path I have specified is right. Please help me.

Thanks in advance

[EDIT-1]

This is the exception I am getting

Message:

@arundeepchohan selenium is not able to access the element I want to extract. I have posted the code in the question. Please take a look. — Sashaank, Sep 08 '20 at 05:17
//span[@class='a-color-secondary') normally we grab the class name on the farthest right which identifies it. — Arundeep Chohan, Sep 08 '20 at 05:22
You could actually just use the entire name for the class if you don't know how to shorten it btw. — Arundeep Chohan, Sep 08 '20 at 05:27
@arundeepchohan Can you post an answer on how to fix this please? — Sashaank, Sep 08 '20 at 05:28
Why don't you get the element by CSS selectors instead, rather than xpath? Does that work, if you try that? — CssProblem, Sep 08 '20 at 04:56
I tried that too. There are multiple elements with the same classes. So selenium is not getting the right info I want — Sashaank, Sep 08 '20 at 04:59
There will always be a way to identify a particular element - admittedly, many big sites try to design their code in ways to make that slightly more difficult (many using computer-generated class names etc), but there should always be something you can reference...the nth-child and nth-of-type selectors are also good. For this particular example, this works: `#snsAccordionRowMiddle .a-spacing-top-base span` — CssProblem, Sep 08 '20 at 07:54

Arundeep Chohan · Accepted Answer · 2020-09-08T05:49:28.333

2

//div[class='a-section a-spacing-none a-spacing-top-base']//span[class='a-size-small a-color-secondary']

XPath could be something like this. You can shorten this.

CSS selector could be and so forth.

.a-section.a-spacing-none.a-spacing-top-base
.a-size-small.a-color-secondary

edited Sep 08 '20 at 05:49

answered Sep 08 '20 at 05:37

Arundeep Chohan

9,779
5
15
32

This gives the same error. Could this have something to do with the way the website is designed? – Sashaank Sep 08 '20 at 05:43

score 1 · Answer 2 · answered Sep 08 '20 at 08:26

I think the reason is xpath expression is not correct.

Take the following element as an example, it means the span has two class:

<span class="a-size-small a-color-secondary">

So, span[@class='a-size-small') will not work.

Instead of this, you can ues xpath as

//span[contains(@class, 'a-size-small') and contains(@class, 'a-color-secondary')]

or cssSelector as

span.a-size-small.a-color-secondary

score 0 · Answer 3 · answered Sep 08 '20 at 06:13

Amazon is updating its content on the basis of the country you are living in, as I have clicked on the link provided by you, there I did not find the element you are looking for simply because the item is not sold here in India.

So in short if you are sitting in India and try to find your element, it is not there, but as you change the location to "United States". it is appearing there.

Solution - Change the location

score -1 · Answer 4 · answered Sep 08 '20 at 09:37

To print the Ships from and sold by Amazon.com of an element you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR and get_attribute():

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.a-section.a-spacing-none.a-spacing-top-base > span.a-size-small.a-color-secondary"))).get_attribute("innerHTML"))

Using XPATH and text attribute:

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='a-section a-spacing-none a-spacing-top-base']/span[@class='a-size-small a-color-secondary']"))).text)

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python

Outro

Link to useful documentation:

get_attribute() method Gets the given attribute or property of the element.
text attribute returns The text of the element.
Difference between text and innerHTML using Selenium

Selenium not extracting info using xpath

[EDIT-1]

4 Answers4

Outro