4

I am building a scraper code and already have been able to read the table and the information that I want. The problem is with the next page link, I have tried using a class name and also and svg tag but the code breaks as the value of the class name changes.

Here is the link of the page

Page to scrape

And the code for the element css that the code runs to click for next page is this

driver.find_element_by_css_selector('#root > div > div > main > div.ez6st4XksKUGZfGdvIhjV > section > div:nth-child(1) > div._1c5cPqlj4FoguvpKXSY69p > div > span:nth-child(3) > svg').click()

It seems that when the value of the class name changes it breaks and changes the element to click and I havent found a way to repeat without changing the element in order to repeat for multiple pages with the same structure.

Thanks

Andersson
  • 51,635
  • 17
  • 77
  • 129
sputnikk1093
  • 43
  • 2

2 Answers2

2

As you can click the span you can also use

from selenium import webdriver
d  = webdriver.Chrome()
url = 'https://super.walmart.com.mx/despensa/enlatados-y-conservas/chiles-enlatados/_/N-10kldy7?%2Fdespensa%2Fenlatados-y-conservas%2Fchiles-enlatados%2F_%2FN-10kldy7%3F%2Fdespensa%2Fenlatados-y-conservas%2Fchiles-enlatados%2F_%2FN-10kldy7%3F%2Fdespensa%2Fenlatados-y-conservas%2Fchiles-enlatados%2F_%2FN-10kldy7%3FNs=product.displayText%7C0&offSet=0&storeId=0000009999&No=40'
d.get(url)
# example number of clicks below
for i in range(2):
    d.find_element_by_xpath("//*[starts-with(@d,'M0')]/parent::*/parent::span").click()
QHarr
  • 83,427
  • 12
  • 54
  • 101
0

You can use below line to click Next button without referring to dynamic class names:

driver.find_element_by_xpath('//span[@value]/following-sibling::span/*[name()="svg"]').click()

The same with CSS-selector:

driver.find_element_by_css_selector('span[value] + span > svg')

Update

from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait as wait

while True:
    try:
        wait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'span[value] + span > svg'))).click()
    except:
        break
Andersson
  • 51,635
  • 17
  • 77
  • 129
  • I'm still unable to figure out how this `name()` works @sir Andersson. Does it only work with tags? – SIM Dec 07 '18 at 08:50
  • @SIM `svg` (and all it's possible descendants) is not from [HTML namespace](https://www.w3.org/1999/xhtml/), but from [SVG namespace](https://www.w3.org/2000/svg), so you cannot use just `//svg` to locate it. You should use `//*[name()="svg"]`. Also it works for attributes. For instance, `//@*[name()="id"]`. Note that in CSS-selector you can locate `svg` as common element... – Andersson Dec 07 '18 at 09:00
  • The way you create xpath is enviable sir. However, the thing is I didn't find any link where I can learn about the usage of `name()` function. Thanks. – SIM Dec 07 '18 at 09:05
  • 1
    @SIM yeah, the [documentation](https://developer.mozilla.org/en-US/docs/Web/XPath/Functions/name) is not informative at all :) You can just remember that if you work with the different namespaces, `name` function is what you need. Note that there is `local-name` function also which works [almost in the same way](https://stackoverflow.com/questions/2462248/what-is-the-difference-between-name-and-local-name) – Andersson Dec 07 '18 at 09:14
  • Thank you Andersson, acutally the code breaks at half ot the process in page 4 rather than keep going to page 6 – sputnikk1093 Dec 09 '18 at 17:09
  • @sputnikk1093 , welcome! You can [accept](https://stackoverflow.com/help/accepted-answer) answer if it solved your problem/[upvote](https://stackoverflow.com/help/privileges/vote-up) if it was useful – Andersson Dec 11 '18 at 06:27