How to scrape data from website using selenium

Question

I am try to scrape data but they will provide me an error this is website link https://www.dastelefonbuch.de/Suche/Zahnarzt

from selenium import webdriver
PATH="C:\Program Files (x86)\chromedriver.exe"
url='https://www.dastelefonbuch.de/Suche/Zahnarzt'
driver =webdriver.Chrome(PATH)
driver.get(url)
vid=driver.find_element_by_xpath("//div[@class='vcard']")
for item in vid:
    title=item.find_element_by_xpath("//div[@class='name']").text
    phone=item.find_element_by_xpath("//span[@class='nr']").text
    website=item.find_element_by_xpath("//i[@class='icon icon_url']").text
    print(title,phone,website)

undetected Selenium · Answer 1 · 2021-11-18T12:06:27.563

1

Instead of vid=driver.find_element_by_xpath("//div[@class='vcard']") which returns a WebElement you need vid=driver.find_element_by_xpath("//div[@class='vcard']") which would return a list to iterate as follows:

from selenium.webdriver.common.by import By

vid = driver.find_elements(By.XPATH, "//div[@class='vcard']")
for item in vid:
    title=item.find_element_by_xpath("//div[@class='name']").text
    phone=item.find_element_by_xpath("//span[@class='nr']").text
    website=item.find_element_by_xpath("//i[@class='icon icon_url']").text
    print(title,phone,website)

edited Nov 18 '21 at 12:06

answered Nov 18 '21 at 11:57

undetected Selenium

183,867
41
278
352

they give these error `NameError: name 'find_elements_by_xpath' is not defined` – Amen Aziz Nov 18 '21 at 12:01
Checkout the answer update and let me know the status – undetected Selenium Nov 18 '21 at 12:06
they give us wong output repeatedly print `one name` many time – Amen Aziz Nov 18 '21 at 12:15
and also not provide website link – Amen Aziz Nov 18 '21 at 12:17
@AmenAziz This answer was based on `find_element_by_xpath()` and `find_elements_by_xpath()` stuff, which I strongly believe have got you started in the right direction. _`repeatedly print one name`_ sounds like a different issue all together and you may like to raise a new question. – undetected Selenium Nov 18 '21 at 12:32

score 0 · Accepted Answer · answered Nov 18 '21 at 12:37

0

When you extract xpath on an item you have to change a bit xpath, saying lookup on current node with ".//"

Change

title=item.find_element_by_xpath("//div[@class='name']").text

to

title=item.find_element_by_xpath(".//div[@class='name']").text

Checkout and feedback if it works

answered Nov 18 '21 at 12:37

Wonka

1,548
1
13
20

it give me the correct but why put `dot` can you tell me – Amen Aziz Nov 18 '21 at 12:44
1

is cause // means global (all html), and .// means from the node (web element, your div). Also just one / means direct childrens. Here you can get more info about it and more complexity Xpaths https://en.wikipedia.org/wiki/XPath – Wonka Nov 18 '21 at 12:53

How to scrape data from website using selenium

2 Answers2