1

I'm trying to scrape some data using selenium. I can get the wanted page to load and open the subpage, but i cannot get the selenium driver to find the xpath from the submenu.

I have tried clicking the subpage with the selenium driver, but again i have a problem getting selenium to use the xpath.

path = r"C:\Program Files\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(path)

#opening website
driver.get("http://elpris.dk")
time.sleep(2)

#setting zipcode to 2200 Nørrebro
driver.find_element_by_xpath("""//*[@id="zip"]""").click()
driver.find_element_by_xpath("""//*[@id="zip"]""").send_keys("2200")

#selecting the dropdown menu
driver.find_element_by_xpath("""//*[@id="btnSelectProfile"]""").click()

#selecting hus 4000 kwh   
driver.find_element_by_xpath("""//*[@id="home"]/div[1]/div/div/form/div/ul/li[4]/span/a""").click()

#clicking on "Find Priser"
driver.find_element_by_xpath("""//*[@id="btnSubmitSearch"]""").click()
time.sleep(2)

#scrolling down to get all the table rows to make sure the full product page is loaded
i = 0
while i < 50:
    driver.find_element_by_xpath("""/html/body""").send_keys(Keys.END)
    i += 1

time.sleep(2)

#counting number of rows
rows = len(driver.find_elements_by_xpath("""//*[@id="results"]/div[2]/div[2]/div/div[2]/div/div/table/tbody/tr"""))

#counting number of col
cols = len(driver.find_elements_by_xpath("""//*[@id="results"]/div[2]/div[2]/div/div[2]/div/div/table/tbody/tr[1]/td"""))

print (rows)
print (cols)

#scroll up
driver.find_element_by_xpath("""/html/body""").send_keys(Keys.HOME)
time.sleep(2)


#entering each indivudual row and getting info
driver.find_element_by_xpath("""//*[@id="results"]/div[2]/div[2]/div/div[2]/div/div/table/tbody/tr[1]//*[@id="btnSeeMore"]""").click()
time.sleep(2)

abonnement = driver.find_element_by_xpath("""/html/body/div[6]/div/div/div/accordion/div/div[2]/div[2]/div/div/div/div[1]/span[3]/div[1]/span[2]""").text()

print(abonnement)

i want to get text from the sub page and then close it and continue to open and grab info from all the subpages.

enter image description here

first off i just need to be able to get som info from the first menu. For example the text in the red circle

an example of this could be the name of the company providing the service.

Rasmus Habel Svärd
  • 27
  • 4
  • Don't use absolute xpath. Check this for more info: https://stackoverflow.com/questions/27183353/what-is-the-difference-between-absolute-and-relative-xpaths-which-is-preferred – return Oct 23 '19 at 08:35
  • At which line of code are you actually stuck? Can you update the question with the _Manual Steps_? – undetected Selenium Oct 23 '19 at 08:57
  • my problem is the last part of the code. I cannot get selenium to output any text from the popup window when it enters the individual row pop ups. ``` driver.find_element_by_xpath("""/html/body/div[6]/div/div/div/accordion/div/div[2]/div[2]/div/div/div/div[1]/span[3]/div[1]/span[2]""").text() ``` – Rasmus Habel Svärd Oct 23 '19 at 09:01

2 Answers2

1

First important thing is to refactor code with best practices in locators. You can find information here and here.
Second, use WebDriverWait instead of sleep.

Here is code with all above to select some elements in opened modal window:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

# you code to click on **SE MERE**

main_details = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".productDetails .main-details p"))).text
price = driver.find_element_by_css_selector(".productDetails .price").text
subscription_monthly_payment = driver.find_element_by_css_selector(".productDetails .contractInfo [tooltip-checker='subscriptionMonthlyPayment'] + .value").text

driver.find_element_by_css_selector(".productDetails .close").click()
Sers
  • 12,047
  • 2
  • 12
  • 31
  • im not 100% sure where you find the css links? the last link .productDetails .contractInfo [tooltip checker='subscriptionMonthlyPayment'] + .value" does not output what i see in the modal. could you explain how you get these links. I have tried with the chropath add on in chrome but without luck – Rasmus Habel Svärd Oct 23 '19 at 12:04
  • What do mean **does not output what i see in the modal**? What value do you get for the first result - modPris? – Sers Oct 23 '19 at 12:08
  • the output i get from this is = what i see when i look at the web page is just to give an exapmle: "26,25kr. (inkl moms)" i will try to update the original post wth a screenshot – Rasmus Habel Svärd Oct 23 '19 at 12:31
  • Just add `.text` to get the text. See in updated answer – Sers Oct 23 '19 at 12:39
  • thx. i am still trying to figure out where you find the css path fx:".productDetails .price" are you using a browser plugin to get this or are you typing yourself? – Rasmus Habel Svärd Oct 23 '19 at 12:50
  • I do not use chropath or other special tools and do not suggest to do it. I'm searching myself using Chrome devtools **Elements** tab. If answer was helpful and works feel free to upvote and accept it. – Sers Oct 23 '19 at 13:01
  • I shared in the answer helpful links, check them. Good luck – Sers Oct 23 '19 at 13:08
0

I could access from developer tools a button name "Shift to this product" on the sub window. You could either give a relative xpath like //button[@id='btnSwitchToProduct'] OR do find by id('btnSwitchToProduct').

You have given the entire path to the element which delays the test script execution. Would suggest you to always use 'id' or 'name' as it is unique value for a page and quickens the test script runs.

ashwinin
  • 81
  • 5