0

Note: This code requires human intervention in between as it is incomplete, and should thus only be run with Jupyter. I am trying to get the last page number of a tripadvisor webpage.

The "Malaysia" and "Switzerland" webpages works fine (urls commented out below) but not the "Hong Kong" one.

from selenium import webdriver #for navigating through the pages
driver = webdriver.Chrome(executable_path=r'C:\\Users\\user\\Downloads\\chromedriver.exe') 
url = "https://www.tripadvisor.com.sg/Hotels-g294217-Hong_Kong-Hotels.html"
#url = "https://www.tripadvisor.com.sg/Hotels-g293951-Malaysia-Hotels.html" 
#url = "https://www.tripadvisor.com.sg/Hotels-g188045-Switzerland-Hotels.html"
driver.get(url) 
driver.implicitly_wait(5)

Human intervention here: Now click on some arbitrary "Check in date", "Check out date" and then click "Update"

last_page_s = driver.find_element_by_css_selector("span.pageNum.last").get_attribute('data-page-number')
last_page = int(last_page_s)
print(last_page)

I'm still a newbie with webscraping so any help is greatly appreciated!!

ping
  • 33
  • 1
  • 5

1 Answers1

-1

To print the last_page number you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR:

    print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "span.separator.cx_brand_refresh_phase2 +a"))).get_attribute('data-page-number'))

  • Using XPATH`:

    print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[contains(@class, 'separator')]//following::a"))).get_attribute('data-page-number'))

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • 1
    Thank you very much! It seems like it was more of my poor choice of css selector ("span.pageNum.last")... But I do still find it very odd that it worked for all webpages except the Hong Kong one. If possible, do you mind telling me the best way to get a css selector (or xpath)? thus far I have been trying to use chrome path and selector gadget but it seems like they don't always work. – ping Sep 09 '20 at 12:46
  • I have done so! Do you mind telling me the best way to get a css selector (or xpath)? Thus far I have been trying to use chrome path and selector gadget but it seems like they don't always work. – ping Sep 09 '20 at 12:50
  • @pingThere is no best practice for _xpath_ and _css_ what matters most is, the locating strategy must be optimized. Chrome path is a good way to start but slowly and gradually you need to write optimised xpath and css on your own through practice. – undetected Selenium Sep 09 '20 at 12:52
  • @ping Dev Tools is the best place where you can validate you locators. See [this](https://stackoverflow.com/questions/62960908/chrome-84-inspect-element-find-results-not-highlighted-in-yellow-like-before/63075040#63075040), [this](https://stackoverflow.com/questions/62945647/why-xpath-does-not-highlighted-the-yellow-mark-in-chrome84/63073849#63073849) and [this](https://stackoverflow.com/questions/63034593/chrome-devtools-not-find-elements-not-search/63035359#63035359) discussion. – undetected Selenium Sep 09 '20 at 12:56
  • That's lovely :) I will explore them. Thank you very much! – ping Sep 09 '20 at 14:34