0

Ive just started to learn python and doing this mini-project, trying to scrap info from a website with beatiful soup, but the phone number is hidden untill i click on the "show" button. How do i do this properly? Website example: https://auto.ria.com/uk/auto_renault_megane_34684935.html

image of show button

Tryed to do it with selenium but it opened each website manualy and clicked on the "show" button, i think this is not right

100yak
  • 13
  • 2
  • Please read [ask] and provide a [mre]. – baduker Jun 20 '23 at 18:32
  • Site scraping is an exercise in reverse engineering. What have you done to find the information you're looking for? Is that information even on the page, or is it fetched dynamically from a separate request when interacting with the page? Is it in the markup, or in JavaScript code? You first need to figure out where that data actually comes from, *then* you'd write code to fetch that data. (Whether manually with custom HTTP requests or whether using a "headless browser" like Selenium to interact with the in-memory page.) – David Jun 20 '23 at 18:39
  • Ive looked trough the source code. Im getting this request when im clicking on the button https://auto.ria.com/users/phones/34684935?hash=USRMkZZYMwXs4JUnz4y0MA&expires=2592000 { "phones": [ { "phoneId": 674716587, "phoneFormatted": "(063) 168 30 38" }, { "phoneId": 679432975, "phoneFormatted": "(095) 083 50 21" } ], "formattedPhoneNumber": "(063) 168 30 38" } but im not sure what i should do with it – 100yak Jun 20 '23 at 18:56
  • also getting a link in the Fetch/XHR https://auto.ria.com/phoneAccessStat?event_id=21&proposal_id=34684935&phone_id=679432975&price=10100&userEmail=&screentype=1&phone_position=1&phone_list_number=2&browser_height=937&browser_width=811&click_action=0 – 100yak Jun 20 '23 at 18:57
  • Does this answer your question? [Python + Selenium: Wait until element is fully loaded](https://stackoverflow.com/questions/50468629/python-selenium-wait-until-element-is-fully-loaded) – undetected Selenium Jun 20 '23 at 19:10

1 Answers1

0

Using Selenium to click on the show link to unmusk the phone number you need to you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following locator strategies:

  • Code block:

    driver.get("https://auto.ria.com/uk/auto_renault_megane_34684935.html")
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "label[for='c-notifier-close']"))).click()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "span.mhide +a"))).click()
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.list-phone a:nth-of-type(2) +div"))).text)

  • Console Output:

    (063) 168 30 38
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • Thanks, i think i found a way to do it work without selenium, but if it doesn't work i will use this. Very helpful – 100yak Jun 20 '23 at 19:12