So I need to scrap a page like this for example and I am using Scrapy + Seleninum to interact with a date-picker calendar.
I realized that if a certain date is available a price shows on the tooltip, and if its not available if you hover on it nothing happens.
Whats the code for me to get the price that appears dynamically when you hover on an available day and also how do I know if its available or not just with the hover?
It is not that straightforward how to approach the problem because of the dynamic nature of the page - you have to use waits here and there and it's tricky to catch the HTML of the dynamic components appearing on click or hover.
Here is the complete working code that would navigate to the page, click the "Check In" input, wait for the calendar to load and report the availability for each of the days in the calendar (it uses the presence of ui-datepicker-unselectable class to determine that). Then, it hovers each cell using the move_to_element() browser action, waits for the tooltip and gets the price:
from selenium import webdriver
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("https://www.airbnb.pt/rooms/265820?check_in=2016-04-26&guests=1&check_out=2016-04-29")
# wait for the check in input to load
wait = WebDriverWait(driver, 10)
elem = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.book-it-panel input[name=checkin]")))
elem.click()
# wait for datepicker to load
wait.until(
EC.visibility_of_element_located((By.CSS_SELECTOR, '.ui-datepicker:not(.loading)'))
)
days = driver.find_elements_by_css_selector(".ui-datepicker table.ui-datepicker-calendar tr td")
for cell in days:
day = cell.text.strip()
if not day:
continue
if "ui-datepicker-unselectable" in cell.get_attribute("class"):
status = "Unavailable"
else:
status = "Available"
price = "n/a"
if status == "Available":
# hover the cell and wait for the tooltip
ActionChains(driver).move_to_element(cell).perform()
price = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.datepicker-tooltip'))).text
print(day, status, price)
Prints:
1 Unavailable n/a
2 Unavailable n/a
3 Unavailable n/a
4 Unavailable n/a
5 Unavailable n/a
6 Unavailable n/a
7 Unavailable n/a
8 Unavailable n/a
9 Unavailable n/a
10 Unavailable n/a
11 Unavailable n/a
12 Unavailable n/a
13 Available €40
14 Unavailable n/a
15 Unavailable n/a
16 Unavailable n/a
17 Unavailable n/a
18 Unavailable n/a
19 Available €36
20 Available €49
21 Unavailable n/a
22 Available €49
23 Unavailable n/a
24 Unavailable n/a
25 Available €40
26 Available €39
27 Available €35
28 Available €37
29 Available €37
30 Available €37
Hi please find the answer
Key Note: we have wait some seconds after click event on calendar, because java-script takes internal processing time on calendar after it get opens.
public static void main(String[] args) throws InterruptedException {
System.setProperty("webdriver.chrome.driver","D:\\eclipseProject\\StackOverFlow\\chromedriver_win32 (1)\\chromedriver.exe");
WebDriver driver = new ChromeDriver();
driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
driver.manage().window().maximize();
Actions act = new Actions(driver);
WebDriverWait wait = new WebDriverWait(driver,30);
driver.get("https://www.airbnb.pt/rooms/265820?check_in=2016-04-26&guests=1&check_out=2016-04-29");
// selecting firstdate picker -- check in
driver.findElement(By.xpath("//*[@class='col-sm-6']/input")).click();
// NOTE: we have to give sleep due to java-script takes internal processing time on calendar after it get opens
Thread.sleep(5000);
// NOTE: calendar is not completely visible hence to make it visible
// scroll a little bit down
((JavascriptExecutor) driver).executeScript("window.scrollBy(0,200)");
// take all calendar dates inside the list
List<WebElement> myhAvDates = driver.findElements(By.xpath("//*[@class='ui-datepicker-calendar']/tbody/tr/td/a[contains(@class, 'ui-state-default')]"));
System.out.println("Size is "+myhAvDates.size());
for(int i=0;i<myhAvDates.size();i++){
wait.until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.xpath("//*[@class='ui-datepicker-calendar']/tbody/tr/td/a")));
System.out.println("Available Date is : " + myhAvDates.get(i).getText());
act.moveToElement(myhAvDates.get(i)).build().perform();
wait.until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector(".datepicker-tooltip")));
WebElement toolTipElement = driver.findElement(By.cssSelector(".datepicker-tooltip"));
System.out.println("Available Date is : " + myhAvDates.get(i).getText() + "==" +"And price is "+ toolTipElement.getText());
myhAvDates = driver.findElements(By.xpath("//*[@class='ui-datepicker-calendar']/tbody/tr/td/a"));
}
}
above code will yield answer like
Size is 10
Available Date is : 13
Available Date is : 13==And price is €40
Available Date is : 19
Available Date is : 19==And price is €36
Available Date is : 20
Available Date is : 20==And price is €49
Available Date is : 22
Available Date is : 22==And price is €49
Available Date is : 25
Available Date is : 25==And price is €40
Available Date is : 26
Available Date is : 26==And price is €39
Available Date is : 27
Available Date is : 27==And price is €35
Available Date is : 28
Available Date is : 28==And price is €37
Available Date is : 29
Available Date is : 29==And price is €37
Available Date is : 30
Available Date is : 30==And price is €37
