Catching text hidden behind a "read more" button with Selenium

Question

Using Selenium for Python, I'm trying to catch the entire description text from this page: realestate.com.kh: Agile Sky Residence

Before clicking on "read more":

After clicking on "read more":

options = webdriver.ChromeOptions()
options.add_argument('headless')
browser = webdriver.Chrome('chromedriver',chrome_options=options)
browser.maximize_window()

browser.implicitly_wait(1)
# click on "Read more", which path is //*[@id="listing-desctiption"]/div[3]/a
browser.find_elements_by_xpath('//*[@id="listing-desctiption"]/div[3]/a')[0].click()
browser.implicitly_wait(1)
# catch the full content of the unwrapped text which path is //*[@id="undefined1"]
descr = browser.find_element_by_xpath('//*[@id="undefined1"]').text
print(descr)

Unfortunately, I get the following error:

selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element ... is not clickable at point (399, 591). Other element would receive the click: ... (Session info: headless chrome=76.0.3809.100)

Am I not proceeding correctly for this kind of situation?

Answer 1

There's problem with headless mode. Default size is 800x600 even maximize_window is used. You can set custom size, in your case 1280x800 is enough:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument('--headless')
browser = webdriver.Chrome(options=options)
wait = WebDriverWait(browser, 5)
browser.set_window_size(1280, 800)

browser.get("https://www.realestate.com.kh/agile-sky-residence/110921/")
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".listing-more-wrapper a"))).click()
description = browser.find_element_by_class_name("description").text
print(description)

Another solution is to use JavaScript to get description text

options = webdriver.ChromeOptions()
options.add_argument('--headless')
browser = webdriver.Chrome(options=options)
browser.maximize_window()

browser.get("https://www.realestate.com.kh/agile-sky-residence/110921/")
description = browser.execute_script("return arguments[0].textContent;",
                                     browser.find_element_by_class_name("description"))
print(description)

Answer 2

To extract the entire description text you need to:

• scrollIntoView() the element with text as Read More and click() on it.
• Then you can use the text property to read the desired text using the following Locator Strategies:

• Code Block:

  from selenium import webdriver
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  
  chrome_options = webdriver.ChromeOptions()
  chrome_options.add_argument('--headless') 
  chrome_options.add_argument("start-maximized")
  driver = webdriver.Chrome(options=chrome_options, executable_path=r'C:\WebDrivers\chromedriver.exe')
  driver.get("https://www.realestate.com.kh/agile-sky-residence/110921/")
  driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//a[contains(., 'Read More')]"))))
  WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[contains(., 'Read More')]/span[@class='icon-down']"))).click()
  print(WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//h2[text()='Agile Sky Residence']//following::div[1]"))).text)
  driver.quit()
  

• Console Output:

  Agile is a Top 10 Developer in China. Over the last 27 years, Agile has excelled across eight major industries including real estate, life services, conservation, education, construction, home management, capital investment, and commercial property management.
  
  The business covers more than 100 cities around the world. Real estate development is the core industry. The nearly 200 projects have expanded to more than 70 cities across China and the world. Agile Sky Residence, a 44-storey high-rise apartment building, is located in the BKK3, one of the most prosperous areas of Phnom Penh. Located at No. 623 Monivong Blvd, it is surrounded by municipal administration, foreign affairs departments, embassies, international organizations, and many international residences.
  Agile Sky Residence is rooted in its prime location. It is convenient and transportation here is easy. What’s more, there are also high-quality schools and medical facilities that offer the best service to expats and Khmer residents in this neighbourhood. For example, it’s just a 10-minute drive to ISPP, CIS International School, BKK High School, Royal University of Law and Economics, Soviet Friendship Hospital, or First Center Polyclinic.
  Agile Sky Residence was designed by Patrick Leung, a famous interior designer from Hong Kong, who has won more than 200 international awards. It is tailored to be the ultimate residence for the new generation by using first-class brand materials and a full range of quality home appliances and furniture. It is a smart option for anyone. The unique plan and innovative facilities of Agile Sky Residence are the reason for its glory. With a beautiful 7-metre-high lobby, a massive parking area, and nine high-speed elevators, the residence delivers perfect hospitality. Agile Sky Residence has a master plan that lays out four-level sky gardens, which create a very functional space of leisure and recreation. A sky bar, sky club, cafe, a book lounge, and even a children’s playground entertain the residents here. Head to the infinity pool if you want a break. Or, for fitness buffs, go to the fitness centre or yoga area. Health and wellbeing are central aspects of living here. Agile Sky Residence hosts nearly a thousand apartments and more than 420 parking spaces.
  The apartments have a wide range of sizes from 39 to 111 square metres. Configurations range from studios to three-bedroom units. The most common type of unit is the 57-square metre one-bedroom unit. The floor plan of each unit has an excellent layout that offers a spacious and comfortable living room and a bedroom to fit the preferences of the people of Asia.