the Python script gets the page and clicks the button that cause reloading the main page content. Then I try to get the page source of the reloaded page, but still having the content of the first one. Probably it should be refreshed somehow.
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(base_listing_url)
time.sleep(10)
driver.find_element_by_css_selector('.fTomoL').click()
time.sleep(10)
print(driver.page_source) #This returns the first page's content
You need to wait for the WebElement of the new page to render within the HTML DOM and you can adapt any of the following approaches:
You can wait for the Page Title to change:
driver.find_element_by_css_selector('.fTomoL').click()
# wait till the page title contains the word Facebook
WebDriverWait(driver, 10).until(EC.title_contains("Facebook"))
print(driver.page_source)
You can wait for a visible element to be visible:
driver.find_element_by_css_selector('.fTomoL').click()
# wait till the page title contains the word Facebook
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//h2[text()='Some Heading']")))
print(driver.page_source)
You can find a couple of relevant detailed discussions in:
