0

zillow picture I have the above image at https://www.zillow.com/homes/for_sale/7132668_zpid/globalrelevanceex_sort/60.780619,-65.522461,4.521666,-125.551758_rect/3_zm/

I cant seem to find the selector for tax history. I tried to use driver wait but the table that is output is the price history not tax history.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import NoSuchElementException

options = webdriver.ChromeOptions()
options.add_argument(f"user-agent={useragent[0]}")
options.add_argument('--proxy-server=%s' % ips[0])
options.add_argument('--incognito')
chromedriver = '~/Downloads/chromedriver'
chromedriver = os.path.expanduser(chromedriver)

driver = webdriver.Chrome(chromedriver, chrome_options=options)
driver.get('https://www.zillow.com/homes/for_sale/7132668_zpid/globalrelevanceex_sort/60.673178,-74.663086,4.653079,-116.323243_rect/3_zm/2_p/')

wait(driver, 10).until(EC.element_to_be_clickable((By.CLASS_NAME, "hdp-collapse"))).click()
table = wait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "div#hdp-tax-history")))
Joey Chao
  • 45
  • 4

1 Answers1

1

Looks like you need some waits (and maybe some clicks to get that tab visible. You can write out the table. The below is just to show how you can access

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

url = 'https://www.zillow.com/homes/for_sale/7132668_zpid/globalrelevanceex_sort/53.566414,-73.081055,17.434511,-118.081055_rect/3_zm/'
d = webdriver.Chrome()
d.get(url)

WebDriverWait(d,20).until(EC.presence_of_element_located((By.ID , 'price-and-tax-history'))).click()
tabs = WebDriverWait(d,5).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".zsg-tab-link")))
tabs[1].click()
print(d.find_element_by_css_selector('#hdp-tax-history table').text)  # just to show present
QHarr
  • 83,427
  • 12
  • 54
  • 101
  • Thanks for the help! Zillow just changed the structure of their website so I have to rewrite all my functions – Joey Chao Feb 08 '19 at 20:44
  • be aware they have also implemented intermittent captcha's – QHarr Feb 08 '19 at 20:48
  • yes it is! thank you, I was wondering if you also know why I get captchas just by using driver.get() without running any automation? – Joey Chao Feb 09 '19 at 21:48
  • There are some good Q&As on this subject on stackoveflow: avoiding detection as bot when using selenium. I remember at least one thread with really good technical detail including changes you can make to the selenium driver source code itself. – QHarr Feb 09 '19 at 22:23
  • Dug this out of my bookmarks: https://stackoverflow.com/questions/33225947/can-a-website-detect-when-you-are-using-selenium-with-chromedriver – QHarr Feb 09 '19 at 23:05
  • see the bits about exposed javascript hooks and pre-defined javascript variables which appear when running with selenium. These are more accurate answers than my original comment though the IP addresses stands. – QHarr Feb 09 '19 at 23:08