from selenium import webdriver
driver= webdriver.Chrome()
driver.get('https://www.realestate.com.au/sold/') #opens normally but anything I search for doesn't load
driver.get('https://www.realestate.com.au/sold/in-alice+springs+-+greater+region%2c+nt%3b/list-1')
The website opens and I don't get any captcha or block but the page doesn't load
It seem some "Flash" problem (you can read the error in "chromedriver" window:
Browser not compatible with given flash manifest
This site use flash and your browser need flash player. By saying your browser I mean selenium browser, not that chrome you open manually.
To install "flash player" in selenium chrome browser, run your script; it will open the browser, then go to https://get.adobe.com/flashplayer/ and install flash player.
This should solve the problem
I guess they are blocking the use of selenium, see the related request is returning 429. Maybe you can a way to escape from this.
A bit of more information about your usecase would have helped us to debug the issue in a better way. However using Selenium I was able to access the website and retrieve the Page Source as follows:
Code Block:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.realestate.com.au/sold/")
print(driver.page_source)
Console Output:
<html class="js rui-burger rui-fancy-selects-on is-desktop app-loaded" lang="en"><head>
<meta http-equiv="x-ua-compatible" content="IE=edge">
<meta charset="utf-8">
<meta name="viewport" content="width=1140">
<meta name="description" content="Discover how much the latest properties have been sold for. Browse sold real estate listings to find auction results and sold prices from all over Australia.">
.
.
.
<script type="text/javascript" async="" charset="utf-8" id="utag_rea-group.sp-only_27" src="//tags.tiqcdn.com/utag/rea-group/sp-only/prod/utag.27.js?utv=ut4.44.201710110106"></script><script type="text/javascript" async="" charset="utf-8" id="utag_rea-group.sp-only_64" src="//tags.tiqcdn.com/utag/rea-group/sp-only/prod/utag.64.js?utv=ut4.44.202005070432"></script><script type="text/javascript" src="https://odb.outbrain.com/utils/get?url=http%3A%2F%2Fwww.realestate.com.au%2Fsold&settings=true&recs=true&widgetJSId=HPR&key=NANOWDGT01&idx=0&version=20009&apv=false&sig=kLvLc8wa&format=html&va=true&et=true&rand=98241&pdobuid=-1&osLang=en-US&winW=1366&winH=657&scrW=1366&scrH=768&adblck=false&secured=true&cmpStat=0&ccpaStat=0&ref=&px=119&py=1959&vpd=1302" charset="UTF-8" async=""></script><script src="https://safeinator.reamedia.com.au/2073b42b71554175cf729737a838c1f1c6d7f862/lib/base.min.js" async="" type="text/javascript"></script></head>
<!--[if IE 8]> <body class="sold rui-preload ie ie8 lt-ie9 lt-ie10" data-rui-select-auto-init="false"> <![endif]-->
<!--[if IE 9]> <body class="sold rui-preload ie ie9 lt-ie10" data-rui-select-auto-init="false"> <![endif]-->
<!--[if gt IE 9]><!-->
<body class="sold" data-rui-select-auto-init="false"><iframe src="https://secure-au.imrworldwide.com/storageframe.html" id="LOCSTORAGE" scrolling="no" name="empty" hidden="true" style="width: 1px; height: 1px; position: absolute; top: -7px; left: -7px; border: 0px;"></iframe>
<!--<![endif]-->
<div class="hero">
<ul class="rui-visuallyhidden rui-accessibility">
<li>
<a href="#rui-main-nav" title="Go directly to the website navigation">skip to Main Navigation</a>
</li>
<li>
<a href="#rui-main-content" title="Go directly to main content">skip to Content</a>
</li>
</ul>
<header class="rui-header rui-header-inverse rui-header-not-fixed rui-clearfix" data-auto-init="false">
<div class="rui-grid rui-header-container">
<div class="rui-clearboth"></div>
<button class="rui-burger-toggle rui-icon rui-icon-navdeck rui-button-no-style rui-desktop">
<span class="rui-visuallyhidden">Menu</span>
<span class="hit-box"></span>
</button>
<nav class="rui-myrea-nav">
<div id="webCollectionsSlot"></div>
<div id="webInboxSlot"></div>
<div id="webInboxOverlay"></div>
<ul class="rui-user-menu logged-out" style="">
<li>
<a href="/my-real-estate/login" title="Sign in">Sign in</a>
</li>
<li>
<a href="/my-real-estate/register" class="rui-button-brand join" title="Create an account">Join</a>
</li>
</ul>
<ul class="rui-user-menu logged-in rui-toggle-wrapper" style="display:none;">
<li>
<a href="#" class="rui-person-guy rui-toggle-link rui-icon rui-icon-user-off"></a>
<div class="rui-account-menu-container">
<ul class="rui-account-menu rui-toggle-container">
<li class="first">
<a href="/saved-properties" title="View saved properties">
<span class="rui-icon rui-icon-save"></span><span>Saved properties</span>
</a>
</li>
<li>
<a href="/property/my-property" title="Track your property">
<span class="rui-icon rui-icon-home"></span><span>Track your property</span>
</a>
</li>
<li>
<a href="https://homeloans.realestate.com.au/my-finances" title="My finances">
<span class="rui-icon rui-icon-commission"></span><span>My finances</span>
</a>
</li>
<li><a href="/my-real-estate/saved-searches-and-email-alerts" title="View property alerts">Saved searches & alerts</a></li>
<li><a href="/my-real-estate/profile" title="View profile">My profile</a></li>
<li><a href="/my-real-estate/account" title="Manage account settings">Account settings</a>
</li>
<li class="last"><a href="/logout.ds" title="Log out">Log out</a></li>
</ul>
</div>
</li>
</ul>
</nav>
</div>
<div class="rui-grid rui-nav-container rui-burger-container">
<a title="realestate.com.au homepage" href="/buy" class="logo">
<img src="//s1.rui.au.reastatic.net/rui-static/img/rea-logo-thin-white-v3.png" alt="realestate.com.au Australia lives here">
</a>
<nav class="rui-main-nav">
<ul>
<li class="rui-nav-tab buy">
<a href="/buy" title="Real estate and property for sale">
<span>Buy</span>
</a>
</li>
.
.
.
<li class="rui-nav-tab commercial">
<a href="http://www.realcommercial.com.au" target="_blank" title="Commercial real estate for sale and lease">
<span>Commercial</span>
</a>
</li>
</ul>
</nav>
</div>
</header>
<div class="hero-content">
<div class="search-form-container">
<form action="listsearchview.ds" class="search-form" name="search-form">
<h1 id="search-form-title">Search sold properties</h1>
<div class="search-container">
<div class="search-inner-container" style="z-index: 1;">
<nav class="search-channel-container">
<a href="/buy" class="rui-search-tab">Buy</a>
<a href="/rent/" class="rui-search-tab">Rent</a>
<a href="/sold/" class="rui-search-tab rui-search-tab-selected">Sold</a>
<a href="/property/" class="rui-search-tab property-value-tab">Property value</a>
<a href="/find-agent" class="rui-search-tab find-agents-tab">Find agents</a>
</nav>
<div class="search-inner-container" style="z-index: 1;">
<div class="rui-search-container search-input-container">
<i class="rui-icon rui-icon-search search-icon"></i>
<input name="where" id="where" tabindex="2" class="rui-input rui-location-box rui-auto-complete-input" data-auto-init="false" data-location-source="consumer-suggest" data-max-results="7" data-client-source="homepage" autocomplete="off" placeholder="Search by state, suburb or postcode" data-multi-select="true" data-suggest-types="suburb,precinct,region,state,postcode" data-auto-submit="false" autofocus=""><div class="clear-text-container"><a class="rui-icon rui-icon-cross" title="Clear text"> </a></div>
<input name="theme" id="theme" type="hidden" value="rea.sold">
<button class="rui-search-button" tabindex="10">
<span class="rui-visually">Search</span>
</button>
<div class="focus-border" style="display: none;"></div>
</div>
</div>
</div>
<section class="search-refinements" id="search-refinements">
<div class="condition property-select-holder">
<div class="select-holder">
<label>All property types</label>
<select class="rui-input rui-select resize-select rui-select-rendered" id="rui-property-type-select-id" name="propertyType" multiple="multiple" style="display: none;">
<option value="All" selected="selected">All property types</option>
<option value="House">House</option>
<option value="unit apartment">Apartment & Unit</option>
<option value="Townhouse">Townhouse</option>
<option value="Villa">Villa</option>
<option value="Land">Land</option>
<option value="Acreage">Acreage</option>
<option value="Rural">Rural</option>
<option value="unitblock">Block of Units</option>
<option value="retire">Retirement Living</option>
</select>
</div>
.
.
.
</body></html>
However, while attempting to invoke the url https://www.realestate.com.au/sold/in-alice+springs+-+greater+region%2c+nt%3b/list-1' it seems Selenium driven Browser Agent is getting detected through the fingerprints
Code Block:
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.realestate.com.au/sold/in-alice+springs+-+greater+region%2c+nt%3b/list-1'")
Console Output:
<html lang="en"><head>
<meta charset="UTF-8">
<link rel="shortcut icon" href="about:blank">
</head>
<body>
<script src="/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/j.js"></script>
<script src="/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/f.js"></script>
<script src="/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/fingerprint/script/kpf.js?url=/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/fingerprint&token=34091693-e55f-5166-d979-67e9426e1030"></script>
</body></html>
You can find a relevant detailed discussions in Chrome browser initiated through ChromeDriver gets detected
