6

I did several hours of research and asked a bunch of people on fiverr who all couldn't solve a a specific problem I have.

I installed Selenium and tried to access a Website. Unfortunately the site won't allow a specific request and doesn't load the site at all. However, if I try to access the website with my "normal" Chrome Browser, it works fine.

I tried several things such as:

  • Different IP's
  • Deleting Cookies
  • Incognito Mode
  • Adding different UserAgents
  • Hiding features which might reveal that a Webdriver is being used

Nothing helped.

Here is a Screenshot of the Error I'm receiving:

enter image description here

And here is the very simple script I'm using:

# coding: utf8
from selenium import webdriver

url = 'https://registrierung.gmx.net/'

# Open ChromeDriver
driver = webdriver.Chrome();
# Open URL
driver.get(url)

If anyone has a solution for that I would highly appreciate it. I'm also willing to give a big tip if someone could help me out here.

Thanks a lot! Stay healthy everyone.

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
nonskill
  • 111
  • 1
  • 5

1 Answers1

2

I took your code modified with a couple of arguments and executed the test. Here are the observations:

  • Code Block:

    from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://registrierung.gmx.net/")
print(driver.page_source)

  • Console Output:

    <html style="" class=" adownload applicationcache blobconstructor blob-constructor borderimage borderradius boxshadow boxsizing canvas canvastext checked classlist contenteditable no-contentsecuritypolicy no-contextmenu cors cssanimations csscalc csscolumns cssfilters cssgradients cssmask csspointerevents cssreflections cssremunit cssresize csstransforms3d csstransforms csstransitions cssvhunit cssvmaxunit cssvminunit cssvwunit dataset details deviceorientation displaytable display-table draganddrop fileinput filereader filesystem flexbox fullscreen geolocation getusermedia hashchange history hsla indexeddb inlinesvg json lastchild localstorage no-mathml mediaqueries meter multiplebgs notification objectfit object-fit opacity pagevisibility performance postmessage progressbar no-regions requestanimationframe raf rgba ruby scriptasync scriptdefer sharedworkers siblinggeneral smil no-strictmode no-stylescoped supports svg svgfilters textshadow no-time no-touchevents typedarrays userselect webaudio webgl websockets websqldatabase webworkers datalistelem video svgasimg datauri no-csshyphens"><head>
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Expires" content="-1">
<meta http-equiv="CacheControl" content="no-cache">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<link rel="shortcut icon" href="data:;base64,iVBORw0KGgo=">

<script type="text/javascript">
(function(){
window["bobcmn"] = "10111111111010200000005200000005200000006200000001249d50ae8200000096200000000200000002300000000300000000300000006/TSPD/300000008TSPD_10130000000cTSPD_101_DID300000005https3000000b0082f871fb6ab200097a0a5b9e04f342a8fdfa6e9e63434256f3f63e9b3885e118fdacf66cc0a382208ea9dc3b70a28002d902f95eb5ac2e5d23ffe409bb24b4c57f9cb8e1a5db4bcad517230d966c75d327f561cc49e16f4300000002TS200000000200000000";
.
.
<script type="text/javascript" src="/TSPD/082f871fb6ab20009afc88ee053e87fea57bf47d9659e73d0ea3c46c77969984660358739f3d19d0?type=11"></script>

<script type="text/javascript">
(function(){
window["blobfp"] = "01010101b00400000100e803000000000d4200623938653464333234383463633839323030356632343563393735363433343663666464633135393536643461353031366131633362353762643466626238663337210068747470733a2f2f72652e73656375726974792e66356161732e636f6d2f72652f0700545350445f3734";window["slobfp"] = "08c3194e510b10009a08af8b7ee6860a22b5726420e697e4";


})();

</script>

<script type="text/javascript" src="/TSPD/082f871fb6ab20009afc88ee053e87fea57bf47d9659e73d0ea3c46c77969984660358739f3d19d0?type=12"></script>
<noscript>Please enable JavaScript to view the page content.<br/>Your support ID is: 11993951574422772310.</noscript>
</head><body>
<style>canvas {display:none;}</style><canvas width="800" height="600"></canvas></body></html>

  • Browser Snapshot:

registrierung

Conclusion

From the Page Source it's quite clear that Selenium driven ChromeDriver initiated Browsing Context gets detected and the navigation is blocked.

I could have dug deeper and provide some more insights but suprisingly now even manually I am unable to access the webpage. Possibly my IP is black-listed now. Once my IP gets whitelisted I will provide more details.

References

You can find a couple of relevant detailed discussions in:

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
  • Will definitely do! You helped my narrowing the request error down which helps a lot. Do you by any chance know if it's possible to bypass that and "hide" that specific Browsing Context in order to access the website normally via Webdriver or is that not possible it all? – nonskill Jul 31 '20 at 18:32
  • 1
    As I already mentioned, my IP seems blocked so I'm unable to proceed and I don't want to risk whitelisting my IP through the back door right now. Please go through the reference discussions which will provide a lot more similar insights. – undetected Selenium Jul 31 '20 at 18:36
  • Will do :) Thanks again – nonskill Jul 31 '20 at 18:44