-1

You probably think that this is a misbehaving crawler problem but this CloudFlare website:

  • Blocks Internet Explorer on Windows 10 desktop (uses Windows API for HTTP)
  • Allows Edge on Windows 10 desktop (uses Windows API for HTTP?)
  • Blocks my crawler on Windows using WinInet Windows API for HTTP
  • Allows my crawler on Mac using Mac API for HTTP

Accordingly to this Internet Explorer 7 on Windows Vista should be support on even free CloudFlare: https://support.cloudflare.com/hc/en-us/articles/203041594-Cloudflare-SSL-cipher-browser-and-protocol-support

I am in preliminary research phase, but it feels like a certificate issue. I have not experienced this problem before on any website when using Windows 10. (But yes, older versions of Windows can have similar problems to certificate problems)

Any ideas? I have not implemeted HTTP2 part of Windows API - is that a possible explanation? Would just be a bit odd to require that so.... But could possibly explain why Edge and not IE works.

Tom
  • 3,587
  • 9
  • 69
  • 124
  • Web-Clients send their type and version in each request-header, web-servers may reject them, e.g. older browsers for their security flaws or incompatible features. See https://stackoverflow.com/questions/15069533/http-request-header-useragent-variable – Sam Ginrich Mar 20 '21 at 19:43
  • I already tried masking perfectly useragent. Behavior with regards to robtos.txt etc. But the thing that is interesting is the site already rejects IE/Win10 connections but not Edge/Win10 connections. Just wondered if anyone experience similar with CloudFlare or certificate problem. (Maybe I am asking wrong place. Maybe better to ask CloudFlare support. Not sure. But I will need to debug this I suppose so just wondered if anyone could point me in right direction to start my investigation) – Tom Mar 20 '21 at 21:05
  • Again a web server may decide over it's trust authority and there should be a way to emulate the Edge. But if a site already rejects IE/Win10, I would not investigate to get there with anything but what already works. – Sam Ginrich Mar 20 '21 at 21:27
  • It is a SSL/cipher issue. But what exactly makes Internet Explorer 10 not work and my crawler when using Windows API not work is confusing when Edge works. But I will research further what the *difference* is, so I have a chance of solving it – Tom Mar 21 '21 at 14:02
  • May be sniffers like Wireshark reveal differences of protocol / resources, else consider automating an Edge. – Sam Ginrich Mar 22 '21 at 08:52

1 Answers1

0

I asked the website owner and the website was setup to only support TLS 1.3 and not accept TLS 1.2 --- TLS 1.3 support is still experimental in Windows 10 (internet options)

Tom
  • 3,587
  • 9
  • 69
  • 124