4

The question title is probably not covering the whole subject as I did quite a lot of research and found a number of weird things.

So first of all, what I'm trying to implement is some kind of a client for a website that would work on a user's behalf (not doing anything illegal, just optimizing some of the user's workflow). I've done this for many websites and it worked fine. However, with the current one there's an issue.

Normally, if I encounter a captcha I just open an embedded Chrome window for the user to pass it. However, with the website I'm talking about it doesn't help as the captcha is not displayed in the browser but is sent to me when I'm mimicking the request browser is sending exactly.

So I tried to investigate the difference between the request sent by the Chrome and by my application using Fiddler. However, even requests sent by a real Chrome face the same captcha if I enable the Fiddler.

I've disabled HTTP/2, SPDY and IPv6 in the Chrome as I thought that could be the difference. It didn't help. I've tried comparing the requests sent by Chrome using the Chrome dev tools - there's no difference, both of them are using HTTP/1.1, both have exactly the same headers, exactly the same cookies (or no cookies, it makes no difference). But whenever I enable Fiddler - the website responds with a captcha.

This is the first time I'm encountering something like this and I'm almost ready to bang my head against a wall as I don't see any possible way for the website to understand that the request is being proxied by Fiddler as it's not adding any custom headers or whatever.

Unless the website is somehow detecting the exact way HTTPS connection is being set up which sounds pretty insane... it shouldn't be possible.

Looking for an advice on how to debug this further.

Update:

I didn't find a solution nor did I understand how does the website in question detect direct connections from Chrome, but managed to find a workaround:

I'm taking the page with a captcha that my code receives from the website and substitute the actual page received by the CEF with that captcha page on the fly, thus allowing the user to pass it.

Since it doesn't answer the original question I'm not posting this as an answer and will leave this question open.

mephisto123
  • 1,400
  • 13
  • 38

1 Answers1

4

The website itself usually does not detect anything. The captcha is usually presented by the anti-dos protection service provider such as Cloudflare.

From my experience such system combine a browser fingerprinting system via JavaScript (get the used web browser name, version and the used OS) with a detection on HTTPS (TLS) level:

In the TLS protocol handshake the client sends a CLIENT_HELLLO message that contains information about the supported TLS versions and cipher suites as well ass other additional data in some TLS extensions (e.g. if it supports HTTP/2).

This handshake can again be fingerprinted. If you now for example use Firefox through Fiddler the browser fingerprint shows Firefox but Fiddler is a .Net application hence the fingerprint indicates that Windows schannel TLS library is used. Both fingerprints mismatch and hence the protection system sends you a redirection to a captcha dialog.

Robert
  • 39,162
  • 17
  • 99
  • 152
  • That makes sense, I've suspected something like that. So basically the only workaround would be to implement my own TLS library that would mimic a real Chrome handshake... Sounds like an insanely complicated task. – mephisto123 Jun 04 '20 at 13:35
  • 1
    @mephisto There are multiple way: manipulate the traffic directly in Chrome before it is sent/received. Or alternatively sometimes it is enough to configure the supported tls versions and cipher suites accordingly in the proxy. Some proxies allow to do so (not sure if this can be done in Fiddler). – Robert Jun 04 '20 at 13:40
  • Thank you. I will mark your answer as accepted one because it answers the main part of my question and provides information on where to look further :) – mephisto123 Jun 04 '20 at 17:34