1

I'm hacking together an amazon api and when only using python requests without proxying, it prompts for a captcha. When routing this python requests traffic through fiddler, it seems to pass without a problem. Is it possible that amazon is fingerprinting python requests and fiddler changes the fingerprint since it's a proxy?

I viewed headers sent from fiddler and python requests and they are the same.

There is no exra proxying/fiddler rules/filters set on fiddler to create a change.

To be clear, all mentioned proxying is only done locally, so it will not change the public ip address.

Thank you!

Zanga
  • 55
  • 3
  • 11
  • 1
    A way to find an answer to your question is to set up your own simple web server (in Python) and submit your request to it, both directly and through the proxy. You'll be able to see exactly how the request and its header are changed by routing it through the proxy and any of those changes *could* be used for fingerprinting. – Grismar Feb 26 '20 at 05:15
  • I kind of tried that, I used [link](https://stackoverflow.com/questions/10588644/how-can-i-see-the-entire-http-request-thats-being-sent-by-my-python-application) to get the headers/parameters sent but I'm starting to think it could be something on the tcp level, beyond my ability. – Zanga Feb 26 '20 at 05:22
  • 3
    You forget the HTTPS layer. By the supported SSL/TLS versions, the enabled ciphers, their order and other subtle differences in the TLS handshake (e.g. in the CLIENT_HELLO message) you can guess how the request was created. Use Wireshark to see the differences. – Robert Feb 26 '20 at 18:51
  • Thank you!! Changing the TLS version of python requests to TLSv1_2 fixed it! – Zanga Feb 27 '20 at 00:52

1 Answers1

2

The reason is that websites are fingerprinting your requests with TLS hello package. There exist libraries like JA3 to generate a fingerprint for each request. They will intentionally block http clients like requests or urllib. If you uses a MITM proxy, because the proxy server create a new TLS connection with the server, the server only sees proxy server's fingerprint, so they will not block it.

If the server only blocks certain popular http libraries, you can simply change the TLS version, then you will have different fingerprint than the default one.

If the server only allows popular real-world browsers, and only accepts them as valid requests, you will need libraries that can simulate browser fingerprints, one of which is curl-impersonate and its python binding curl_cffi.

pip install curl_cffi

from curl_cffi import requests

# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome101")

print(r.json())
# output: {'ja3_hash': '53ff64ddf993ca882b70e1c82af5da49'
# the fingerprint should be the same as target browser
ospider
  • 9,334
  • 3
  • 46
  • 46