11

I am building a website recorder that acts like a proxy, in order to test web scrapers on an ongoing basis. It is split into three Docker containers, all on GNU/Linux: (1) a proxy, (2) an API and request queue, and (3) a simple web app.

It works fine for HTTP sites: I click a button in the web app, this makes a request to the API container, and that adds something to an internal request queue, which then requests the site via the proxy. The proxy records the site as it passes through.

However, I'd forgotten that one cannot record HTTPS site traffic, and now I've come to implement this, I've found that proxies just use the CONNECT verb, and then act as a data exchanger between the client and the target. I believe I cannot replay the same data chunks as part of the encryption uses a randomised, throwaway, symmetric key (however I have a script suitable for testing this, so I will do so just for the educational value!).

So, I was wondering if my fetching client could give up enough secrets for the proxy system to decode the byte-stream? I am using Wget to do the fetch, which I guess would be using OpenSSL. It does not need to be Wget though: if I were using a PHP script with file_get_contents with a stream context, can I ask the openssl module for the decryption keys?

(To be fair, I will probably not solve the problem in this fashion even if it is possible, I just thought it would be really interesting to learn a bit more about TLS. In practice, I will record a "null" entry against all secure websites in the proxy, and require the requesting service to notify the proxy of header/body data via an API call, so it can be later played back. They will of course have plaintext copies of these items).

halfer
  • 19,824
  • 17
  • 99
  • 186
  • If this were possible, I'd be very concerned. – miken32 Apr 04 '17 at 21:52
  • Why @miken32? Encryption is, for most practical purposes, unbreakable because the client does not emit secrets to crackers who would want them. However, in a closed system, if it can be asked to give up those secrets with permission, surely that can be used for decryption without worrying that the encryption system is flawed in general? – halfer Apr 04 '17 at 21:54
  • I've added a bounty, to tempt people into answering. It's a fun question without a real purpose, but I think it's interesting. If this is possible, I expect answers will say that one would have to write/modify ones own client, so as to be able to fish out the necessary secrets from OpenSSL. Of course, I am not looking for working code, as that would be rather broad - just the techniques involved. Does OpenSSL permit all secrets to be obtained by a client, without recompiling OpenSSL? – halfer Apr 06 '17 at 11:23
  • 1
    By the way, if this is for a commercial project, I'd recommend using one of the existing HTTPS debugging proxies to do this, rather than attempting to write your own. I know of at least https://www.fiddlerbook.com/fiddler/help/httpsdecryption.asp and https://www.charlesproxy.com/documentation/proxying/ssl-proxying/ that do exactly this, including TLS unwrapping. If this is for a hobby or educational project to learn about HTTPS then good on you and carry on! – Rich Apr 11 '17 at 08:50
  • 1
    Good advice, thanks @Rich. The purpose is for a commercial project, but it's just for testing rather than production, and in any case I'm avoiding the need to handle TLS connections in the proxy by modifying them in the client and correcting that in the proxy later. I did try a ready-made solution (Wiremock) but it didn't work as I intended, and I had to back out a fair bit of work. At least if I write it, I will be able to modify it `:-)`. – halfer Apr 11 '17 at 10:14

1 Answers1

12

Yes, I think you have a couple of options here.

HTTPS is specifically designed to thwart "Man in the Middle" attacks and eavesdroppers, which is essentially what you are trying to achieve. You can break some of its assumptions though, and defeat it.

At the start of the SSL connection, 1. the remote server presents its public key and its cert, 2. the client verifies the cert and 3. sends a session key encrypted with the server's public key. For more details, see e.g. An overview of the SSL or TLS handshake

You have two possible ways to circumvent this protection, in the scenario you describe:

1. Rewrite the TLS data, replacing the server's cert and key with your own

Since you control the communications channel, you could substitute the server's public key & cert with one that you control, at step (1). If you then ask the client to skip step (2) using the --no-check-certificate argument to wget, you can then have full access to the encrypted data.

This is how the Fiddler debugging proxy allows access to HTTPS traffic, see https://www.fiddlerbook.com/fiddler/help/httpsdecryption.asp

2. Retrieve the session key from the client application

Since the client app knows the session key, if you could extract it, you could then decrypt the stream. I think this is what you had in mind in the question.

wget itself has no options to allow logging the session key (see "HTTPS (SSL/TLS) Options"), but it does look like its TLS library, "GnuTLS" has a debugging option that will do what you want, see "Debugging and auditing" in the GnuTLS docs:

SSLKEYLOGFILE When set to a filename, GnuTLS will append to it the session keys in the NSS Key Log format. That format can be read by wireshark and will allow decryption of the session for debugging.

Try setting the SSLKEYLOGFILE environment variable to a filename, and see if wget will then log your TLS session keys to that file? You might need to recompile wget with a debug build of GnuTLS. I haven't tried this myself.

Rich
  • 15,048
  • 2
  • 66
  • 119
  • Excellent, number 2 looks very close to what I had in mind, but number 1 is very interesting - thank you! I need to do some reading here... – halfer Apr 09 '17 at 18:48