5

Setup/Environment:

In our PHP application, we sometimes need to make HTTPS requests from PHP to other servers. The setup in question is as follows:

  • We are using PHP stream wrappers for doing the HTTP requests (using Guzzle HTTP). We are doing this because stream wrappers support using the Windows Certficiate Store for certificate verification.
  • The server runs on Windows.
  • We use a proxy on for the HTTPS requests.
  • The firewalls are configured to allow
    1. Access to the servers we are doing our requests to.
    2. Access to all certificate revocation lists relevant for the certificates used.

Our problem:

Sometimes, out of the blue, our HTTPS requests fail, with certificate validation errors. This problem persists, until someone opens a remote desktop session to the server and requests the very same URL we are trying to query in the servers Internet Explorer. After that, our PHP application can do its requests as it should.

Question:

What is the problem here? And what can we do to analyse this further?

Jost
  • 5,948
  • 8
  • 42
  • 72
  • 1
    What kind of certificate validation errors do you have? – Jan Garaj Dec 14 '19 at 19:08
  • 1
    Just this message: `PHP Warning: fsockopen(): SSL operation failed with code 1. OpenSSL Error messages: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed` – Jost Dec 15 '19 at 20:37
  • 1
    Is root CA and all intermediate CAs (maybe proxy's CA) imported in the Root store? Could you provide PHP version and output from `print_r(openssl_get_cert_locations());`? – Jan Garaj Dec 15 '19 at 20:56
  • 1
    All CAs are imported (certificate management by the Domain), and there are no certificates in any of the paths listed in `openssl_get_cert_locations()`, which is correct. The stream wrapper functions use the Windows Certificate store to get the needed certificates, which contains all the necessary certificates, and the user running PHP has all the necessary permissions (at least we thing so) to access them. – Jost Dec 16 '19 at 05:28
  • 1
    Have you informed Guzzle to use https proxy? [DOCS](http://docs.guzzlephp.org/en/latest/request-options.html#proxy) Also [verify](http://docs.guzzlephp.org/en/latest/request-options.html#verify) is a good section to read. –  Dec 16 '19 at 20:53
  • Yes, Guzzle uses the proxy - doing the requests ***works***!!! Only sometimes, after some time, it stops working, and starts working again after requesting the same URL in the Internet Explorer on the server. – Jost Dec 17 '19 at 08:18
  • There are proxies, then there's traffic sniffing/analyzing proxies mitm. Is there a chance that the request is terminating locally in the organization and the sign certificate tied to domain and the windows login process? – vhoang Dec 20 '19 at 13:21

3 Answers3

2

If that were a Guzzle problem, it would happen every time.

However, do try to issue the same HTTPS call using cURL to both verify this is the case, and see if by any chance the cURL request also temporarily clears the issue, just as Internet Explorer does.

But this rather looks like a caching problem - the PHP server request is not able to properly access (priming certificates) the Certificate Store, it is only able to use its services after someone else has gained access, and only as long as the cache does not expire. To be sure this is the case, simply issue calls periodically and mark the time elapsed between user logging in and using IE, and Guzzle calls starting to fail. If I am right, that time will always be the same.

It could be a permission problem (I think it probably is, but what permissions to give to what, that I'm at a loss to guess at). Maybe calls aren't allowed unless fresh CRLs for that URL are available, and PHP doesn't get them). This situation could also either be fixed temporarily by running a IE connection attempt to the same URL from a PowerShell script launched by PHP in case of error, or (more likely, and hopefully) attempting to run said script will elicit some more informative error message.

update

I have looked into how PHP on Windows handles TLS through Guzzle, and nothing obvious came out. But I found an interesting page about TLS/SSL quirks.

More interestingly, I also found out several references on how PHP ends up using Schannel for TLS connections, and how Windows and specifically Internet Explorer have a, let us say, cavalier attitude about interoperability. So I would suggest you try activating the Schannel log on Windows and see whether anything comes out of it.

Additionally, on the linked page there is a reference to a client cache being used, and the related page ends up here ("ClientCacheTime").

LSerni
  • 55,617
  • 10
  • 65
  • 107
  • I can rule out that our HTTP requests don't fetch the CRLs, because we had to add specific firewall rules to allow fetching the CRLs. Before doing that, the requests did not go through at all. But you are right - this looks like a caching problem, but what cache could that be? What permissions would be needed to fill that cache from PHP? I was hoping someone would have had the same problem and knows :/ – Jost Dec 20 '19 at 13:36
  • 1
    The references in your update are just what I was hoping for - we will investigate further an I'll tell how it worked out - probably in 2-3 weeks. Thanks :-) – Jost Dec 22 '19 at 15:55
1

Its not an application problem.

I am 99% sure this is routing problem and in some circumstances packets are dropped in the router. I would look at the network, change the environment or, if possible, do some network sniffing or monitoring.

If You have a decent network infrastructure, You can do SNPM traps for request count and timeout data collecting (from routers and switches) and ingest it in Elastic APM. This would give You quite detailed time-series analysis.

Sakvojage
  • 21
  • 6
  • Doing network sniffing etc. is hard for us, as this occurs on a customers on-premise server to which we don't have access. – Jost Dec 20 '19 at 15:14
1

You can see this https://github.com/guzzle/guzzle/issues/394 verifyis the problem. and if you make the verify to be false that will make your system expose to security attack.

// Use the system's CA bundle (this is the default setting)
$client->request('GET', '/', ['verify' => true]);

// Use a custom SSL certificate on disk.
$client->request('GET', '/', ['verify' => '/path/to/cert.pem']);

// Disable validation entirely (don't do this!).
$client->request('GET', '/', ['verify' => false]);

These are the Request Options and you can see how to do the SSL certificate verification. They describe the issue as the following

Not all system's have a known CA bundle on disk. For example, Windows and OS X do not have a single common location for CA bundles. When setting "verify" to true, Guzzle will do its best to find the most appropriate CA bundle on your system. When using cURL or the PHP stream wrapper on PHP versions >= 5.6, this happens by default. When using the PHP stream wrapper on versions < 5.6, Guzzle tries to find your CA bundle in the following order:

Check if openssl.cafile is set in your php.ini file.

Check if curl.cainfo is set in your php.ini file.

Check if /etc/pki/tls/certs/ca-bundle.crt exists (Red Hat, CentOS, Fedora; provided by the ca-certificates package)

Check if /etc/ssl/certs/ca-certificates.crt exists (Ubuntu, Debian; provided by the ca-certificates package)

Check if /usr/local/share/certs/ca-root-nss.crt exists (FreeBSD; provided by the ca_root_nss package)

Check if /usr/local/etc/openssl/cert.pem (OS X; provided by homebrew)

Check if C:\windows\system32\curl-ca-bundle.crt exists (Windows)

Check if C:\windows\curl-ca-bundle.crt exists (Windows)

The result of this lookup is cached in memory so that subsequent calls in the same process will return very quickly. However, when sending only a single request per-process in something like Apache, you should consider setting the openssl.cafile environment variable to the path on disk to the file so that this entire process is skipped

See also and how to ignore invalid ssl certificate errors in-guzzle 5 and guzzle-request-fails

I_Al-thamary
  • 3,385
  • 2
  • 24
  • 37
  • 1
    The linked ticket is about using cURL, but we are using PHP stream wrappers as backend for Guzzle, so it does not apply here. – Jost Dec 20 '19 at 15:13
  • @Jost I only show the problem and you can see the other link and the same problem in python https://stackoverflow.com/questions/10667960/python-requests-throwing-sslerror. The problem is in SSL certificate – I_Al-thamary Dec 20 '19 at 15:15
  • @Jost I recommend you to read http://docs.guzzlephp.org/en/stable/request-options.html#verify – I_Al-thamary Dec 21 '19 at 04:46