HTTP appending GET input twice after enabling SSL Certification (HTTPS)

Question

We have upgraded our hosting platform with latest tech stack which includes PHP updates from Version 7.0 to 7.3 and enabled SSL certification.

After the upgrade, one of our user authentication method has failed though, it was working till the hosting platform upgrade.

Here is copy of PHP code - codecheck.php,

<html>
<body>
<?php 
        $header = "Content-Type: application/json";
        header($header);

        $code = $_GET["code"];

        $codelistFile  = "./codelist.txt";
        $codeList = file( $codelistFile, FILE_SKIP_EMPTY_LINES);

        $codelistOutput = sprintf('%s%s', $code, "\r\n" );

        file_put_contents( $codelistFile, $codelistOutput, FILE_APPEND); 
?>
</body>
</html>

Here is result of codelist.txt before the platform upgrade (with PHP version 7.0)

65cafead50f6d205d66f90c74f1683344ca86c8cc60fc0370c278ecb880da5c8
6e85e436538335da64f6e9172bd4191686e591aa390cca69acb9346668a48bd5

Here is result of codelist.txt after the platform upgrade (with PHP version 7.3)

774cad9dd07761fe79db8baa9370a3dd84abca558c73c1f46b39e7c996a26d70?code=774cad9dd07761fe79db8baa9370a3dd84abca558c73c1f46b39e7c996a26d70
f10bb27fb82b0d539d3607012655012764c60794cc656aa6912eccc16d927a82?code=f10bb27fb82b0d539d3607012655012764c60794cc656aa6912eccc16d927a82

Here is value of code repeated along with 'code' text itself hence the value of 'code' does not match when it compared.

Here is what I can see in ssl_access log files, ssl_access.log-20190629:79.1.200.79 - - [29/Jun/2019:07:46:24 +0100] "GET /codelist.php?code=ae21250db8b20cac3b7016e6d36a63de5846d537f032ed841a3e5c9121202cf4?code=ae21250db8b20cac3b7016e6d36a63de5846d537f032ed841a3e5c9121202cf4 HTTP/1.1" 200 19 "-" "Registration"

From this log file, I can see all GET requests to server appending the data twice.

I would expect it would be something like,

example.com/?code=123456789 but not as example.com/?code=123456789?code=123456789

I am very new to PHP and HTTPS stuff, please help to figure out the issue. Thank you.

Here is an update:

As suggested, the issue seems to be more with SSL re-writing,

Here is code from desktop app where the app will connect and check the code with the server,

C++:


CString RegistrationServer::Uri( CString page, CString code )
{
    CString sServer;

    sServer.Format("http://www.mywebsite,com/%s?code=%s", page, code); 

   //Here page=codecheck.php and code = 10;

    return sServer;
}

Here is log when submitted through desktop app,

27.62.66.34 - - [30/Jun/2019:21:55:51 +0100] "GET /codecheck.php?code=10?code=10 HTTP/1.1" 200 - "-" "Hack-o-Matic ver 0.01"

I can simulate the same request through web browser as below,

https://www.mywebsite/codecheck.php?code=10

Here is log when submitted through web browser,

27.62.66.34 - - [30/Jun/2019:21:46:28 +0100] "GET /codecheck.php?code=10 HTTP/1.1" 200 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"

You can see the difference in both the request is http vs https.

When the request is coming from desktop app, the code data is appended twice which uses http.

It appears that changing desktop app to have https will help fix the issue but that's something that we can't do anything with desktop app.

So we have to relay on fix from Server side but our hosting company doesn't seem to understand the problem exactly.

They keep analysing the issue since last 3 days and coming up some fixes like googleapi call fixes but that's not helping to fixing up our real issue.

I'm not sure if I'm missing some better phrases/terms to explain this issue to them better. Please let me know if there is better way to explain the issue to our hosting company.

If nothing working out, Can I ask them to remove SSL certification?

Another Update:

Here is response from our hosting company,

We have this referred to our engineers and they confirmed that this only happens when calling http and not https. You need to use https now since you have enabled SSL.

Latitude-E6540:~$ curl -I http://www.mywebsite.com/codecheck.php?code=10 
HTTP/1.1 301 Moved Permanently
Server: nginx/1.15.8
Date: Mon, 01 Jul 2019 11:03:47 GMT
Content-Type: text/html
Content-Length: 169
Connection: keep-alive
Location: https://www.mywebsite.com/codecheck.php?code=10?code=10
Strict-Transport-Security: max-age=15768000

Our engineers made some tests and they were not able to replicate when they set to https.

Latitude-E6540:~$ curl -I https://www.mywebsite.com/codecheck.php?code=10 
HTTP/1.1 200 OK
Server: nginx/1.15.8
Date: Mon, 01 Jul 2019 11:03:35 GMT
Content-Type: application/json
Connection: keep-alive
Strict-Transport-Security: max-age=15768000

Here is log from server,

213.171.217.184 - - [01/Jul/2019:12:03:35 +0100] "HEAD /usage7.php?code=10 HTTP/1.1" 200 - "-" "curl/7.58.0"

They confirmed that this looks to be something with your local software settings as this only seems to get in the case of "after submitting the requests through browser, HTTP GET data is not appended twice but when the same is submitted through their desktop software, the HTTP GET data is appended twice"

What I wanted to ask you is, from below curl output itself where I can see the code is appended twice when request is made with http, Does this having any clue to spot where the issue resides?

Location: https://www.mywebsite/codecheck.php?code=10?code=10

Are there any errors/notices in the server logs? Not really enough info with the posted code to know, I suspect the issue is elsewhere. I _seriously doubt_ that is at the point of `$_GET`. — Paul T., Jun 29 '19 at 23:51
`$_GET["code"]` means that argument "code" was sent to the html page. Ex. http://example.com/?code=123456789. Here `$_GET["code"]` will equal 123456789. So the value that is written in the file is supposed to be the value of that argument. — Nic3500, Jun 30 '19 at 00:11
I don't find any error in logs. here is what I can see in ssl_access log files, ssl_access.log-20190629:79.1.200.79 - - [29/Jun/2019:07:46:24 +0100] "GET /codelist.php?code=ae21250db8b20cac3b7016e6d36a63de5846d537f032ed841a3e5c9121202cf4?code=ae21250db8b20cac3b7016e6d36a63de5846d537f032ed841a3e5c9121202cf4 HTTP/1.1" 200 19 "-" "Registration" — Thi, Jun 30 '19 at 00:12
Even with this SSL log, I can see the value for code twice and the same written to the file. I would expect it would be something like example.com/?code=123456789 but not as example.com/?code=123456789?code=123456789. — Thi, Jun 30 '19 at 00:19
we have not done anything on client side, it's still the same client before and after the upgrade. The platform OS is linux before and after the upgrade and only the PHP version changed and SSL certification enabled for the site. — Thi, Jun 30 '19 at 00:22
Thi, please review my latest update to my answer. It looks very much like a Server HTTP Host routing issue. — Martin, Jun 30 '19 at 00:35
For the update about doubling the code, `FILE_APPEND` is specified with the write, could that be part of the trouble? — Paul T., Jun 30 '19 at 04:30
As per hosting provider, they say it's because website making http (not https) calls to the google api for css and other things. They have advised that I need to ensure that any code that relates to http is switched to https. Here is google api call which I can see in our every html page which using http, Do you think changing it to https will fix up the issue in all of our html pages? — Thi, Jun 30 '19 at 09:57
This does not appear to be a PHP problem in the first place. Go check what URL rewriting might be set up, this appears to go wrong in some place. — 04FS, Jul 01 '19 at 07:49
The rewriting is configured in your http server, you need to check there (however I am not sure how the double code would result in the changed output you have described) — eckes, Jul 01 '19 at 10:51
Thank you for the update. I am not very experienced with C++ coding so have added the tag to the question. The method that needs to be employed is that the `RegistrationServer::Uri` checks if the `code=` value exists already in the URL string, only adding it if it does not already exist. Good luck! — Martin, Jul 01 '19 at 12:33
`If nothing working out, Can I ask them to remove SSL certification?` I'm afraid this is absolutely not the way this problem should be resolved!! — Martin, Jul 01 '19 at 12:34
There's not enough information here. There's either a bug in your desktop app, or a bug in your redirect rules, and neither are shown. Since the problem started when you enabled SSL, and there's evidently some feature somewhere on the server redirecting http:// requests to https://, I'd say the problem is likely to be there. If you don't know what this feature/mechanism is, you need to have a conversation with the sysadmin who does. A full trace of the original HTTP request might help narrow it down though. Certainly this is unlikely to have anything to do with PHP. — Lightness Races in Orbit, Jul 01 '19 at 12:40
@Lightness Races in Orbit , the desktop app uses http and I think, updating desktop app to https will fix the issue but we can't change desktop app now. We have no access to how redirect is configured in our as we are in shared hosting and all those configuration stuffs are controlled by hosting company. — Thi, Jul 01 '19 at 13:42
@Thi: Then you should talk to the hosting company. The HTTP request trace would still help. Can you provide it? (hint: Wireshark!) — Lightness Races in Orbit, Jul 01 '19 at 13:46

score 2 · Answer 1 · edited Jun 20 '20 at 09:12

How to solve PHP upgrade errors:

Post-event, how to find, diagnose and fix errors apparently caused by PHP updates?

1) Check your scripts for PHP Errors.

2) Check changes to your php.ini file caused by updates.
Depending on your system and upgrade method, the php.ini file may be adjusted or even a new default one. Read the Migration Notes to see if this may apply to you. You will need to review and explore what's changed. Also manually compare your reserved/backup php.ini with the current/new live one.

3) Read the PHP Migration notes for each version you have upgraded into and then out of
(These are best done from oldest to newest).

4) Read the corresponding PHP Changelog(s) and search this text (it's loooong) for the functions you've found be failing in step (1).

For your specific instances; your code is of a very low quality (you are sending HTTP heders after you are sending HTML code) so the issue may well be caused by PHP upgrading an already existing error from E_WARNING to E_ERROR, or similar.

Low quality code is most easily fixed by turing on error_reporting(E_ALL); either in the scripts or in the php.ini and reading the resulting error logs.

Good Luck.

Update

Even with this SSL log, I can see the value for code twice and the same written to the file. I would expect it would be something like
example.com/?code=123456789 but not as
example.com/?code=123456789?code=123456789.

The sign you have two ? means you should be exploring the code that sets the code= value, please update your question with this information, how is code set?

Your issue may be with your HTTP Host routing, Apache, Nginx, etc., your HTTP Host is possibly double loading, first the HTTP_ page and then secondly redirecting on to the HTTPS page with the original query string appended, thus appending twice.

I think one or both of the above is where your problem lies.

Update 2:

Comment by Thi:

Here is what my hosting company responded, "as per our engineers the cause of the logs is de to the website making http (not https) calls to the google api for css and other things. They have advised that you need to ensure that any code that relates to http is switched to https." - There is below line in all of our html pages and have changed it to https but it didn't help <link href="fonts.googleapis.com/…" rel="stylesheet" type="text/css">

This relates to what I reference above about checking your server routing for HTTP and HTTPS protocols.

Solutons:

1) Update all your outgoing links to https:// (or simply //) so:

<link href="//fonts.googleapis.com/..." rel="stylesheet" type="text/css">

will always connect securely, if loaded securely.

2) Use Content Security Policy (CSP) Upgrade Insecure Requests flag to do just that; to force all http:// links within your website to be turned into https:// links by the client browser.

In your .htaccess, or equivilant file:

Content-Security-Policy: upgrade-insecure-requests;

However, insecure calls to 3rd party resources will NOT be the cause of your code block being appended to your URL twice.

Here is what my hosting company responded, "as per our engineers the cause of the logs is de to the website making http (not https) calls to the google api for css and other things. They have advised that you need to ensure that any code that relates to http is switched to https." --- There is below line in all of our html pages and have changed it to https but it didn't help — Thi, Jun 30 '19 at 19:12
@Thi thanks for the info. Please can you do as requested and **edit** your question to include the code that generates the `?code=` string. Thank you. — Martin, Jul 01 '19 at 09:27
Thanks for the update, I'd added more details to the question. As stated, the issue looks like something to do with HTTPS rewriting but I'm here helpless to fix up the issue as I've to relay on hosting company — Thi, Jul 01 '19 at 10:49
@Thi: So you pay a company to host your site, but have no way to talk to them about how that's done and configured? I hope you're not paying too much... — Lightness Races in Orbit, Jul 01 '19 at 13:46

HTTP appending GET input twice after enabling SSL Certification (HTTPS)

1 Answers1

How to solve PHP upgrade errors:

Update

Update 2: