PHP equivalent of C# SHA1 Unicode hashing

Question

I have a running C# app with user authentication. I encrypt the passwords with the SHA1Managed class using Encoding.Unicode. Now, we are developing a new PHP web app and it seems that PHP sha1 method uses ASCII, so the encrypted passwords are slightly different.

I cannot change my C# code as it is running in a number of different servers and all user passwords are already encrypted. This cannot be changed as it would imply asking all users their passwords (which is not viable).

I have read the following solutions, however I am not able to solve my problem with them:

1. C# SHA-1 vs. PHP SHA-1...Different Results? In this one they suggest using ASCIIEncoding. However, as I stated before, this cannot be changed.
2. SHA1: different results for PHP and C# and Generate a PHP UTF-16 SHA1 hash to match C# method these one use base64 encoding, and that is not compatible with my C# encoding (I think)
3. php equivalent to following sha1 encryption c# code this one also uses ASCIIEncoding.

This is my C# code which CANNOT be changed:

byte[] msgBytes = Encoding.Unicode.GetBytes(data);
SHA1Managed sha1 = new SHA1Managed();
byte[] hashBytes = sha1.ComputeHash(msgBytes);

StringBuilder hashRet = new StringBuilder("");
foreach (byte b in hashBytes)
    hashRet.AppendFormat("{0:X}", b);

return hashRet.ToString();

This is the PHP code implemented so far:

$str=mb_convert_encoding($password.$SALT, 'UTF-16LE');
$result=strtoupper(sha1($str));

The result obtained for an example string "1981" in C# D4CEDCADB729F37C30EBF41BC8F2929AF526AD3

In PHP I get: D4CEDCADB729F37C30EBF41BC8F29209AF526AD3

As can be seen, the only difference between each one is the 0 (zero) char. This happens in all strings, but the 0 appears in different locations.

EDIT

As https://stackoverflow.com/users/1839439/dharman stated, results can't be reproduced exactly as my string has a SALT. However, the results should be different even without the SALT.

SOLUTION A user gave a suggestion which solved my problem. Unfortunately, he deleted the answer before I could post that it worked. The problem is in C# when I am parsing the bytes to string in the StringBuilder. The format is {0:X} and any leading 0 in the byte is ignored. For example a 01 would be parsed to 1, a 0D would be parsed to D.

Therefore, I had to iterate over pairs of the result obtained in PHP, and remove leading zeros in there.

The final code in PHP is the following:

$str=mb_convert_encoding($password.$SALT, 'UTF-16LE');
$result=strtoupper(sha1($str));
$array = str_split($result, 2);
foreach ($array as &$valor) {
    $valor = ltrim($valor, '0');
}
unset($valor);
$result = implode($array);

Answer 1

You won't like my answer, but here it is.
First of all the problem is not with PHP, PHP is simple and it is working correctly. You have got a serious bug in your C# code. It needs fixing one way or the other as soon as possible.

This cannot be changed as it would imply asking all users their passwords (which is not viable).

Unfortunately this is what must happen. You can do it without causing mass panic though. I assume your DB has some kind of way of knowing whether the password has been reset by administrators; if not then you need to add such column (preferably of type timestamp). Next time the user logs in, they must provide their password, because you have reset all of them, so take that password and rehash it properly and store the new hash in the database. It would be wise to use a new column for the new hash, or least have a way of identifying the corrected hashes. The column should be at least VARCHAR(60), but best would be to have it VARCHAR(255), which should accommodate all popular hashes.

Of course, SHA1 is not a suitable hashing method for passwords, even if using salts. Since you need to rehash all passwords anyway a good idea would be to switch to bcrypt or a similar secure hashing function.

Because your original application introduced this bug, you require some workarounds. Supporting the buggy behaviour in the new application does not seem like a good idea, so I would advise not to look for workarounds in PHP. You could of course try to mangle the passwords in the same way as you suggested in the question, but that is just going to drag the same bug into the new application without fixing it at all.

Before you start doing anything, you should analyse how many of your passwords are damaged in the database. The correct SHA1 hash length should be 40 characters. If you can find the number of passwords which are less than 40, you are going to know which and how many passwords need to be fixed.

Fixing the passwords is going to be difficult, but definitely worth it.

A note on character encoding:
PHP uses UTF-8 encoding most of the time. It is the most common encoding used on the web, and I would recommend to use that for your strings. When it comes to hashing it doesn't matter, in which encoding the string is, because hashes are calculated on bytes. This is the reason why you are casting your C# string to bytes in UTF-16 with Encoding.Unicode.GetBytes(data).