ASP.NET: Sign URL with public and private keys from files

Question

I want to update Google AMP cache, so I need to sign an URL as described here.

My main issue: I'm struggling massively with how I should get my certificates/keys and how to include them in my code below. I just can't find any all covering instructions for Windows and IIS.

I have been reading these posts:

1. Using /update-cache requests to update AMP pages
2. How can I sign a file using RSA and SHA256 with .NET?

I don't want to use my computer's certificate store as described in the second post. I'd rather use files on disk for both public and private keys. From my production server IIS, I exported my certificate to a .pfx file, from which I then extracted the private key and certificate using the instructions on the bottom of this site. The server.key contains -----BEGIN RSA PRIVATE KEY-----, which If I use that to load into the privateCert variable in the code below throws error Cannot find the requested object.

What I have gotten from my SSL provider: www_example_com.crt, www_example_com.p7b, the certificate code (see below).

Steps I've taken:

1. I created test-public-key.cer by opening www_example.com.crt and using the Copy to file wizard to copy it to a base64 encoded .cer file.
2. I saved certificate code I received from my SSL provider as file test-private-key.cer.

When I run the following code I get error

Object reference not set to an instance of an object.

on line key.FromXmlString(privateCert.PrivateKey.ToXmlString(True))

Dim publicCert As X509Certificate2 = New X509Certificate2("C:\temp\_certificates\test-public-key.cer")
Dim privateCert As X509Certificate2 = New X509Certificate2("C:\temp\_certificates\test-private-key.cer")

'Round-trip the key to XML and back, there might be a better way but this works
Dim key As RSACryptoServiceProvider = New RSACryptoServiceProvider
key.FromXmlString(privateCert.PrivateKey.ToXmlString(True))

'Create some data to sign
Dim data() As Byte = System.Text.Encoding.Unicode.GetBytes(signatureUrl)

'Sign the data
Dim sig() As Byte = key.SignData(data, CryptoConfig.MapNameToOID("SHA256"))

Dim AMPURLSignature As String = EncodeTo64(sig.ToString)

'Lastly, the verification can be done directly with the certificate's public key without need for the reconstruction as we did with the private key:
key = CType(publicCert.PublicKey.Key, RSACryptoServiceProvider)
If Not key.VerifyData(data, CryptoConfig.MapNameToOID("SHA256"), sig) Then
    Throw New CryptographicException
End If

EncodeTo64 function

Public Shared Function EncodeTo64(ByVal toEncode As String) As String
    Dim toEncodeAsBytes As Byte() = System.Text.ASCIIEncoding.ASCII.GetBytes(toEncode)
    Dim returnValue As String = System.Convert.ToBase64String(toEncodeAsBytes)
    Return returnValue
End Function

certificate code

-----BEGIN CERTIFICATE-----
MIIF (...) DXuJ
-----END CERTIFICATE-----

UPDATE 1

I was able to generate a mysite.pfx file by following the export steps on this page. In the wizard I made sure to select "Yes, export the private key" and I added a password. The rest of the steps I followed verbatim.

I then also ran these commands:
openssl pkcs12 -in mysite.pfx -nocerts -out private-key-VPS.pem
penssl pkcs12 -in mysite.pfx -clcerts -nokeys -out certificate-VPS.pem

I ended up with private-key-VPS.pem and a certificate-VPS.pem files

I'm aware the steps to get the mysite.pfx are slightly different than what @CodeFuller described, but so far so good?

I then added code:

Dim certificate As X509Certificate2 = New X509Certificate2("C:\temp\_certificates\prodserverV2\mysite.pfx", "mypwd")
Dim rsa As RSA = certificate.GetRSAPrivateKey()
Dim data() As Byte = System.Text.Encoding.Unicode.GetBytes(signatureUrl)
Dim sig() As Byte = rsa.SignData(data, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1)

Dim AMPURLSignature As String = EncodeTo64(sig.ToString)

But there I get 4 errors:

GetRSAPrivateKey' is not a member of 'X509Certificate2'.
'SignData' is not a member of 'RSA'.
'HashAlgorithmName' is not declared. It may be inaccessible due to its protection level.
'RSASignaturePadding' is not declared. It may be inaccessible due to its protection level.

UPDATE 2

Thanks @CodeFuller! I targeted framework 4.6.1 and now seem to be 1 step further. I end up with an URL like this: https://www-mysite-com.cdn.ampproject.org/update-cache/c/s/www.mysite.com/articles/270/newarticle1/amp?amp_action=flush&_ts=1522939248&_url_signature=U30zdGVtLkJ5uGVbRQ==. How can I now check if it's a valid URL? I'm checking section "Generate the RSA key" on this page, but I'm confused, since I actually already just coded these steps or not? How can I check whether the URL I now end up with is valid?

UPDATE 3

Ok, I tried your new code. Still get the URL signature verification error. I tried with both the /amp URL of my article and without the /amp part in my URL. Both result in the same URL signature verification error.

I noticed when I print the final URL to my website (see code below), the URL reads:

https://www-toptrouwen-nl.cdn.ampproject.org/update-cache/c/s/www.toptrouwen.nl/artikelen/132/het-uitwisselen-van-de-trouwringen-hoe-voorkom-je-bloopers/amp?amp_action=flush&_ts=1523094395&_url_signature=U3lzdGVrLkn5dGVbXQ==

Notice that where the parameters should be amp_ts and amp_url_signature, they now are _ts and _url_signature respectively.

I tried editing the URL before I do the call to Google by manually renaming parameters _ts and _url_signature to amp_ts and amp_url_signature. But I guess that would result in a difference between the signature and the actual URL. Could it be that somehow my code botches the & character and therefore when I rename these manually later it always result in a signature verification? Do you see what I could fix in my code?

BTW: I tried replacing & with %26 in code-behind before signing the URL but then I get a Google 404 error:

The requested URL /update-cache/c/s/www.toptrouwen.nl/artikelen/132/het-uitwisselen-van-de-trouwringen-hoe-voorkom-je-bloopers/amp?amp_action=flush%26amp_ts=1523094395%26amp_url_signature=U3lzdGVrLkJ1dGVbXQ== was not found on this server. That’s all we know.

My code:

Dim ampBaseUrl As String = "https://www-toptrouwen-nl.cdn.ampproject.org"
Dim signatureUrl As String = "/update-cache/c/s/www.toptrouwen.nl/artikelen/132/het-uitwisselen-van-de-trouwringen-hoe-voorkom-je-bloopers/amp?amp_action=flush&amp_ts=" + tStamp

Dim rsa As RSA = certificate.GetRSAPrivateKey()
Dim data() As Byte = System.Text.Encoding.ASCII.GetBytes(signatureUrl)
Dim sig() As Byte = rsa.SignData(data, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1)

Dim AMPURLSignature As String = EncodeTo64(sig.ToString)
Dim url As String = ampBaseUrl + signatureUrl + "&amp_url_signature=" + AMPURLSignature

ltStatus.Text = "AMP URL:<a target='_blank' href='" + url + "'>" + url + "</a>"

Also, I'm sure this page exists in Google AMP cache, since I can see and request it in Google's search results on my mobile device.

UPDATE 4

I'm getting close I think and also getting some extra help, see here: https://github.com/ampproject/amphtml/issues/14483#issuecomment-380549060

What I'm trying now to make it easier for others to test as well: Instead of depending on my SSL I now ran the following commands to get a public and private key

openssl genrsa 2048 > private-key.pem
openssl rsa -in private-key.pem -pubout >public-key.pem

I now have files private-key.pem and public-key.pem

I'll rename public-key.pem to apikey.pub and place that on https://example.com/.well-known/amphtml/apikey.pub

I want to take the easiest approach recommended by @CodeFuller and create a .pfx file that I can then load into a variable of type X509Certificate2. When I run this command:
openssl pkcs12 -export -out keys.pfx -inkey private-key.pem -in public-key.pem

I get the error: unable to load certificates

But this time I don't have a .crt file, only a public-key.pem. How can I get a .pfx file? I already checked here.

Answer 1

I saved certificate code I received from my SSL provider as file test-private-key.cer

...

certificate code

-----BEGIN CERTIFICATE-----
MIIF (...) DXuJ
-----END CERTIFICATE-----

The file stored in a format

-----BEGIN CERTIFICATE----- 
MIIF (...) DXuJ 
-----END CERTIFICATE-----

is a certificate which basically contains a public key. It does not contain private key. That's why when you create instance of X509Certificate2 from such file, it's HasPrivateKey property is set to False and PrivateKey returns Nothing, and following statement expectedly throws NullReferenceException:

privateCert.PrivateKey.ToXmlString(True)

In order to sign the data, you need a private key. Private keys have the following format

-----BEGIN RSA PRIVATE KEY-----
MIICXQ...B7Bou+
-----END RSA PRIVATE KEY-----

Such private keys are usually stored in *.key or *.pem (Privacy Enhanced Mail) files. There is no built-in way to load instance of X509Certificate2 from pem file. There are a lot of code samples available how to do it, you will find them in the question linked above. However the easiest solution will be to create pfx file (containing both private and public keys). Then you could easily load pfx with corresponding constructor of X509Certificate2.

Creation of pfx file is very easy with SSL tool. If private.key contains private key (-----BEGIN RSA PRIVATE KEY-----) and public.crt contains public key (-----BEGIN CERTIFICATE-----), they you could create pfx file with the following command:

openssl pkcs12 -export -out keys.pfx -inkey private.key -in public.crt

You will be asked to enter the password. This password will also be used when you load the key to X509Certificate2:

Dim certificate As X509Certificate2 = New X509Certificate2("d:\CodeFuller\_days\2018.04.05\keys.pfx", "Password here")

Now HasPrivateKey property is set to True and PrivateKey returns the instance of RSACryptoServiceProvider.

UPDATE

Regarding this code:

'Round-trip the key to XML and back, there might be a better way but this works
Dim key As RSACryptoServiceProvider = New
RSACryptoServiceProvider
key.FromXmlString(privateCert.PrivateKey.ToXmlString(True))

The instance of RSACryptoServiceProvider is actually stored in certificate.PrivateKey so you could avoid above code and replace it with:

Dim provider As RSACryptoServiceProvider = certificate.PrivateKey

However your current SignData() call will not work:

Dim sig() As Byte = key.SignData(data, CryptoConfig.MapNameToOID("SHA256"))

This will throw following exception:

System.Security.Cryptography.CryptographicException: 'Invalid algorithm specified.'

The root cause is that RSACryptoServiceProvider does not support SHA256. That's why I suggest replacing it with RSACng in the following way:

Dim rsa As RSA = certificate.GetRSAPrivateKey()
Dim data() As Byte = System.Text.Encoding.Unicode.GetBytes(signatureUrl)
Dim sig() As Byte = rsa.SignData(data, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1)

Method GetRSAPrivateKey was added to X509Certificate2 class only since .NET Framework 4.6, so consider upgrading if you get following error:

error BC30456: 'GetRSAPrivateKey' is not a member of 'X509Certificate2'

UPDATE 2 (regarding URL validation)

The page you referenced contains openssl command for verifying the signature:

openssl dgst -sha256 -signature signature.bin -verify public-key.pem url.txt

However in your case it will be just a sanity check, because you have just generated the signature with a valid procedure. So answering your question:

How can I check whether the URL I now end up with is valid?

The best check is just to send request to AMP Cache with signed URL and check the response. I haven't used AMP Cache before but I believe it will respond with some HTTP error if the signature is invalid.

UPDATE 3 (regarding failed signature verification)

Update AMP Content page contains following command line for signing the URL:

echo -n >url.txt '/update-cache/c/s/example.com/article?amp_action=flush&amp_ts=1484941817' cat url.txt | openssl dgst -sha256 -sign private-key.pem >signature.bin

I have compared result signature built by this command with the signature calculated by the code from my answer. It turned out that they differ. I have researched the possible root cause and found that the problem is caused by the way we get URL bytes. Currently it's:

Dim data() As Byte = System.Text.Encoding.Unicode.GetBytes(signatureUrl)

However we should sign the URL represented in ASCII. So replace above line with:

Dim data() As Byte = System.Text.Encoding.ASCII.GetBytes(signatureUrl)

Now both signatures, from openssl utility and the code above, matches. If after the fix you still get URL signature verification error from Google AMP, then the problem will be with the input URL passed for signing.

UPDATE 4 (Getting PFX from private and public keys)

Generate private key:

openssl genrsa 2048 > private-key.pem

Generate public key:

openssl rsa -in private-key.pem -pubout > public-key.pem

Create certificate signing request:

openssl req -new -key private-key.pem -out certificate.csr

Create certificate:

openssl x509 -req -days 365 -in certificate.csr -signkey private-key.pem -out public.crt

You will be asked here for some certificate fields, e.g. Country Name, Organization Name, etc. It does not really matter which values you use, since you need this certificate for test purposes.

Create pfx file:

openssl pkcs12 -export -out keys.pfx -inkey private-key.pem -in public.crt

Answer 2

This is the code I use for signing a string with a certificate from disk. I modified it for use in an url. Maybe it will be of help to you.

public string signString(string originalString)
{
    //load the certificate from disk
    X509Certificate2 cert = loadCertificateFromFile("myCert.pfx", "myPassword");

    //get the associated CSP and private key
    using (RSACryptoServiceProvider csp = (RSACryptoServiceProvider)cert.PrivateKey)
    using (SHA1Managed sha1 = new SHA1Managed())
    {
        //hash the data
        UnicodeEncoding encoding = new UnicodeEncoding();

        byte[] data = encoding.GetBytes(originalString);

        byte[] hash = sha1.ComputeHash(data);

        //sign the hash
        byte[] signed = csp.SignHash(hash, CryptoConfig.MapNameToOID("SHA1"));

        //convert to base64 and encode for use in an URL
        return Server.UrlEncode(Convert.ToBase64String(signed));

        //or return a regular string
        //return Encoding.Default.GetString(signed);
    }
}


public bool verifyString(string originalString, string signedString)
{
    //convert back from base64 and url encoded string
    byte[] signature = Convert.FromBase64String(Server.UrlDecode(signedString));

    //or a regular string
    //byte[] signature = Encoding.Default.GetBytes(signedString);

    //load the certificate from disk
    X509Certificate2 cert = loadCertificateFromFile("myCert.pfx", "myPassword");

    //get the associated CSP and private key
    using (RSACryptoServiceProvider csp = (RSACryptoServiceProvider)cert.PrivateKey)
    using (SHA1Managed sha1 = new SHA1Managed())
    {
        //hash the data
        UnicodeEncoding encoding = new UnicodeEncoding();

        byte[] data = encoding.GetBytes(originalString);

        byte[] hash = sha1.ComputeHash(data);

        //sign the hash
        byte[] signed = csp.SignHash(hash, CryptoConfig.MapNameToOID("SHA1"));

        //verify the signature
        return csp.VerifyHash(hash, CryptoConfig.MapNameToOID("SHA1"), signature);
    }
}


public X509Certificate2 loadCertificateFromFile(string path, string password)
{
    //get the absolute path
    string absolutePath = Server.MapPath(path);

    //for non-website users, use this
    //string absolutePath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, path);

    if (!string.IsNullOrEmpty(password))
    {
        return new X509Certificate2(absolutePath, password, X509KeyStorageFlags.MachineKeySet | X509KeyStorageFlags.Exportable);
    }
    else
    {
        return new X509Certificate2(absolutePath);
    }
}

Usage:

string originalString = "This is a test string";
string signedString = signString(originalString);
bool stringIsCorrect = verifyString(originalString, signedString);

I now see you use VB. You can use this to convert the code http://www.carlosag.net/tools/codetranslator. However it may need some modification afterwards as I don't know the accuracy of the converter.