OpenCSP failed with error code 2148073494

Question

We started seeing this exception occur intermittently when our Azure App Service was moved to an App Service Environment. We're using Identity Server 4, and the exception occurs during token signing. This can be seen in the call stack below.

The signing cert is never saved to a cert store, it's loaded from a database as a byte array:

new X509Certificate2(rawData, password,
                    X509KeyStorageFlags.MachineKeySet |
                    X509KeyStorageFlags.PersistKeySet);

These references appear to be the same issue:

.NET Core X509Certificate2.PrivateKey throws nte_bad_keyset error

https://github.com/dotnet/corefx/issues/2583

Edit: Initially we thought this was something specific to the App Service Env in Azure, but now we're seeing the exception in standard App Services. As a test we created an Azure web job that loads a certificate from a byte array and creates JWT tokens (which require a private key for signing). This repros the error. Also of note is the Azure Web job won't even run when certain flags are passed to the X509Certificate2 ctor (e.g. X509KeyStorageFlags.UserKeySet).

Stack trace is the same each time:

System.Security.Cryptography.CryptographicException:
   at Internal.NativeCrypto.CapiHelper.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
   at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeProvHandle()
   at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeKeyHandle()
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 keySize, CspParameters parameters, Boolean useDefaultKeySize)
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(CspParameters parameters)
   at Internal.Cryptography.Pal.CertificatePal.<>c.<GetRSAPrivateKey>b__59_0(CspParameters csp)
   at Internal.Cryptography.Pal.CertificatePal.GetPrivateKey[T](Func`2 createCsp, Func`2 createCng)
   at Internal.Cryptography.Pal.CertificatePal.GetRSAPrivateKey()
   at Internal.Cryptography.Pal.CertificateExtensionsCommon.GetPrivateKey[T](X509Certificate2 certificate, Predicate`1 matchesConstraints)
   at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_PrivateKey()
   at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_HasPrivateKey()
   at Microsoft.IdentityModel.Tokens.AsymmetricSignatureProvider.HasPrivateKey(SecurityKey key)
   at Microsoft.IdentityModel.Tokens.AsymmetricSignatureProvider..ctor(SecurityKey key, String algorithm, Boolean willCreateSignatures)
   at Microsoft.IdentityModel.Tokens.CryptoProviderFactory.CreateSignatureProvider(SecurityKey key, String algorithm, Boolean willCreateSignatures)
   at Microsoft.IdentityModel.Tokens.CryptoProviderFactory.CreateForSigning(SecurityKey key, String algorithm)
   at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.CreateEncodedSignature(String input, SigningCredentials signingCredentials)
   at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.WriteToken(SecurityToken token)
   at IdentityServer4.Services.DefaultTokenCreationService.CreateJwtAsync(JwtSecurityToken jwt)
   at IdentityServer4.Services.DefaultTokenCreationService.<CreateTokenAsync>d__3.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.Services.DefaultTokenService.<CreateSecurityTokenAsync>d__9.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at IdentityServer4.ResponseHandling.TokenResponseGenerator.<CreateAccessTokenAsync>d__10.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.ResponseHandling.TokenResponseGenerator.<ProcessTokenRequestAsync>d__8.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.ResponseHandling.TokenResponseGenerator.<ProcessAsync>d__6.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.Endpoints.TokenEndpoint.<ProcessTokenRequestAsync>d__6.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.Endpoints.TokenEndpoint.<ProcessAsync>d__5.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.Hosting.IdentityServerMiddleware.<Invoke>d__3.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.Hosting.FederatedSignOutMiddleware.<Invoke>d__6.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.Hosting.AuthenticationMiddleware.<Invoke>d__2.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.<Invoke>d__7.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer4.Hosting.BaseUrlMiddleware.<Invoke>d__2.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ApplicationInsights.AspNetCore.ExceptionTrackingMiddleware.<Invoke>d__4.MoveNext()

score 1 · Accepted Answer · answered Mar 07 '17 at 17:41

1

To resolve the issue we switched from X509 certificates (X509Certificate2) to RSA generated keys (RSACryptoServiceProvider). Identity Server 4 supports both. The exception no longer occurs.

answered Mar 07 '17 at 17:41

J. Andrew Laughlin

1,926
3
21
33

Can you provide a sample of how you got the RSA stuff to work? – mithun_daa Mar 28 '17 at 15:48
2

@mithun_daa essentially we generate an RSA key pair (http://stackoverflow.com/a/42006084/393179), convert the key to a Microsoft.IdentityModel.Tokens.SecurityKey, and hand that to Identity Server 4 when it requests signing keys and validation keys. – J. Andrew Laughlin Mar 30 '17 at 17:25

score 0 · Answer 2 · edited May 23 '17 at 12:02

0

That error means that somewhere after the certificate learned where the key was being stored it got deleted.

Maybe something is calling CngKey.Delete
Maybe something is cloning it into an RSACryptoServiceProvider and setting PersistKeyInCsp to false.
Maybe something is just cleaning up the machine keys directory after the fact.
The other (very rare) known case is where two threads load the PFX in parallel (they don't have to be in the same process) with one of them not having PersistKeySet asserted at import time.

If you're loading and discarding the certificates you really shouldn't set PersistKeySet. Every time you open a PFX another file is created on disk for each private key contained therein... those files get cleaned up normally (as long as the process doesn't abnormally terminate), but PersistKeySet prevents that cleanup from happening.

If nothing in code analysis reveals the two obvious key deletion locations (PersistKeyInCsp=false being the sneaky one) then you'll have to follow the filesystem auditing recommendations in .NET Core X509Certificate2.PrivateKey throws nte_bad_keyset error.

edited May 23 '17 at 12:02

Community

1
1

answered Feb 08 '17 at 01:59

bartonjs

30,352
2
71
111

thanks for the quick response. I've removed the `X509KeyStorageFlags.PersistKeySet` flag, but the exception still occurs. – J. Andrew Laughlin Feb 08 '17 at 23:37
bartonjs: I found another question where you mention that a `X509Certificate2` can be created a with flag to indicate the private key should not be saved to disk. How is this done in .NET Core?http://stackoverflow.com/questions/40536601/azure-web-application-new-x509certificate2-causing-system-security-cryptograph – J. Andrew Laughlin Feb 08 '17 at 23:47
You'll probably need to turn on auditing, then. For ephemeral load, which is only available from nightly/unofficial builds: Reference the current dailies of .NET Core and change your target TFM from netstandardxxx to netcoreapp11. Then `X509KeyStorageFlags.EphemeralKeySet` should become available. – bartonjs Feb 09 '17 at 03:02

Eugene Voynov · Answer 3 · 2017-07-26T13:39:37.467

0

Maybe you didn't give IIS access to your private keys. I had same error, but it was necessary to use X509 Certs instead of keys. I've just granted a permission to read private keys from cert for IIS_IUSRS on MMC. It helps for me.

There's a full answer from @thames: How to give ASP.NET access to a private key in a certificate in the certificate store?

UPD:

Step-by-step guide:

Create/purchase cert with private key
Import cert to "Local Computer" account (not "Current User"). Check "Allow private key to be exported" in Import Wizard.
Open MMC Console and add Certificates Snap-In. Choose "Computer Account".
Find your cert and right-click on it. Choose "All Tasks" > "Manage Private Keys"
Add Read access to your key for user IIS_IUSRS or IIS AppPool\<AppPoolName> (depends on IIS version and config. It's better to use "Advanced" button to show all possible options)
Restart your application

edited Jul 26 '17 at 13:39

answered Jul 26 '17 at 11:46

Eugene Voynov

350
4
11

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - [From Review](/review/low-quality-posts/16834792) – Govind Samrow Jul 26 '17 at 12:34
This link contains only step-by-step guide. I think there's enough info to answer author question. But ok, i'll edit an answer – Eugene Voynov Jul 26 '17 at 13:26
The Question is about identityServer, which using certificates differently to IIS – Michael Freidgeim Sep 07 '17 at 14:06
It helps me with IS4 running on IIS, thats why i suggest this answer – Eugene Voynov Sep 08 '17 at 07:31

OpenCSP failed with error code 2148073494

3 Answers3