.Net Core 7 Identity Server 6 JWT Auth Bearer Token invalid after server restarts

Question

This should be so simple, but I'm pulling my hair out trying to figure out why this wont work. I'm at the point where I am considering abandoning Duende Identity Server and moving to something else like Auth0.

I have Identity server working correctly, but for the life of me I can't get the token to stay valid after a server restart. I recently switched from using cookies to authenticate to JWT tokens since I am now supporting more than one front end application with this server. Using Cookies this worked totally fine with the following code in startup:

            services.AddDataProtection()
                .SetApplicationName("App-Name")
                .PersistKeysToAzureBlobStorage(connectionString, containerName, blobName);

I'm using .Net core 7 and Duende Identity Server 6 and Postgres with Entity Framework as a database. The data protection code is still there, but no longer seems to be working with JWT tokens.

My DB Context looks like this:

    public class ApplicationDbContext : ApiAuthorizationDbContext<ApplicationUser>
    {}

and my relevant startup code looks like this:

            services.AddDbContext<ApplicationDbContext>(
                options =>
                    options.UseNpgsql(
                            Configuration.GetConnectionString("db-connection-string"),
                            x =>
                            {
                                x.MigrationsHistoryTable("__efmigrationshistory", "public");
                                x.MigrationsAssembly("API");
                            })
                        .UseLowerCaseNamingConvention());

            StartupDataProtection.Configure(services, Configuration, _webHostEnvironment);

            services.AddDatabaseDeveloperPageExceptionFilter();
            services.AddIdentity<ApplicationUser, IdentityRole>()
                .AddRoleManager<RoleManager<IdentityRole>>()
                .AddEntityFrameworkStores<ApplicationDbContext>()
                .AddDefaultTokenProviders()
                .AddDefaultUI();

            services.AddIdentityServer(
                    options =>
                    {
                        if(_webHostEnvironment.IsProduction())
                        {
                            options.LicenseKey = Configuration.GetSection("IdentityServer").GetValue<string>("LicenseKey");
                        }
                    })
                .AddApiAuthorization<ApplicationUser, ApplicationDbContext>(
                    options =>
                    {
                        options.IdentityResources["openid"].UserClaims.Add(JwtClaimTypes.Name);
                        options.ApiResources.Single().UserClaims.Add(JwtClaimTypes.Name);
                        options.IdentityResources["openid"].UserClaims.Add(JwtClaimTypes.Role);
                        options.ApiResources.Single().UserClaims.Add(JwtClaimTypes.Role);
                    });

            services.AddAuthentication(
                    options =>
                    {
                        options.DefaultAuthenticateScheme = JwtBearerDefaults.AuthenticationScheme;
                        options.DefaultChallengeScheme = JwtBearerDefaults.AuthenticationScheme;
                        options.DefaultScheme = JwtBearerDefaults.AuthenticationScheme;
                    })
                .AddIdentityServerJwt()
                .AddJwtBearer(
                    options =>
                    {
                        options.Authority = Configuration["JWT:ValidIssuer"];
                        options.SaveToken = true;
                        options.TokenValidationParameters = new TokenValidationParameters
                        {
                            ValidateIssuer = true,
                            ValidateAudience = true,
                            ValidateIssuerSigningKey = true,
                            ValidAudience = Configuration["JWT:ValidAudience"],
                            ValidIssuer = Configuration["JWT:ValidIssuer"],
                            IssuerSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(Configuration["JWT:Secret"])),
                            //This makes roles work as claims
                            NameClaimType = JwtClaimTypes.Name,
                            RoleClaimType = JwtClaimTypes.Role
                        };
                        //This configures JWT Tokens to work with SignalR
                        options.Events = new JwtBearerEvents
                        {
                            OnMessageReceived = context =>
                            {
                                StringValues accessToken = context.Request.Query["access_token"];

                                // If the request is for our hub...
                                PathString path = context.HttpContext.Request.Path;
                                if(!string.IsNullOrEmpty(accessToken) &&
                                   path.StartsWithSegments("/hubs"))
                                {
                                    // Read the token out of the query string
                                    context.Token = accessToken;
                                }

                                return Task.CompletedTask;
                            }
                        };
                    });

My understanding is that the call to AddApiAuthorization<ApplicationUser, ApplicationDbContext>() internally calls .AddOperationalStore<TContext>() which should setup the application to use the EF Database as the key store. And I believe this is my source of confusion.

I am not 100% sure if it's set up to use EF Core with Postgres to use the Key store or not. I am also not 100% sure if the DataProtection code that I had previously with Cookies is still required or not.

And the final issue: This seems to work totally fine locally. It's just on my test and production servers that I am getting the invalid token every time I restart the application. If it matters - in production/test the application is running in docker containers and in Kubernetes in Azure Kubernetes Service.

I know this is a lot of information, but if anyone has any ideas I'd love to hear it. For now, every time I deploy everyone's tokens stop working with the error www-authenticate Bearer error="invalid_token" coming back in the header.

Answer 1

OK first off, I've never used Identity Server, but I might be able to point you in a direction that helps.

Firstly go to your JSON Web Key Set endpoint which should be hosted at /.well-known/jwks.json. Alternatively you might be able to navigate to the JSON Web Key Set endpoint from the OpenID Configuration endpoint at /.well-known/openid-configuration.

In the JSON Web Key Set endpoint, there should one or more keys. Each of the keys in the list should have a Key IDs a.k.a. kid.

Now authenticate in your identity server to obtain an access token.

Copy and debug the token to https://jwt.io (or you could just base64 decode it, but jwt.io is probably easier).

There should be a kid header claim in your JWT.

There should be a matching kid in the JSON Web Key Set endpoint. Essentially, this is how resource servers validate JWTs - generally they'll lazily retrieve they public key information (from the JSON Web Key Set endpoint) associated to the Key ID of the JWT.

Now restart your identity server.

Open up the JSON Web Key Set endpoint again (after the restart) and check that the kid from your access token (that was minted before the identity server restart) is still in the list. If not, this would be the problem - for some reason your signing key kid must be getting reset when the identity server is restarted...

EDIT - I just noticed you're using symmetric signing keys (i.e. IssuerSigningKey = new SymmetricSecurityKey...). Personally I've always used asymmetric keys, so I'm not actually sure what will be in JSON Web Key Set endpoint with symmetric signing keys... I would assume that the symmetric signing keys isn't there in clear text or there's some kind of authorization required to access this page... otherwise it's a serious security defect. Anyways, I suspect the issue you're trying to solve is probably with the key ID and not the key itself...