ABP Framework HttpApi.Host failure in K8S deployment with nginx reverse proxy

Question

I have deployed the ABP framework in a Kubernetes cluster.

Following deployments are present:

• Redis
• MSSql
• AuthServer
• HttpApi.Host
• Nginx ingress / reverse proxy with https termination, no encryption within the cluster.

Hence, the AuthServer, HttpApi.Host are listening on port 80 / http, and the nginx is listening on https. The configuration / Helm values are following:

use-forwarded-headers: "true"
use-proxy-protocol: "true"
use-gzip: "true"

Everything well so far, and after deployment I can enter the Swagger and Authorize:

This can be confirmed when check the AuthServer logs:

[19:12:39 INF] CORS policy execution successful.
[19:12:39 INF] The request URI matched a server endpoint: Token.
[19:12:39 INF] The token request was successfully extracted: {
  "grant_type": "authorization_code",
  "code": "[redacted]",
  "client_id": "foobar_Swagger",
  "redirect_uri": "https://api.staging.foobar.io/swagger/oauth2-redirect.html"
}.
[19:12:39 INF] The token request was successfully validated.

However, now I would like to use the Swagger to ensure that the connection towards the endpoints are working correctly, so I try the first GET endpoint:

As you can see, there is a 500 response. Looking the logs at the HttpApi.Host pod:

--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
[19:12:52 ERR] Connection id "0HMSQJRK38VSM", Request id "0HMSQJRK38VSM:00000005": An unhandled exception was thrown by the application.
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: '[PII of type 'System.String' is hidden. For more details, see https://aka.ms/IdentityModel/PII.]'.
 ---> System.IO.IOException: IDX20804: Unable to retrieve document from: '[PII of type 'System.String' is hidden. For more details, see https://aka.ms/IdentityModel/PII.]'.
 ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
 ---> System.Net.Sockets.SocketException (104): Connection reset by peer
   --- End of inner exception stack trace ---

So it seems that the HttpApi can not connect to the AuthServer, since it is http according to the stacktrace above "The SSL connection could not be established, see inner exception"

It seems like the HttpApi.Host are connecting via http inside the cluster but the AuthServer does not like it?

Please give me some advices here, thanks in advance.

Answer 1

First of all, you can disable PII to see detailed information which is helpful to see the exact error. Simply add:

Microsoft.IdentityModel.Logging.IdentityModelEventSource.ShowPII = true;

to the ConfigureServices method of your service.

The problem you are having is probably related to finding the correct discovery endpoint (to validate the token) in the internal network. Your application is authenticated against the Authorization URL but it is trying to reach the same URL inside the k8s network to reach the discovery endpoint. However it needs to make a request to internal service name instead of the DNS.

We have added OIDC support to the Volo.Abp.Swashbuckle package after v7.4-rc.1 that you can define different metadata (discovery endpoint) address than the Issuer (Authorization URL).

You can use the .AddAbpSwaggerGenWithOidc() method. Keep on mind that discovery url should be the real DNS and the authority should be the k8s service name.

Edit: You can check the docs at: ABP Swagger Integration and you can find a sample at: ABP Swagger Demo

You can set metadata as: https://myauthserver.staging.mycompany.com/.well-known/openid-configuration. After v7.4-rc.2, setting metadata as https://myauthserver.staging.mycompany.com without full path should be enough.

You can also set the RequireHttpsMetadata to false since the discovery enpoint for token validation will be done to an endpoint something like http://myprojectname-st-authserver instead of a real DNS like: https://myauthserver.staging.mycompany.com

Answer 2

May you can set AuthServer.Authority from https to http, and then set AuthServer.RequireHttpsMetadata to false.

Cause you use the self sign cert, and you do not trust the cert's private key, then the http client can't verify the cert.