OIDC reverse proxy sidecar in kubernetes

Question

I'm trying to secure java applications on kubernetes.

For a simple Springboot app with permitAll, I choose openresty (nginx) with lua-resty-openidc as reverse proxy.

One example that illustrates mostly what I'm trying to do : https://medium.com/@lukas.eichler/securing-pods-with-sidecar-proxies-d84f8d34be3e

It "works" in localhost, but not on kubernetes.

Here's my nginx.conf :

worker_processes  1;

error_log  logs/error.log;
error_log  logs/error.log  notice;
error_log  logs/error.log  debug;

events {
    worker_connections  1024;
}

http {
    lua_package_path '~/lua/?.lua;;';

    resolver ${dns.ip};

    lua_ssl_trusted_certificate /ssl/certs/chain.pem;
    lua_ssl_verify_depth 5;

    lua_shared_dict discovery 1m;
    lua_shared_dict jwks 1m;
    lua_shared_dict introspection 10m;

    lua_shared_dict sessions 10m;

    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;

    keepalive_timeout  65;

    server {
        server_name  localhost;
        listen       80;

        location /OAuth2Client {
            access_by_lua_block {
                local opts = {
                    discovery = "${openam-provider}/.well-known/openid-configuration",
                    redirect_uri = "http://localhost:8080/OAuth2Client/authorization-code/callback",
                    client_id = "myClientId",
                    client_secret = "myClientSecret",
                    scope = "openid profile email",
                }

                local res, err = require("resty.openidc").authenticate(opts)

                if err then
                    ngx.status = 500
                    ngx.say(err)
                    ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
                end

                ngx.req.set_header("Authorization", "Bearer " .. res.access_token)
                ngx.req.set_header("X-USER", res.id_token.sub)
            }

            proxy_pass  http://localhost:8080/OAuth2Client;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

So in local, as my nginx and my springboot app are running on localhost, the redirections are working.

Now, when I deploy it on kubernetes with this, the browser doesn't map localhost with the internal container ip.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: oauth2-client-deployment
spec:
  selector:
    matchLabels:
      app: OAuth2Client
  replicas: 2
  template:
    metadata:
      labels:
        app: OAuth2Client
    spec:
      #hostAliases:
      #- ip: "127.0.0.1"
      #  hostnames:
      #  - "oauth2-client.local"
      containers:
      - name: oauth2-client-container
        image: repo/oauth2-client-springboot:latest
        env:
          - name: SPRING_PROFILES_ACTIVE
            value: dev
        envFrom:
          - secretRef: 
              name: openam-client-secret
          - secretRef: 
              name: keystore-java-opts
        volumeMounts:
          - name: oauth2-client-keystore
            mountPath: "/projected"
            readOnly: true
        ports:
          - containerPort: 8080
      - name: oauth2-sidecar
        image: repo/oauth2-sidecar:latest
        ports:
          - containerPort: 80
      volumes:
        - name: oauth2-client-keystore
          projected:
            sources:
              - secret:
                   name: keystore-secret
                   items:
                    - key: keystore.jks
                      path: keystore.jks
              - secret:
                   name: truststore-secret
                   items:
                    - key: truststore.jks
                      path: truststore.jks
      imagePullSecrets:
      - name: regcred

---

apiVersion: v1
kind: Service
metadata:
  name: oauth2-client-service-sidecar
spec:
  selector:
    app: OAuth2Client
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

So how could I map this localhost ? I don't want the app container to be exposed as there's no security on it, that's why I used the nginx as sidecar and the service only targets it. How to tell nginx to redirect_uri and proxypass to the app container ip ?

And subsidiary question : as nginx doesn't accept env variables, how should I do to make it generic, so apps could provide their own redirect_uri that should be used in nginx.conf ?

Another subsidiary question : the command ngx.req.set_header("Authorization", "Bearer " .. res.access_token) doesn't seem to work, as I don't see any Authorization header in my request in my app...

Answer 1

Configure your service with type ClusterIP to be reachable only internally, then use the fqdn in your services to reach the service without IP dependency.

apiVersion: v1
kind: Service
metadata:
  name: oauth2-client-service-sidecar
spec:
  selector:
    app: OAuth2Client
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

Then use oauth2-client-service-sidecar.<namespacen>.cluster.local in your nginx configuration to reach the service:

proxy_pass  http://oauth2-client-service-sidecar.<namespacen>.cluster.local/OAuth2Client;