Nginx ingress sends private IP for X-Real-IP to services

Question

I have created a Nginx Ingress and Service with the following code:

apiVersion: v1
kind: Service
metadata:
  name: myservice
spec:
  type: ClusterIP
  selector:
    name: my-app
  ports:
    - port: 8000
      targetPort: 8000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myingress
  annotations:
    kubernetes.io/ingress.class: nginx
  labels:
    name: myingress
spec:
  rules:
  - host: mydomain.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: my-service
            port: 
              number: 8000

Nginx ingress installed with: helm install ingress-nginx ingress-nginx/ingress-nginx.

I have also enabled proxy protocols for ELB. But in nginx logs I don't see the real client ip for X-Forwarded-For and X-Real-IP headers. This is the final headers I see in my app logs:

X-Forwarded-For:[192.168.21.145] X-Forwarded-Port:[80] X-Forwarded-Proto:[http] X-Forwarded-Scheme:[http] X-Real-Ip:[192.168.21.145] X-Request-Id:[1bc14871ebc2bfbd9b2b6f31] X-Scheme:[http]

How do I get the real client ip instead of the ingress pod IP? Also is there a way to know which headers the ELB is sending to the ingress?

Are you sure `192.168.21.145` is the ELB IP and not the nginx ingress controller pod IP? — whites11, Jul 12 '21 at 11:30

whites11 · Answer 1 · 2021-07-12T13:24:07.663

One solution is to use externalTrafficPolicy: Local (see documentation).

In fact, according to the kubernetes documentation:

Due to the implementation of this feature, the source IP seen in the target container is not the original source IP of the client. ... service.spec.externalTrafficPolicy - denotes if this Service desires to route external traffic to node-local or cluster-wide endpoints. There are two available options: Cluster (default) and Local. Cluster obscures the client source IP and may cause a second hop to another node, but should have good overall load-spreading. Local preserves the client source IP and avoids a second hop for LoadBalancer and NodePort type services, but risks potentially imbalanced traffic spreading.

If you want to follow this route, update your nginx ingress controller Service and add the externalTrafficPolicy field:

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress-controller
spec:
  ...
  externalTrafficPolicy: Local

A possible alternative could be to use Proxy protocol (see documentation)

Thanks, but it seems that externalTrafficPolicy doesn't work with ClusterIP. I changed ClusterIP to NodePort and added externalTrafficPolicy, But still getting the same result. — Hamid, Jul 12 '21 at 12:44
My mistake. You have to do this in the nginx ingress controller service, not your backend service — whites11, Jul 12 '21 at 13:23

score 6 · Answer 2 · answered Sep 15 '21 at 01:45

The proxy protocol should be enabled in the ConfigMap for the ingress-controller as well as the ELB.

L4 uses proxy-protocol

For L7 use use-forwarded-headers

# configmap.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  use-forwarded-headers: "true"
  use-proxy-protocol: "true"

https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#proxy-protocol

Would this also work with UDP? Because for some reason it does not. I just still get the same "local" IP and not the real public IP — Mohammed Noureldin, Sep 19 '22 at 14:24

score 4 · Answer 3 · answered Oct 08 '21 at 20:14

Just expanding on @strongjz answer.

By default, the Load balancer that will be created in AWS for a Service of type LoadBalancer will be a Classic Load Balancer, operating on Layer 4, i.e., proxying in the TCP protocol level.

For this scenario, the best way to preserve the real ip is to use the Proxy Protocol, because it is capable of doing this at the TCP level.

To do this, you should enable the Proxy Protocol both on the Load Balancer and on Nginx-ingress.

Those values should do it for a Helm installation of nginx-ingress:

  controller:
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    config:
      use-proxy-protocol: "true"
      real-ip-header: "proxy_protocol"

The service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" annotation will tell the aws-load-balancer-controller to create your LoadBalancer with Proxy Protocol enabled. I'm not sure what happens if you add it to a pre-existing Ingress-nginx, but it should work too.

The use-proxy-protocol and real-ip-header are options passed to Nginx, to also enable Proxy Protocol there.

Reference:

It depends if the default helm template is used or the AWS-specific manifest. The AWS specific manifest deploys an NLB by default. https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.3.0/deploy/static/provider/aws/deploy.yaml — strongjz, Aug 05 '22 at 17:32

score 0 · Answer 4 · answered Apr 16 '23 at 09:49

If you used helm to install ingress-nginx then run following command to make ingress-nginx forward client ip

helm upgrade --install ingress-nginx-chart ingress-nginx/ingress-nginx --set controller.service.externalTrafficPolicy=Local

If it works do upvote it, not to help me but to help other seekers who face this problems :)

Nginx ingress sends private IP for X-Real-IP to services

4 Answers4