Get visitors real IP in PHP app running behind Kubernetes LoadBalancer service in GCE

Question

I am having issues getting the visitors real IP in my PHP app. I have Kubernetes running in Google Container Engine (master: 1.4.8, node: 1.4.7).

Service definition:

apiVersion: v1
kind: Service
metadata:
    name: app-service
spec:
    type: LoadBalancer # spawning google loadbalancer
    selector:
        name: app # running simple php/nginx container
    ports:
        - port: 80
          targetPort: 80

How can it be that the X-Forwarded-For headers etc. don't get passed through to my php app? I am only getting back the source ip (in php REMOTE_ADDR), which is 10.0.1.1. In Google Cloud I can see the service is using a layer 4 load balancer. Could this be the issue that the real source ip is lost and the X-Forwarded-For header never gets set?

If someone could explain me what is going on, that would be super helpful!

For what its worth, I am using the following nginx configuration in my app container:

location ~ \.php$ {
    fastcgi_pass php-upstream;
    fastcgi_split_path_info ^(.+\.php)(/.*)$;
    include fastcgi_params;
    fastcgi_param  SCRIPT_FILENAME  $realpath_root$fastcgi_script_name;
    fastcgi_param DOCUMENT_ROOT $realpath_root;
}

EDIT I have put my whole application behind CloudFlare, so it is now pointing from CloudFlare http proxy -> GCE Load Balancer. And somehow the X-Forwarded-For headers and all are present! For me it seems like the issue is with the GCE Load Balancer, it is somehow unable to set those headers?

Answer 1

A new feature was added to Kubernetes 1.5:

Due to the implementation of this feature, the source IP for sessions as seen in the target container will not be the original source IP of the client. This is the default behavior as of Kubernetes v1.5. However, starting in v1.5, an optional beta feature has been added that will preserve the client Source IP for GCE/GKE environments. This feature will be phased in for other cloud providers in subsequent releases.

More details are available here and boil down to adding an annotation for services of type loadbalancer:

$ kubectl annotate service loadbalancer service.beta.kubernetes.io/external-traffic=OnlyLocal

This will open a healthcheck port on the node, to verify if service endpoints are available on the node.

More details on how this issue is rolled out to other clusters seems to be available on kubernetes/features issue tracking this feature.

Note: Question seems similar to StackOverflow/Kubernetes-not-preserving-source-ip