Exposing bare metal kubernetes cluster to internet

Question

I am trying to setup own single-node kubernetes cluster on bare metal dedicated server. I am not that experienced in dev-ops but I need some service to be deployed for my own project. I already did a cluster setup with juju and conjure-up kubernetes over LXD. I have running cluster pretty fine.

$ juju status

Model                         Controller                Cloud/Region         Version  SLA          Timestamp
conjure-canonical-kubern-3b3  conjure-up-localhost-db9  localhost/localhost  2.4.3    unsupported  23:49:09Z

App                    Version  Status  Scale  Charm                  Store       Rev  OS      Notes
easyrsa                3.0.1    active      1  easyrsa                jujucharms  195  ubuntu
etcd                   3.2.10   active      3  etcd                   jujucharms  338  ubuntu
flannel                0.10.0   active      2  flannel                jujucharms  351  ubuntu
kubeapi-load-balancer  1.14.0   active      1  kubeapi-load-balancer  jujucharms  525  ubuntu  exposed
kubernetes-master      1.13.1   active      1  kubernetes-master      jujucharms  542  ubuntu
kubernetes-worker      1.13.1   active      1  kubernetes-worker      jujucharms  398  ubuntu  exposed

Unit                      Workload  Agent  Machine  Public address  Ports           Message
easyrsa/0*                active    idle   0        10.213.117.66                   Certificate Authority connected.
etcd/0*                   active    idle   1        10.213.117.171  2379/tcp        Healthy with 3 known peers
etcd/1                    active    idle   2        10.213.117.10   2379/tcp        Healthy with 3 known peers
etcd/2                    active    idle   3        10.213.117.238  2379/tcp        Healthy with 3 known peers
kubeapi-load-balancer/0*  active    idle   4        10.213.117.123  443/tcp         Loadbalancer ready.
kubernetes-master/0*      active    idle   5        10.213.117.172  6443/tcp        Kubernetes master running.
  flannel/1*              active    idle            10.213.117.172                  Flannel subnet 10.1.83.1/24
kubernetes-worker/0*      active    idle   7        10.213.117.136  80/tcp,443/tcp  Kubernetes worker running.
  flannel/4               active    idle            10.213.117.136                  Flannel subnet 10.1.27.1/24

Entity  Meter status  Message
model   amber         user verification pending

Machine  State    DNS             Inst id        Series  AZ  Message
0        started  10.213.117.66   juju-b03445-0  bionic      Running
1        started  10.213.117.171  juju-b03445-1  bionic      Running
2        started  10.213.117.10   juju-b03445-2  bionic      Running
3        started  10.213.117.238  juju-b03445-3  bionic      Running
4        started  10.213.117.123  juju-b03445-4  bionic      Running
5        started  10.213.117.172  juju-b03445-5  bionic      Running
7        started  10.213.117.136  juju-b03445-7  bionic      Running

I also deployed Hello world application to output some hello on port 8080 inside the pod and nginx-ingress for it to re-route the traffic to this service on specified host.

NAME                               READY   STATUS    RESTARTS   AGE
pod/hello-world-696b6b59bd-fznwr   1/1     Running   1          176m

NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/example-service   NodePort    10.152.183.53   <none>        8080:30450/TCP   176m
service/kubernetes        ClusterIP   10.152.183.1    <none>        443/TCP          10h

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello-world   1/1     1            1           176m

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/hello-world-696b6b59bd   1         1         1       176m

When I do curl localhost as expected I have connection refused, which looks still fine as it's not exposed to cluster. when I curl the kubernetes-worker/0 with public address 10.213.117.136 on port 30450 (which I get from kubectl get all)

$ curl 10.213.117.136:30450
Hello Kubernetes!

Everything works like a charm (which is obvious). When I do

curl -H "Host: testhost.com" 10.213.117.136
Hello Kubernetes!

It works again like charm! That means ingress controller is successfully routing port 80 based on host rule to correct services. At this point I am 100% sure that cluster works as it should.

Now I am trying to access this service over the internet externally. When I load <server_ip> obviously nothing loads as it's living inside own lxd subnet. Therefore I was thinking forward port 80 from server eth0 to this IP. So I added this rule to iptables

sudo iptables -t nat -A PREROUTING -p tcp -j DNAT --to-destination 10.213.117.136 (For the sake of example let's route everything not only port 80). Now when I open on my computer http://<server_ip> it loads!

So the real question is how to do that on production? Should I setup this forwarding rule in iptables? Is that normal approach or hacky solution and there is something "standard" which I am missing? The thing is to add this rule with static worker node will make the cluster completely static. IP eventually change, I can remove/add units to workers and it will stop working. I was thinking about writing script which will obtain this IP address from juju like this:

$ juju status kubernetes-worker/0 --format=json | jq '.machines["7"]."dns-name"'
"10.213.117.136"

and add it to IP-tables, which is more okay-ish solution than hardcoded IP but still I feel it's a tricky and there must be a better way.

As last idea I get to run HAProxy outside of the cluster, directly on the machine and just do forwarding of traffic to all available workers. This might eventually also work. But still I don't know the answer what is the correct solution and what is usually used in this case. Thank you!

Answer 1

So the real question is how to do that on production?

The normal way to do this in a production system is to use a Service.

The simplest case is when you just want your application to be accessible from outside on your node(s). In that case you can use a Type NodePort Service. This would create the iptables rules necessary to forward the traffic from the host IP address to the pod(s) providing the service.

If you have a single node (which is not recommended in production!), you're ready at this point.

If you have multiple nodes in your Kubernetes cluster, all of them would be configured by Kubernetes to provide access to the service (your clients could use any of them to access the service). Though, you'd have to solve the problem of how the clients would know which nodes are available to be contacted...

There are several ways to handle this:

• use a protocol understood by the client to publish the currently available IP addresses (for example DNS),

• use a floating (failover, virtual, HA) IP address managed by some software on your Kubernetes nodes (for example pacemaker/corosync), and direct the clients to this address,

• use an external load-balancer, configured separately, to forward traffic to some of the operating nodes,

• use an external load-balancer, configured automatically by Kubernetes using a cloud provider integration script (by using a Type LoadBalancer Service), to forward traffic to some of the operating nodes.