microk8s Broken K8s Dashboard and Kubeflow Dashboard

Question

I'm using microk8s in an Ubuntu 18.04 LTS VM, 3 cores, 60 GB storage, 12 GB of memory. I followed the instructions from microk8s website here to install it.

$ snap install microk8s --classic --channel=1.18/stable
$ sudo microk8s start
$ sudo microk8s enable dns storage dashboard
$ sudo microk8s enable kubeflow

This took some time but everything got started. I have 110 pods running. When Kubeflow finished installing I got

Operator pods ready.
Waiting for service pods to become ready.

Congratulations, Kubeflow is now available.
The dashboard is available at http://10.64.140.43.xip.io/

    Username: admin
    Password: VIVGI9KB7GEX6JNAQJXZTXD97S42XD

Which did not work. When I put that URL into my browser, I get a white screen. I then tried to access the kubernetes dashboard and was able to login with my token but it says it cannot find any resources. It doesn't show anything in namespaces, nodes, or anything.

The namespace is still at default because I cannot get it to show kubeflow. The documentation says that kubeflow doesn't work for kubernetes 1.18 but if you install an older version of microk8s it doesn't give you the option to enable kubeflow. That also doesn't explain why the dashboard is not working. I don't really know what to check to fix this issue. I tried using the internal IP address for the kubeflow ambassador (kubeflow load balancer) but that didn't seem to work either.

here is more detail on the ambassador service

sudo microk8s kubectl -n kubeflow describe service/ambassador
Name:                     ambassador
Namespace:                kubeflow
Labels:                   juju-app=ambassador
Annotations:              juju.io/controller: 7fe60455-b041-4b5e-8c85-0c8155b0f52d
                          juju.io/model: f635bf6f-c598-4d5c-841b-e798f520a898
                          metallb.universe.tf/address-pool: default
Selector:                 juju-app=ambassador
Type:                     LoadBalancer
IP:                       10.152.183.67
LoadBalancer Ingress:     10.64.140.43
Port:                     ambassador  80/TCP
TargetPort:               80/TCP
NodePort:                 ambassador  30085/TCP
Endpoints:                10.1.40.17:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason        Age                 From                Message
  ----    ------        ----                ----                -------
  Normal  IPAllocated   58m                 metallb-controller  Assigned IP "10.64.140.43"
  Normal  nodeAssigned  2m5s (x5 over 56m)  metallb-speaker     announcing from node "tatooine"

Answer 1

I am running MicroK8s 1.18.2 + Kubeflow on Ubuntu 18.04.3 LTS natively (not in a VM environment) and had the same issue. Luckily I was able to solve the issue and would like to share my solution with you. It turned out that my laptop was not able to resolve the URL to the dashboard (10.64.140.43.xip.io), so I added this address into the hosts file.

My installation routine:

> # Install MicroK8s
> $ sudo snap install microk8s --classic --channel=1.18/stable  

> # Set IP forwarding
> $ sudo apt-get update -qq  
> $ sudo apt-get install -qq -y iptables-persistent  
> $ sudo iptables -P FORWARD ACCEPT

> # Add xip.io adress to hosts   
> $ sudo -- sh -c "echo '10.64.140.43\t10.64.140.43.xip.io' >>
> /etc/hosts"

> # Check if MicroK8s is Running
> $ microk8s status --wait-ready | grep microk8s  
> $ microk8s kubectl get nodes
> $ microk8s kubectl get services

> # Set kubectl alias for MicroK8s 
> $ sudo snap alias microk8s.kubectl kubectl 

> # Activate MicroK8s Add-ons (DNS, Storage, K8s-Dashboard)
> $ microk8s.enable dns storage dashboard

> # (OPTIONAL) Activate GPU Devices for Nvidia GPUs
> $ microk8s.enable gpu 

> # Activate Kubeflow 
> $ microk8s.enable kubeflow    

> # Check if MicroK8s Add-ons are Running 
> $ microk8s status --wait-ready

Depending on your hardware and internet connection, it might take some time until all services are ready. If all services are up, simply open http://10.64.140.43.xip.io in your browser and enter the user (admin) and your generated password.

I hope this works for you as well. :)

Answer 2

So I had the same issue and spent some time on it but found how to do it thanks to the code there https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/enable.kubeflow.sh

After the enable kubeflow, the kubeflow dashboard is available at http://10.64.140.43.xip.io/, but this is a private IP, unreachable outside of your local network of your VM. You cant' fix that with a proxy (I tried) because some URL are absolute (with the domain name).

Let's say your VM is reachable from outside at public IP A.B.C.D. You need to change the kubeflow hostname to A.B.C.D.xip.io

So basically you need to change the hostname in several juju apps:

sudo microk8s juju config dex-auth public-url=http://A.B.C.D.xip.io
sudo microk8s juju config oidc-gatekeeper public-url=http://A.B.C.D.xip.io
sudo microk8s juju config ambassador juju-external-hostname=A.B.C.D.xip.io

I hope this helps,