0

I have a dask cluster running in minikube in a remote VM (Oracle Linux, 64GB, 8 core). The VM connects through proxy to external networks.

I want to access the Jupyter notebook provided by Dask from my browser running in my local Mac.

I would like to understand what are the options available for me to setup this connection.

Here is what I tried:

minikube start --driver=docker --base-image="gcr.io/k8s-minikube/kicbase:v0.0.29" --memory 32768 --cpus 6 
minikube tunnel
helm install mydask dask/dask --set scheduler.serviceType=LoadBalancer --set jupyter.serviceType=LoadBalancer

While this does provide external-ip to the 'mydask-jupyter' service, the IP is not in the same subnet as my VM. Therefore it is not publicly accessible.

Next I tried starting minikube as below:

minikube start --driver=none

However running into other errors:

  initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.23.1:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem": exit status 1
stdout:
[init] Using Kubernetes version: v1.23.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'

stderr:
    [WARNING Firewalld]: firewalld is active, please ensure ports [8443 10250] are open or your cluster may not function correctly
    [WARNING Swap]: swap is enabled; production deployments should disable swap unless testing the NodeSwap feature gate of the kubelet
    [WARNING Hostname]: hostname "my-dask" could not be reached
    [WARNING Hostname]: hostname "my-dask": lookup my-dask on <<IP>> no such host
error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.23.1: output: Trying to pull repository k8s.gcr.io/kube-apiserver ... 
Get "https://k8s.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
...

My Questions:

  1. What configurations are required to fix this?
  2. Is there a better alternative to minikube that will be convenient for this use-case?

Thank you.

John Subas
  • 81
  • 1
  • 11
  • can I ask why? minikube is intended as a testing resource for accurately emulating a full kubernetes environment while providing the debugging & pricing convenience of being on a local machine/laptop. if you actually want to leverage a kubernetes dask cluster, perhaps you want kubernetes, not minikube? if you're actually trying to set up testing, I think this is likely too complex a question for SO, or if not, will require a full [mre]. debugging the issues with dask+jupyter in docker in minikube in VM in proxy in cloud is unlikely to be reproducible and may be a better github issue. – Michael Delgado Jan 23 '22 at 02:12
  • or maybe more directly - you say "Is there a better alternative to minikube that will be convenient for this use-case?", but what is your use case? what's your goal? – Michael Delgado Jan 23 '22 at 02:15
  • The use-case is that in order to avoid configuring a multi-node cluster that is tedious to setup and manage networking and permissions, I want to set up a single node test cluster that can run multiple pods/containers on it's own but only provides some external access by means of a Jupyter notebook. The intent is simply to test Dask. Though I mentioned Dask my use-case is very generic such as: How do I get a minikube container to be accessible by internet through the IP of the VM on which it is running? – John Subas Jan 24 '22 at 01:14
  • sounds good - just checking re: use case. in this case, this sounds like it's not a dask question and just a question about how to expose an app running within minikube to the public cloud given your setup. Maybe check out [this question?](https://stackoverflow.com/questions/40767164/expose-port-in-minikube) – Michael Delgado Jan 24 '22 at 04:00

0 Answers0