Kubernetes pods not starting, running behind a proxy

Question

I am running kubernetes on minikube, I am behind a proxy, so I had set the env variables(HTTP_PROXY & NO_PROXY) for docker in /etc/systemd/system/docker.service.d/http-proxy.conf. I was able to do docker pull but when I run the below example

kubectl run hello-minikube --image=gcr.io/google_containers/echoserver:1.4 --port=8080
kubectl expose deployment hello-minikube --type=NodePort
kubectl get pod

pod never starts and I get the error

desc = unable to pull sandbox image \"gcr.io/google_containers/pause-amd64:3.0\"

docker pull gcr.io/google_containers/echoserver:1.4 works fine

Answer 1

I ran into the same problem and am sharing what I learned after making a couple of wrong turns. This is with minikube v0.19.0. If you have an older version you might want to update.

Remember, there are two things we need to accomplish:

1. Make sure kubctl does not go through the proxy when connecting to minikube on your desktop.
2. Make sure that the docker daemon in minikube does go through the proxy when it needs to connect to image repositories.

First, make sure your proxy settings are correct in your environment. Here is an example from my .bashrc:

export {http,https,ftp}_proxy=http://${MY_PROXY_HOST}:${MY_PROXY_PORT}
export {HTTP,HTTPS,FTP}_PROXY=${http_proxy}
export no_proxy="localhost,127.0.0.1,localaddress,.your.domain.com,192.168.99.100"
export NO_PROXY=${no_proxy}

A couple things to note:

1. I set both lower and upper case. Sometimes this matters.
2. 192.168.99.100 is from minikube ip. You can add it after your cluster is started.

OK, so that should take care of kubectl working correctly. Now we have the next issue, which is making sure that the Docker daemon in minikube is configured with your proxy settings. You do this, as mentioned by PMat like this:

$ minikube delete
$ minikube start --docker-env HTTP_PROXY=${http_proxy} --docker-env HTTPS_PROXY=${https_proxy} --docker-env NO_PROXY=192.168.99.0/24

To verify that theses settings have taken, do this:

$ minikube ssh -- systemctl show docker --property=Environment --no-pager

You should see the proxy environment variables listed.

Why do the minikube delete? Because without it the start won't update the Docker environment if you had previously created a cluster (say without the proxy information). Maybe this is why PMat did not have success passing --docker-env to start (or maybe it was on older version of minikube).

Answer 2

I was able to fix it myself. I had Docker on my host and there is Docker in Minikube. Docker in Minukube had issues I had to ssh into minikube VM and follow this post

Cannot download Docker images behind a proxy and it all works nows,

There should be a better way of doing this, on starting minikube i have passed docker env like below, which did not work

minikube start --docker-env HTTP_PROXY=http://xxxx:8080 --docker-env HTTPS_PROXY=http://xxxx:8080 
--docker-env NO_PROXY=localhost,127.0.0.0/8,192.0.0.0/8 --extra-config=kubelet.PodInfraContainerImage=myhub/pause:3.0

I had set the same env variable inside Minikube VM, to make it work

Answer 3

It looks like you need to add the minikube ip to no_proxy:

export NO_PROXY=$no_proxy,$(minikube ip)

see this thread: kubectl behind a proxy