0

I have an active kubernetes cluster inside Minikube VM (using VirtualBox as driver), so for deploying new containers I am able to download the images as this connection is already laid out using istio service, now if I ssh into my minikube VM first of all I am not able to wget https content but http contents are connected after setting proxies and no_proxies but if I want to access any link from inside of my containers, say simple pod with python image and urllib library and I want to connect from inside this pod and then print the contents from any link (eg.http://python.org) I am not able to do so, all I am getting is no route to host error in logs which points to some problem with the connection due to proxies.

def basic():
    import urllib.request
    print("inside basic funtion")
    with urllib.request.urlopen('http://python.org/') as response:
        html = response.read()
        print(html)

this is the python code I am running from inside my container as a pipeline component.

Most recent error I got-

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/urllib/request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/local/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/usr/local/lib/python3.7/http/client.py", line 928, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/local/lib/python3.7/socket.py", line 727, in create_connection
    raise err
  File "/usr/local/lib/python3.7/socket.py", line 716, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<string>", line 11, in <module>
  File "<string>", line 3, in basic
  File "/usr/local/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.7/urllib/request.py", line 525, in open
   response = self._open(req, data)
  File "/usr/local/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/usr/local/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.7/urllib/request.py", line 1345, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/local/lib/python3.7/urllib/request.py", line 1319, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 110] Operation timed out>

I have started minikube as-

minikube start --cpus 6 --memory 12288 --disk-size=80g --extra-config=apiserver.service-account-issuer=api --extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/apiserver.key --extra-config=apiserver.service-account-api-audiences=api --kubernetes-version v1.14.0

after setting the env variables as well.

Update: I created a different container just to check the curl from inside the component as- (I am using kfp libraries for creating containers)

def curl_op(text):
    return dsl.ContainerOp(
        name='curl',
        image='tutum/curl',
        command=['sh', '-c'],
        arguments=['curl -x http://<proxy-server>:<proxy-port> "$0"', text]
    )

so using the above argument I am able to connect to external links, which again makes it certain that i need to create the containers with proxies set.

Chinmay Singh
  • 41
  • 12
  • 1
    Since you are saying that `http` works but `https` doesn't, I think that you have add the port `8443` to your `Service` in the k8s `yml` file. Also, please post you `yml` files. – Felipe Jan 07 '21 at 07:47
  • Did you take a look https://stackoverflow.com/questions/43728355/kubernetes-pods-not-starting-running-behind-a-proxy https://stackoverflow.com/questions/53894597/how-to-use-proxy-in-minikube-on-ubuntu-vm ? – Malgorzata Jan 07 '21 at 12:16
  • Yeah, I have checked these, unfortunately not the problem in my case, I am able to connect to external links from inside my Minikube VM and hence can pull the docker images and such to create pods, the problem is when I try to connect from inside the pods, which in my case are part of pipeline component, the case with urllib is just the basic test case I created, more general test case was that I was trying to download csv data from inside these python image pods and then do some manipulations where again I was not able to download csv file as they are external links. – Chinmay Singh Jan 07 '21 at 14:43
  • @Felipe Actually port and nodes are set correctly as the http problem is particular to wget, I can curl both http and https protocol websites, the problem is only when trying to connect from inside the pods. – Chinmay Singh Jan 08 '21 at 05:50
  • Try to set /proc/sys/net/bridge/bridge-nf-call-iptables file with 1 if its not set already inside the vm. There are some issues about that with minikube. That may help. sudo echo "1" > /proc/sys/net/bridge/bridge-nf-call-iptables – Luffy Jan 08 '21 at 06:37
  • Strangely all these files are blank and read-only by default - /proc/sys/net/bridge/bridge-nf-call-iptables , /etc/kubernetes/manifests (all the yaml files inside- etcd.yaml , kube-controller-manager.yaml , kube-apiserver.yaml , kube-scheduler.yaml). Can't do the chmod too to change the permissions. – Chinmay Singh Jan 08 '21 at 07:54

1 Answers1

0

So for running the above python code I mentioned as pipeline component. I added the environment variables using the os library and this individual piece was able to connect to external networks. Updated python code-

def basic():
    import urllib.request
    import os
    proxy = 'http://proxy-path:port'

    os.environ['http_proxy'] = proxy 
    os.environ['HTTP_PROXY'] = proxy
    os.environ['https_proxy'] = proxy
    os.environ['HTTPS_PROXY'] = proxy

    print("inside basic funtion")
    with urllib.request.urlopen('http://python.org/') as response:
        html = response.read()
        print(html)

And if the docker image is created from scratch without taking help of pipeline library function then we need to just add the env details into our dockerfile the usual way after the base image call-

ENV HTTP_PROXY http://proxy-path:port
ENV HTTPS_PROXY http://proxy-path:port
Chinmay Singh
  • 41
  • 12