8

I am configuring ArgoCD, and all the pods are Running state like below.

$ kubectl get pods -n argocd -o wide
NAME                              READY   STATUS    RESTARTS   AGE    IP               NODE        NOMINATED NODE   READINESS GATES
argocd-application-controller     1/1     Running   0          138m   172.16.195.218   worker-1    <none>           <none>
argocd-applicationset-controller  1/1     Running   0          138m   172.16.195.216   worker-1    <none>           <none>
argocd-dex-server                 1/1     Running   0          138m   172.16.59.213    worker-2    <none>           <none>
argocd-notifications-controlle    1/1     Running   0          138m   172.16.195.217   worker-1    <none>           <none>
argocd-redis                      1/1     Running   0          138m   172.16.59.214    worker-2    <none>           <none>
argocd-repo-server                1/1     Running   0          46m    172.16.59.216    worker-2    <none>           <none>
argocd-server                     1/1     Running   0          138m   172.16.59.215    worker-2    <none>           <none>

But when I create a new app, ArgoCD shows the following error.

Unable to create application: application spec for test is invalid: InvalidSpecError: repository not accessible: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup argocd-repo-server on 10.96.0.10:53: read udp 172.16.59.215:50498->10.96.0.10:53: i/o timeout"

This error occurs when not only using private git repository but also public github repository. And curl to git repository from worker-2 node is OK.

Maybe it seems that the connection from argocd-server to argocd-repo-server is timed out. But I cannot understand why this problem occurs.

My Environment:

  • Rocky Linux 8.5
  • kubeadm (1 Master Node + 2 Worker Nodes)
  • Calico as CNI
Ellisein
  • 878
  • 6
  • 17

4 Answers4

4

According to your logs you've probably faced IP connectivity issue with DNS server. Apparently, due to inability to resolve domain names argocd-server cannot initiate connection to argocd-repo-server.

A general plan to you how to troubleshoot such issues:

  1. be sure that your DNS pod is up & running in your K8s cluster.
  2. be sure your pod has IP connectivity with K8s DNS server.
  3. be sure your pod has access to UDP/53 of K8s DNS server.
  4. be sure DNS entry that your pod's asking for is resolved, so argocd-repo-server is resolved to an IP address.

See that pretty explained guide for more details - kubernetes.io.

mdraevich
  • 116
  • 6
1

I had the same issue and after hours debugging, I found it is because I installed core-dns after I install argocd.

So I just reset the whole cluster, and installed core-dns firstly then argocd, the issue was gone.

Ron
  • 6,037
  • 4
  • 33
  • 52
0

I don't have calico based config but had the same issue when I started messing up with the argo server service.

With a clean setup, I used NodePort configuration for server instead of LoadBalancer. Without further tunnelling the portal together with argo cli, both worked with the registry connection. Would recommend to try the same.

pebkm
  • 1
0

I was having the same problem, I switched to the namespace of argocd and issued kubectl rollout restart deployment to restart all pods and that solved the problem.

ssaid
  • 65
  • 8