0

Update 2022-03-22: I could isolate the problem to the Cluster Autoscaler and not enough pod "slots" left on a node. No solution still. For a detailed analysis see https://github.com/tektoncd/pipeline/issues/4699

I have an EKS cluster running with aws-ebs controller. Now I want to use tekton on this cluster. Tekton has an affinity assistant which should schedule pods on the same node if they share a workspace (aka volumeClaim). Sadly, this does not seem to work for me, as I randomly get an error from my node stating didn't match pod affinity rules and didn't find available persistent volume to bind even there is a volumne existing. After debugging, I found that the persistentVolumes created from time to time are in a different region and on another host than the pod which is spanned.

Does somebody know how to still use “automatic” aws-ebs provisioning with tekton on EKS or something similar, making this work? My fallback solution would be to try S3 as a storage ... but I assume this maybe not the best solution as I have many small files from a git repository. Just provisioning a volume and then running pods only on this one node, is not the solution I would opt for. Even this is better than nothing :)

Any help would be appreciated! If more information is needed, please add a comment and I try to follow up.

-Thanks a lot!

  • Sounds like you need to use larger nodes, since the nodes becomes full of pods? – Jonas Mar 22 '22 at 17:41
  • So far the autoscaler added nodes when there was the need, but this time it seems the request did not work properly. –  Mar 22 '22 at 19:22
  • When you use the Affinity Assistant, the pods for the run cannot run on other nodes - so the autoscaler does not help in this case - you need larger nodes or disable the Affinity Assistant. – Jonas Mar 22 '22 at 19:36
  • What happens if I disable the Affinity Assistant? The documentation is not that extensive on this topic. –  Mar 22 '22 at 19:39
  • The pipeline will take longer time to run. – Jonas Mar 22 '22 at 19:41

1 Answers1

0

You get this events:

0/6 nodes are available: 2 Too many pods

Your node is essentially full. When you use Tekton Pipelines with the Affinity Assistant, all pods in the run will be scheduled to the same pod.

If you want to run Tekton Pipelines with space for just few pods per node, then you should disable the Affinity Assistant in the config map.

Jonas
  • 121,568
  • 97
  • 310
  • 388
  • I need to share the workspace between the pod as the first clones the repo which is then built in the second—can I achieve this without the Affinity Assistant? –  Mar 22 '22 at 19:21
  • Yes, you need to use a workspace and a PVC for that. The Affinity Assistant is only an optimization. – Jonas Mar 22 '22 at 19:34
  • What is the affinity assistant pod then used for? I thought it is really just for the persistence volume allocation. –  May 13 '22 at 11:02
  • It helps you to run two parallel jobs to use a shared workspace at the same time. – Jonas May 13 '22 at 16:07