Pods terminated immediately from Jenkins to Kubernetes cluster

Question

I have jenkins running on Windows as a service on 127.0.0.1:8080
and minikube running on 192.168.99.101
below is the pipeline for Jenkins job

podTemplate(
  activeDeadlineSeconds: 240,
  name: 'default',
  inheritFrom: 'default',
  nodeSelector: 'key1=value1,kubernetes.io/hostname=minikube',
  containers: [
  containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat' ),
  containerTemplate(name: 'jnlp', image: 'jenkins/jnlp-slave:3.35-2-alpine', args: '${computer.jnlpmac} ${computer.name}')
]) {

node(POD_LABEL) {
    stage('Get a Maven project') {
        git 'https://github.com/<project>.git/'
        container('maven') {
            stage('Build a Maven project') {
                sh 'mvn -B -gs ./settings.xml clean install'
            }
        }
    }
  }
}

The following are the logs for from the kubernetes:

$ kubectl get -a pods --watch
Flag --show-all has been deprecated, will be removed in an upcoming release
NAME                  READY     STATUS    RESTARTS   AGE
default-7r0n8-b2sxx   0/2       Pending   0          0s
default-7r0n8-b2sxx   0/2       Pending   0         0s
default-7r0n8-b2sxx   0/2       ContainerCreating   0         0s
default-7r0n8-b2sxx   2/2       Running   0         0s
default-7r0n8-b2sxx   1/2       Error     0         1s
default-7r0n8-18cv6   0/2       Pending   0         0s
default-7r0n8-18cv6   0/2       Pending   0         0s
default-7r0n8-18cv6   0/2       ContainerCreating   0         0s
default-7r0n8-18cv6   2/2       Running   0         0s
default-7r0n8-18cv6   1/2       Error     0         2s
default-7r0n8-0kz80   0/2       Pending   0         0s
default-7r0n8-0kz80   0/2       Pending   0         0s
default-7r0n8-0kz80   0/2       ContainerCreating   0         0s
default-7r0n8-0kz80   2/2       Running   0         0s
default-7r0n8-0kz80   1/2       Error     0         1s

The following is the console's output from the jenkins job

The pods are getting killed and recreated each time.

Please Can anyone help how to resolve this issue?
Any help would be appreciated.

pod logs

also let me know , if some other fields are required in pipeline .. — elliot alderson, Jul 26 '20 at 10:52
I had some issues to connect pods after upgrade the plugin , once I revert to 1.25.4 it works , maybe it's your issue ? did you enable to connect pods before or it's the first time ? — Mor Lajb, Jul 26 '20 at 12:28
please note that jenkins is running outside local k8 cluster . this is my first time setup — elliot alderson, Jul 26 '20 at 12:29
did you able to run pods before , or it's your first setup ? — Mor Lajb, Jul 26 '20 at 12:30
yes this is my first setup, which minikube version you used? — elliot alderson, Jul 26 '20 at 12:34
i am still facing the same issue , the pods keep on getting created and destroyed , for the 1.25.4 version — elliot alderson, Jul 26 '20 at 12:57
the slave pod comes for a while and vanishes off := Still waiting to schedule task default-llqqz-18ltt’ is offline — elliot alderson, Jul 26 '20 at 13:01
Could you provide logs from thoses poses? `kubectl logs ` or this `job` YAML `kubectl get job -o yaml` ? Is it possible to upgrade Kubernetes version? — PjoterS, Jul 27 '20 at 12:50
theres nothing much in the logs , pods get created and then gets killed immediately. also the agents dont start automatically , i have to manually downloaded .jnlp but it gets killed and another one starts. — elliot alderson, Jul 29 '20 at 16:10

Debashish Hota · Answer 1 · 2020-07-31T07:20:49.687

4

I have the same problem since yesterday. Navigate to Configure Clouds --> kubernetes --> Pod Template. Set Pod Retention as 'Always'. Pods will stay in error status. You can get the pod logs with command kubectl logs --all-containers and see the error reason. In my case Jenkins URL and Jenkins Tunnel is not set properly.

Refer to https://www.youtube.com/watch?v=DCkzdsffeh0

It gives details for jenkins and kubernetes plugin configuration

edited Jul 31 '20 at 07:20

answered Jul 31 '20 at 06:29

Debashish Hota

61
4

Did you setup the environment same as mine?running Jenkins as windows service, outside minikube cluster – elliot alderson Jul 31 '20 at 18:08
No, I have setup jenkins in GKE. jnlp container in the agent pod is not able to connect to Jenkis so new agent pod are getting created. I believe it is related to the jenkins URL. In my case after I provided correct Jenkins URL it worked for me. For jenkins as window service check if setting Jenkins URL to http://:8080 works. – Debashish Hota Aug 01 '20 at 16:42
Please try the following: 1. For the agent pod created from jenkins, type kubectl exec -it <> -- bash. 2. Inside the pod bash, curl http://<>. (e.g http://<>). The curl command which works is the correct Jenkins url. Agent pod will be able to connect back to jenkins. Sorry I could not find time to set up minikube and check the exact url. Also try with http://:8080 in curl command. If you succeed please let me know – Debashish Hota Aug 03 '20 at 17:37
The pod gets created and killed immediately, same thing repeats again and again as shown above. – Abhishek D K Aug 04 '20 at 18:48
the issue is inbound docker-agent is trying to ping jenkins master 127.0.0.1:8080, but this is not possible because containers are isolated from host applications. – elliot alderson Aug 05 '20 at 14:10

Pods terminated immediately from Jenkins to Kubernetes cluster

1 Answers1

Linked