How to fix "process apparently never started in ..." error in Jenkins pipeline?

Question

I am getting the strange error below in my Jenkins pipeline

[Pipeline] withDockerContainer
acp-ci-ubuntu-test does not seem to be running inside a container
$ docker run -t -d -u 1002:1006 -u ubuntu --net=host -v /var/run/docker.sock:/var/run/docker.sock -v /home/ubuntu/.docker:/home/ubuntu/.docker -w /home/ubuntu/workspace/CD-acp-cassandra -v /home/ubuntu/workspace/CD-acp-cassandra:/home/ubuntu/workspace/CD-acp-cassandra:rw,z -v /home/ubuntu/workspace/CD-acp-cassandra@tmp:/home/ubuntu/workspace/CD-acp-cassandra@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** quay.io/arubadevops/acp-build:ut-build cat
$ docker top 83d04d0a3a3f9785bdde3932f55dee36c079147eb655c1ee9d14f5b542f8fb44 -eo pid,comm
[Pipeline] {
[Pipeline] sh
process apparently never started in /home/ubuntu/workspace/CD-acp-cassandra@tmp/durable-70b242d1
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
[Pipeline] }
$ docker stop --time=1 83d04d0a3a3f9785bdde3932f55dee36c079147eb655c1ee9d14f5b542f8fb44
$ docker rm -f 83d04d0a3a3f9785bdde3932f55dee36c079147eb655c1ee9d14f5b542f8fb44
[Pipeline] // withDockerContainer

The corresponding stage in Jenkins pipeline is

    stage("Build docker containers & coreupdate packages") {
        agent {
            docker {
                image "quay.io/arubadevops/acp-build:ut-build"
                label "acp-ci-ubuntu"
                args "-u ubuntu --net=host -v /var/run/docker.sock:/var/run/docker.sock -v $HOME/.docker:/home/ubuntu/.docker"
              }
          }
          steps {
              script {
                 try {
                    sh "export CI_BUILD_NUMBER=${currentBuild.number}; cd docker; ./build.sh; cd ../test; ./build.sh;"
                    ciBuildStatus="PASSED"
                 } catch (err) {
                    ciBuildStatus="FAILED"
                 }
              }
          }
      }

What could be the reasons why the process is not getting started within the docker container? Any pointers on how to debug further are also helpful.

Answer 1

This error means the Jenkins process is stuck on some command.

Some suggestions:

• Upgrade all of your plugins and re-try.
• Make sure you've the right number of executors and jobs aren't stuck in the queue.
• If you're pulling the image (not your local), try adding alwaysPull true (next line to image).
• When using agent inside stage, remove the outer agent. See: JENKINS-63449.
• Execute org.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true in Jenkins's Script Console to debug.
• When the process is stuck, SSH to Jenkins VM and run docker ps to see which command is running.
• Run docker ps -a to see the latest failed runs. In my case it tried to run cat next to custom CMD command set by container (e.g. ansible-playbook cat), which was the invalid command. The cat command is used by design. To change entrypoint, please read JENKINS-51307.
• If your container is still running, you can login to your Docker container by docker exec -it -u0 $(docker ps -ql) bash and run ps wuax to see what's doing.
• Try removing some global variables (could be a bug), see: parallel jobs not starting with docker workflow.

Answer 2

The issue is caused by some breaking changes introduced in the Jenkins durable-task plugin v1.31.

Source:

https://issues.jenkins-ci.org/browse/JENKINS-59907 and https://github.com/jenkinsci/durable-task-plugin/blob/master/CHANGELOG.md

Solution: Upgrading the Jenkins durable-task plugin to v1.33 resolved the issue for us.

Answer 3

I had this same problem and in my case, it was related to the -u <user> arg passed to the agent. In the end, changing my pipeline to use -u root fixed the problem.

---

In the original post, I notice a -u ubuntu was used to run the container:

docker run -t -d -u 1002:1006 -u ubuntu ... -e ******** quay.io/arubadevops/acp-build:ut-build cat

I was also using a custom user, one I've added when building the Docker image.

agent {
  docker {
    image "app:latest"
    args "-u someuser"
    alwaysPull false
    reuseNode true
  }
}
steps {
  sh '''
    # DO STUFF
  '''
}

Starting the container locally using the same Jenkins commands works OK:

docker run -t -d -u 1000:1000 -u someuser app:image cat
docker top <hash> -eo pid,comm
docker exec -it <hash> ls  # DO STUFF

But in Jenkins, it fails with the same "process never started.." error:

$ docker run -t -d -u 1000:1000 -u someuser app:image cat
$ docker top <hash> -eo pid,comm
[Pipeline] {
[Pipeline] unstash
[Pipeline] sh
process apparently never started in /home/jenkins/agent/workspace/branch@tmp/durable-f5dfbb1c

For some reason, changing it to -u root worked.

agent {
  docker {
    image "app:latest"
    args "-u root"      # <=-----------
    alwaysPull false
    reuseNode true
  }
}

Answer 4

If you have upgraded the durable-task plugin to 1.33 or later and it still won't work, check if there's an empty environment variable configured in your pipeline or stored in the Jenkins configuration (dashed) and remove it:

Answer 5

In addition to kenorb's answer:

• Check permissions inside the container you are running in and the Jenkins directory on the build host.

I am running custom Docker containers and after several hours of debugging, I found that after trying to execute what Jenkins was trying to execute inside the running container (by exec into the container, running echo "$(ps waux)", and executing those sh -c commands one by one). I found Jenkins couldn't create the log file inside the container due to a mismatch in UID and GID.

Answer 6

If you are running Jenkins inside of Docker and using a DinD container for Jenkins running Docker jobs, make sure you mount your Jenkins data volume to /var/jenkins_home in the service providing the Docker daemon. The log creation is actually being attempted by the daemon, which means the daemon container needs access to the volume with the workspace that is being operated on.

Example snippet for docker-compose.yml:

services:
  dind:
    container_name: dind-for-jenkins
    privileged: true
    image: docker:stable-dind
    volumes:
      - 'jenkins-data:/var/jenkins_home'

Answer 7

This has eaten my life! I tried every imaginable solution on at least 10 SO posts, and in the end it was because my pipeline had spaces in its name. :|

So I changed "let's try scripting" with "scripts_try" and it just worked.

Answer 8

Building a Jenkins job which runs within a Docker container, and ran into this same error. The version of the Durable-Task plugin is at v1.35, so that was not the issue. My issue was ... my job was trying to run a chmod -R 755 *.sh command, and the active user within the container did not have sufficient permissions to execute chmod against those files. Would have expected Jenkins to fail the job here, but launching the container using an ID which did have permissions to run the chmod command got past this error.

Answer 9

In my case the problem was related to using Kubernetes agents:

agent {
    kubernetes {
        cloud 'cloud'
        namespace 'namespace'
        yamlFile '.ci/build-pod.yaml'
    }
}

I needed to add runAsUser to pod definition, so that Jenkins assumes a user that has the required permissions within the image:

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: jnlp
      image: custom-build-image:latest
      args:
        - jenkins-slave
      tty: false
      workingDir: /home/jenkins
  securityContext:
    runAsUser: 1000