Jenkins build Docker daemon not running on kubernetes cluster

Question

I'm new to Dev Ops and trying to build my code using Jenkins and upload it on the kubernetes cluster which is hosted on the IBM cloud. But when I run the Docker run command in the Jenkins script I keep getting this error. Installed all the latest plugins and

+ docker run hello-world
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.

Here's the Jenkins script which I don't know is right or wrong. I searched a couple of articles and question. They all were not giving me a positive result. Tried this Jenkins Docker in Docker on GCP/Kubernetes.

podTemplate(
    cloud: "kubernetes",
    label:"mypod",
    containers:[
        containerTemplate(
            name:"nodejs",
            image:"node",
            ttyEnabled:true,
            command:'cat',
            alwaysPullImage: true,
            resourceRequestCpu: '200m',
            resourceRequestMemory: '100Mi',
        ),
        containerTemplate(
            name:"docker",
            image:"",
            ttyEnabled:true,
            command:'cat',
            alwaysPullImage: true,
            resourceRequestCpu: '200m',
            resourceRequestMemory: '100Mi',
        ),
        containerTemplate(
            name:"helm",
            image:"alpine/helm",
            ttyEnabled:true,
            command:'cat',
            alwaysPullImage: true,
            resourceRequestCpu: '200m',
            resourceRequestMemory: '100Mi',
        )
    ],
    volumes:[
        hostPathVolume(hostPath: '/var/run/docker.sock', mountPath: '/var/run/docker.sock')
    ]
){
    node("mypod"){
        def commitId
        stage ("Fetch repo"){
            checkout scm
            commitId = sh(script: 'git rev-parse --short HEAD',returnStdout:true).trim()
        }
        stage ("Installing packages"){
            container("nodejs"){
                sh 'npm install'
            }
        }
        stage ("Build"){
            container("nodejs"){
                sh 'npm run build'
            }
        }
        def repository
        stage ("Docker"){
            container('docker'){
                docker.withRegistry("https://us.icr.io/api","ibm-cloud"){
                    sh "docker run hello-world"
                }
            }
        }
        stage ("Deploy"){
            container ("helm"){
                sh 'helm version'
            }
        }
    }
}

This is the deployment file of my Jenkins pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-uat
  labels:
    app: jenkins
    chart: jenkins-5.0.18
    release: jenkins-uat
    heritage: Helm
spec:
  selector:
    matchLabels:
      app: jenkins
      release: jenkins-uat
  template:
    metadata:
      labels:
        app: jenkins
        chart: jenkins-5.0.18
        release: jenkins-uat
        heritage: Helm
    spec:      
      securityContext:
        fsGroup: 1001
      containers:
        - name: jenkins
          image: docker.io/bitnami/jenkins:2.235.1-debian-10-r7
          imagePullPolicy: "IfNotPresent"
          securityContext:
            runAsUser: 1001
          env:
            - name: JENKINS_USERNAME
              value: "hlpjenkin"
            - name: JENKINS_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: jenkins-uat
                  key: jenkins-password
            - name: JENKINS_HOME
              value: "/opt/bitnami/jenkins/jenkins_home"
            - name: DISABLE_JENKINS_INITIALIZATION
              value: "no"
          ports:
            - name: http
              containerPort: 8080
            - name: https
              containerPort: 8443
          livenessProbe:
            httpGet:
              path: /login
              port: http
            initialDelaySeconds: 180
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 6
          readinessProbe:
            httpGet:
              path: /login
              port: http
            initialDelaySeconds: 30
            periodSeconds: 5
            timeoutSeconds: 3
            successThreshold: 1
            failureThreshold: 3
          resources:
            limits: {}
            requests:
              cpu: 300m
              memory: 512Mi
          volumeMounts:
            - name: jenkins-data
              mountPath: /bitnami/jenkins
      volumes:
        - name: jenkins-data
          persistentVolumeClaim:
            claimName: jenkins-uat

Answer 1

So I have installed Jenkins as a container in my k8s cluster :) and managed to reproduce the same error:

docker run --rm hello-world
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.

How to fix it.

In order to fix you definitely need to have access to the Docker on your K8s Node. Very good explanation of how that works was given by jpetazzo.

Technically you do not need "Docker in Docker" (that is the "full Docker setup" in Docker). You just want to be able to run Docker from your CI system, while this CI system itself is in a container. So that that your CI system like Jenkins can start containers.

So when you start your CI container (Jenkins or other), instead of hacking something together with Docker-in-Docker, start it with the access to /var/run/docker.sock on main host.

Below you can see the part of my Yamls that a responsible for that.
That allows my CI container to have access to the Docker socket, and CI container will, therefore, be able to start containers.

Except that instead of starting “child” containers, it will start “sibling” containers, but that is perfectly fine in our context.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
...
spec:
  template:
    spec:
      containers:
      - env:
        volumeMounts:
        - mountPath: /var/run/docker.sock
          name: docker-sock
      ...
      volumes:
      - hostPath:
          path: /var/run/docker.sock
          type: File
        name: docker-sock

So in my case, the pipeline I've created produces the following logs:

####pipeline

pipeline {
    agent any

    stages     {
        stage('second_stage'){
            steps{
                sh 'docker run --rm hello-world'
            }
        }
    }
}

####logs

+ docker run --rm hello-world

Hello from Docker!

Answer 2

I had this similar problem and I fixed this by enabling my user to be part of docker group and execute docker. This happens when your user is unable to find docker.

You need follow the post installation steps after installing docker.

1. Create the docker group sudo groupadd docker

2. Add your user to the docker group. sudo usermod -aG docker $USER

3. Restart docker service sudo service docker stop and sudo service docker start

4. Exit/Logout from current user and Log back in to verify

Answer 3

So I see a couple of problems in your podtemplate.

First of all, for docker container, you didn't specify any image. You should use a docker image in this container. Create your own container with docker installed in it or you can use https://hub.docker.com/r/volaka/ibm-cloud-cli this image. It includes ibmcloud cli, kubectl, helm and docker for kubernetes automation on IBM Cloud.

Second thing is that I think it is related with Jenkins Kubernetes. Once you create a podTemplate in a pipeline, even if you edit the template, sometimes the changes are not seen in the latest pod. I had this kind of error so I deleted and recreated the pipeline with the edited podTemplate. I am saying this because even if you have declared your volume binding in podTemplate, I don't see it in the created pod's yaml. So I recommend you to recreate your pipeline with your final podTemplate.

I have created a detailed walkthrough about how to install, configure and automate Jenkins pipelines on IBM Kubernetes Service. Feel free to check it. https://volaka.gitbook.io/jenkins-on-k8s/