How to force pull docker images in DC OS?

Question

For docker orchestration, we are currently using mesos and chronos to schedule job runs.

Now, we dropped chronos and try to set it up via DCOs, using mesos and metronome.

In chronos, I could activate force pulling a docker image via its yml config:

container:
  type: docker
  image: registry.example.com:5001/the-app:production
  forcePullImage: true

Now, in DC/OS using metronome and mesos, I also want it to force it to always pull the up-to-date image from the registry, instead of relying on its cached version.

Yet the json config for docker seems limited:

"docker": {
  "image": "registry.example.com:5001/the-app:production"
},

If I push a new image to the production tag, the old image is used for the job run on mesos.

Just for the sake of it, I tried adding the flag:

"docker": {
  "image": "registry.example.com:5001/my-app:staging",
  "forcePullImage": true
},

yet on the put request, I get an error:

http PUT example.com/service/metronome/v1/jobs/the-app < app-config.json 

HTTP/1.1 422 Unprocessable Entity
Connection: keep-alive
Content-Length: 147
Content-Type: application/json
Date: Fri, 12 May 2017 09:57:55 GMT
Server: openresty/1.9.15.1

{
    "details": [
        {
            "errors": [
                "Additional properties are not allowed but found 'forcePullImage'."
            ],
            "path": "/run/docker"
        }
    ],
    "message": "Object is not valid"
}

How can I achieve that the DC OS always pulls the up-to-date image? Or do I have to always update the job definition via a unique image tag?

Answer 1

The Metronome API doesn't support this yet, see https://github.com/dcos/metronome/blob/master/api/src/main/resources/public/api/v1/schema/jobspec.schema.json

Answer 2

As this is currently not possible I created a feature request asking for this feature.

---

In the meantime, I created workaround to be able to update the image tag for all the registered jobs using typescript and request-promise library.

Basically I fetch all the jobs from the metronome api, filter them by id starting with my app name, and then change the docker image, and issue for each changed job a PUT request to the metronome api to update the config.

Here's my solution:

const targetTag = 'stage-build-1501'; // currently hardcoded, should be set via jenkins run
const app = 'my-app';
const dockerImage = `registry.example.com:5001/${app}:${targetTag}`;

interface JobConfig {
    id: string;
    description: string;
    labels: object;
    run: {
        cpus: number,
        mem: number,
        disk: number,
        cmd: string,
        env: any,
        placement: any,
        artifacts: any[];
        maxLaunchDelay: 3600;
        docker: { image: string };
        volumes: any[];
        restart: any;
    };
}

const rp = require('request-promise');

const BASE_URL = 'http://example.com';
const METRONOME_URL = '/service/metronome/v1/jobs';
const JOBS_URL = BASE_URL + METRONOME_URL;

const jobsOptions = {
    uri: JOBS_URL,
    headers: {
        'User-Agent': 'Request-Promise',
    },
    json: true,
};

const createJobUpdateOptions = (jobConfig: JobConfig) => {
    return {
        method: 'PUT',
        body: jobConfig,
        uri: `${JOBS_URL}/${jobConfig.id}`,
        headers: {
            'User-Agent': 'Request-Promise',
        },
        json: true,
    };
};

rp(jobsOptions).then((jobs: JobConfig[]) => {
    const filteredJobs = jobs.filter((job: any) => {
        return job.id.includes('job-prefix.'); // I don't want to change the image of all jobs, only for the same application
    });

    filteredJobs.map((job: JobConfig) => {
        job.run.docker.image = dockerImage;
    });

    filteredJobs.map((updateJob: JobConfig) => {
        console.log(`${updateJob.id} to be updated!`);
        const requestOption = createJobUpdateOptions(updateJob);
        rp(requestOption).then((response: any) => {
            console.log(`Updated schedule for ${updateJob.id}`);
        });
    });

});

Answer 3

I had a similar problem where my image repo was authenticated and I could not provide the necessary auth info using the metronome syntax. I worked around this by specifying 2 commands instead of the directly referencing the image.

docker --config /etc/.docker pull
docker --config /etc/.docker run

Answer 4

I think the "forcePullImage": true should work with the docker dictionary.

Check: https://mesosphere.github.io/marathon/docs/native-docker.html

Look at the "force pull option".