5

We are moving our build process outside our OpenShift 3.11 cluster and have noticed that when wanting to rollback to an older deployment the docker image used is not the old one but the newest one known to OpenShift.

I saw for images built by an OpenShift source build that this worked, but for our externally built images which are pushed to the openshift external docker registry and then oc new-app registry/foo/bar:master (which creates an image stream for this) this does not work.

Can this be achieved at all?

Do I need extra metadata on my docker image?

Do I need to tell OpenShift more about my image?

Note: Opened https://github.com/openshift/origin/issues/23754 in Origin - the open source project below the Redhat stuff - about this

Thorbjørn Ravn Andersen
  • 73,784
  • 33
  • 194
  • 347

2 Answers2

3

Use immutable tags.

As you are addressing your images with a tag called master, I assume the you are currently using mutable tags that get updated with every image build. With this, you can just “roll back” by reverting your commit and re-building your old image, but then you still get an updated base image and different metadata.

If you design your whole release process around immutable tags you can do exactly what you want. To achieve that, use one of the following approaches:

(1) Address your image via image digest. In that case, you need a registry that keeps your images even if they are not tagged. For example, Quay.io does not keep untagged images and you can only pull the latest version. To avoid that, just add a dummy tag, e.g. the image digest.

(2) Release your images with semantic versioning, e.g. by using Angular commit messages and adding semantic-release to your build pipeline. Make sure that the base image you are using is immutable, too. To deploy the images to the our cluster, use a GitOps workflow with a tool like Flux CD. Now you can easily roll back by reverting a commit in your GitOps repo.

(3) Reconsider whether you need rollback at all. Alternatively, you can use canary deployments with Istio that comes with OpenShift 4. Instead of rolling back, you just deploy two versions of your app at the same time, then you use Istio to route the traffic to the one or the other until you are happy.

Hendrik M Halkow
  • 2,208
  • 15
  • 24
  • We are using an OpenShift external docker registry. We may also use a Habor instance instead. OpenShift 4 is not an option. If I understand you correctly, you essentially say that I need to manually (or using a non-OpenShift tool) keep track of the images to deploy and hence therefore manually deploy them so the deployment descriptor contains the unique tag given by me that rollback needs to locate the old image? Is there no way that OpenShift can do this given sufficient hints at one time? – Thorbjørn Ravn Andersen Sep 16 '19 at 10:18
  • The `sufficient hints` can only be the tags mentioned at [https://docs.openshift.com/container-platform/3.11/architecture/core_concepts/builds_and_image_streams.html]. They include the image digest I mentioned in (1). Have these been created? And does `oc rollback` work? – Hendrik M Halkow Sep 16 '19 at 17:46
  • The image digest tags have not been created explicitly. Unfortunately I do not have time to investigate further before the bounty runs out, så I have awarded it to you on good faith, and I'll come back to it later. Thanks. – Thorbjørn Ravn Andersen Sep 17 '19 at 09:56
0

For rolling back to a specific version you just need to specify the version, newest will be rolled back to automatically if you dont specify deployment version.

DOCs

Image change triggers on the deployment configuration are disabled as part of the rollback to prevent unwanted deployments soon after the rollback is complete. To re-enable the image change triggers:

$ oc deploy <deployment_config> --enable-triggers

To roll back to aspecific version:

$ oc rollback <deployment_config> --to-version=1

ProllyGeek
  • 15,517
  • 9
  • 53
  • 72
  • I am looking at the rollback button in the GUI which for images built by a Build Config can roll back to that specific image, but appears to for images imported from the external registry not to do that but use the newest docker image which for me is an error. Are we talking about the same thing? – Thorbjørn Ravn Andersen Sep 09 '19 at 11:30
  • I think we are talking about same thing except that I invoke command directly from CLI, do you have any traces for what happens when you click GUI button? – ProllyGeek Sep 09 '19 at 11:35
  • 1
    From what I read you are talking about the deployment descriptor version, that works! The problem is that the deployment back then referred to an older image in the image stream than the one being used when the old deploiyment descriptor is being rolled back to. Like "deployment 2" but using "image 5" instead of "image 2". – Thorbjørn Ravn Andersen Sep 09 '19 at 14:32
  • @ThorbjørnRavnAndersen ah ok so wrong image is pulled. something like this issue? https://github.com/openshift/origin/issues/13302 – ProllyGeek Sep 09 '19 at 14:39
  • Doesn't look like it. I will open an issue though, just to see if it _is_ a bug. – Thorbjørn Ravn Andersen Sep 09 '19 at 14:44
  • Well I am interested in this issue, so if you get time we can review together step by step how your process works. – ProllyGeek Sep 09 '19 at 14:45
  • I opened this issue: https://github.com/openshift/origin/issues/23754 No reaction so far. – Thorbjørn Ravn Andersen Sep 16 '19 at 10:19