4

I work at a large organization that runs hundreds of jobs in a shared Jenkins cluster.

My Jenkins job needs to run integration tests against untrusted code running inside Docker containers. I am fearful that that when my Jenkins job gets terminated abruptly (e.g. job aborted or times out) I will be left with orphaned containers.

I have tried https://github.com/moby/moby/issues/1905 and ulimits does not work for me (this is because it only works for containers that run bash, and I cannot guarantee that mine will do so).

I tried https://stackoverflow.com/a/26351355/14731 but --lxc-conf is not a recognized option for Docker for Windows (this needs to run across all platforms supported by docker).

Any ideas?

Gili
  • 86,244
  • 97
  • 390
  • 689

2 Answers2

2

Well you can have a cleanup command in the first and last step of your job, for example, first clean old deads, then rename the existing contailer to old_$jobname and kill it

  • docker container prune -f
  • docker rename $jobname old$jobname
  • docker kill old$jobname do whatever you need

launch your new container - docker run --name $jobname$

martinkenneth
  • 142
  • 1
  • 10
  • What happens if my Jenkins job is killed, and is never run again? Point is, I can't guarantee that I will be around long enough to shut down the container myself. – Gili Nov 02 '17 at 21:41
  • in that case, run the container with -i (interactive) instead of -d (detached or daemon I guess ) and launch your tests as the command like: `docker run -i --name go_web_tests ${JOB_NAME} sh test.sh` the container will die when process complete, then, if you are worried of dead containers, you can clean up on the next build – martinkenneth Nov 03 '17 at 13:51
  • Unfortunately, I don't think this will work in my case. My unit test depends on multiple containers (e.g. mysql and wordpress) so I can only launch one of them in interactive mode. – Gili Nov 07 '17 at 16:19
2

By the looks of things, people are handling this outside of docker.

They are adding Jenkins post-build steps that clean up orphaned docker containers on aborted or failed builds.

See Martin Kenneth's build script as an example.

Gili
  • 86,244
  • 97
  • 390
  • 689