Say I have a trivial container based on the ubuntu:latest. Now there is a security update and ubuntu:latest is updated in the docker repo .
How would I know my local image and its containers are running behind?
Is there some best practice for automatically updating local images and containers to follow the docker repo updates, which in practice would give you the same niceties of having unattended-upgrades running on a conventional ubuntu-machine
We use a script which checks if a running container is started with the latest image. We also use upstart init scripts for starting the docker image.
#!/usr/bin/env bash
set -e
BASE_IMAGE="registry"
REGISTRY="registry.hub.docker.com"
IMAGE="$REGISTRY/$BASE_IMAGE"
CID=$(docker ps | grep $IMAGE | awk '{print $1}')
docker pull $IMAGE
for im in $CID
do
LATEST=`docker inspect --format "{{.Id}}" $IMAGE`
RUNNING=`docker inspect --format "{{.Image}}" $im`
NAME=`docker inspect --format '{{.Name}}' $im | sed "s/\///g"`
echo "Latest:" $LATEST
echo "Running:" $RUNNING
if [ "$RUNNING" != "$LATEST" ];then
echo "upgrading $NAME"
stop docker-$NAME
docker rm -f $NAME
start docker-$NAME
else
echo "$NAME up to date"
fi
done
And init looks like
docker run -t -i --name $NAME $im /bin/bash
You can use Watchtower to watch for updates to the image a container is instantiated from and automatically pull the update and restart the container using the updated image. However, that doesn't solve the problem of rebuilding your own custom images when there's a change to the upstream image it's based on. You could view this as a two-part problem: (1) knowing when an upstream image has been updated, and (2) doing the actual image rebuild. (1) can be solved fairly easily, but (2) depends a lot on your local build environment/practices, so it's probably much harder to create a generalized solution for that.
If you're able to use Docker Hub's automated builds, the whole problem can be solved relatively cleanly using the repository links feature, which lets you trigger a rebuild automatically when a linked repository (probably an upstream one) is updated. You can also configure a webhook to notify you when an automated build occurs. If you want an email or SMS notification, you could connect the webhook to IFTTT Maker. I found the IFTTT user interface to be kind of confusing, but you would configure the Docker webhook to post to https://maker.ifttt.com/trigger/`docker_xyz_image_built`/with/key/`your_key`.
If you need to build locally, you can at least solve the problem of getting notifications when an upstream image is updated by creating a dummy repo in Docker Hub linked to your repo(s) of interest. The sole purpose of the dummy repo would be to trigger a webhook when it gets rebuilt (which implies one of its linked repos was updated). If you're able to receive this webhook, you could even use that to trigger a rebuild on your side.
A 'docker way' would be to use docker hub automated builds. The Repository Links feature will rebuild your container when an upstream container is rebuilt, and the Webhooks feature will send you a notification.
It looks like the webhooks are limited to HTTP POST calls. You'd need to set up a service to catch them, or maybe use one of the POST to email services out there.
I haven't looked into it, but the new Docker Universal Control Plane might have a feature for detecting updated containers and re-deploying.
One of the ways to do it is to drive this through your CI/CD systems. Once your parent image is built, have something that scans your git repos for images using that parent. If found, you'd then send a pull request to bump to new versions of the image. The pull request, if all tests pass, would be merged and you'd have a new child image based on updated parent. An example of a tool that takes this approach can be found here: https://engineering.salesforce.com/open-sourcing-dockerfile-image-update-6400121c1a75 .
If you don't control your parent image, as would be the case if you are depending on the official ubuntu image, you can write some tooling that detects changes in the parent image tag or checksum(not the same thing, tags are mutable) and invoke children image builds accordingly.
I had the same issue and thought it can be simply solved by a cron job calling unattended-upgrade daily.
My intention is to have this as an automatic and quick solution to ensure that production container is secure and updated because it can take me sometime to update my images and deploy a new docker image with the latest security updates.
It is also possible to automate the image build and deployment with Github hooks
I've created a basic docker image with that automatically checks and installs security updates daily (can run directly by docker run itech/docker-unattended-upgrade ).
I also came across another different approach to check if the container needs an update.
My complete implementation:
Dockerfile
FROM ubuntu:14.04
RUN apt-get update \
&& apt-get install -y supervisor unattended-upgrades \
&& rm -rf /var/lib/apt/lists/*
COPY install /install
RUN chmod 755 install
RUN /install
COPY start /start
RUN chmod 755 /start
Helper scripts
install
#!/bin/bash
set -e
cat > /etc/supervisor/conf.d/cron.conf <<EOF
[program:cron]
priority=20
directory=/tmp
command=/usr/sbin/cron -f
user=root
autostart=true
autorestart=true
stdout_logfile=/var/log/supervisor/%(program_name)s.log
stderr_logfile=/var/log/supervisor/%(program_name)s.log
EOF
rm -rf /var/lib/apt/lists/*
ENTRYPOINT ["/start"]
start
#!/bin/bash
set -e
echo "Adding crontab for unattended-upgrade ..."
echo "0 0 * * * root /usr/bin/unattended-upgrade" >> /etc/crontab
# can also use @daily syntax or use /etc/cron.daily
echo "Starting supervisord ..."
exec /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf
Edit
I developed a small tool docker-run that runs as docker container and can be used to update packages inside all or selected running containers, it can also be used to run any arbitrary commands.
Can be easily tested with the following command:
docker run --rm -v /var/run/docker.sock:/tmp/docker.sock itech/docker-run exec
which by default will execute date command in all running containers and display the results. If you pass update instead of exec it will execute apt-get update followed by apt-get upgrade -y in all running containers
You would not know your container is behind without running docker pull. Then you'd need to rebuild or recompose your image.
docker pull image:tag
docker-compose -f docker-compose.yml -f production.yml up -d --build
The commands can be put in a script along with anything else necessary to complete the upgrade, although a proper container would not need anything additional.
Here is a simplest way to update docker container automatically
Put the job via $ crontab -e:
0 * * * * sh ~/.docker/cron.sh
Create dir ~/.docker with file cron.sh:
#!/bin/sh
if grep -Fqe "Image is up to date" << EOF
`docker pull ubuntu:latest`
EOF
then
echo "no update, just do cleaning"
docker system prune --force
else
echo "newest exist, recompose!"
cd /path/to/your/compose/file
docker-compose down --volumes
docker-compose up -d
fi
Another approach could be to assume that your base image gets behind quite quickly (and that's very likely to happen), and force another image build of your application periodically (e.g. every week) and then re-deploy it if it has changed.
As far as I can tell, popular base images like the official Debian or Java update their tags to cater for security fixes, so tags are not immutable (if you want a stronger guarantee of that you need to use the reference [image:@digest], available in more recent Docker versions). Therefore, if you were to build your image with docker build --pull, then your application should get the latest and greatest of the base image tag you're referencing.
Since mutable tags can be confusing, it's best to increment the version number of your application every time you do this so that at least on your side things are cleaner.
So I'm not sure that the script suggested in one of the previous answers does the job, since it doesn't rebuild you application's image - it just updates the base image tag and then it restarts the container, but the new container still references the old base image hash.
I wouldn't advocate for running cron-type jobs in containers (or any other processes, unless really necessary) as this goes against the mantra of running only one process per container (there are various arguments about why this is better, so I'm not going to go into it here).
Dependency management for Docker images is a real problem. I'm part of a team that built a tool, MicroBadger, to help with this by monitoring container images and inspecting metadata. One of its features is to let you set up a notification webhook that gets called when an image you're interested in (e.g. a base image) changes.
There are a lot of answers here, but none of them suited my needs. I wanted an actual answer to the asker's #1 question. How do I know when an image is updated on hub.docker.com?
The below script can be run daily. On first run, it gets a baseline of the tags and update dates from the HUB registry and saves them locally. From then out, every time it is run it checks the registry for new tags and update dates. Since this changes every time a new image exists, it tells us if the base image has changed. Here is the script:
#!/bin/bash
DATAPATH='/data/docker/updater/data'
if [ ! -d "${DATAPATH}" ]; then
mkdir "${DATAPATH}";
fi
IMAGES=$(docker ps --format "{{.Image}}")
for IMAGE in $IMAGES; do
ORIGIMAGE=${IMAGE}
if [[ "$IMAGE" != *\/* ]]; then
IMAGE=library/${IMAGE}
fi
IMAGE=${IMAGE%%:*}
echo "Checking ${IMAGE}"
PARSED=${IMAGE//\//.}
if [ ! -f "${DATAPATH}/${PARSED}" ]; then
# File doesn't exist yet, make baseline
echo "Setting baseline for ${IMAGE}"
curl -s "https://registry.hub.docker.com/v2/repositories/${IMAGE}/tags/" > "${DATAPATH}/${PARSED}"
else
# File does exist, do a compare
NEW=$(curl -s "https://registry.hub.docker.com/v2/repositories/${IMAGE}/tags/")
OLD=$(cat "${DATAPATH}/${PARSED}")
if [[ "${VAR1}" == "${VAR2}" ]]; then
echo "Image ${IMAGE} is up to date";
else
echo ${NEW} > "${DATAPATH}/${PARSED}"
echo "Image ${IMAGE} needs to be updated";
H=`hostname`
ssh -i /data/keys/<KEYFILE> <USER>@<REMOTEHOST>.com "{ echo \"MAIL FROM: root@${H}\"; echo \"RCPT TO: <USER>@<EMAILHOST>.com\"; echo \"DATA\"; echo \"Subject: ${H} - ${IMAGE} needs update\"; echo \"\"; echo -e \"\n${IMAGE} needs update.\n\ndocker pull ${ORIGIMAGE}\"; echo \"\"; echo \".\"; echo \"quit\"; sleep 1; } | telnet <SMTPHOST> 25"
fi
fi
done;
You will want to alter the DATAPATH variable at the top, and alter the email notification command at the end to suit your needs. For me, I have it SSH into a server on another network where my SMTP is located. But you could easily use the mail command, too.
Now, you also want to check for updated packages inside the containers themselves. This is actually probably more effective than doing a "pull" once your containers are working. Here's the script to pull that off:
#!/bin/bash
function needsUpdates() {
RESULT=$(docker exec ${1} bash -c ' \
if [[ -f /etc/apt/sources.list ]]; then \
grep security /etc/apt/sources.list > /tmp/security.list; \
apt-get update > /dev/null; \
apt-get upgrade -oDir::Etc::Sourcelist=/tmp/security.list -s; \
fi; \
')
RESULT=$(echo $RESULT)
GOODRESULT="Reading package lists... Building dependency tree... Reading state information... Calculating upgrade... 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded."
if [[ "${RESULT}" != "" ]] && [[ "${RESULT}" != "${GOODRESULT}" ]]; then
return 0
else
return 1
fi
}
function sendEmail() {
echo "Container ${1} needs security updates";
H=`hostname`
ssh -i /data/keys/<KEYFILE> <USRER>@<REMOTEHOST>.com "{ echo \"MAIL FROM: root@${H}\"; echo \"RCPT TO: <USER>@<EMAILHOST>.com\"; echo \"DATA\"; echo \"Subject: ${H} - ${1} container needs security update\"; echo \"\"; echo -e \"\n${1} container needs update.\n\n\"; echo -e \"docker exec ${1} bash -c 'grep security /etc/apt/sources.list > /tmp/security.list; apt-get update > /dev/null; apt-get upgrade -oDir::Etc::Sourcelist=/tmp/security.list -s'\n\n\"; echo \"Remove the -s to run the update\"; echo \"\"; echo \".\"; echo \"quit\"; sleep 1; } | telnet <SMTPHOST> 25"
}
CONTAINERS=$(docker ps --format "{{.Names}}")
for CONTAINER in $CONTAINERS; do
echo "Checking ${CONTAINER}"
if needsUpdates $CONTAINER; then
sendEmail $CONTAINER
fi
done
UPDATE: Use Dependabot - https://dependabot.com/docker/
BLUF: finding the right insertion point for monitoring changes to a container is the challenge. It would be great if DockerHub would solve this. (Repository Links have been mentioned but note when setting them up on DockerHub - "Trigger a build in this repository whenever the base image is updated on Docker Hub. Only works for non-official images.")
While trying to solve this myself I saw several recommendations for webhooks so I wanted to elaborate on a couple of solutions I have used.
Use microbadger.com to track changes in a container and use it's notification webhook feature to trigger an action. I set this up with zapier.com (but you can use any customizable webhook service) to create a new issue in my github repository that uses Alpine as a base image.
Track the RSS feed for git commits to an upstream container. ex. https://github.com/gliderlabs/docker-alpine/commits/rootfs/library-3.8/x86_64. I used zapier.com to monitor this feed and to trigger an automatic build of my container in Travis-CI anytime something is committed. This is a little extreme but you can change the trigger to do other things like open an issue in your git repository for manual intervention.
I'm not going into the whole question of whether or not you want unattended updates in production (I think not). I'm just leaving this here for reference in case anybody finds it useful. Update all your docker images to the latest version with the following command in your terminal:
# docker images | awk '(NR>1) && ($2!~/none/) {print $1":"$2}' | xargs -L1 docker pull
Premise to my answer:
Approach
Additionally, the base image can be upgraded/ the container with a complete new base image can be built at regular intervals, as the maintainer feels necessary
Advantages
Above Answers are also correct
There are two Approach
- Use webhooks
- Run script for every specific minute to get fresh pull of docker images
I am just sharing script may be it will helpful for you! You can use it with cronjob, I tried succesfully on OSX
#!/bin/bash
##You can use below commented line for setting cron tab for running cron job and to store its O/P in one .txt file
#* * * * * /usr/bin/sudo -u admin -i bash -c /Users/Swapnil/Documents/checkimg.sh > /Users/Swapnil/Documents/cron_output.log 2>&1
# Example for the Docker Hub V2 API
# Returns all images and tags associated with a Docker Hub organization account.
# Requires 'jq': https://stedolan.github.io/jq/
# set username, password, and organization
# Filepath where your docker-compose file is present
FILEPATH="/Users/Swapnil/Documents/lamp-alpine"
# Your Docker hub user name
UNAME="ur username"
# Your Docker hub user password
UPASS="ur pwd"
# e.g organisation_name/image_name:image_tag
ORG="ur org name"
IMGNAME="ur img name"
IMGTAG="ur img tag"
# Container name
CONTNAME="ur container name"
# Expected built mins
BUILDMINS="5"
#Generally cronjob frequency
CHECKTIME="5"
NETWORKNAME="${IMGNAME}_private-network"
#After Image pulling, need to bring up all docker services?
DO_DOCKER_COMPOSE_UP=true
# -------
echo "Eecuting Script @ date and time in YmdHMS: $(date +%Y%m%d%H%M%S)"
set -e
PIDFILE=/Users/Swapnil/Documents/$IMGNAME/forever.pid
if [ -f $PIDFILE ]
then
PID=$(cat $PIDFILE)
ps -p $PID > /dev/null 2>&1
if [ $? -eq 0 ]
then
echo "Process already running"
exit 1
else
## Process not found assume not running
echo $$
echo $$ > $PIDFILE
if [ $? -ne 0 ]
then
echo "Could not create PID file"
exit 1
fi
fi
else
echo $$ > $PIDFILE
if [ $? -ne 0 ]
then
echo "Could not create PID file"
exit 1
fi
fi
# Check Docker is running or not; If not runing then exit
if docker info|grep Containers ; then
echo "Docker is running"
else
echo "Docker is not running"
rm $PIDFILE
exit 1
fi
# Check Container is running or not; and set variable
CONT_INFO=$(docker ps -f "name=$CONTNAME" --format "{{.Names}}")
if [ "$CONT_INFO" = "$CONTNAME" ]; then
echo "Container is running"
IS_CONTAINER_RUNNING=true
else
echo "Container is not running"
IS_CONTAINER_RUNNING=false
fi
# get token
echo "Retrieving token ..."
TOKEN=$(curl -s -H "Content-Type: application/json" -X POST -d '{"username": "'${UNAME}'", "password": "'${UPASS}'"}' https://hub.docker.com/v2/users/login/ | jq -r .token)
# get list of repositories
echo "Retrieving repository list ..."
REPO_LIST=$(curl -s -H "Authorization: JWT ${TOKEN}" https://hub.docker.com/v2/repositories/${ORG}/?page_size=100 | jq -r '.results|.[]|.name')
# output images & tags
echo "Images and tags for organization: ${ORG}"
echo
for i in ${REPO_LIST}
do
echo "${i}:"
# tags
IMAGE_TAGS=$(curl -s -H "Authorization: JWT ${TOKEN}" https://hub.docker.com/v2/repositories/${ORG}/${i}/tags/?page_size=100 | jq -r '.results|.[]|.name')
for j in ${IMAGE_TAGS}
do
echo " - ${j}"
done
#echo
done
# Check Perticular image is the latest or not
#imm=$(curl -s -H "Authorization: JWT ${TOKEN}" https://hub.docker.com/v2/repositories/${ORG}/${IMGNAME}/tags/?page_size=100)
echo "-----------------"
echo "Last built date details about Image ${IMGNAME} : ${IMGTAG} for organization: ${ORG}"
IMAGE_UPDATED_DATE=$(curl -s -H "Authorization: JWT ${TOKEN}" https://hub.docker.com/v2/repositories/${ORG}/${IMGNAME}/tags/?page_size=100 | jq -r '.results|.[]|select(.name | contains("'${IMGTAG}'")).last_updated')
echo "On Docker Hub IMAGE_UPDATED_DATE---$IMAGE_UPDATED_DATE"
echo "-----------------"
IMAGE_CREATED_DATE=$(docker image inspect ${ORG}/${IMGNAME}:${IMGTAG} | jq -r '.[]|.Created')
echo "Locally IMAGE_CREATED_DATE---$IMAGE_CREATED_DATE"
updatedDate=$(date -jf '%Y-%m-%dT%H:%M' "${IMAGE_UPDATED_DATE:0:16}" +%Y%m%d%H%M%S)
createdDate=$(date -jf '%Y-%m-%dT%H:%M' "${IMAGE_CREATED_DATE:0:16}" +%Y%m%d%H%M%S)
currentDate=$(date +%Y%m%d%H%M%S)
start_date=$(date -jf "%Y%m%d%H%M%S" "$currentDate" "+%s")
end_date=$(date -jf "%Y%m%d%H%M%S" "$updatedDate" "+%s")
updiffMins=$(( ($start_date - $end_date) / (60) ))
if [[ "$updiffMins" -lt $(($CHECKTIME+1)) ]]; then
if [ ! -d "${FILEPATH}" ]; then
mkdir "${FILEPATH}";
fi
cd "${FILEPATH}"
pwd
echo "updatedDate---$updatedDate" > "ScriptOutput_${currentDate}.txt"
echo "createdDate---$createdDate" >> "ScriptOutput_${currentDate}.txt"
echo "currentDate---$currentDate" >> "ScriptOutput_${currentDate}.txt"
echo "Found after regular checking time -> Docker hub's latest updated image is new; Diff ${updiffMins} mins" >> "ScriptOutput_${currentDate}.txt"
echo "Script is checking for latest updates after every ${CHECKTIME} mins" >> "ScriptOutput_${currentDate}.txt"
echo "Fetching all new"
echo "---------------------------"
if $IS_CONTAINER_RUNNING ; then
echo "Container is running"
else
docker-compose down
echo "Container stopped and removed; Network removed" >> "ScriptOutput_${currentDate}.txt"
fi
echo "Image_Created_Date=$currentDate" > ".env"
echo "ORG=$ORG" >> ".env"
echo "IMGNAME=$IMGNAME" >> ".env"
echo "IMGTAG=$IMGTAG" >> ".env"
echo "CONTNAME=$CONTNAME" >> ".env"
echo "NETWORKNAME=$NETWORKNAME" >> ".env"
docker-compose build --no-cache
echo "Docker Compose built" >> "ScriptOutput_${currentDate}.txt"
if $DO_DOCKER_COMPOSE_UP ; then
docker-compose up -d
echo "Docker services are up now, checked in" >> "ScriptOutput_${currentDate}.txt"
else
echo "Docker services are down, checked in" >> "ScriptOutput_${currentDate}.txt"
fi
elif [[ "$updatedDate" -gt "$createdDate" ]]; then
echo "Updated is latest"
start_date=$(date -jf "%Y%m%d%H%M%S" "$updatedDate" "+%s")
end_date=$(date -jf "%Y%m%d%H%M%S" "$createdDate" "+%s")
diffMins=$(( ($start_date - $end_date) / (60) ))
if [[ "$BUILDMINS" -lt "$diffMins" ]]; then
if [ ! -d "${FILEPATH}" ]; then
mkdir "${FILEPATH}";
fi
cd "${FILEPATH}"
pwd
echo "updatedDate---$updatedDate" > "ScriptOutput_${currentDate}.txt"
echo "createdDate---$createdDate" >> "ScriptOutput_${currentDate}.txt"
echo "currentDate---$currentDate" >> "ScriptOutput_${currentDate}.txt"
echo "Found after comparing times -> Docker hub's latest updated image is new; Diff ${diffMins} mins" >> "ScriptOutput_${currentDate}.txt"
echo "Actual image built time is less i.e. ${diffMins} mins than MAX expexted BUILD TIME i.e. ${BUILDMINS} mins" >> "ScriptOutput_${currentDate}.txt"
echo "Fetching all new" >> "ScriptOutput_${currentDate}.txt"
echo "-----------------------------"
if $IS_CONTAINER_RUNNING ; then
echo "Container is running"
else
docker-compose down
echo "Container stopped and removed; Network removed" >> "ScriptOutput_${currentDate}.txt"
fi
echo "Image_Created_Date=$currentDate" > ".env"
echo "ORG=$ORG" >> ".env"
echo "IMGNAME=$IMGNAME" >> ".env"
echo "IMGTAG=$IMGTAG" >> ".env"
echo "CONTNAME=$CONTNAME" >> ".env"
echo "NETWORKNAME=$NETWORKNAME" >> ".env"
docker-compose build --no-cache
echo "Docker Compose built" >> "ScriptOutput_${currentDate}.txt"
if $DO_DOCKER_COMPOSE_UP ; then
docker-compose up -d
echo "Docker services are up now" >> "ScriptOutput_${currentDate}.txt"
else
echo "Docker services are down" >> "ScriptOutput_${currentDate}.txt"
fi
elif [[ "$BUILDMINS" -gt "$diffMins" ]]; then
echo "Docker hub's latest updated image is NOT new; Diff ${diffMins} mins"
echo "Docker images not fetched"
else
echo "Docker hub's latest updated image is NOT new; Diff ${diffMins} mins"
echo "Docker images not fetched"
fi
elif [[ "$createdDate" -gt "$updatedDate" ]]; then
echo "Created is latest"
start_date=$(date -jf "%Y%m%d%H%M%S" "$createdDate" "+%s")
end_date=$(date -jf "%Y%m%d%H%M%S" "$updatedDate" "+%s")
echo "Docker hub has older docker image than local; Older than $(( ($start_date - $end_date) / (60) ))mins"
fi
echo
echo "------------end---------------"
rm $PIDFILE
Here is my docker-compose file
version: "3.2"
services:
lamp-alpine:
build:
context: .
container_name: "${CONTNAME}"
image: "${ORG}/${IMGNAME}:${IMGTAG}"
ports:
- "127.0.0.1:80:80"
networks:
- private-network
networks:
private-network:
driver: bridge
have you tried this: https://github.com/v2tec/watchtower. it's a simple tool running in docker container watching other containers, if their base image changed, it will pull and redeploy.
A simple and great solution is shepherd
