Consul watch with critical consul checks

Question

So I have a consul check that watches over a container and is designed to go critical when the container is stopped. I want to create a consul watch that will run a script after the check has gone critical, or after several critical responses (for example if my check sends 5 critical responses I want it to run a script).

Here is the json for my working check and my guess as to what I my watch might look like:

{

 // this check works
 "checks": [
    {
        "id": "docker_stuff",
        "name": "curl test",
        "notes": "curls the docker container",
        "script": "/scripts/docker.py",
        "interval": "1s"
    }
 ],

 //this watch doesn't work
 "watches": [
    {
        "Node": "client2",
        "CheckID": "docker-stuff",
        "Name": "docker-stuff-watch",
        "Status": "critical",
        "Status_amt": "5",
        "handler": "/scripts/new-docker.sh",
        "Output": "container relaunched",
    }
 ]

}

What do I need to change in my watch to get it working?

Would I also need to use a consul event to watch my health check and then trigger a consul watch (of the event type) that runs my /scripts/new-docker.sh script? If so then how would I make a consul event that watches over my health check? For example if this was my consul check, watch and event, what would I need to change to get this working?

{
 "checks": [
    {
        "id": "docker_stuff",
        "name": "curl test",
        "notes": "curls the docker container",
        "script": "/scripts/docker.py",
        "interval": "1s"
    }
 ],
 "watches": [
    {     
        "type": "event", 
        "name": "docker-stuff-watch",
        "handler": "/scripts/new-docker.sh"
    }
 ],
 "events": [
    {
        "Node": "client2",
        "CheckID": "docker-stuff",
        "Name": "docker-stuff-event",
        "Status": "critical",
        "Status_amt": "5",
        "Output": "container relaunched",
    }
 ]
}

score 1 · Answer 1 · answered Oct 17 '16 at 15:44

What do I need to change in my watch to get it working?

Are there any errors? Make sure your watch handler '/scripts/new-docker.sh' is consuming STDIN that Consul will be sending, even if it is throwing it away to /dev/null, otherwise the process will wait forever for it to be consumed

Something like

while read -r -t 0; do read -r; done

score 0 · Answer 2 · answered Jun 27 '16 at 20:48

I would recommend considering an upgrade to the next version of Docker 1.12 (release candidate at the moment). The new concept of services can be used to state the desired number of containers to be run.

https://docs.docker.com/engine/swarm/swarm-tutorial/deploy-service/

There's also a new HEALTHCHECK directive in the Dockerfile that enables you to bundle a check script with the container image.

These new features might enable you to replace the functionality you've had to implement using consul.

Consul watch with critical consul checks

2 Answers2