docker-compose healthcheck for pentaho data integration (pdi)

Question

I am building my custom pdi image using docker. I could build image and ran it without any issues. Now I need to add healthcheck for my pdi container. Can anyone suggest me a healthcheck command?

I tried to use,

healthcheck:
      test: /home/data-integration/kitchen.sh -file="/home/jobs/my.kjb" -level=Basic && echo $? || exit 1

but gives an error as, ERROR: Invalid interpolation format for "healthcheck" option in service "pentaho": "/home/data-integration/kitchen.sh -file="/home/jobs/my.kjb" -level=Basic && echo $? || exit 1"

If I use healthcheck command as below it become unhealthy even there are no any errors.

healthcheck:
      test: /home/data-integration/kitchen.sh -file="/home/jobs/my.kjb" -level=Basic || exit 1

if I find it from docker inspect containerID,

I do not use carte or anything or any UIs. I just unzip the pdi zip file and want to run my pdi job to a given schedule. my entrypoint.sh file is as below,

#!/bin/sh
## entrypoint.sh
/home/data-integration/kitchen.sh -file="/home/jobs/my.kjb" -level=Basic
tail -f /dev/null

When I manually run job file and check echo $? it gives 0 as output if job is success. how to use it correctly in to docker-compose healthcheck?

TheQueenIsDead · Answer 1 · 2021-08-04T20:05:35.203

Being a new user, I cannot comment yet, so I hope this answer gives you something to think about.

Food for thought

Per the Docker documentation on healthchecks, the format is as stated: https://docs.docker.com/engine/reference/builder/#healthcheck

I'm not familiar with your application specifically, but if there is any startup required, then setting a delay to give the container time to initialize may be helpful.

I also see that you're using the same command in your entrypoint script that you are using to healthcheck.

Healthchecks should typically not be the same thing as the running process, and instead should be used to ensure the running process is working correctly. The docs highlight this, as does this blogpost, detailing how to check that a web app is alive by pinging the server.

Another note is that if your entrypoint tails dev null, you will not get the logs of the running process through docker logs. If you want to schedule a task to run often in a container, I recommend wrapping your command in a while loop which calls the command, or using an external orchestrator like Kubernetes Cron Jobs (Edit: Or even a crontab on the host that calls docker run)

Fix

Finally, if you're set on simply fixing the immediate issue of formatting, you need to escape the $ character in the healthcheck, like so:

      test: /home/data-integration/kitchen.sh -file="/home/jobs/my.kjb" -level=Basic && echo $$? || exit 1

Other issues akin to this are: How can I escape a $ dollar sign in a docker compose file?

And it links to the docs on variable escaping here: https://docs.docker.com/compose/compose-file/compose-file-v3/#variable-substitution

Thank you very much for your valuable ideas. I used infinite while loop with sleep and found a healthcheck command using exit codes. I will post my answer also here. Your comments were really helped for me. Thanks again! — Prabuddha Kulatunga, Aug 05 '21 at 01:39

Prabuddha Kulatunga · Accepted Answer · 2021-08-05T20:16:23.910

Found the healthcheck for pdi container and I will post here since this can help for others.

Basically, When a job executed without any errors it return 0 as exit code. But when there is an error it usually returns 1 as exit code as I found.

So, checked exit code status for the needed pdi job execution command and used it for healthcheck as below sample,

create healthcheck.sh file and copy it to your container,(in here, I copied it to /home/scripts/ path inside my container.)

#!/bin/sh
set -e

## execute job 
/home/data-integration/kitchen.sh -file="/home/jobs/my.kjb" -level=Basic
 
## Job executed without errors? Use exit status of the job command ##
if [ $? -eq 0 ]
then
  echo "Success"
  exit 0
else
  echo "Failure" >&2
  exit 1
fi

Then run healthcheck.sh file in docker-compose.yml (used 2.3 docker-compose.yml version)

healthcheck:
      test: ./home/scripts/healthcheck.sh
      interval: 55s
      timeout: 50s
      retries: 3
      start_period: 9m

Note:

Make sure to run chmod +x for healthcheck.sh file after copy it to the container using Dockerfile, before run the docker-compose.yml file OR change healthcheck command as,

healthcheck:
      test: chmod +x /home/scripts/healthcheck.sh && ./home/scripts/healthcheck.sh
      interval: 55s
      timeout: 50s
      retries: 3
      start_period: 9m

For my docker image, I unzipped pdi-ce-9.1.0.0-324.zip file and executed the job file repeatedly using a entrypoint.sh file to do my ETL process as a schedule. Used java:8-jre-alpine image to unzip it.
New entrypoint.sh edited according to @TheQueenIsDead suggested with infinite while loop to run pdi job repeatedly,

#!/bin/sh
## entrypoint.sh
while :
do
    /home/data-integration/kitchen.sh -file="/home/jobs/my.kjb" -level=Basic
    sleep 120
done

References:

docker-compose healthcheck for pentaho data integration (pdi)

2 Answers2

Food for thought

Fix