1

Essentially, I have a perl workflow that runs a suite of java tools via the system command (picard in GATK4 for those of you who are familiar with Broad Institute's software). These gatk4 commands return code 0 if successful. When I kick off the perl workflow manually (RHE7.6), it all seems to flow through fine.

However, the Gatk4 processes that return a 0 upon success, fail when the same workflow is run as part of a cron-job like so:

0,15,30 * * * * /usr/bin/flock -n /tmp/QAQC.lock -c "/seq/QAQC/bin/launch_QAQC.pl 1>>launch_QAQC.log 2>>launch_QAQC.err"

I suspect that something about running flock with the –c option is causing those jobs (and therefore the perl pipeline) to abort.

Any insights that you can provide would be very much appreciated.

Thanks

EDIT: Figured out the solution, in that the problem is not with flock. Cron wasn't seeing the same env variables as the interactive shell.

sirisha sunkara
  • 21
  • 3
  • can you please tell what error message you are getting in launch_QAQC.err file – amit bhosale Jun 16 '20 at 06:34
  • can you please remove ',' and execute script for example 0 15 30 * * check this url https://crontab.guru/#0_15_30_*_* – amit bhosale Jun 16 '20 at 06:44
  • as per corntab syntax [Minute] [hour] [Day_of_the_Month] [Month_of_the_Year] [Day_of_the_Week] [command] so check your current syntax https://tecadmin.net/crontab-in-linux-with-20-examples-of-cron-schedule/ – amit bhosale Jun 16 '20 at 06:45
  • try running as the same user and with the same current directory as via cron? verify no environment variable differences (check %ENV in the perl script)? have you tried without the flock? – ysth Jun 16 '20 at 10:05
  • What do you mean by "abort"? What's logged to STDERR? What the exit code of the process? – ikegami Jun 16 '20 at 14:49
  • @ysth has most likely cause of problems, cron doesn't run with the same env variables as an interactive shell + CWD is also a factor. The answer is to standardise your environment for your production jobs in some way. Some approaches are described in https://stackoverflow.com/questions/2229825/where-can-i-set-environment-variables-that-crontab-will-use/10657111 It may also be worth adding `.$USER` suffix to your lock file to ensure it is unique based on the process owner. This pattern will also protect anyone who copies your technique and needs to run multiple jobs with same name/identifier. – KevinJWalters Jun 21 '20 at 00:45
  • It's rare but `ulimit` settings can also be different. It's more common this changes behaviour rather than stops it running, e.g. core files might not be produced or file descriptor limits could be different and cause very subtle capacity issues. – KevinJWalters Jun 21 '20 at 00:47
  • @KevinJWalters that's it. There was a hidden issue with the GATK4 command, in that it runs Rscript under the hood. My cron process was unable to find the path to the Rscript, which was otherwise visible to the perl workflow that runs the GATK4 command. Cron did not see the same env variables as the interactive shell. Exporting the path in the cron setup fixed it. Thanks for the pointers. – sirisha sunkara Jul 17 '20 at 17:54

0 Answers0