Script behaves differently when run from cron job and from command line using django manage.py when using a cron supervisor

Question

I know crons run in a different environment than command lines, but I'm using absolute paths everywhere and I don't understand why my script behaves differently. I believe it is somehow related to my cron_supervisor which runs the django "manage.py" within a sub process.

Cron:

0 * * * * /home/p1/.virtualenvs/prod/bin/python /home/p1/p1/manage.py cron_supervisor --command="/home/p1/.virtualenvs/prod/bin/python /home/p1/p1/manage.py envoyer_argent"

This will call the cron_supervisor, and it's call the script, but the script won't be executed as it would if I would run:

/home/p1/.virtualenvs/prod/bin/python /home/p1/p1/manage.py envoyer_argent

Is there something particular to be done for the script to be called properly when running it through another script?

Here is the supervisor, which basically is for error handling and making sure we get warned if something goes wrong within the cron scripts themselves.

import logging
import os
from subprocess import PIPE, Popen

from django.core.management.base import BaseCommand

from command_utils import email_admin_error, isomorphic_logging
from utils.send_slack_message import send_slack_message

CURRENT_DIR = os.path.dirname(os.path.abspath(__file__))
PROJECT_DIR = CURRENT_DIR + '/../../../'

logging.basicConfig(
  level=logging.INFO,
  filename=PROJECT_DIR + 'cron-supervisor.log',
  format='%(asctime)s %(levelname)s: %(message)s',
  datefmt='%Y-%m-%d %H:%M:%S'
)


class Command(BaseCommand):
  help = "Control a subprocess"

  def add_arguments(self, parser):
    parser.add_argument(
      '--command',
      dest='command',
      help="Command to execute",
    )
    parser.add_argument(
      '--mute_on_success',
      dest='mute_on_success',
      action='store_true',
      help="Don't post any massage on success",
    )

  def handle(self, *args, **options):
    try:
      isomorphic_logging(logging, "Starting cron supervisor with command \"" + options['command'] + "\"")
      if options['command']:
        self.command = options['command']
      else:
        error_message = "Empty required parameter --command"
        # log error
        isomorphic_logging(logging, error_message, "error")
        # send slack message
        send_slack_message("Cron Supervisor Error: " + error_message)
        # send email to admin
        email_admin_error("Cron Supervisor Error", error_message)
        raise ValueError(error_message)

      if options['mute_on_success']:
        self.mute_on_success = True
      else:
        self.mute_on_success = False

      # running process
      process = Popen([self.command], stdout=PIPE, stderr=PIPE, shell=True)
      output, error = process.communicate()

      if output:
        isomorphic_logging(logging, "Output from cron:" + output)

      # check for any subprocess error
      if process.returncode != 0:
        error_message = 'Command \"{command}\" - Error \nReturn code: {code}\n```{error}```'.format(
          code=process.returncode,
          error=error,
          command=self.command,
        )
        self.handle_error(error_message)

      else:
        message = "Command \"{command}\" ended without error".format(command=self.command)
        isomorphic_logging(logging, message)
        # post message on slack if process isn't muted_on_success
        if not self.mute_on_success:
          send_slack_message(message)
    except Exception as e:
      error_message = 'Command \"{command}\" - Error \n```{error}```'.format(
        error=e,
        command=self.command,
      )
      self.handle_error(error_message)

  def handle_error(self, error_message):
    # log the error in local file
    isomorphic_logging(logging, error_message)
    # post message in slack
    send_slack_message(error_message)
    # email admin
    email_admin_error("Cron Supervisor Error", error_message)

Example of script not executed properly when being called by the cron, through the cron_supervisor:

# -*- coding: utf-8 -*-

import json
import logging
import os

from django.conf import settings
from django.core.management.base import BaseCommand

from utils.lock import handle_lock

logging.basicConfig(
  level=logging.INFO,
  filename=os.path.join(settings.BASE_DIR, 'crons.log'),
  format='%(asctime)s %(levelname)s: %(message)s',
  datefmt='%Y-%m-%d %H:%M:%S'
)
class Command(BaseCommand):
  help = "Envoi de l'argent en attente"

  @handle_lock
  def handle(self, *args, **options):

    logging.info("some logs that won't be log (not called)")

logging.info("Those logs will be correcly logged")

Additionally, I have another issue with the logging which I don't quite understand either, I specify to store logs within cron-supervisor.log but they don't get stored there, I couldn't figure out why. (but that's not related to my main issue, just doesn't help with debug)

Answer 1

Your cron job can't just run the Python interpreter in the virtualenv; this is completely insufficient. You need to activate the env just like in an interactive environment.

0 * * * * . /home/p1/.virtualenvs/prod/bin/activate; python /home/p1/p1/manage.py cron_supervisor --command="python /home/p1/p1/manage.py envoyer_argent"

This is already complex enough that you might want to create a separate wrapper script containing these commands.

Without proper diagnostics of how your current script doesn't work, it's entirely possible that this fix alone is insufficient. Cron jobs do not only (or particularly) need absoute paths; the main differences compared to interactive shells is that cron jobs run with a different and more spare environment, where e.g. the shell's PATH, various library paths, environment variables etc can be different or missing altogether; and of course, no interactive facilities are available.

The system variables will hopefully be taken care of by your virtualenv; if it's correctly done, activating it will set up all the variables (PATH, PYTHONPATH, etc) your script needs. There could still be things like locale settings which are set up by your shell only when you log in interactively; but again, without details, let's just hope this isn't an issue for you.

The reason some people recommend absolute paths is that this will work regardless of your working directory. But a correctly written script should work fine in any directory; if it matters, the cron job will start in the owner's home directory. If you wanted to point to a relative path from there, this will work fine inside a cron job just as it does outside.

As an aside, you probably should not use subprocess.Popen() if one of the higher-level wrappers from the subprocess module do what you want. Unless compatibility with legacy Python versions is important, you should probably use subprocess.run() ... though running Python as a subprocess of Python is also often a useless oomplication. See also my answer to this related question.