Run huey task queue in the background with supervisor on Elastic Beanstalk

Question

I am trying to run a huey task queue on elastic beanstalk that is needed by my Flask app. But there is no built in way to run huey as a daemon process. The author of huey has advised to run huey with supervisor (this link) and since elastic beanstalk already uses supervisor, I thought we could just add the program to be managed by supervisor. But I am not sure how to do this programatically. Currently, I am using the container_commands (ref link) key in the config file to run this but elastic beanstalk gives me a timeout error after sometime as it runs in the foreground. Below is the config file I am using.

packages:
  yum:
    gcc: []
    gcc-c++: []
    gcc-gfortran: []
    htop: []
    make: []
    wget: []
    atlas-devel: []
    lapack-devel: []
commands:
  01enable_swap:
    command:
      - sudo dd if=/dev/zero of=/var/swap1 bs=1M count=1024
      - sudo mkswap /var/swap1
      - sudo chmod 644 /var/swap1
      - sudo swapon /var/swap1
    cwd: /home/ec2-user
  02install_redis:
    command:
      - wget "http://download.redis.io/redis-stable.tar.gz"
      - tar -xvzf redis-stable.tar.gz
      - rm redis-stable.tar.gz
      - cd redis-stable
      - sudo make
      - sudo make install
    cwd: /home/ec2-user
container_commands:
  01download_nltk_packages:
    command: "python install_resources.py"
  02run_redis:
    command: "redis-server --host 127.0.0.1 --port 6379 --daemonize yes"
  03run_huey:
    command: "huey_consumer jupiter.huey"

Here's what I want to achieve:
1. huey should run as a background process when my Flask app is deployed.
2. supervisor should handle automatic start/stop of the huey process.

score 0 · Accepted Answer · answered Sep 06 '18 at 20:29

I solved this problem by doing the following in an ebextensions file called 002_supervisor.conf. This is for django but I'm sure it could be adapted for flask.

Create a supervisor config file
Create a supervisor init.d file
Create a huey.conf file to be loaded by supervisor

files:
   /usr/local/etc/supervisord.conf:
    mode: "000755"
    owner: root
    group: root
    content: |
        [unix_http_server]
        file=/tmp/supervisor.sock   ; (the path to the socket file)

        [supervisord]
        logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
        pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
        nodaemon=false               ; (start in foreground if true;default false)

        [rpcinterface:supervisor]
        supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

        [supervisorctl]
        serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket

        [include]
        files = /usr/local/etc/*.conf

        [inet_http_server]
        port = 127.0.0.1:9001
   /etc/init.d/supervisord:
    mode: "000755"
    owner: root
    group: root
    content: |
        #!/bin/bash

        # Source function library
        . /etc/rc.d/init.d/functions

        # Source system settings
        if [ -f /etc/sysconfig/supervisord ]; then
            . /etc/sysconfig/supervisord
        fi

        # Path to the supervisorctl script, server binary,
        # and short-form for messages.
        supervisorctl=/usr/local/bin/supervisorctl
        supervisord=${SUPERVISORD-/usr/local/bin/supervisord}
        prog=supervisord
        pidfile=${PIDFILE-/tmp/supervisord.pid}
        lockfile=${LOCKFILE-/var/lock/subsys/supervisord}
        STOP_TIMEOUT=${STOP_TIMEOUT-60}
        OPTIONS="${OPTIONS--c /usr/local/etc/supervisord.conf}"
        RETVAL=0

        start() {
            echo -n $"Starting $prog: "
            daemon --pidfile=${pidfile} $supervisord $OPTIONS
            RETVAL=$?
            echo
            if [ $RETVAL -eq 0 ]; then
                touch ${lockfile}
                $supervisorctl $OPTIONS status
            fi
            return $RETVAL
        }

        stop() {
            echo -n $"Stopping $prog: "
            killproc -p ${pidfile} -d ${STOP_TIMEOUT} $supervisord
            RETVAL=$?
            echo
            [ $RETVAL -eq 0 ] && rm -rf ${lockfile} ${pidfile}
        }

        reload() {
            echo -n $"Reloading $prog: "
            LSB=1 killproc -p $pidfile $supervisord -HUP
            RETVAL=$?
            echo
            if [ $RETVAL -eq 7 ]; then
                failure $"$prog reload"
            else
                $supervisorctl $OPTIONS status
            fi
        }

        restart() {
            stop
            start
        }

        case "$1" in
            start)
                start
                ;;
            stop)
                stop
                ;;
            status)
                status -p ${pidfile} $supervisord
                RETVAL=$?
                [ $RETVAL -eq 0 ] && $supervisorctl $OPTIONS status
                ;;
            restart)
                restart
                ;;
            condrestart|try-restart)
                if status -p ${pidfile} $supervisord >&/dev/null; then
                  stop
                  start
                fi
                ;;
            force-reload|reload)
                reload
                ;;
            *)
                echo $"Usage: $prog {start|stop|restart|condrestart|try-restart|force-reload|reload}"
                RETVAL=2
            esac

            exit $RETVAL
   "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_huey.sh" :
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash

      # Get django environment variables
      env=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
      env=${env%?}

      # Create huey configuration script
      hueyconf="[program:huey]
      ; Set full path to celery program if using virtualenv
      command=/opt/python/current/app/production.py run_huey
      user=nobody
      numprocs=1
      stdout_logfile=/var/log/huey.log
      stderr_logfile=/var/log/huey.log
      autostart=true
      autorestart=true
      startsecs=10

      ; Need to wait for currently executing tasks to finish at shutdown.
      ; Increase this if you have very long running tasks.
      stopwaitsecs = 60

      ; When resorting to send SIGKILL to the program to terminate it
      ; send SIGKILL to its whole process group instead,
      ; taking care of its children as well.
      killasgroup=true
      environment=$env"

      # Create the celery supervisord conf script
      echo "$hueyconf" | tee /usr/local/etc/huey.conf

      # Update supervisord in cache without restarting all services
      /usr/local/bin/supervisorctl reread
      /usr/local/bin/supervisorctl update

      # Start/Restart huey through supervisord
      /usr/local/bin/supervisorctl -c /usr/local/etc/supervisord.conf restart huey

commands:
  01_start_supervisor:
    command: '/etc/init.d/supervisord restart'
    leader_only: true

Run huey task queue in the background with supervisor on Elastic Beanstalk

1 Answers1