28

We use PHP gearman workers to run various tasks in parallel. Everything works just fine, and I have silly little shell script to spin them up when I want them. Being a programmer (and therefore lazy), I wanted to see if I could spin these up via an upstart script.

I figured out how to use the instance stanza, so I could start them with an instance number:

description "Async insert workers"
author      "Mike Grunder"

env SCRIPT_PATH="/path/to/my/script"

instance $N

script
    php $SCRIPT_PATH/worker.php
end script

And this works great, to start them like so:

sudo start async-worker N=1
sudo start async-worker N=2

The way I want to use these workers is to spin up some number of them (maybe one per core, etc), and I would like to do this on startup. To be clear, I don't need the upstart script to detect the number of cores. I'm happy to just say "do 8 instances", but that's why I want multiple running. Is there a way for me to use the "start on" clause in an upstart script to do this automatically?

For example, start instance 1, 2, 3, 4? Then have them exit on shutdown properly?

I suppose I could hook this into an init.d script, but I was wondering if upstart can handle something like this, or if anyone has figured out this issue.

Cheers guys!

mkgrunder
  • 941
  • 1
  • 7
  • 13

1 Answers1

40

What you need is a bootstrap task that runs on startup and iterates over all your worker jobs, starting each one.

#/etc/init/async-workers-all.conf

start on runlevel [2345]

task

env NUM_WORKERS=8

script
  for i in `seq 1 $NUM_WORKERS`
  do
    start async-worker N=$i
  done
end script

The key is to make this a task, which tells upstart to let the task run to completion before emitting any events for it. See http://upstart.ubuntu.com/cookbook/#task and http://upstart.ubuntu.com/cookbook/#instance

cwick
  • 26,132
  • 12
  • 37
  • 40
  • Thanks so much! I tried to work it out in a similar manner, but I was missing the `task` bit, so the first iteration of the loop would just block. Awesome. :) – mkgrunder Jan 09 '12 at 01:42
  • 11
    You should also checkout http://stackoverflow.com/questions/12084025/restarting-upstart-instance-prcesses that allows to not only start multiple instances, but also stop them. – Evgeny Oct 23 '12 at 15:17
  • I've read the documentation but I still don't get it, why is that loop blocking in its first iteration unless it's a task? /cc @mkgrunder – Mahn Aug 29 '14 at 10:55