0

With the aid of this question here, I've been busy taking over an woefully undocumented gearman setup from a previous dev. Funnily enough, I've never seen such a woefully poor excuse for documentation as the gearman official website either, so have spent a while writing my own for the repo.

Anyway, I'm doing ok regardless as I'm persistent, so all good, except my staging site, which is unusable as it will not load memcache, due to stuck jobs.

So on to monitor, I run:

 gearadmin  --status | sort -n | column -t

Production workers show 

0 0 26

all done, great!

On staging, I was left with 

2550 0 0

...basically every product failing.

So I ran 

/usr/bin/gearman -t 1000 -n -w -f what_i_need_to_clear  > /dev/null

This seemingly clears a job queue. lol. Almost. Now I have 

5 4 4

How do I clear those? as the jobs just back up behind it. I just want to reset my staging to 

0 0 0

I'm using a federated table setup with PHP/MySQL/Sphinx/Memcache

Can anyone shed some light on this, or show me how to debug those jobs?

Shadow
  • 33,525
  • 10
  • 51
  • 64
John Lewis
  • 318
  • 1
  • 10
  • Is gearman configured for persistent storage (backed by a `queue` table in your database)? – Kris Peeling Jan 27 '20 at 13:19
  • Hi Kris and thanks for chiming in. Yes, it is. It's also an empty table currently with production jobs running all the time and completing as normal. – John Lewis Jan 29 '20 at 07:32
  • 1
    OK when you get stuck jobs and they are backing up, you have to delete the corresponding entries from the `queue` table, then restart gearman server process. This will clear the jobs for good and they shouldn't come back. – Kris Peeling Jan 30 '20 at 17:56

1 Answers1

1

This is an old question I know but in case anyone else has the same question in the future... the answer to why your final jobs didn't clear was that had workers running which were working on those jobs.

The three columns in the status output:

5 4 4

That means 5 jobs in the queue, 4 jobs in progress, 4 workers available.

So while your gearman -w > /dev/null was consuming jobs, 4 jobs were locked by the other 4 workers. The fact you're left with 5 I guess was probably a race condition, one job got returned to the queue by another worker while your drain was running.

You can avoid this by stopping the other workers while you drain the queue, or like Kris said in the comments, stop gearmand, clear the persistent storage and restart gearmand.

Colin Pickard
  • 45,724
  • 13
  • 98
  • 148