3

I have been trying to make the uwsgi python spooler work properly for quite some time. I have a setup in which I run a django application with two worker processes. I have tried setting a cron spooler (and a timer spooler) to run a task every ten minutes, but no matter what configuration of settings I've tried, it always seems to register the signal multiple times, and running the task multiple times.

This is how I run uwsgi:

#!/bin/bash
sudo uwsgi --emperor /etc/uwsgi/vassals --uid http --gid http --enable-threads --pidfile=/tmp/uwsgi.pid --daemonize=/var/log/uwsgi/uwsgi.log

This is my uwsgi vassal config in /etc/uwsgi/vassals/django.ini:

[uwsgi]

chdir           = /home/user/django
module          = django.wsgi

master          = true
processes       = 2
socket          = /tmp/uwsgi-django.sock
vacuum          = true
pidfile         = /tmp/uwsgi-django.pid
daemonize      = /home/user/django/log.log
env = DJANGO_SETTINGS_MODULE=django.settings
#lazy-apps = false
#lazy = false
spooler = %(chdir)/tasks
#spooler-processes = 1
#import = django-app/spooler.py
#spooler-import = django-app/spooler.py
shared-import = django-app/spooler.py

(I have changed some of the path names for privacy reasons). The lines that are commented out are various attempts at making it not duplicate my signals, but every time it seems to register the signal twice, and sometimes even thrice (presumably in both the workers and the single spooler process).

[uwsgi-signal] signum 0 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 1 registered (wid: 1 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 1 registered (wid: 2 modifier1: 0 target: default, any worker)

Does anyone know why this is happening, and how to properly prevent it?

This is the spooler.py file:

@cron(-10, -1, -1, -1, -1)
def periodicUpdate(signal):
    print "Running cron job..."
    _getStats()

also tried

@timer(600)
def periodicUpdate(signal):
    print "Running cron job..."
    _getStats()

I also tried adding target='spooler' to the timer/cron-decorator, but it did not seem to many any difference.

eirikrye
  • 281
  • 3
  • 6

2 Answers2

3

Are you sure you do not have other signals registered in django.wsgi, settings.py or other django-related file ? --shared-import will only load things one time (in the master).

Btw i do not get what you are trying to accomplish. This is not how the spooler is supposed to work, and even if you want to use it as a signal handler target you have to specify it when you register signals (with target='spooler' in the decorator)

roberto
  • 12,723
  • 44
  • 30
  • 1
    All I am trying to accomplish is to run a script every ten minutes. It does run every ten minutes, however it seems to be run two or three times instead of once as I would expect. I don't understand how I am misusing the spooler in any way. It seems to match the documentation use cases. Even if I put target='spooler' in the decorator, it is run multiple times (e.g. it prints "Running cron job..." and populates my database with duplicate data) – eirikrye Jan 21 '14 at 08:35
  • If I add target='spooler', and use spooler-import to import the file containing the cronjob into the spooler, I get an error saying that "only the master and the workers can register signal handlers" and no signals are registered or executed. If I remove the target, and use shared-import or import, the cron/timer signal is initially registered once, but whenever I load one of the pages in my django application (e.g. I "wake up" one of the worker processes) it re-registers it. – eirikrye Jan 21 '14 at 09:45
  • the spooler works by running enqueued tasks asynchronously without bothering workers. So, its "classic" usage would be having your signal handler enqueuing a request in the spooler. Your problem is that you are somewhat re-registering the signal. I suggest you to restart from the beginning with a simple app (an hello world WSGI with a @cron entry in the same file) and once all is working as expected, try to catch the problem with your django app. – roberto Jan 21 '14 at 10:27
0

While this is an old question, I couldn't find the answer elsewhere. I used this solution with Flask, but it should be similar with Django.

During initialization (prefork mode) you need to register a signal.

uwsgi.register_signal(26, "spooler", periodicUpdate)

Then the timer should look like this:

@timer(600, target='spooler')
def periodicUpdate(signal):
    print "Running cron job..."
    _getStats()

As for the comments:

The error 'only the master and the workers can register signal handlers' is correct because you haven't register any signal.

The issue with the

'whenever I load one of the pages in my django application it re-registers it'

can probably happen because its worker is calling the method (periodicUpdate) once. That's why the signal must be register before the workers are being spawned.

user3816638
  • 43
  • 9