Processing some file before starting app and reacting for every change

Question

I have a file containing some data – data.txt (existing in proper localization). I would like that django app processes this file before starting app and reacts for every change (without restart). What is the best way to do it?

How do you want the app to react, and what do you mean by “processing” the file? Processing *before* starting the app doesn't seem to make a lot of sense. Why not process data during request-response cycle in a view function? (If the data is heavy, you can use some async task queue to process data and reflect the progress in DB, then just return appropriate error in HTTP response from view function if the data is not ready yet.) — Anton Strogonoff, Apr 20 '13 at 16:19
This file is used to store extra information about app. The data wouldn't be heavy and this conception is not mine :) — David Silva, Apr 20 '13 at 16:29
By "processing" I mean modest updates on DB. By "react" I mean detection of changes in the file and new processing. — David Silva, Apr 20 '13 at 16:37

score 3 · Accepted Answer · answered Apr 24 '13 at 11:29

For startup you can write middleware that does what you want in init and afterwards raise django.core.exceptions.MiddlewareNotUsed from the init, so the django will not use it for any request processing. docs

And middleware init will be called at startup, not at the first request. As for react to file changes you can use https://github.com/gorakhargosh/watchdog ( an example of usage can be found here). So you can either start it somewhere in middleware too, or if its only db updates you can create a separate script(or django management command) that will be run via supervisor or something like this and will monitor this file and update the db.

score 1 · Answer 2 · answered Apr 22 '13 at 20:51

An option could be pynotify that monitors the filesystem for changes, it works only on Linux though.

Otherwise look at the code of runserver command that seems to do the same thing (the code of the autoreload module is here).

To run a command before starting an app I suppose you can write some code in the settings module.

score 1 · Answer 3 · answered Apr 25 '13 at 07:46

maybe you could put a object in the settings which will lookup to the file for every change. ... ie : make a class who will load the file and reload this one if he is modified

class ExtraConfigWatcher(object):
    def __init__(self, file):
        self.file = file
        self.cached = dict()
        self.last_date_modified = None

    def update_config(self):
        """
        update the config by reloading the file
        """
        if has_been_modified(self.file, self.last_date_modified):
            # regenerate the config with te file.
            self.cached = get_dict_with_file(self.file)
            self.last_date_modified = time.time()
    def __getitem__(self, *args, **kwargs):
        self.update_config()
        return self.cached.__getitem__(*args, **kwargs)

    def __setitem__(self, *args, **kwargs):
        raise NotImplemented("you can't set config into this")

in settings.py: initialize this object

EXTRA_CONFIG = ExtraConfigWatcher("path/to/the/file.dat")

in myapps/views.py: import settings and use EXTRA_CONFIG

from django.conf import settings
def dosomthing(request):
    if settings.EXTRA_CONFIG["the_data_from_the_file"] == "foo":
        # bouhh

score 1 · Answer 4 · edited May 23 '17 at 11:43

A while ago I was trying to find a mechanism to "hot-swap" Python modules. While that is not exactly what you need, maybe you can use the implementation I proposed, and monitor your configuration file for modifications and act accordingly.

The code I proposed is the following (I did not use inotify because I am working in an NFS file system):

import imp
import time
import hashlib
import threading
import logging

logger = logging.getLogger("")

class MonitorThread(threading.Thread):
    def __init__(self, engine, frequency=1):
        super(MonitorThread, self).__init__()
        self.engine = engine
        self.frequency = frequency
        # daemonize the thread so that it ends with the master program
        self.daemon = True 

    def run(self):
        while True:
            with open(self.engine.source, "rb") as fp:
                fingerprint = hashlib.sha1(fp.read()).hexdigest()
            if not fingerprint == self.engine.fingerprint:
                self.engine.notify(fingerprint)
            time.sleep(self.frequency)

class Engine(object):
    def __init__(self, source):
        # store the path to the engine source
        self.source = source        
        # load the module for the first time and create a fingerprint
        # for the file
        self.mod = imp.load_source("source", self.source)
        with open(self.source, "rb") as fp:
            self.fingerprint = hashlib.sha1(fp.read()).hexdigest()
        # turn on monitoring thread
        monitor = MonitorThread(self)
        monitor.start()

    def notify(self, fingerprint):
        logger.info("received notification of fingerprint change ({0})".\
                        format(fingerprint))
        self.fingerprint = fingerprint
        self.mod = imp.load_source("source", self.source)

    def __getattr__(self, attr):
        return getattr(self.mod, attr)

def main():
    logging.basicConfig(level=logging.INFO, 
                        filename="hotswap.log")
    engine = Engine("engine.py")
    # this silly loop is a sample of how the program can be running in
    # one thread and the monitoring is performed in another.
    while True:
        engine.f1()
        engine.f2()
        time.sleep(1)

if __name__ == "__main__":
    main()

Processing some file before starting app and reacting for every change

4 Answers4