Django: Execute code only for `manage.py runserver`, not for `migrate`, `help` etc

Question

We are using Django as backend for a website that provides various things, among others using a Neural Network using Tensorflow to answer to certain requests.

For that, we created an AppConfig and added loading of this app config to the INSTALLED_APPS in Django's settings.py. This AppConfig then loads the Neural Network as soon as it is initialized:

settings.py:

INSTALLED_APPS = [
...
    'bert_app.apps.BertAppConfig',
]

.../bert_apps/app.py:

class BertAppConfig(AppConfig):
    name = 'bert_app'
    if 'bert_app.apps.BertAppConfig' in settings.INSTALLED_APPS:
        predictor = BertPredictor() #loads the ANN.

Now while that works and does what it should, the ANN is now loaded for every single command run through manage.py. While we of course want it to be executed if you call manage.py runserver, we don't want it to be run for manage.py migrate, or manage.py help and all other commands.

I am generally not sure if this is the proper way how to load an ANN for a Django-Backend in general, so does anybody have any tips how to do this properly? I can imagine that loading the model on startup is not quite best practice, and I am very open to suggestions on how to do that properly instead.

However, there is also some other code besides the actual model-loading that also takes a few seconds and that is definitely supposed to be executed as soon as the server starts up (so on manage.py runserver), but also not on manage.py help (as it takes a few seconds as well), so is there some quick fix for how to tell Django to execute it only on runserver and not for its other commands?

Your code snippets look like you have been mixing Django and Flask code. — Klaus D., Nov 30 '20 at 11:07
"I am generally not sure if this is the proper way how to load" -> generally the proper way to initialize any extras is from your AppConfig. Note however, that this will be done before forking. Depending on the web server you use, you can have multiple processes, and they might be killed in the middle. So if your neural network should persist some data in RAM across requests, then you might require a completely different approach. — Art, Nov 30 '20 at 11:27
@KlausD. Why do you think so? What looks like Flask code to you? — Chris Stenkamp, Nov 30 '20 at 11:31
@Art So you mean, generally what we did here is correct as long as we don't use multiple processes? What would you suggest in case we would? My original suggestion was to serve the model as microservice, but the team told me that would be overengineering... — Chris Stenkamp, Nov 30 '20 at 11:31
@ChrisStenkamp well, you can also always shoot yourself in the foot without multiprocessing :) What you did is correct if you don't need to persist changed ANN state between the requests. First, answer the following question: do requests update your ANN state in memory such, that you want to see this updated state when you make your next web request? If no, continue what you're doing. If yes, well, things get more complicated: you need to run ANN separately (e.g. in a separate manage.py command) and use some form of IPC/RPC to communicate to it. — Art, Nov 30 '20 at 11:44

Art · Accepted Answer · 2020-11-30T11:51:20.913

6

I had a similar problem, solved it with checking argv.

class SomeAppConfig(AppConfig):

    def ready(self, *args, **kwargs):
        is_manage_py = any(arg.casefold().endswith("manage.py") for arg in sys.argv)
        is_runserver = any(arg.casefold() == "runserver" for arg in sys.argv)

        if (is_manage_py and is_runserver) or (not is_manage_py):
            init_your_thing_here()

Now a bit closer to the if not is_manage_py part: in production you run your web server with uwsgi/uvicorn/..., which is still a web server, except it's not run with manage.py. Most likely, it's the only thing that you will ever run without manage.py

Use AppConfig.ready() - it's intended for it:

Subclasses can override this method to perform initialization tasks such as registering signals. It is called as soon as the registry is fully populated. - [django documentation]

To get your AppConfig back, use:

from django.apps import apps
apps.get_app_config(app_name)
# apps.get_app_configs()  # all

edited Nov 30 '20 at 11:51

answered Nov 30 '20 at 11:22

Art

2,235
18
34

1

Thank you, this seems to work! :) One thing however: So far, we don't even create an instance of the AppConfig (as you see in the question, `predictor` is simply an attribute of the class). So when we use it , we use `from bert_app.apps import BertAppConfig`, `BertAppConfig.predictor.do_stuff()` elsewhere. In your answer, the AppConfig seems to be OOP. How would we access the instance of it lateron? Singleton-Pattern maybe? And when will `ready()` be called by Django? – Chris Stenkamp Nov 30 '20 at 11:40
@ChrisStenkamp django will create an instance of it for you. Use the `ready()` method, it's specifically made for it. To retrieve the object you created inside of your `AppConfig`, just get your `AppConfig`, of even store this into some module-wide (global) variable (better not in the `apps.py` but somewhere else). And please, please don't implement your own singletons in Python - there are modules for it. I updated the answer with infor about `AppConfig`. – Art Nov 30 '20 at 11:54

score 0 · Answer 2 · answered Nov 30 '20 at 11:21

This is another way, in your manage.py will have something probably look like this

def main():
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'slambook.settings')
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)
    # check if has runserver
    if `runserver` in sys.argv:
        #execute your custom function


if __name__ == '__main__':
    main()

you can check sys.argv if it have runserver, if so then execute your script or function

But doing it this way, nothing is loaded on other commands than runserver, and for example the `help` commands simply exits without returning anything.. — Chris Stenkamp, Nov 30 '20 at 11:41

Django: Execute code only for `manage.py runserver`, not for `migrate`, `help` etc

2 Answers2

Linked