Gunicorn reflect changed code dynamically

Question

I am developing a django web application where a user can modify the code of certain classes, in the application itself, through UI using ace editor (think of as gitlab/github where you can change code online). But these classes are ran by django and celery worker at some point.

Once code changes are saved, the changes are not picked by django due to gunicorn but works fine with celery because its different process. (running it locally using runserver works fine and changes are picked by both django and celery).

Is there a way to make gunicorn reflects the changes of certain directory that contain the classes without reloading the whole application? and if reloading is necessary, is there a way to reload gunicorn's workers one-by-one without having any downtime?

the gunicron command:

/usr/local/bin/gunicorn config.wsgi --bind 0.0.0.0:5000 --chdir=/app

The wsgi configuration file:

import os
import sys

from django.core.wsgi import get_wsgi_application

app_path = os.path.abspath(os.path.join(
    os.path.dirname(os.path.abspath(__file__)), os.pardir))
sys.path.append(os.path.join(app_path, 'an_application'))

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.production")

application = get_wsgi_application()

have a look at this thread https://superuser.com/questions/181517/how-to-execute-a-command-whenever-a-file-changes it may be useful for you — cizario, Nov 06 '20 at 17:00
Thank you @cizario this helpful, but I am kinda hoping for a cleaner way. — Coderji, Nov 06 '20 at 17:11
***".... without reloading the whole application"*** ; reloading can be done, *"somehow"*, but, I don't think reloading can be done *without reloading the whole application* @Coderji — JPG, Nov 09 '20 at 07:27
in fact `gunicorn` has an option `--reload` (@see https://docs.gunicorn.org/en/latest/settings.html#debugging) to reload on code changes **BUT** it's **NOT** intended for **PRODUCTION** environment. — cizario, Nov 09 '20 at 08:17
@Coderji have a look at this similar project https://www.djangosites.org/s/django-visual-herokuapp-com/, but unfortunately it doesn't auto-relaod the project upon code changes. anyway it may help you improving your project — cizario, Nov 09 '20 at 08:42
I would just trigger a rolling release of whole project instead. — Krzysztof Szularz, Nov 10 '20 at 19:29
You could write the code chunk to a file and then run it on runtime using python's `eval()`. It's not really a safe operation, but should do the job. — João Victor Monte, Nov 12 '20 at 01:28
@KrzysztofSzularz this would be interesting, I couldn't find that gunicorn capable of doing a rolling release. do you have something in mind — Coderji, Nov 12 '20 at 16:21
@JoãoVictorMonte I am afraid that I will eventually rely on `eval` which I was trying to avoid. — Coderji, Nov 12 '20 at 16:21
I think it would be useful if you could give a specific example on where and when this dynamically modified classes would be used. There certainly are several approaches to this, but pros and cons depend on the specifics of the use case. — ivissani, Nov 13 '20 at 02:11
You could set `gunicorn` to autoreload on source change using `watchdog`? https://stackoverflow.com/a/19502993/10625611 — Qumber, Nov 13 '20 at 05:23
@Coderji I was thinking of triggering the "regular" of the app, like in Kubernetes or however you're hosting the app. — Krzysztof Szularz, Nov 13 '20 at 09:11
@JoãoVictorMonte I think I am going with your answer and since you suggested it first, then I will give you the bounty. Can you write an answer. — Coderji, Nov 15 '20 at 12:22
@Coderji I think you should give the bounty to Aman Garg, because exec is more appropriate and he built a very good formated answear. I'm grateful for the recognize tho. =) — João Victor Monte, Nov 15 '20 at 15:03

score 1 · Answer 1 · answered Nov 13 '20 at 15:13

1

The reload option is "intended for development". There's no strong wording saying you shouldn't use it in production. The reason you shouldn't use it in production is because people make typos, change in one file, may need several other changes in others, etc etc. So, you can make your site inaccessible and then you don't have a working app to fix it again.

For a dev, that's no problem as you look at the logs/output in your shell and restart it. This is why @Krzysztof's suggestion is the best one. Push the code changes to your repo, make it go through the CI/CD and switch over the pod. If CI fails, then CD won't happen so you're good.

Of course, that's a scope far too large for a Q&A site.

answered Nov 13 '20 at 15:13

Thank you for your answer, I will give the watching files a chance, tho I was trying to find another solution. The way it is currently implemented is to push to git and go through CI/CD to update. but release of the application to production happens every 2 weeks (before each sprint) and for these kind of classes, we cannot wait for such long time. and cherry picking the updates to push to release branch is not really a good approach we found at the moment. Therefore, the client suggested to modify these classes on the fly. – Coderji Nov 15 '20 at 11:39
If these are bugfixes, then teach your team to use bug fixing branches, branched off current production branch. I don't understand why some classes need changing in short time for other reasons. Are you using `choices`, where you should be using foreign keys? Can you give an example of such a model that needs frequent updates? – Nov 15 '20 at 16:55
The application is hosting ML models were each model has its own script for pre/post processing for each model. Even though the model call can be generalized and uploaded, you cannot generalize the processing part completely. Additionally, the algorithms are not all python, they can be C++ and matlab which the "code data scientist change" can call matlab runtime or Boost class. Eventually, we are versioning the each model, the new version might contain new processings that they need to update, but you still need to keep the old version so the requesters don't break their pipeline. – Coderji Nov 15 '20 at 17:19

Aman Garg · Answer 2 · 2020-11-13T18:30:53.507

Why not save the code in a separate text file or database and the relevant method can simply load the code dynamically as a string and execute it using exec()?

Let say you have a function function1 which can be edited by a user. When the user submits the changes, process the input (separate out the functions so that you know which function has what definition), and save them all individually, like function1, function2 etc., in a database or a text file as strings.

One you need to execute function1, just load its value that you saved and use exec to execute the code.

This way, you won't need to reload gunicorn since all workers will always fetch the updated function definition at run time!

Something in the lines of:


def function1_original():
    # load function definition
    f = open("function1.txt", "r")
    
    # execute the string
    exec(f.read())  # this will just load the function definition
    function1()  # this will execute the user defined function

So the user will define:

def function1():
    # user defined code
    # blah blah
    ...

score 0 · Accepted Answer · answered Jan 12 '21 at 13:58

I was able to solve this by changing the extension of the python scripts to anything but .py

Then I loaded these files using the following function:

from importlib import util
from immportlib.machinary import SourceFileLoader

def load_module(module_name, modele_path):
    module_path = path.join(path.dirname(__file__),  "/path/to/your/files{}.anyextension".format(module_name))
    spec = util.spec_from_loader(module_name,
                                 SourceFileLoader(module_name, module_path))
    module = util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module

In this case, they are not loaded by Gunicorn in RAM and I was able to apply the changes on fly without the need to apply eval or exec functiong.

Gunicorn reflect changed code dynamically

3 Answers3