Why am I very-occasionally getting an ImportError using django-constance?

Question

Once every few hundred thousand requests I see one of these:

ImportError at /
cannot import name 'Config' from partially initialized module 'constance.base' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/constance/base.py)

I cannot identify any rhyme or reason. It doesn't correspond with accessing constance admin, just randomly occurs. My best guess is it's something to do with the LazyObject in constance's __init__.py, and maybe random race-conditions in restarting expired gunicorn workers or something?!

I'm using:

Python 3.9.2
Django 3.2
django-constance = {extras = ["database"],version = "==2.8.*"}
"constance" and "constance.backends.database" in INSTALLED_APPS (at top)
CONSTANCE_BACKEND = "constance.backends.database.DatabaseBackend"
"constance.context_processors.config" in TEMPLATES[0]["OPTIONS"]["context_processors"]

All my code does is from constance import config and access config attributes in the standard way in python code and Django templates.

For what it's worth, we've been using django-constance on this site for years, but never saw this error until we upgraded to 2.8.0 (from 2.6.0). We were using Django 3.1 when it first appeared, but has also occurred since upgrading to 3.2.

I cannot find any similar error reports on https://github.com/jazzband/django-constance/

Any ideas what could be causing this and how to resolve it?

Errors like this can be caused by module naming conflicts ([1](https://stackoverflow.com/questions/66276332/python-attributeerror-partially-initialized-module), [2](https://stackoverflow.com/questions/59762996/how-to-fix-attributeerror-partially-initialized-module) to name just a few examples). Have you looked into that? — Thomas, Aug 28 '21 at 08:16
Yes, I've searched for that possibility but don't think it is the case. Also because it is reporting `constance.base` and I definitely don't have any files named that. — DrMeers, Aug 28 '21 at 12:04

FazeL · Answer 1 · 2021-09-02T08:21:13.780

2

This was a bug in constance that has been solved in this pull request.

The root of the problem is that prior to this pull request constance testsuite didn't run tests on Django 3.2, which is been fixed and the changes that caused your error are also outed and fixed.

That means in constance's __init__.py, now there is an if clause separating current handling of importation in Django 3.2 from the older Django versions.

edited Sep 02 '21 at 08:21

answered Sep 02 '21 at 08:14

FazeL

916
2
13
30

Hmm, I had already seen that as-yet-unreleased commit, but I don't see how it could fix the issue. Surely it just addresses [the deprecation of the variable](https://docs.djangoproject.com/en/3.2/releases/3.2/#automatic-appconfig-discovery) but wouldn't actually affect import mechanisms as Django will still use `constance.apps.ConstanceConfig` automatically, it just doesn't need to be told explicitly to do so after Django 3.2. – DrMeers Sep 02 '21 at 11:58
@DrMeers An `import django` was added to top and there was an explicitly check for django to be lower than 3.2, It isn't far fetched to assume there was a bug with the new `AppConfig` system that required django import and not defining the variable since it is very prone to circular imports. – FazeL Sep 03 '21 at 10:36
2

I am the author of that PR and I can say for sure that it did not fix any bug. All I did was run tests over django 3.2 and add a django version check to avoid warnings. `RemovedInDjango41Warning: 'constance' defines default_app_config = 'constance.apps.ConstanceConfig'. Django now detects this configuration automatically. You can remove default_app_config.` – Mogost Sep 03 '21 at 11:12
I've managed to somewhat diagnose the issue today -- see my answer below, and reported GitHub issue – DrMeers Sep 15 '21 at 01:01

score 2 · Answer 2 · answered Sep 15 '21 at 00:56

OK, I've managed to reproduce the issue and track down the likely cause.

I managed to trigger it once using Django's runserver, on the first request handled, but in hundreds of subsequent restarts/attempts I couldn't make it recur.

I then instead ran a couple of local gunicorn worker threads with a low requests-per-worker threshold, and spammed the local port with rapid-fire requests, and sure enough the ImportError occurs every now and then.

The issue seems to be this:

constance.__init__ uses a django.utils.functional.LazyObject which, once lazily evaluated, imports constance.base and instantiates a Config from that module.
constance.base.Config.__init__ uses constance.utils.import_module_attr to import the backend specified in the project settings, which in turn uses importlib.import_module to import, in my case, constance.backends.database.DatabaseBackend.
constance.backends.database.__init__ also imports config from ... (constance), thus potentially creating an import loop.

It seems there is a rare race condition in which LazyConfig._setup tries to import constance.base.Config while constance.base hasn't be fully initialised. I injected some debugging statements to demonstrate, and the buggy sequence looks like this:

[2021-09-15 09:35:10 +1000] [13504] [INFO] Booting worker with pid: 13504
BEGIN constance.__init__.LazyConfig._setup() vars(self): {'_wrapped': <object object at 0x108da8340>}
BEGIN constance.__init__.LazyConfig._setup() vars(self): {'_wrapped': <object object at 0x108da8340>}
constance.base imported
constance.base.Config.__init__: importing constance.backends.database.DatabaseBackend
ImportError at /
cannot import name 'Config' from partially initialized module 'constance.base' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/constance/base.py) 
EXIT constance.__init__.LazyConfig._setup() vars(self): {'_wrapped': <constance.base.Config object at 0x10fa7ea30>}

It seems to only happen when _setup is called twice in a row like that, before constance.base is imported, then the two threads seem to race from that point.

I guess I'll open a GitHub issue on the django-constance repository to resolve the issue.

Reported here: https://github.com/jazzband/django-constance/issues/452 — DrMeers, Sep 15 '21 at 01:00
Oh, I see, there are stuff that doesn't work right out of the box with gthreads, and it doesn't give a real advantage except concurrent handling more idle things(which is big, but not enough to pay for a thread-safe code). Most solutions I saw either use sync, or gevent/eventlet, since in between there is little. — FazeL, Sep 23 '21 at 21:52
Well it did also occur using Django's runserver, so I'm not convinced it's gthread-specific. — DrMeers, Sep 24 '21 at 06:30

Why am I very-occasionally getting an ImportError using django-constance?

2 Answers2