When and where do I create configurable custom groups in Django without running in "no such table: auth_group" on startup?

Question

I'm still learning and working on an existing Django app (v1.10.) for which I am trying to switch existing custom groups to making them dynamically configurable, so that the app can be used in different contexts. I have an ini file, which includes my dynamic groups:

; Custom groups and permission settings
[platform_group_list]
TEAM = policy_team
LEAD = policy_team_lead

[platform_group_value_list]
TEAM = Policy Team Member
LEAD = Policy Team Lead

which I parse in settings.py:

PLATFORM_GROUP_LIST = raw_parser.items("platform_group_list")
PLATFORM_GROUP_VALUE_LIST = raw_parser._sections["platform_group_value_list"]

The original app created groups and permissions inside migrations. I commented out the part there as it kept throwing errors regarding missing permissions and thought that maybe it would be better to do this in admin.py since this is where the groups "belong". So I have:

# admin.py (removed custom permissions, not relevant atm)
from django.contrib import admin
from django.contrib.auth.admin import UserAdmin as BaseUserAdmin
from django.contrib.contenttypes.models import ContentType
from django.contrib.auth.models import Group, Permission, User
from django.contrib.auth import models as auth_models
from django.core.exceptions import ObjectDoesNotExist
from django.conf import settings
from django.utils.translation import ugettext as _

from account.models import SignupCodeResult
from .models import UserConfig

# BACKCOMPAT
# add staff to the backcompat groups and vice versa, then remove the 
# backcompat groups again. Used to be in migrations, fails when setting up
# where custom permissions don't exist. Switch to configurable groups

def create_group(name):
  return Group.objects.get_or_create(name=name)

def delete_group(name):
  try:
    Group.objects.get(name=name).delete()
  except ObjectDoesNotExist:
    # pass
    raise Exception("admin: trying to delete non-existing group")

def get_permission(app, name):
  try:
    return Permission.objects.get(content_type__app_label=app, codename=name)
  except:
    return False

def backcompat_init_teams_and_permissions():
  for group_title in settings.BACKCOMPAT_GROUP_LIST:
    new_group, created = create_group(group_title)

    if created:
      for user in User.objects.filter(is_staff=True, is_active=True):
        user.groups.add(new_group)

      for permission_name in settings.BACKCOMPAT_PERMISSION_LIST:
        new_permission = get_permission("initproc", permission_name)
        if new_permission:
          new_group.permissions.add(new_permission)

def backcompat_reverse_teams_and_permissions():
  for group_title in settings.BACKCOMPAT_GROUP_LIST:
    for user in User.objects.filter(groups__name=group_title, is_active=True):
      user.is_staff = True
      user.save()
  delete_group(group_title)

# new and dynamic
def create_custom_groups_and_permissions():
  for (group_key, group_title) in settings.PLATFORM_GROUP_LIST:
    new_group, created = create_group(settings.PLATFORM_GROUP_VALUE_LIST[group_key])
    if created:
      new_group.save()

class UserConfigInline(admin.StackedInline):
  model = UserConfig
  can_delete = False
  verbose_name_plural = 'config'

# Define a new User admin
class UserAdmin(BaseUserAdmin):
  inlines = (UserConfigInline, )

# Re-register UserAdmin
admin.site.unregister(User)
admin.site.register(User, UserAdmin)
admin.site.register(SignupCodeResult)
admin.site.register(Permission)

# XXX where to call all of these?

# backcompat - move existing staff to custom group and custom group to staff
backcompat_init_teams_and_permissions()
backcompat_reverse_teams_and_permissions()

# define custom groups as per init.ini
create_custom_groups_and_permissions()

This works fine when the app is up and running, but when I set it up from scratch, I cannot call runserver, makemigrations or migrate as the all fail on

return Group.objects.get_or_create(name=name)

with django.db.utils.OperationalError: no such table: auth_group.

I figured this means the table isn't created when I'm already trying to access it.

I can get my app to work by commenting out the above calls plus another error when trying to access groups in forms.py (stack below in case useful). Once commented out, I can call migrate and when I start my app and uncommented again, groups (and permissions) are correctly added. So for me this is just a "chicken and egg" problem.

Question:
When and where should I create my custom groups to not run into no such table: auth_group from accessing the table before its created? I could move everything back into migrations, but I don't think it's the right/ideal place. admin.py seems "too early" and I didn't find something like a signal that tells me "all database tables are set".

Thanks for helping!

Here is the error stacktrace:

Traceback (most recent call last):
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 337, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such table: auth_group

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "abstimmungstool/manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/core/management/base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/core/management/base.py", line 342, in execute
    self.check()
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/core/management/base.py", line 374, in check
    include_deployment_checks=include_deployment_checks,
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/core/management/base.py", line 361, in _run_checks
    return checks.run_checks(**kwargs)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/core/checks/registry.py", line 81, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/core/checks/urls.py", line 14, in check_url_config
    return check_resolver(resolver)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/core/checks/urls.py", line 24, in check_resolver
    for pattern in resolver.url_patterns:
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/urls/resolvers.py", line 313, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/urls/resolvers.py", line 306, in urlconf_module
    return import_module(self.urlconf_name)
  File "/usr/local/demo/venv/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/demo/abstimmungstool/voty/urls.py", line 38, in <module>
    ) + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/conf/urls/__init__.py", line 50, in include
    urlconf_module = import_module(urlconf_module)
  File "/usr/local/demo/venv/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/demo/abstimmungstool/voty/initadmin/urls.py", line 13, in <module>
    from . import views
  File "/usr/local/demo/abstimmungstool/voty/initadmin/views.py", line 33, in <module>
    from .forms import (UploadFileForm, LoginEmailOrUsernameForm, UserEditForm,
  File "/usr/local/demo/abstimmungstool/voty/initadmin/forms.py", line 93, in <module>
    class UserModerateForm(forms.ModelForm):
  File "/usr/local/demo/abstimmungstool/voty/initadmin/forms.py", line 140, in UserModerateForm
    widget=forms.CheckboxSelectMultiple,choices=[(x.id, x.name) for x in Group.objects.all()],
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __iter__
    self._fetch_all()
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/db/models/query.py", line 1087, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/db/models/query.py", line 54, in __iter__
    results = compiler.execute_sql()
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/demo/venv/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 337, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: auth_group

I did, it throws the same error as above, `no such table: auth_group` — frequent, Jul 18 '18 at 12:01
ok, per this [comment](https://stackoverflow.com/a/36224148), the current `admin.py` is not the right place, this is just the error I received. — frequent, Jul 18 '18 at 18:41

score 0 · Answer 1 · answered Jul 19 '18 at 08:53

After some research I found that a custom management command fits best blog, docs). I created a set_groups_and_permissions command like so:

# -*- coding: utf-8 -*-
from django.core.management import BaseCommand
from django.contrib.contenttypes.models import ContentType
from django.contrib.auth.models import Group, Permission, User
from django.contrib.auth import models as auth_models
from django.core.exceptions import ObjectDoesNotExist
from django.conf import settings
from django.utils.translation import ugettext as _

# BACKCOMPAT
# add all staff to the defined backcompat role and vice versa, then remove the 
# backcompat groups again. Used to be in migrations, fails when setting up
# where custom permissions don't exist. Plus, custom groups should be
# configurable, so we replicate what was done here, then switch to configurable
# custom groups and permissions
def create_group(name):

  # returns group, created (True/False)
  return Group.objects.get_or_create(name=name)

def delete_group(name):
  try:
    Group.objects.get(name=name).delete()
  except ObjectDoesNotExist:
    # pass
    raise Exception("initadmin: trying to delete non-existing group")

def get_permission(app, name):
  try:
    return Permission.objects.get(content_type__app_label=app, codename=name)
  except:
    return False

def backcompat_init_teams_and_permissions():
  for group_title in settings.BACKCOMPAT_ROLE_LIST:
    new_group, created = create_group(group_title)

    if created:
      for user in User.objects.filter(is_staff=True, is_active=True):
        user.groups.add(new_group)

      for permission_name in settings.BACKCOMPAT_PERMISSION_LIST:
        new_permission = get_permission("initproc", permission_name)
        if new_permission:
          new_group.permissions.add(new_permission)

def backcompat_reverse_teams_and_permissions():
  for group_title in settings.BACKCOMPAT_ROLE_LIST:
    for user in User.objects.filter(groups__name=group_title, is_active=True):
      user.is_staff = True
      user.save()
  delete_group(group_title)

def create_custom_groups_and_permissions():
  for (group_key, group_title) in settings.PLATFORM_GROUP_LIST:
    new_group, created = create_group(settings.PLATFORM_GROUP_VALUE_LIST[group_key])
    if created:
      new_group.save()

# ---------------------------------- Command -----------------------------------
class Command(BaseCommand):
  help = "Create groups, permissions as defined in settings.py"

  def handle(self, *args, **options):

    backcompat_init_teams_and_permissions()
    backcompat_reverse_teams_and_permissions()
    create_custom_groups_and_permissions()

    print("Groups, Permissions created.")

Regarding the other error on my forms.py, I removed the call to Groups.objects.all from my widget-choices and populate it on runtime.

class UserGiveGroupPrivilegeForm(forms.ModelForm):

  class Meta:
    model = get_user_model()
    fields = ["groups"]

  def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
    for field in iter(self.fields):
      self.fields[field].widget.attrs.update({
        'class': 'form-control'
      })

  groups = forms.MultipleChoiceField(
    #widget=forms.CheckboxSelectMultiple,choices=[(x.id, x.name) for x in Group.objects.all()],
    widget=forms.CheckboxSelectMultiple,
    required=False,
    label=_("Groups"),
    help_text=_("Please select the group(s) this user should belong to.")
  )
  action = forms.CharField(
    max_length=24,
    widget=forms.HiddenInput(),
    initial="give_group_privileges"
  )

Not optimal, but ok.

Alors with the above I can setup my app from scratch without any errors django.db.utils.OperationalError: no such table: auth_group from calls to tables before they are created.

When and where do I create configurable custom groups in Django without running in "no such table: auth_group" on startup?

1 Answers1