0

I have joined a team working on a Django project, and I am trying to setup the local mysql database schema. This project has existed for a long time, so there are existing migrations.

I have created the empty database in mysql, but when I try to apply migrations to create the schema, it complains that the tables don't exist yet.

$ ./manage.py migrate

Traceback (most recent call last):
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute
    return self.cursor.execute(query, args)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 250, in execute
    self.errorhandler(self, exc, value)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
    raise errorvalue
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 247, in execute
    res = self._query(query)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 411, in _query
    rowcount = self._do_query(q)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 374, in _do_query
    db.query(q)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/connections.py", line 277, in query
    _mysql.connection.query(self, query)
_mysql_exceptions.ProgrammingError: (1146, "Table 'api_test.headset_product' doesn't exist")

I've also tried syncdb, migrate --run-syncdb, and makemigrations with the same results. I've also tried loaddata with our fixtures, with the same results.

I've read Django : Table doesn't exist, but I didn't drop tables in this case (no tables have ever existed), and I don't want to recreate all the migrations (I'm not the only person on this project).

I have a related project that uses a local postgresql database, and it doesn't seem to have this problem. I can run migrate and it does just what I expect, so it feels that I've run into a problem specific to this codebase, but we aren't certain how to debug it.

"full output of makemigrations, including the full typed command."

$ ./manage.py makemigrations
[WARNING] statsd not configured

[WARNING] SMTP not configured; emails will not be sent

[WARNING] V1 database access not configured

Traceback (most recent call last):
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute
    return self.cursor.execute(query, args)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 250, in execute
    self.errorhandler(self, exc, value)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
    raise errorvalue
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 247, in execute
    res = self._query(query)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 411, in _query
    rowcount = self._do_query(q)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 374, in _do_query
    db.query(q)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/connections.py", line 277, in query
    _mysql.connection.query(self, query)
_mysql_exceptions.ProgrammingError: (1146, "Table 'reign_api_test.headset_product' doesn't exist")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./manage.py", line 14, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/management/base.py", line 332, in execute
    self.check()
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/management/base.py", line 364, in check
    include_deployment_checks=include_deployment_checks,
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/management/base.py", line 351, in _run_checks
    return checks.run_checks(**kwargs)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/checks/registry.py", line 73, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/checks/urls.py", line 13, in check_url_config
    return check_resolver(resolver)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/core/checks/urls.py", line 23, in check_resolver
    return check_method()
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/urls/resolvers.py", line 397, in check
    for pattern in self.url_patterns:
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/utils/functional.py", line 36, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/urls/resolvers.py", line 536, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/utils/functional.py", line 36, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/urls/resolvers.py", line 529, in urlconf_module
    return import_module(self.urlconf_name)
  File "/Users/rnapier/.pyenv/versions/3.6.6/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/rnapier/clients/Jaybird/Source/reign-api/reign_api/reign_api/urls.py", line 13, in <module>
    import newsfeed.urls
  File "/Users/rnapier/clients/Jaybird/Source/reign-api/reign_api/newsfeed/urls.py", line 2, in <module>
    from newsfeed import views
  File "/Users/rnapier/clients/Jaybird/Source/reign-api/reign_api/newsfeed/views.py", line 15, in <module>
    from newsfeed import models, serializers
  File "/Users/rnapier/clients/Jaybird/Source/reign-api/reign_api/newsfeed/serializers.py", line 27, in <module>
    class NewsfeedSubscriptionSerializer(serializers.Serializer):
  File "/Users/rnapier/clients/Jaybird/Source/reign-api/reign_api/newsfeed/serializers.py", line 28, in NewsfeedSubscriptionSerializer
    PRODUCTS = [f for f in headset.models.Product.objects.values_list("name", flat=True)]
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/models/query.py", line 272, in __iter__
    self._fetch_all()
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/models/query.py", line 1179, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/models/query.py", line 178, in __iter__
    for row in compiler.results_iter(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size):
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1017, in results_iter
    results = self.execute_sql(MULTI, chunked_fetch=chunked_fetch, chunk_size=chunk_size)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1066, in execute_sql
    cursor.execute(sql, params)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/utils.py", line 100, in execute
    return super().execute(sql, params)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute
    return self.cursor.execute(query, args)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 250, in execute
    self.errorhandler(self, exc, value)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
    raise errorvalue
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 247, in execute
    res = self._query(query)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 411, in _query
    rowcount = self._do_query(q)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/cursors.py", line 374, in _do_query
    db.query(q)
  File "/Users/rnapier/.pyenv/versions/api-3.6.6/lib/python3.6/site-packages/MySQLdb/connections.py", line 277, in query
    _mysql.connection.query(self, query)
django.db.utils.ProgrammingError: (1146, "Table 'reign_api_test.headset_product' doesn't exist")

"model structure for headset_product"

class Product(models.Model):
    """
    products supported by the Headset API
    """

    name = models.CharField(max_length=128, unique=True)

    def __str__(self) -> str:
        return "<Product {s.id} {s.name}>".format(s=self)

    __repr__ = __str__
Rob Napier
  • 286,113
  • 34
  • 456
  • 610
  • Show us the full output of `makemigrations`, including the full typed command. – John Gordon Nov 03 '18 at 16:50
  • Post db configuration in your settings – a_k_v Nov 03 '18 at 16:53
  • post model structure for headset_product. Problem in the model structure can also cause this problem – a_k_v Nov 03 '18 at 16:59
  • @JohnGordon Updated. @a_k_v I updated with headset_product, but I'm not certain which db configuration you mean (my Django is still a bit shaky). The environment includes `DATABASE_URL=mysql://reign_api_test:reign_api_test@127.0.0.1:3306/reign_api_test` – Rob Napier Nov 03 '18 at 17:08
  • (My teammates have working environments, but they've had them working for a long time and don't remember exactly how they got there.) – Rob Napier Nov 03 '18 at 17:09
  • Are all of the previous migrations files present in the `migrations` subdirectory of your app? – John Gordon Nov 03 '18 at 17:11

1 Answers1

3

The problem is in reign_api/newsfeed/serializers.py, which is trying to make a query at class or module level. This should always be avoided in Django, as the query will be run at first import - before the migrations on a new system will have a chance to run. But this will be causing other problems too, in that that query will only be executed once per process, and will therefore not update when new data is added or changed.

Your colleagues should fix this problem, either by refactoring to move the query into a method, or probably better by using a more appropriate serializer field that takes a queryset and does the query when called.

Daniel Roseman
  • 588,541
  • 66
  • 880
  • 895