MySQL database breaks while running too many tests in django

Question

I have a Django project with hundreds of tests and after adding about 50 more tests, when I run all of the tests at once using python manage.py test, almost all of the tests raise the following error:

Traceback (most recent call last):
  File ".../venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
    cursor.execute(sql, params)
  File ".../venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File ".../venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File ".../venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 79, in _execute
    self.db.validate_no_broken_transaction()
  File ".../venv/lib/python3.9/site-packages/django/db/backends/base/base.py", line 437, in validate_no_broken_transaction
    raise TransactionManagementError(
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

During handling of the above exception, another exception occurred:
  File ".../venv/lib/python3.9/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File ".../venv/lib/python3.9/site-packages/django/db/models/query.py", line 653, in first
    for obj in (self if self.ordered else self.order_by('pk'))[:1]:
  File ".../venv/lib/python3.9/site-packages/django/db/models/query.py", line 274, in __iter__
    self._fetch_all()
  File ".../venv/lib/python3.9/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File ".../venv/lib/python3.9/site-packages/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File ".../venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1145, in execute_sql
    cursor.close()
  File ".../venv/lib/python3.9/site-packages/MySQLdb/cursors.py", line 83, in close
    while self.nextset():
  File ".../venv/lib/python3.9/site-packages/MySQLdb/cursors.py", line 137, in nextset
    nr = db.next_result()
MySQLdb._exceptions.OperationalError: (2006, '')

But if I run the tests individually, all of the tests run normally and succeed. So I'm assuming there is an issue with the database when it runs several tests at once, but I can't figure out what it is. I have already tried the solutions on this question but my problem doesn't seem to be the same. I also made this question but I decided to make another one since I did not know at the time that the problem is probably related to the amount of tests in the database.

Does anyone know how to solve this problem?

MySQL error logs:

2022-08-04T15:17:47.816192Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 1104
2022-08-04T15:17:47.841975Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-04T15:17:48.487357Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-04T15:17:48.814069Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-04T15:17:48.814094Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-04T15:17:48.866968Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-04T15:17:48.867002Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30-0ubuntu0.22.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
2022-08-05T02:50:34.190929Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-08-05T02:50:35.768640Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.22.04.1)  (Ubuntu).
2022-08-05T16:01:36.811311Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 1119
2022-08-05T16:01:36.830991Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-05T16:01:37.492375Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-05T16:01:37.799605Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-05T16:01:37.799627Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-05T16:01:37.824311Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-05T16:01:37.824331Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30-0ubuntu0.22.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
2022-08-06T00:32:10.132946Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-08-06T00:32:12.839068Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.22.04.1)  (Ubuntu).
2022-08-06T21:28:21.218813Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 1134
2022-08-06T21:28:21.243315Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-06T21:28:21.867888Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-06T21:28:22.188538Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-06T21:28:22.188561Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-06T21:28:22.209764Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-06T21:28:22.209797Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30-0ubuntu0.22.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
2022-08-08T11:51:47.706954Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-08-08T11:51:48.491678Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.22.04.1)  (Ubuntu).
2022-08-08T11:51:50.428184Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 1115
2022-08-08T11:51:50.444404Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-08T11:51:50.956962Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-08T11:51:51.322340Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-08T11:51:51.322369Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-08T11:51:51.359037Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-08T11:51:51.359047Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30-0ubuntu0.22.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
2022-08-08T21:36:15.186489Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-08-08T21:36:17.194038Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 11  user: 'root'.
2022-08-08T21:36:17.194179Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 852  user: 'root'.
2022-08-08T21:36:17.194293Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 853  user: 'root'.
2022-08-08T21:36:18.316093Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.22.04.1)  (Ubuntu).
2022-08-08T21:42:06.980028Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 5459
2022-08-08T21:42:06.992543Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-08T21:42:07.380065Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-08T21:42:07.648877Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-08T21:42:07.648897Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.                                                
2022-08-08T21:42:07.674744Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock                                             
2022-08-08T21:42:07.674766Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30-0ubuntu0.22.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).              
2022-08-04T15:17:47.816192Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 1104
2022-08-04T15:17:47.841975Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-04T15:17:48.487357Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-04T15:17:48.814069Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-04T15:17:48.814094Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-04T15:17:48.866968Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-04T15:17:48.867002Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30-0ubuntu0.22.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
2022-08-05T02:50:34.190929Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-08-05T02:50:35.768640Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.22.04.1)  (Ubuntu).
2022-08-05T16:01:36.811311Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 1119
2022-08-05T16:01:36.830991Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-05T16:01:37.492375Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-05T16:01:37.799605Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-05T16:01:37.799627Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-05T16:01:37.824311Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-05T16:01:37.824331Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30-0ubuntu0.22.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
2022-08-06T00:32:10.132946Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-08-06T00:32:12.839068Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.22.04.1)  (Ubuntu).
2022-08-06T21:28:21.218813Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 1134
2022-08-06T21:28:21.243315Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-06T21:28:21.867888Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-06T21:28:22.188538Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-06T21:28:22.188561Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-06T21:28:22.209764Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-06T21:28:22.209797Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30-0ubuntu0.22.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
2022-08-08T11:51:47.706954Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-08-08T11:51:48.491678Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.22.04.1)  (Ubuntu).
2022-08-08T11:51:50.428184Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 1115
2022-08-08T11:51:50.444404Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-08T11:51:50.956962Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-08T11:51:51.322340Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-08T11:51:51.322369Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-08T11:51:51.359037Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-08T11:51:51.359047Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30-0ubuntu0.22.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
2022-08-08T21:36:15.186489Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-08-08T21:36:17.194038Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 11  user: 'root'.
2022-08-08T21:36:17.194179Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 852  user: 'root'.
2022-08-08T21:36:17.194293Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 853  user: 'root'.
2022-08-08T21:36:18.316093Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.22.04.1)  (Ubuntu).
2022-08-08T21:42:06.980028Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 5459
2022-08-08T21:42:06.992543Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-08T21:42:07.380065Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-08T21:42:07.648877Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-08T21:42:07.648897Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.                                                
2022-08-08T21:42:07.674744Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock                                             
2022-08-08T21:42:07.674766Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30-0ubuntu0.22.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).

MySQL configurations:

[mysqld]
user        = mysql
# pid-file  = /var/run/mysqld/mysqld.pid
# socket    = /var/run/mysqld/mysqld.sock
# port      = 3306
# datadir   = /var/lib/mysql


# tmpdir        = /tmp
#
bind-address        = 127.0.0.1
mysqlx-bind-address = 127.0.0.1
key_buffer_size     = 16M
# max_allowed_packet    = 64M
# thread_stack      = 256K

# thread_cache_size       = -1

myisam-recover-options  = BACKUP

# max_connections        = 151

# table_open_cache       = 4000

max_allowed_packet = 640M
wait_timeout = 28800000
interactive_timeout = 28800000
innodb_log_file_size = 256MB
innodb_buffer_pool_size = 12000MB

mysql> SELECT @@global.wait_timeout, @@session.wait_timeout, @@global.max_allowed_packet, @@global.innodb_log_file_size, @@global.innodb_buffer_pool_size;
+-----------------------+------------------------+-----------------------------+-------------------------------+----------------------------------+
| @@global.wait_timeout | @@session.wait_timeout | @@global.max_allowed_packet | @@global.innodb_log_file_size | @@global.innodb_buffer_pool_size |
+-----------------------+------------------------+-----------------------------+-------------------------------+----------------------------------+
|              28800000 |               28800000 |                   671088640 |                     268435456 |                      12884901888 |
+-----------------------+------------------------+-----------------------------+-------------------------------+----------------------------------+
1 row in set (0,00 sec)

I'm using python 3.9, Django 3.2, and MySQL 8.0.23. My OS is Ubuntu 22.04

I'm not very experienced in MySQL, but the only error logs I found are under /var/logs/mysql/error.log, and there is not a lot of relevant information. I will be posting it anyway. If there are other MySQL logs that you want to see please let me know — Romero Cartaxo, Aug 08 '22 at 21:50
These logs are form the moment that `python manage.py test` was running? — Hector Vido, Aug 09 '22 at 22:33
Hum... I deleted my answer. Now I saw that 2006 error was after the first exception. I think is is not related to MySQL, but django. — Hector Vido, Aug 19 '22 at 17:45

score 0 · Accepted Answer · answered Sep 05 '22 at 20:27

I figured it out. Some of my tests had the following structure

class MyTest(TestCase):

@classmethod
def setUpClass(cls):
    super().setUpClass()
    ...
def setUp(self):
    ...
def test_something(self):
   ...

In my update, I had made significant changes and some of these changes broke the code inside the setUpClass method. When that happened the database broke somehow and all of the tests after this one would fail as well, because there was no database anymore.

To prevent this use the setUpTestData method. This method is called by the setUpClass, which means it will run once before all the tests in the file, but I will not break if there is some error inside it. (You also don't have to call super when using this method).

MySQL database breaks while running too many tests in django

1 Answers1