0

I have a problem: I can't do anything in a database.

It will return the following directly to me when any action is currently performed, But this error seems to be only targeted at one of my databases:

ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    8
Current database: gomeet

ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/home/mysql/software/mysql.sock' (111)
ERROR: 
Can't connect to the server

enter image description here

It seems that I send any instruction in the gomeet database and dataserver.err will return me a similar error:

InnoDB: Error: trying to access page number 0 in space 3090,
InnoDB: space name gomeet/api_worker_face_back,
InnoDB: which is outside the tablespace bounds.
InnoDB: Byte offset 0, len 16384, i/o type 10.
InnoDB: If you get this error at mysqld startup, please check that
InnoDB: your my.cnf matches the ibdata files that you have in the
InnoDB: MySQL server.
2021-05-18 17:04:07 7f0fe3206700  InnoDB: Assertion failure in thread 139706211788544 in file fil0fil.cc line 5603
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
09:04:07 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=40
max_threads=1500
thread_count=40
connection_count=40
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 604688 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x177301d0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f0fe3205ea0 thread_stack 0x40000
/home/mysql/software/bin/mysqld(my_print_stacktrace+0x2c)[0x8f016c]
/home/mysql/software/bin/mysqld(handle_fatal_signal+0x364)[0x66cb64]
/lib64/libpthread.so.0(+0xf5f0)[0x7f13417d25f0]
/lib64/libc.so.6(gsignal+0x37)[0x7f13407d1337]
/lib64/libc.so.6(abort+0x148)[0x7f13407d2a28]
/home/mysql/software/bin/mysqld[0xaa1074]
/home/mysql/software/bin/mysqld[0xa6d899]
/home/mysql/software/bin/mysqld[0xa565b1]
/home/mysql/software/bin/mysqld[0xa3e751]
/home/mysql/software/bin/mysqld[0x9ee9b6]
/home/mysql/software/bin/mysqld[0x952b18]
/home/mysql/software/bin/mysqld[0x959e20]
/home/mysql/software/bin/mysqld(_ZN7handler7ha_openEP5TABLEPKcii+0x33)[0x5aa5c3]
/home/mysql/software/bin/mysqld(_Z21open_table_from_shareP3THDP11TABLE_SHAREPKcjjjP5TABLEb+0x61c)[0x7705cc]
/home/mysql/software/bin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP18Open_table_context+0xc8b)[0x69fc0b]
/home/mysql/software/bin/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy+0x839)[0x6a7d09]
/home/mysql/software/bin/mysqld(_Z30open_normal_and_derived_tablesP3THDP10TABLE_LISTj+0x4a)[0x6a878a]
/home/mysql/software/bin/mysqld(_Z18mysqld_list_fieldsP3THDP10TABLE_LISTPKc+0x25)[0x7222d5]
/home/mysql/software/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2103)[0x6f66f3]
/home/mysql/software/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x1ed)[0x6be7dd]
/home/mysql/software/bin/mysqld(handle_one_connection+0x39)[0x6be829]
/home/mysql/software/bin/mysqld(pfs_spawn_thread+0x140)[0x9374d0]
/lib64/libpthread.so.0(+0x7e65)[0x7f13417cae65]
/lib64/libc.so.6(clone+0x6d)[0x7f134089988d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f0f6000c078): is an invalid pointer
Connection ID (thread ID): 21
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
210518 17:04:07 mysqld_safe Number of processes running now: 0
210518 17:04:07 mysqld_safe mysqld restarted
2021-05-18 17:04:07 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-05-18 17:04:07 3479 [Note] Plugin 'FEDERATED' is disabled.
2021-05-18 17:04:07 3479 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-05-18 17:04:07 3479 [Note] InnoDB: The InnoDB memory heap is disabled
2021-05-18 17:04:07 3479 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-05-18 17:04:07 3479 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-05-18 17:04:07 3479 [Note] InnoDB: Using CPU crc32 instructions
2021-05-18 17:04:07 3479 [Note] InnoDB: Initializing buffer pool, size = 12.0G
2021-05-18 17:04:08 3479 [Note] InnoDB: Completed initialization of buffer pool
2021-05-18 17:04:08 3479 [Note] InnoDB: Highest supported file format is Barracuda.
2021-05-18 17:04:08 3479 [Note] InnoDB: Log scan progressed past the checkpoint lsn 474510929998
2021-05-18 17:04:08 3479 [Note] InnoDB: Database was not shutdown normally!
2021-05-18 17:04:08 3479 [Note] InnoDB: Starting crash recovery.
2021-05-18 17:04:08 3479 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-05-18 17:04:08 3479 [Note] InnoDB: Restoring possible half-written data pages 
2021-05-18 17:04:08 3479 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 474516172800
InnoDB: Doing recovery: scanned up to log sequence number 474519100469
2021-05-18 17:04:08 3479 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 8103845, file name mysql-bin.002311
2021-05-18 17:04:09 3479 [Note] InnoDB: 128 rollback segment(s) are active.
2021-05-18 17:04:10 3479 [Note] InnoDB: Waiting for purge to start
2021-05-18 17:04:10 3479 [Note] InnoDB: 5.6.16 started; log sequence number 474519100469
2021-05-18 17:04:10 3479 [Note] Recovering after a crash using mysql-bin
2021-05-18 17:04:10 3479 [Note] Starting crash recovery...
2021-05-18 17:04:10 3479 [Note] Crash recovery finished.
2021-05-18 17:04:10 3479 [Note] Server hostname (bind-address): '*'; port: 3306
2021-05-18 17:04:10 3479 [Note] IPv6 is available.
2021-05-18 17:04:10 3479 [Note]   - '::' resolves to '::';
2021-05-18 17:04:10 3479 [Note] Server socket created on IP: '::'.
2021-05-18 17:04:10 3479 [Note] Event Scheduler: Loaded 0 events
2021-05-18 17:04:10 3479 [Note] /home/mysql/software/bin/mysqld: ready for connections.
Version: '5.6.16-log'  socket: '/home/mysql/software/mysql.sock'  port: 3306  Source distribution

my.cnf:

[mysqld]

# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.

innodb_buffer_pool_size = 12G
#innodb_additional_mem_pool_size=128M

# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin

# These are commonly set, remove the # and set as required.
# basedir = .....
datadir = /home/mysql/mysqldb
# port = .....

server_id = 94
log-bin=mysql-bin
expire_logs_days = 60
binlog-do-db=gomeet
binlog-do-db=gomeetLog
binlog_format=mixed

#innodb_file_per_table=1
#thread_concurrency=8

back_log=600

#innodb_force_recovery = 1
#innodb_file_format=Barracuda


#innodb_log_file_size=1024M

#innodb_strict_mode=0

#innodb_page_size=32K

#sort_buffer_size=8M
#read_buffer_size=8M
#read_rnd_buffer_size=8M
#table_open_cache=2048
max_allowed_packet=64M
#tmp_table_size=2G


#innodb_log_file_size=148M
# socket = .....
max_connections=1500

wait_timeout = 600

interactive_timeout = 600

# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values# join_buffer_size = 128M
# join_buffer_size = 8M
# read_rnd_buffer_size = 2M 

sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
                                                                           

If I operate in the program api_worker_face_back MySQL will be triggered to restart once.

2021-05-18 03:16:10 5328 [ERROR] InnoDB: The OS said file flush did not succeed
2021-05-18 03:16:10 7f6a83305700  InnoDB: Operating system error number 5 in a file operation.
InnoDB: Error number 5 means 'Input/output error'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
2021-05-18 03:16:10 5328 [ERROR] InnoDB: File (unknown): 'flush' returned OS error 105. Cannot continue operation
210518 03:16:11 mysqld_safe Number of processes running now: 0
210518 03:16:11 mysqld_safe mysqld restarted
2021-05-18 03:16:11 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-05-18 03:16:11 5389 [Note] Plugin 'FEDERATED' is disabled.
2021-05-18 03:16:11 7f2d6eefe740 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
2021-05-18 03:16:11 5389 [Note] InnoDB: Using atomics to ref count buffer pool pages
2021-05-18 03:16:11 5389 [Note] InnoDB: The InnoDB memory heap is disabled
2021-05-18 03:16:11 5389 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-05-18 03:16:11 5389 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-05-18 03:16:11 5389 [Note] InnoDB: Using CPU crc32 instructions
2021-05-18 03:16:11 5389 [Note] InnoDB: Initializing buffer pool, size = 12.0G
2021-05-18 03:16:11 5389 [Note] InnoDB: Completed initialization of buffer pool
2021-05-18 03:16:11 5389 [Note] InnoDB: Highest supported file format is Barracuda.
2021-05-18 03:16:11 5389 [Note] InnoDB: Log scan progressed past the checkpoint lsn 474072587915
2021-05-18 03:16:11 5389 [Note] InnoDB: Database was not shutdown normally!
2021-05-18 03:16:11 5389 [Note] InnoDB: Starting crash recovery.
2021-05-18 03:16:11 5389 [Note] InnoDB: Reading tablespace information from the .ibd files...
2021-05-18 03:16:11 5389 [Note] InnoDB: Restoring possible half-written data pages
2021-05-18 03:16:11 5389 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 474077830656
InnoDB: Doing recovery: scanned up to log sequence number 474083073536
InnoDB: Doing recovery: scanned up to log sequence number 474084218025
InnoDB: Transaction 7266397060 was in the XA prepared state.
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 0 row operations to undo
InnoDB: Trx id counter is 7266397440
2021-05-18 03:16:11 5389 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 8891015, file name mysql-bin.000851
2021-05-18 03:16:13 5389 [Note] InnoDB: 128 rollback segment(s) are active.
InnoDB: Starting in background the rollback of uncommitted transactions
2021-05-18 03:16:13 7f2a0da01700  InnoDB: Rollback of non-prepared transactions completed
2021-05-18 03:16:13 5389 [Note] InnoDB: Waiting for purge to start
2021-05-18 03:16:13 5389 [Note] InnoDB: 5.6.16 started; log sequence number 474084218025
2021-05-18 03:16:13 5389 [Note] Recovering after a crash using mysql-bin
2021-05-18 03:16:13 5389 [Note] Starting crash recovery...
2021-05-18 03:16:13 7f2d6eefe740  InnoDB: Starting recovery for XA transactions...
2021-05-18 03:16:13 7f2d6eefe740  InnoDB: Transaction 7266397060 in prepared state after recovery
2021-05-18 03:16:13 7f2d6eefe740  InnoDB: Transaction contains changes to 1 rows
2021-05-18 03:16:13 7f2d6eefe740  InnoDB: 1 transactions in prepared state after recovery
2021-05-18 03:16:13 5389 [Note] Found 1 prepared transaction(s) in InnoDB
2021-05-18 03:16:13 5389 [Note] Crash recovery finished.
2021-05-18 03:16:13 5389 [Note] Server hostname (bind-address): '*'; port: 3306
2021-05-18 03:16:13 5389 [Note] IPv6 is available.
2021-05-18 03:16:13 5389 [Note]   - '::' resolves to '::';
2021-05-18 03:16:13 5389 [Note] Server socket created on IP: '::'.
2021-05-18 03:16:13 5389 [Note] Event Scheduler: Loaded 0 events
2021-05-18 03:16:13 5389 [Note] /home/mysql/software/bin/mysqld: ready for connections.
Version: '5.6.16-log'  socket: '/home/mysql/software/mysql.sock'  port: 3306  Source distribution
Haceral
  • 41
  • 7
  • @matigo I use this version: mysql Ver 14.14 Distrib 5.6.16, for Linux (x86_64) using EditLine wrapper, It doesn't seem to be table being full. I still have 234G left on my disk – Haceral May 18 '21 at 09:31
  • thank, I tried to remove the `gomeet/api_worker_face_back` table file mentioned in the log, and then restart mysql, and everything was restored as before. – Haceral May 19 '21 at 02:50
  • There are more than 100 million pieces of data in this table, and the growth rate is very fast (there are problems in the program, which leads to inserting data into this table all the time). It seems that there are also interesting things. If I connect and operate the `gomeet` database in the program, as long as I don't operate the `api_worker_face_back` table, everything can work normally, but once I operate the `api_worker_face_back` table, MySQL will be triggered to restart once – Haceral May 19 '21 at 02:50
  • 1
    There you go. If `api_worker_face_back` is causing a problem, then there's a *very* good change that table has become corrupt. You will need to repair it during the next maintenance window, which I suggest scheduling very soon if the growth rate is high. You may be able to do some preliminary investigations with `ANALYZE TABLE api_worker_face_back;`, then plan accordingly. The most effective way would be to dump the data for this table, `DROP`, `CREATE`, then import. – matigo May 19 '21 at 02:55
  • I added the error log that triggered the restart, which I think is the main source of the error – Haceral May 19 '21 at 02:57
  • Error 105 *often* points to hardware failure, such as a bad sector on a disk. – matigo May 19 '21 at 02:59
  • You are right. Thank you very much for your help and your advice – Haceral May 19 '21 at 03:24

1 Answers1

0

max_allowed_packet=64M Adding this line into my.cnf file solves your problem.

This is useful when the columns have large values, which cause the issues, you can find the explanation here.

On Windows this file is located at: "C:\ProgramData\MySQL\MySQL Server 5.6"

On Linux (Ubuntu): /etc/mysql

Check this out: https://dev.mysql.com/doc/refman/8.0/en/replication-features-max-allowed-packet.html

Kashyap
  • 48
  • 3