Mysql Server Down frequently

Question

below is a log of mysql

MYSQL shutdown frequent and i cant solve the problem

image file

2018-02-26T08:15:08.301271Z 591 [Warning] IP address '192.168.1.4' has been resolved to the host name '192.168.1.4', which resembles IPv4-address itself.

2018-02-26T08:15:08.395035Z 596 [Warning] IP address '192.168.1.4' has been resolved to the host name '192.168.1.4', which resembles IPv4-address itself.

2018-02-26T08:19:23.208784Z 0 [ERROR] InnoDB: Operating system error number 995 in a file operation.

2018-02-26T08:19:23.208784Z 0 [ERROR] InnoDB: The error means that the I/O operation has been aborted because of either a thread exit or an application request. Retry attempt is made.

2018-02-26 15:19:23 0x25b0 InnoDB: Assertion failure in thread 9648 in file fil0fil.cc line 5789

InnoDB: Failing assertion: err == DB_SUCCESS

InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html

InnoDB: about forcing recovery.

08:19:23 UTC - mysqld got exception 0x80000003 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

Attempting to collect some information that could help diagnose the problem.

As this is a crash and something is definitely wrong, the information collection process might fail.

key_buffer_size=8388608

read_buffer_size=65536

max_used_connections=48

max_threads=200

thread_count=7

connection_count=7

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 74620 K bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

7f68cb05ea2 mysqld.exe!my_errno()

7f68cea9919 mysqld.exe!my_wildcmp_mb()

7f68cea8810 mysqld.exe!my_wildcmp_mb()

7f68cc05ac8 mysqld.exe!?reserve@?$vector@EV?$allocator@E@std@@@std@@QEAAX_K@Z()

7f68cc2c49a mysqld.exe!?reserve@?$vector@EV?$allocator@E@std@@@std@@QEAAX_K@Z()

7f68cbc4e94 mysqld.exe!?reserve@?$vector@EV?$allocator@E@std@@@std@@QEAAX_K@Z()

7fefff81842 KERNEL32.DLL!BaseThreadInitThunk()

7ff012ac3f1 ntdll.dll!RtlUserThreadStart()

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

2018-02-26T08:19:36.819511Z 0 [Note] C:\Program Files\MySQL\MySQL Server 5.7\bin\mysqld.exe (mysqld 5.7.11-log) starting as process 7620 ...

2018-02-26T08:19:36.835137Z 0 [Note] InnoDB: Mutexes and rw_locks use Windows interlocked functions

2018-02-26T08:19:36.835137Z 0 [Note] InnoDB: Uses event mutexes

2018-02-26T08:19:36.835137Z 0 [Note] InnoDB: _mm_lfence() and _mm_sfence() are used for memory barrier

2018-02-26T08:19:36.835137Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3

2018-02-26T08:19:36.835137Z 0 [Note] InnoDB: Adjusting innodb_buffer_pool_instances from 8 to 1 since innodb_buffer_pool_size is less than 1024 MiB

2018-02-26T08:19:36.835137Z 0 [Note] InnoDB: Number of pools: 1

2018-02-26T08:19:36.835137Z 0 [Note] InnoDB: Not using CPU crc32 instructions

2018-02-26T08:19:36.897642Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M

2018-02-26T08:19:36.897642Z 0 [Note] InnoDB: Completed initialization of buffer pool

2018-02-26T08:19:36.944522Z 0 [Note] InnoDB: Highest supported file format is Barracuda.

2018-02-26T08:19:36.960149Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 100862484017

2018-02-26T08:19:36.960149Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 100862486308

2018-02-26T08:19:36.975774Z 0 [Note] InnoDB: Database was not shutdown normally!

2018-02-26T08:19:36.975774Z 0 [Note] InnoDB: Starting crash recovery.

2018-02-26T08:19:37.444572Z 0 [Note] InnoDB: Starting an apply batch of log records to the database...

InnoDB: Progress in percent: 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

2018-02-26T08:19:37.975874Z 0 [Note] InnoDB: Apply batch completed

2018-02-26T08:19:39.585408Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"

2018-02-26T08:19:39.585408Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables

2018-02-26T08:19:39.585408Z 0 [Note] InnoDB: Setting file '.\ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

2018-02-26T08:19:40.101086Z 0 [Note] InnoDB: File '.\ibtmp1' size is now 12 MB.

2018-02-26T08:19:40.101086Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.

2018-02-26T08:19:40.101086Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.

2018-02-26T08:19:40.101086Z 0 [Note] InnoDB: Waiting for purge to start

2018-02-26T08:19:40.163591Z 0 [Note] InnoDB: 5.7.11 started; log sequence number 100862486308

2018-02-26T08:19:40.163591Z 0 [Note] InnoDB: Loading buffer pool(s) from C:\ProgramData\MySQL\MySQL Server 5.7\Data\ib_buffer_pool

2018-02-26T08:19:40.163591Z 0 [Note] Plugin 'FEDERATED' is disabled.

2018-02-26T08:19:40.491749Z 0 [Warning] Failed to set up SSL because of the following SSL library error: SSL context is not usable without certificate and private key

2018-02-26T08:19:40.491749Z 0 [Note] Server hostname (bind-address): '*'; port: 3306

2018-02-26T08:19:40.491749Z 0 [Note] IPv6 is available.

2018-02-26T08:19:40.491749Z 0 [Note] - '::' resolves to '::';

2018-02-26T08:19:40.491749Z 0 [Note] Server socket created on IP: '::'.

2018-02-26T08:19:40.757400Z 0 [Note] Event Scheduler: Loaded 0 events

2018-02-26T08:19:40.757400Z 0 [Note] C:\Program Files\MySQL\MySQL Server 5.7\bin\mysqld.exe: ready for connections.

Version: '5.7.11-log' socket: '' port: 3306 MySQL Community Server (GPL)

2018-02-26T08:19:45.164087Z 0 [Note] InnoDB: Buffer pool(s) load completed at 180226 15:19:45

Could you post results of A) SHOW GLOBAL VARIABLES LIKE 'thread%'; and B) SHOW GLOBAL STATUS LIKE 'thread%'; and C) SHOW GLOBAL STATUS LIKE 'uptime%'; ? It looks like THREAD CREATION caused the crash. It would also be helpful to know how much RAM is on your server, please. — Wilson Hauck, Mar 06 '18 at 19:44
not yet. i still stuck and my database server still shutdown. — sanseru, Mar 07 '18 at 09:17
Do you know which QUERY is causing the crash? If so, please post in the original Question near the top. Thanks for the pastebin. Give me an hour to look over your situation. — Wilson Hauck, Mar 07 '18 at 11:50
While I am creating a list of changes, please use MySQLCalculator.com and you will see why you are crashing. In the per connection area, you have EXCESSIVE RAM requests, consistently. Every one of those will be REMOVED to allow defaults to work for you. There will be additional suggestions. Check back in 2 hours, please. — Wilson Hauck, Mar 07 '18 at 12:08
Possibly helpful https://stackoverflow.com/questions/226172/how-do-i-repair-an-innodb-table — Jaydee, Mar 07 '18 at 13:14

score 0 · Answer 1 · answered Mar 03 '18 at 10:35

Suggestions for your my.cnf/ini [mysqld] section

#sort_buffer_size=~256K  # lead to allow DEFAULT to work for you
max_connections=100  # from 200 since max_used_connection were 48 since start

derived from your original posted information of 2/26/2018. These changes will lower RAM requirements and we do not know how much RAM you have on your server.

score 0 · Accepted Answer · answered Mar 07 '18 at 13:03

0

Suggestions for your my.cnf/ini [mysqld] section

Every x Connection listed in MySQLCalculator.com must be REMOVED from your my.cnf-ini to allow DEFAULTS to serve you.

thread_cache_size=100  # from 10 REFMAN v5.7 5.1.5 for CAP of 100 suggested
innodb_io_capacity=800  # from 200 to enable higher capacity
lock_wait_timeout=300  # from 31536000, who wants to wait ONE Year?
eq_range_index_dive_limit=20  $ from 200 not found in 20, is missing
expire_logs_days=5  # from 0 so you have limited historical logs
key_buffer_size=1M  # from 8M  you had key_blocks_used of 2
innodb_buffer_pool_instances=8  # from 1 to avoid mutex contention
innodb_buffer_pool_size=8G  # from 128M until you need more for data volume
innodb_log_buffer_size=8M  # from 134M - can not be > innodb_log_file_size
innodb_lru_scan_depth=128  # from 1024  see REFMAN for why
innodb_page_cleaners=64  # from 1 will be limited to be = innodb_buffer_pool_instances
innodb_print_all_deadlocks=ON  # from OFF - check error log DAILY
innodb_read_io_threads=64  # from 4 see dba.stackexchange.com Q 5666 9/12/11
innodb_thread_concurrency=0  # in 5666 Rolando explains these 3 values
innodb_io_threads=64  # from 4  and how the combination enables multi-core
innodb_stats_sample_pages=32  # from 8 for more accurate cardinality
#max_allowed_packet=1G   # leading # to disable for DEFAULT size

if you NEED larger size for LOCAL INFILE, in your SESSION, SET @max_allowed_packet=nnnnnnnnnnnn up to 1G which is the MAX

max_seeks_for_key=32  # from a huge number, do not waste CPU past 32
max_write_lock_count=16  # from a huge number, allow RD after nn LOCKS
wait_timeout=3600  # from 8 hours, not touched in 1 HR release rscrs, log in again

Good luck.

answered Mar 07 '18 at 13:03

Wilson Hauck

2,094
1
11
19

Thanks for the suggestion. I will try your suggestion, every update will be told to you – sanseru Mar 08 '18 at 00:40
if i change my.ini in my MYSQL server and an error occurs. and i back to restore my.ini to default. can it will running well like before? sorry for bad english – sanseru Mar 08 '18 at 00:43
Thanks for the acknowledgment. Looking forward to Summary in a few days. Would like to know after all are applied and running for 5 days. – Wilson Hauck Mar 08 '18 at 00:44
Yes, that is true, keep your last 5 ini's at least, so you can get back to yesterday, or the day before. – Wilson Hauck Mar 08 '18 at 00:44
Keep your last 5 useful ini's. Name them so you know when saved. I name my.ini my-201803070707.ini so I know the date and time, last edited and saved. They use very little space. – Wilson Hauck Mar 08 '18 at 00:47
And when you sort the folder, they are in order. – Wilson Hauck Mar 08 '18 at 00:47
Post your complete error log, if you have a problem and I should be able to assist and offer advice to overcome the my.cnf-ini issue. – Wilson Hauck Mar 08 '18 at 00:50
https://pastebin.com/2kugerKA @wilsonhauck can you check this. i cant execution myini because right now is office hours – sanseru Mar 08 '18 at 02:22
@sanseru Thank you for the opportunity to review your .ini. Do NOT try it, as it is. It will fail just as it has been. Give me 24 hours to get back to you, please. I will rearrange your ini to test outside of office hours. Ask your chat people for a copy of my chat session yesterday, please. From Wilson dot Hauck Thank You – Wilson Hauck Mar 08 '18 at 09:27
I've tried the data in your pastebin and remove note . and an error occurs check this https://pastebin.com/Dr0aUqtp. my server running 24/7 and in office hours shutdown frequently because user input data. but in the night or weekend server is very smooth not have issue. – sanseru Mar 08 '18 at 15:37
@sanseru My humble apologies, innodb_io_threads SHOULD BE innodb_write_io_threads Please correct it and file/save. – Wilson Hauck Mar 08 '18 at 17:19
Your error logs will be available for 5 days after the ini is running. Have you identified the query with bad data that is causing the server to crash? It should be in your error logs that will be saved for 5 days. Weekends always quiet, no one is working like Mon-Fri. If you already know the query causing the crash, please post it. – Wilson Hauck Mar 08 '18 at 17:24
I've tried setting you my.ini and it works. thank you very much and I will provide the latest updates after this configuration runs on office hours. and i can't find query is causing the server to crash. because to many query enter database when office hours. or you have trick to find out which query is causing error – sanseru Mar 10 '18 at 16:49
OK thank you for your help. noted friday afternoon. – sanseru Mar 10 '18 at 19:03
@sanseru Please post results of this query, we will go from there, SELECT table_name, table_rows FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = 'performance_schema'; – Wilson Hauck Mar 13 '18 at 07:07
ok , I will send the result. and your suggestion my.ini working well. 2 days not have isue. thanks you – sanseru Mar 14 '18 at 01:16
@sanseru Great news. How is response time compared to a week ago? – Wilson Hauck Mar 14 '18 at 15:59
this performance schema result https://pastebin.com/ib6vXKe2. response time fast and better than a week ago. no mysql shutdown problem. I am very grateful to you for helping me. for the global status report I will send when office hours expire. once again I am very grateful to you – sanseru Mar 16 '18 at 04:01
@sanseru Great, now we do not have to find that query that was killing MySQL. Please post in pastebin.com, SHOW GLOBAL STATUS; and SHOW GLOBAL VARIABLES; and SHOW ENGINE INNODB STATUS; Have a GREAT weekend. – Wilson Hauck Mar 16 '18 at 08:23
check this https://pastebin.com/PdkdT7nG. you too Have a great weekend :) good to see you – sanseru Mar 16 '18 at 11:38
@sanseru Analysis of 6 days GLOBAL STATUS data provided indicates, A) com_begin count 158, com_commit 3,350,022; why so many commits? B) com_rollback 25,910; average of 1 rollback every 19 seconds. 1 minute (and NO more than 1 minute) would likely contain clues of a rollback and what caused it. C) com_stmt_prepare counted 136, com_stmt_close counted 134. Unless they are paired, resources from 2 prepares were still in use when GLOBAL STATUS was prepared on Friday. Please post (to pastebin.com) your slow query log from today (Monday). Thank you. – Wilson Hauck Mar 19 '18 at 09:44
@sanseru correction to B) above. B) com_rollback 25,910; average of 1 rollback every 19 seconds. 1 minute of General_log (and NO more than 1 minute) would likely contain clues of a rollback and what caused it. Post companion part of the 'error log' for that time of day, please which now has Innodb_print_all_deadlocks turned ON. – Wilson Hauck Mar 19 '18 at 13:14
yes i have skype. but my english not very good. add me lukmansanseru / lukmansanseru@hotmail.com I dont know why so many commits because, i handle this server in error situation. https://pastebin.com/iTh0GC1h – sanseru Mar 21 '18 at 02:52
@sanseru Could you post text results of 1) SHOW INDEX FROM claim; 2) SHOW INDEX FROM currency; 3) SHOW INDEX FROM member_limit_layer; 4) SHOW INDEX FROM member; 5) SHOW INDEX FROM member_product; 6) SHOW INDEX FROM diagnosis; 7) SHOW INDEX FROM product_layer_limit; 8) SHOW INDEX FROM product; 9) SHOW INDEX FROM member_limit_layer; 10) SHOW INDEX FROM diagnosis_set; 11) SHOW INDEX FROM client; 12) SHOW INDEX FROM client_category; 13) SHOW INDEX FROM member_limit_layer; (a duplicate) 14) SHOW INDEX FROM case_status; Thanks – Wilson Hauck Mar 21 '18 at 12:58

Mysql Server Down frequently

2 Answers2