0

When I'm using https://httpstatus.io/ to check the HTTP status of let's stay 30 URLs, the server crashes and only the first 5-6 images return 200, the others return a server error.

Using "top" command in Apache at the moment I trigger the request of 30 URLs, 5-6 instances of php-cgi appear at the top, each loading the CPU with over 5%. I have a Wordpress website.

I'm guessing these two are related to each other? The fact that MySQL's maximum memory usage is dangerously high and the php-cgi overloads the CPU resulting in server crashes?

My server settings:

vCPU/s:1 vCPU
RAM:2048.00 MB
Storage: 64 GB NVMe

The MySQL Tuner:

perl mysqltuner.pl
 >>  MySQLTuner 2.0.9
         * Jean-Marie Renouard <jmrenouard@gmail.com>
         * Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.pl/
 >>  Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.7.40-log
[OK] Operating on 64-bit architecture

-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/log/mysqld.log exists
[--] Log file: /var/log/mysqld.log (5M)
[OK] Log file /var/log/mysqld.log is not empty
[OK] Log file /var/log/mysqld.log is smaller than 32 Mb
[OK] Log file /var/log/mysqld.log is readable.
[!!] /var/log/mysqld.log contains 4595 warning(s).
[!!] /var/log/mysqld.log contains 1470 error(s).
[--] 126 start(s) detected in /var/log/mysqld.log
[--] 1) 2022-11-26T16:51:46.008951Z 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 2) 2022-11-26T14:25:04.411192Z 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 3) 2022-11-26T14:00:44.505181Z 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 4) 2022-11-26T13:41:24.726384Z 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 5) 2022-11-26T13:30:30.052935Z 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 6) 2022-11-26T13:14:52.946860Z 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 7) 2022-11-26T13:10:52.597525Z 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 8) 2022-11-26T13:06:44.314567Z 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 9) 2022-11-26T12:59:06.893587Z 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 10) 2022-11-26T12:56:43.639611Z 0 [Note] /usr/sbin/mysqld: ready for connections.
[--] 47 shutdown(s) detected in /var/log/mysqld.log
[--] 1) 2022-11-26T16:51:45.487321Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 2) 2022-11-26T14:25:03.877227Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 3) 2022-11-26T14:00:44.020059Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 4) 2022-11-26T14:00:41.779016Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 5) 2022-11-26T13:37:23.080832Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 6) 2022-11-26T13:37:22.580337Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 7) 2022-11-26T13:37:22.111923Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 8) 2022-11-26T13:37:21.666683Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 9) 2022-11-26T13:37:21.130343Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 10) 2022-11-26T13:30:29.577245Z 0 [Note] /usr/sbin/mysqld: Shutdown complete

-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA
[--] Data in MyISAM tables: 200.8M (Tables: 137)
[--] Data in InnoDB tables: 16.0K (Tables: 1)
[OK] Total fragmented tables: 0

-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.

-------- Views Metrics -----------------------------------------------------------------------------

-------- Triggers Metrics --------------------------------------------------------------------------

-------- Routines Metrics --------------------------------------------------------------------------

-------- Security Recommendations ------------------------------------------------------------------
[OK] No Role user detected
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] There is no basic password file list!

-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 14m 56s (5K q [6.276 qps], 182 conn, TX: 48M, RX: 731K)
[--] Reads / Writes: 99% / 1%
[--] Binary logging is disabled
[--] Physical Memory     : 1.8G
[--] Max MySQL memory    : 13.4G
[--] Other process memory: 0B
[--] Total buffers: 682.0M global + 260.6M per thread (50 max threads)
[--] Performance_schema Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[!!] Maximum reached memory usage: 1.7G (93.83% of installed RAM)
[!!] Maximum possible memory usage: 13.4G (746.15% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (0/5K)
[OK] Highest usage of available connections: 8% (4/50)
[OK] Aborted connections: 0.55% (1/182)
[!!] CPanel and Flex system skip-name-resolve should be on
[OK] Query cache is disabled by default due to mutex contention on multiprocessor machines.
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 1K sorts)
[OK] No joins without indexes
[OK] Temporary tables created on disk: 13% (544 on disk / 4K total)
[OK] Thread cache hit rate: 97% (4 created / 182 connections)
[OK] Table cache hit rate: 61% (8K hits / 14K requests)
[OK] table_definition_cache (912) is greater than number of tables (418)
[OK] Open file limit used: 1% (780/40K)
[OK] Table locks acquired immediately: 100% (5K immediate / 5K locks)

-------- Performance schema ------------------------------------------------------------------------
[!!] Performance_schema should be activated.
[--] Sys schema is installed.

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.

-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.8% (24.1M used / 128.0M cache)
[OK] Key buffer size / total MyISAM indexes: 128.0M/11.6M
[OK] Read Key buffer hit rate: 99.1% (80K cached / 734 reads)
[OK] Write Key buffer hit rate: 100.0% (166 cached / 166 writes)

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 512.0M / 16.0K
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (6.25%): 16.0M * 2 / 512.0M should be equal to 25%
[OK] InnoDB buffer pool instances: 1
[--] Number of InnoDB Buffer Pool Chunk: 4 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 97.50% (15884 hits / 16291 total)
[!!] InnoDB Write Log efficiency: 0% (2 hits / 0 total)
[OK] InnoDB log waits: 0.00% (0 waits / 2 writes)

-------- Aria Metrics ------------------------------------------------------------------------------
[--] Aria Storage Engine not available.

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: ROW
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Check warning line(s) in /var/log/mysqld.log file
    Check error line(s) in /var/log/mysqld.log file
    MySQL was started within the last 24 hours: recommendations may be inaccurate
    Reduce your overall MySQL memory footprint for system stability
    Dedicate this server to your database for highest performance.
    name resolution is enabled due to cPanel doesn't support this disabled.
    Performance schema should be activated for better diagnostics
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read this:
Variables to adjust:
  *** MySQL's maximum memory usage is dangerously high ***
  *** Add RAM before increasing MySQL buffer variables ***
    skip-name-resolve=0
    performance_schema=ON
    key_buffer_size (~ 25M)
    innodb_log_file_size should be (=64M) if possible, so InnoDB total log files size equals 25% of buffer pool size.

CNF

[mysqld]
performance-schema=0

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

innodb_buffer_pool_size = 500M
innodb_flush_log_at_trx_commit  = 1
innodb_log_file_size = 16M
# innodb_additional_mem_pool_size = 32M
innodb_buffer_pool_instances = 1
innodb_log_buffer_size = 10M

max_connections = 50
wait_timeout = 3600
interactive_timeout = 3600

join_buffer_size = 2M
sort_buffer_size = 2M
read_rnd_buffer_size = 2M
key_buffer_size = 128M
max_allowed_packet=268435456
read_rnd_buffer_size = 256K
# thread_stac = 256K
thread_cache_size = 8
open_files_limit = 3000
table_open_cache = 1024
default-storage-engine = MyISAM
innodb_file_per_table = 1

query_cache_type = 0
query_cache_limit = 0
query_cache_size = 0

max_heap_table_size = 32M
tmp_table_size = 32M

general_log_file = /var/log/mysql/mysql.log
general_log = 0

slow-query-log = 1
slow-query-log-file = /var/log/mysql/mysql-slow.log
long_query_time = 1
log-queries-not-using-indexes = 1

I found something about using "slow-query-log" to debug the causes but I'm super not with this stuff and I couldn't understand much of it.

If someone can guide me I'd very much appreciate it!

  • "*the server crashes*" What server is crashing? If it's your Wordpress site, what is it doing? How is it using MySQL? – Schwern Nov 26 '22 at 18:40
  • I don't know if the Apache or MySQL is crashing but the server becomes unresponsive due to the CPU overload. I'm guessing I need to tune MySQL? – RocketBrian Nov 26 '22 at 18:51
  • Whatever is causing the crash, it's not because of the theoretical max memory usage of MySQL. The estimate given by Mysqltuner is totally wrong. I wish people would stop using that script. I explain in more detail in these past answers: https://stackoverflow.com/a/50823508/20860 and https://stackoverflow.com/a/26210951/20860. – Bill Karwin Nov 26 '22 at 19:08
  • @RocketBrian I would first look at what your Wordpress site is doing. With only 2G of memory, my first guess would be that Wordpress is using up main memory and the server is [thrashing](https://en.wikipedia.org/wiki/Virtual_memory#Thrashing) swapping memory between disk and RAM. – Schwern Nov 26 '22 at 19:14
  • Hi both, I really don't know where to start from. I'm really clueless. What commands should I use to debug what is happening when those redirects are triggered? Specifically, I changed the domain address of one of my domains and redirected everything to a new domain. When I check the HTTPS status of 30+ old URLs (which redirect to the new domain) the server either times out or crashes. I guess those redirects trigger processes that overkill the CPU. If I check with "top" command when I trigger redirects, 4-5 new php-cgi processes appear that overload the CPU with 5-6% each. – RocketBrian Nov 26 '22 at 21:18
  • Are the "images" stored on the filesystem? Or in the database? – Rick James Nov 27 '22 at 06:27
  • How to summarize the slowlog and provide it to us to explain: [_SlowLog_](http://mysql.rjweb.org/doc.php/mysql_analysis#slow_queries_and_slowlog) – Rick James Nov 27 '22 at 06:29
  • Hi Rick, the images get stored on the file system and a corresponding entry gets saved in the database as well. – RocketBrian Nov 27 '22 at 08:00
  • @RocketBrian I would check 1) [php-cgi can have very poor performance](https://blog.layershift.com/which-php-mode-apache-vs-cgi-vs-fastcgi/), 2) check for [full table scans on large tables](https://dev.mysql.com/doc/refman/8.0/en/table-scan-avoidance.html), 3) check for anywhere you're [fetching all results at once](https://www.php.net/manual/en/pdostatement.fetchall.php), instead of one by one on a loop. – Schwern Nov 27 '22 at 21:01
  • Posting result of SHOW FULL PROCESSLIST; from your 'OVERLOAD' world may be helpful. – Wilson Hauck Nov 28 '22 at 03:03
  • Hi both, I know php-cgi has poor performance but that's not the case here (I guess). One php-cgi process consumes a maximum 5% CPU which wouldn't be a problem. The problem is that every incoming redirect triggers a php-cgi request, which overloads the CPU. I'm not sure if this is the correct behavior. I mean, every person could theoretically kill my website if it triggers more than 30-40 redirect requests. Also, this only happens with redirect requests (like when 30 requests redirect from the old domain to the new one). If those requests are on the same origin it doesn't happen. – RocketBrian Nov 28 '22 at 09:23
  • Here's what I mean: https://ibb.co/rQzxYTd . This is happens at the moment I check 30-40 URLs with the mentioned tool (all requests are 301 redirects) – RocketBrian Nov 28 '22 at 11:47
  • @RocketBrian You might want to ask this on [Wordpress.SE](https://wordpress.stackexchange.com). – Schwern Nov 29 '22 at 01:17
  • Thanks, Schwern, will do. In the meantime, I found out why php-cgi consumes that much CPU. Tracking the process with strace I found that each process calls every PHP file I have on. That takes ages and consumes too much memory. Why is this happening? Given that it is only happening when I request an image URL and not an actual page URL. Why is every PHP file loaded when it's loading images? – RocketBrian Nov 29 '22 at 09:39
  • 1
    @RocketBrian That's something you'll have to ask the Wordpress folks, but at a guess, it's because your images are being routed through PHP and, because you're using php-cgi, the whole application has to re-load itself each request. Try FastCGI, it keeps the PHP code loaded between requests. You can also configure your web server to serve images, and other static content, directly rather than run them through PHP. – Schwern Nov 29 '22 at 17:50
  • Why are there any "redirects"? What is the web server's limit on concurrent 'children'? – Rick James Nov 29 '22 at 20:38
  • 1
    HTTP should reach for each image -- PHP should not be involved. Are you using `` ? – Rick James Nov 29 '22 at 20:40
  • Moving to FastCGI and PHP-FPM did the trick. Thank you Schwern. Hi Rick, I have an old domain that redirects to the new one, and this behavior happened when there were multiple incoming redirects from the old to the new domain. – RocketBrian Nov 30 '22 at 07:54
  • @RocketBrian Glad you got it fixed! – Schwern Nov 30 '22 at 17:32

0 Answers0