3

While working on upgrading to PHP 5.5 I've faced a problem with php-fpm.

I've created a new web node with the new PHP version and I'm signing it in on the load balancer to test it in prod. Everything works out fine until a lot of php-fpm processes gets stuck out of sudden (after ~1 hour). Hereafter I receives timeouts.

I've changed distribution from Debian to Ubuntu and upgraded to PHP 5.5. Besides that the new server is the same as before (when everything worked/works).

I've tried signing in two new web nodes and they will get stuck at the same time. I think it can be related to sudden slow responses from database (since that's the only thing they share).

However, I would like php-fpm not to crash, so it will work when database is OK again.

To fix it now I can restart php-fpm service but that's not a solid plan.

ps aux | grep php shows me:

root     13919  0.0  0.4 423472 19736 ?        Ss   Dec08   0:10 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)
www-data 14087  0.0  3.1 474972 127284 ?       S    Dec08   0:54 php-fpm: pool www
www-data 14088  0.0  2.3 446944 94956 ?        S    Dec08   0:50 php-fpm: pool www
www-data 14089  0.1  2.2 443816 92644 ?        S    Dec08   1:03 php-fpm: pool www
www-data 14090  0.0  3.0 474208 124716 ?       S    Dec08   0:44 php-fpm: pool www
www-data 14092  0.0  3.1 478284 129060 ?       S    Dec08   0:41 php-fpm: pool www
www-data 14099  0.0  2.5 459700 105224 ?       S    Dec08   0:14 php-fpm: pool www
www-data 14111  0.0  2.0 439876 84952 ?        S    Dec08   0:06 php-fpm: pool www
www-data 14114  0.0  0.4 427220 18472 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14115  0.0  0.4 427220 18380 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14116  0.0  0.4 427184 18936 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14117  0.0  0.4 427204 18904 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14118  0.0  0.4 427220 18372 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14119  0.0  0.4 427184 18832 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14120  0.0  0.4 427188 18960 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14121  0.0  0.4 427220 18384 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14122  0.0  0.4 427200 18932 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14123  0.0  0.4 427236 18488 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14124  0.0  0.4 427220 18392 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14125  0.0  0.4 427220 18384 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14126  0.0  0.4 427204 18820 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14127  0.0  0.4 427220 18484 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14128  0.0  0.4 427204 18920 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14129  0.0  0.4 427188 18904 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14130  0.0  0.4 427220 18372 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14131  0.0  0.4 427188 18920 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14132  0.0  0.4 427220 18372 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14133  0.0  0.4 427204 18936 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14134  0.0  0.4 427220 18388 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14135  0.0  0.4 427220 18368 ?        S    Dec08   0:00 php-fpm: pool www
www-data 14136  0.0  0.4 427184 18936 ?        S    Dec08   0:00 php-fpm: pool www

As you can see 24 processes has been running for 0 seconds (htop tells me it's around 2ms). They will stay like that forever. All of them appeared at the same time.

My config:

pid = /var/run/php5-fpm.pid
error_log = /var/log/php5-fpm.log
log_level = debug
user = www-data
group = www-data
listen = 127.0.0.1:9000
pm = dynamic
pm.max_children = 30
pm.start_servers = 5
pm.min_spare_servers = 3
pm.max_spare_servers = 7
pm.max_requests = 500

Also, I never see anything in the php-fpm.log (just NOTICE: configuration file /etc/php5/fpm/php-fpm.conf test is successful)

Anybody?

EDIT: I have tried running both ondemand and dynamic process manager

Morten Petersen
  • 83
  • 1
  • 6
  • Related: https://serverfault.com/questions/833968/nginx-php-fpm-7-0-hangs-on-some-requests More info to check: https://shubhamjain.co/2015/09/10/debugging-stuck-php-fpm-process-with-strace/ – abchk1234 Jun 03 '19 at 04:32

1 Answers1

-1

I can't leave a comment, although you should check your log file related to php:

( /var/www/ )

( (hostname)/logs/error_log )

and

( /var/log/php5-fpm.log )
Dick Faps
  • 1
  • 3
  • `/var/log/php5-fpm.log` is empty. My php error logs doesn't log anything of interest (only errors from my application). – Morten Petersen Dec 09 '14 at 09:21
  • try this in your shell: `for i in {1..100}; do php-fpm ; killall php-fpm; done` – Dick Faps Dec 09 '14 at 09:27
  • 3
    Well, it's easy for me to kill them and get it to work again. I can just restart the php-fpm service. But it only helps me until it crashes again. I'm looking for a reason for this happening. – Morten Petersen Dec 09 '14 at 09:30
  • It could be a conflict with another process depending on the result of the test I mentioned. If you get a segmentation fault then that is the likely culprit. Also consider running php5-fpm as a daemon (http://ma.ttias.be/a-better-way-to-run-php-fpm/) – Dick Faps Dec 09 '14 at 09:32
  • Sorry. I get: No command 'php-fpm' found, did you mean: Command 'php5-fpm' from package 'php5-fpm' (universe) php-fpm: command not found php-fpm: no process found. – Morten Petersen Dec 09 '14 at 09:39
  • Correcting as suggested give me: [09-Dec-2014 10:40:29] ERROR: failed to open error_log (/var/log/php5-fpm.log): Permission denied (13) [09-Dec-2014 10:40:29] ERROR: failed to post process the configuration [09-Dec-2014 10:40:29] ERROR: FPM initialization failed php5-fpm(13919): Operation not permitted – Morten Petersen Dec 09 '14 at 09:41
  • With sudo: [09-Dec-2014 10:41:39.211419] ERROR: pid 15941, fpm_sockets_new_listening_socket(), line 194: unable to bind listening socket for address '127.0.0.1:9000': Address already in use (98) [09-Dec-2014 10:41:39.211543] ERROR: pid 15941, fpm_init(), line 72: FPM initialization failed – Morten Petersen Dec 09 '14 at 09:41
  • Ah, that won't work I guess. Your problem sounds similar to this really https://github.com/gplessis/dotdeb-php5/issues/9#issuecomment-8160255, or php5-fpm is incorrectly being started and/or already running as another process (your error indicates that it's already running and the ownership permissions are not sufficient for controlling it). If you `ps aux | grep php5-fpm` does it tell you the owner/group? – Dick Faps Dec 09 '14 at 10:00
  • You can see that in the main question – Morten Petersen Dec 09 '14 at 10:06
  • Yes, but I can't see what the permissions are for writing to `/var/log/php5-fpm.log` for example - I would imagine you need to allow www-data higher privileges. – Dick Faps Dec 09 '14 at 10:11
  • -rw------- 1 root root 400922 Dec 9 11:19 php5-fpm.log – Morten Petersen Dec 09 '14 at 10:21
  • A lot of stuff was written in the log after trying the cmds you suggested btw. – Morten Petersen Dec 09 '14 at 10:21
  • You might want to have the permissions set to be `-rw-rw---- 1 root www-data`. – Dick Faps Dec 09 '14 at 10:26
  • I think the issue really is just a matter of ironing out your configuration a bit. Once you have the correct permissions set you'll have the info in your log that will spell out where the problem areas are. [This guide](http://www.howtoforge.com/how-to-install-nginx-with-php5-and-php-fpm-and-mysql-on-centos-6.5-p2) was written for php5-fpm / nginx and centOS, although you can apply the same methodology to ubuntu. It highlights some important things to note concerning nginx. I think you'll find it will helpful, for it shows similar pitfalls like you've posted. – Dick Faps Dec 09 '14 at 10:39
  • One more thing — if you want to use unix sockets this is extremely important: http://stackoverflow.com/q/23443398/4339382, and http://stackoverflow.com/q/24325695/4339382. The guide I linked before already mentions that at the very end, but these two answers go into more detail. – Dick Faps Dec 09 '14 at 10:51