3

We have a Wordpress based website that gets about 1.5 million pageviews per day. Maximum concurrent sessions reach up to 5000 users per minute at times.

We have one web and one database server. Specs for both machines are as follows:

CPU: 2 x Intel(R) Xeon(R) CPU X5650 @ 2.66GHz
12M Cache / 6 Cores / 12 Threads (x2)
Memory: 24GB
Drive(s)(size/GB) - 4x240gb SSDs RAID 10

We run on Nginx, use FastCGI, and also use APC for caching, along with simple page caching with WP Super Cache.

Current Nginx configuration is as follows:

worker_processes  24;
worker_rlimit_nofile 200000;
worker_connections  4000;

FastCGI settings are as follows:

FcgidMaxProcesses 200
FcgidProcessLifeTime 3600
MaxProcessCount 100
FcgidIOTimeout 400
FcgidIdleTimeout 600
FcgidIdleScanInterval 90
FcgidBusyTimeout 400
FcgidBusyScanInterval 80
ErrorScanInterval 3
ZombieScanInterval 3

Server fails and we get Nginx errors (502-504) during high traffic. I am wondering if this is related to a bad configuration.

Your assistance is much appreciated.

mightypixel
  • 141
  • 1
  • 7
  • This might be better answered on Server Fault. – Jason McCreary Nov 18 '13 at 16:22
  • OK, thanks, I'll post it there. – mightypixel Nov 18 '13 at 16:25
  • Do you use php5-fpm / fastcgi ? Maybe you may increase the amount of phpfastcgi workers instead, I don't think the error is coming from nginx, but from PHP or other thinks, you might aswell get them for some lockwait / timeouts. Could you provide more information, from syslog, dmesg, and error.log ? – Daniel W. Nov 18 '13 at 16:31
  • @DanFromGermany We do use FastCGI indeed, sorry I haven't mentioned it. When the website fails -albeit temporarily-, the error.log is full of entries such as `connect() failed (111: Connection refused) while connecting to upstream`. I have updated my original question with FastCGI settings. Thanks. – mightypixel Nov 18 '13 at 16:47
  • Have a look at Varnish as a cache server, although not the answer to your question I've seen a lot of improvements after installing it on client sites. Real simple to set up, too. – John Dorean Nov 18 '13 at 16:50
  • @mightypixel can you investigate if it's an error of the client connecting to nginx, or nginx connecting to the fastcgi? There is also a TCP limit on the linux kernel, might this be a problem? http://stackoverflow.com/questions/410616/increasing-the-maximum-number-of-tcp-ip-connections-in-linux – Daniel W. Nov 18 '13 at 17:00
  • @DanFromGermany This is what I have on the TCP limits: `net.ipv4.ip_local_port_range = 32768 61000 net.ipv4.tcp_fin_timeout = 60`. On the first question, I would appreciate if you could guide me in the right direction; how would I know the difference. Which logs I can check to figure out. (quite a newbie on the subject, sorry.) – mightypixel Nov 18 '13 at 17:09
  • type in console `dmesg` to get kernel alerts, other logs are in `cd /var/log/` there is a folder maybe `httpd` or `nginx`, `mysql`, `php....` also a file `syslog[.log]`, use `cat` or `tail -f /var/log/syslog` – Daniel W. Nov 18 '13 at 17:11
  • @ChrisWhite I've been meaning to set up Varnish but I have read that it doesn't play nice with Nginx? Would that not be your experience? Thanks. – mightypixel Nov 18 '13 at 17:15
  • @DanFromGermany No errors in `dmseg` or `syslog` but `httpd error_log` is full of entries such as: `Line 13166: [Mon Nov 18 16:24:06 2013] [error] FastCGI process 9601 still did not exit, terminating forcefully Line 13167: [Mon Nov 18 16:24:06 2013] [error] FastCGI process 9120 still did not exit, terminating forcefully Line 13168: [Mon Nov 18 16:24:06 2013] [error] FastCGI process 9340 still did not exit, terminating forcefully Line 13303: [Mon Nov 18 16:29:20 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting` – mightypixel Nov 18 '13 at 17:24
  • You could try raising MaxClients value but butch your RAM with `htop` , `free` or `top`. And maybe the nginx KeepAlive can be adjusted. Smaller values release ressources faster but takes another connection on reconnect of the client. I have 8 seconds there. Also try to google for the best settings for php-fpm-workers and connection setting. If you are using localhost, make sure to use unix-sockets instead of tcp. – Daniel W. Nov 19 '13 at 08:29
  • @DanFromGermany Thanks, I'll fiddle with the settings as per your suggestions. – mightypixel Nov 19 '13 at 14:13

1 Answers1

2

"...it makes sense for us to run multiple workers, usually 1 worker per CPU core. Though, anything above 2-4 workers is overkill as nginx will hit other bottlenecks before the CPU becomes an issue and usually you’ll just have idle processes"

Check out this blog: http://blog.martinfjordvald.com/2011/04/optimizing-nginx-for-high-traffic-loads/

This post might be helpful as well: Tuning nginx worker_process to obtain 100k hits per min

Community
  • 1
  • 1
Hunter
  • 146
  • 4