Apache to slow to responde, but CPU and memory not max out

Question

The problem

2 apache servers have a long response time, but I do not see CPU or memory max out.

Details

I have 2 apache server servering static content for client.

• This web site has a lot of traffic.
• At high traffic I have ~10 request per second (html, css, js, images).
• Each HTML is making 30 other request to the servers for loading js, css, and images.
• Safari developer tool show that 2MB of that is getting transfer each time I hit a html page
• These two server are running on Amazon Web Service
• both instances are m1.large (2 CPUS, 7.5 RAM)
• I'm serving images in the same server
• server are in US but a lot of traffic comes from Europe

I tried

• changing from prefork to worker
• increasing processses
• increasing threads
• increasing time out

I'm running benchmarks with ab (apachebench) and I do not see improvement.

My question are:

• Is it possible that serving the images and large resorouces like js (400k) might be slowing down the server?
• Is it possible that 5 request per second per server is just too much traffic and there is no tuning I can do, so only solution is to add more servers?
• does amazon web services have a problem with bandwidth?

New Info

My files are being read from a mounted directory on GlusterFS

Metrics collected with ab (apache bench) run on a EC2 instance on same network

Connections: 500
Concurrency: 200

Server with files on mounted directory (files on glusterfs)
    Request per second: 25.26
    Time per request: 38.954
    Transfer rate: 546.02

Server without files on mounted directory (files on local storage)
    Request per second: 1282.62
    Time per request: 0.780
    Transfer rate: 27104.40

New Question

Is it possible that a reading the resources (htmls, js, css, images) from a mounted directory (NFS or GlusterFS) might slow down dramatically the performance of Apache?

Thanks

Answer 1

It is absolutely possible (and indeed probable) that serving up large static resources could slow down your server. You have to have Apache worker threads open the entire time that each one of these pieces of content are being downloaded. The larger the file, the longer the download, and the longer you have to hold a thread open. You might be reaching your max threads limits before reaching any sort of memory limitations you have set for Apache.

First, I would recommend getting all of your static content off of your server and into Cloudfront or similar CDN. This will make it to where your web server will only have to worry about the primary web requests. This might take the requests per second (and related number of open Apache threads) down from 10 request/second to like .3 requests/second (based on your 30:1 ratio of primary requests to secondary content requests).

Reducing the number of requests you are serving by over an order of magnitude will certainly help server performance and possibly allow you to reduce down to a single server (or if you still want multiple servers - which is a good idea) possibly reduce the size of your servers.

One thing you will find that basically all high volume websites have in common is that they leave the business of serving up static content to a CDN. Once you get to the point of being a high volume site, you must absolutely consider this (or at least serve static content from different servers using Nginx, Lighty, or some other web server better suited for serving static content than Apache is).

After offloading your static traffic, then you can really start with worrying about tuning your web servers to handle the primary requests. When you get to that point, you will need to know a few things:

• The average memory usage for a single request thread
• The amount of memory that you have allocated to Apache (maybe 70-80% of overall instance memory if this is dedicated Apache server)
• The average amount of time it takes your application to respond to requests

Based on that, it is a pretty simple formula to make a good starting point for tuning your max thread settings.

Say you had the following:

Apache memory: 4000KB
Avg. thread memory: 20KB
Avg. time per request: 0.5 s

That means your configuration could handle request throughput as follows:

100 requests/second = 4000kb / (20kb * 0.5 seconds/request )

Since each request averages 0.5s, you could assume that you would need 50 threads to handle this throughput.

Obviously, you would want to set you max threads higher then 50 to account for request spikes and such, but at least this gives you a good place to start.

Answer 2

1. Try to start/stop the instance. This will move you to a different host. If the host your instance is on is having any issues, that will mitigate it.
2. Beyond checking system load numbers, take a look at memory usage, IO and CPU usage.
3. Look at your system log to see if anything produced an error that may explain the current situation.

Checkout Eric J. answer in this thread Amazon EC2 Bitnami Wordpress Extremely Slow