2

I have a strange problem I cannot handle. I don't know what to do, I already set the php.ini to the following:

max_execution_time = 120
memory_limit = 1024M
output_buffering = on
compression is turned off

The webserver is idling at about 95% and has free memory of 6GB.

I also tuned apache MPM:

mpm_prefork_module:
StartServers          500
MinSpareServers       500
MaxSpareServers      1000
MaxClients          5500
MaxRequestsPerChild   0

But this all does not help at all (I also tried different values).

I need to handle about 3000 incoming API requests per minute. I need to return about 50KB for each.

The server can handle that amount of incoming requests. I tested it. It just can't throw all the data out. There is a kind of throttling. But when testing the bandwith, I get the full 100MBit.

Here is my problem:

When I download a static binary file via apache2, I have a speed of 12000 KiloByte/sec, so that's nearly the whole 100MBit connection.

I created a php file that does all the API stuff, but not returning the result. It just returns some random data of a specific size. Now I am loading this file/data with many threads (1000 at the time) from a different server.

Now I checked how many requests the server handles per minute. I calculated the transfer rate per seconds.

      0 byte = about 5000
   1000 byte = about 3000 =  50 Kilobyte/sec
  10000 byte = about 1600 = 266 Kilobyte/sec
  50000 byte = about  430 = 358 Kilobyte/sec
 100000 byte = about  337 = 561 Kilobyte/sec
 500000 byte = about   69 = 567 Kilobyte/sec

This shows that the server can handle the demanded amount of requests (it could handle 5000 per minute instead of the 3000 I need), when not returning the data. When I return the 50KB I need, then I only get 430 requests per minute. It doesn't matter if I return the random or the real data, using the real PHP file ore just a dummy returning some random 50KB data. This does not make any difference.

What can I do to solve the php throttling issue?

Here is some sourcecode that will lead to the same throttling issue:

<?
$chars = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";

for ($i = 0; $i < 50000; $i++) 
{
  echo $chars[rand(0, strlen($chars))];
}
?>
Chris
  • 4,325
  • 11
  • 51
  • 70
  • Are you comparing the downloading of one file (~12M) in Apache with lots of small files (50K) in PHP? If so, you should try a test of downloading lots of small files from Apache, otherwise you're changing two variables at the same time, and thus the test is not fair. I should think hundreds/thousands of requests for a 50K file has a much lower throughput that one request for a large file, since connections have their own timing overhead. – halfer Jan 20 '13 at 22:56
  • Yes, the download test of the single file was just to test the bandwidth of the server. To test the amount of concurrent connections I did the other test. – Chris Jan 20 '13 at 22:59
  • @Chris if you are testing a static vs dynamic transfer, the test is not fair. What is your CPU usage during the dynamic transfer tests? – Petah Jan 20 '13 at 23:01
  • I have about 95% idle and 6GB of free memory. My server is bored, but it doesn't use the amount of memory or CPU to do something. – Chris Jan 20 '13 at 23:02

1 Answers1

0

There could be several factors at play here:

PHP's Overhead

Even for a basic script that just splats out a file, PHP has to initialize its self. There's a cost to that.

Memory Overhead

Chances are, your PHP script is causing the entire 50KB of data to be copied into memory (cost to that too). Unless you're jumping through hoops to stream the result, this is most likely happening.

Buffering

Similar to above, if you are not streaming/chunking the result - PHP has to load the entire chunk of data into memory before emitting it to the requester.

Unknowns

There's a lot of moving parts in PHP's guts, and it's probably not safe to make many assumptions about where the slowdown really is without dissecting and profiling your specific script.

Solving it

Web servers like apache and nginx are well optimized for serving static files. Let them do that; they will always be faster than introducing a layer of indirection between them.

I don't know enough about your specific problem, but general guidelines:

  • Have the PHP script do basic processing, and then redirect to a URL that is served directly by your web server.
  • If you need a security layer; consider setting up throwaway URLs or a server module that can quickly check whether the requester has access (but doesn't have to manage w/ streaming file contents)
Nevir
  • 7,951
  • 4
  • 41
  • 50
  • To add to this, mod_xsendfile is good for putting a auth layer in front of static files: https://tn123.org/mod_xsendfile/ – Petah Jan 20 '13 at 23:05
  • @Nevir I edited my post to post some code to reproduce the error. I have the problem with a simple php like this (I broke it down to just the transfer issue). When you say that apache handles better static data, I could write the api result to a static file, using a random filename. Then give this filename to the user and say that the user should do a second request to download the result of the api request. Now apache can serve this data as a static file. – Chris Jan 20 '13 at 23:08
  • Well, your ideal is to avoid `echo`ing anything from PHP, and instead doing a [`http_redirect`](http://php.net/manual/en/function.http-redirect.php). Writing to a temp file is likely to get you into trouble though (how long does it need to stay around? when do you clean it up? etc) – Nevir Jan 20 '13 at 23:16
  • I.e. if the result data doesn't change very frequently, you should try to cache them directly, and redirect to the cache. If the data _does_ change frequently, then you need to optimize your PHP script to stream the file data back as efficiently as possible – Nevir Jan 20 '13 at 23:17
  • My data changed for every request. For that reason I can test it very good with random data. Building the whole 50KB string and then echoing it at whole does not change anything :( I really don't know what to optimize. I get the same problems with my posted 5 lines of code. As the output changes every time it is very good comparable to my original code. I don't know why the CPU is 95% idle and I have free memory of 6GB. There is also enough free bandwidth, so where is the problem for apache/php? – Chris Jan 20 '13 at 23:25
  • But if php has problems with throwing the data out, would it help to write the data to a temp file and doing an http_redirect to it? The data is only valid for one request. Then I would do a cron job to delete all data older than two minutes. Do you think programming that would help? – Chris Jan 20 '13 at 23:27
  • Take a look at http://stackoverflow.com/questions/6914912/streaming-a-large-file-using-php for the streaming part of it. But yeah, if you can't find a reasonable method of streaming the file via PHP, temp files seem reasonable (but make sure you're thinking through the edge cases - what happens if your cron job deletes the file while it's being streamed out? what if the requester has a really slow connection? etc) – Nevir Jan 20 '13 at 23:36
  • Okay, I tested this. But there was no enhancement. Then I just made a plain 50KB.htm file, put it in the www folder and downloaded it with multiple threads like above. I only can get about 4-5 files per seconds (280 per minute). Again there is free memory and now 99% CPU power left. So I think it is apache itself that is throttling me. Maybe because all requests are coming from the same IP. Also I found in access.log, that there are 10-30 requests in one second, then for some seconds no requests, then again 20-30 requests and so on. This also looks like a kind of rate limit. – Chris Jan 21 '13 at 02:47