How to speed up file_get_contents?

Question

Here's my code:

$language = $_GET['soundtype'];
$word = $_GET['sound'];
$word = urlencode($word);
if ($language == 'english') {
    $url = "<the first url>";
} else if ($language == 'chinese') {
    $url = "<the second url>";
}
$opts = array(
  'http'=>array(
    'method'=>"GET",
    'header'=>"User-Agent: <my user agent>"
  )
);
$context = stream_context_create($opts);
$page = file_get_contents($url, false, $context);
header('Content-Type: audio/mpeg');
echo $page;

But I've found that this runs terribly slow.

Are there any possible methods of optimization?

Note: $url is a remote url.

What's the size of the file and many seconds does it take to fetch? Divide one by the other and you'll find the bandwidth. If it's unexpectedly slow, either you or the remote side needs more! — Paul Dixon, Dec 03 '12 at 09:08
Define "slow". Two seconds? Twenty seconds? Two minutes? What are you using to measure? What are you comparing it against? Is it also slow at the command line? How about other methods, like curl? What have you done to test and troubleshoot? — Charles, Dec 03 '12 at 09:13
Then why don't you just downloas the two files to your server and serve these files instead of the remote ones ? — Intrepidd, Dec 03 '12 at 09:39

MrCode · Accepted Answer · 2012-12-03T10:07:27.870

9

It's slow because file_get_contents() reads the entire file into $page, PHP waits for the file to be received before outputting the content. So what you're doing is: downloading the entire file on the server side, then outputting it as a single huge string.

file_get_contents() does not support streaming or grabbing offsets of the remote file. An option is to create a raw socket with fsockopen(), do the HTTP request, and read the response in a loop, as you read each chunk, output it to the browser. This will be faster because the file will be streamed.

Example from the Manual:

$fp = fsockopen("www.example.com", 80, $errno, $errstr, 30);
if (!$fp) {
    echo "$errstr ($errno)<br />\n";
} else {

    header('Content-Type: audio/mpeg');

    $out = "GET / HTTP/1.1\r\n";
    $out .= "Host: www.example.com\r\n";
    $out .= "Connection: Close\r\n\r\n";
    fwrite($fp, $out);
    while (!feof($fp)) {
        echo fgets($fp, 128);
    }
    fclose($fp);
}

The above is looping while there is still content available, on each iteration it reads 128 bytes and then outputs it to the browser. The same principle will work for what you're doing. You'll need to make sure that you don't output the response HTTP headers which will be the first few lines, because since you are doing a raw request, you will get the raw response with headers included. If you output the response headers you will end up with a corrupt file.

edited Dec 03 '12 at 10:07

answered Dec 03 '12 at 09:09

MrCode

63,975
10
90
112

1

Could you please provide an example? `fsockopen()` is Greek to me. – Lucas Dec 03 '12 at 09:10
7

Please, please, *please* don't write *yet another* socket-based HTTP client. There are at least two PHP native ways to make HTTP requests and *dozens* of high-quality, well-tested pure-PHP HTTP libraries out there that can do all the things you need and more. – Charles Dec 03 '12 at 09:19
@Charles, I don't intend to write a HTTP client. The question is why is the current code slow or how to speed it up, and the example code is used to illustrate exactly that. Of course you can use any HTTP library you can find but it must support streaming (not all do). There are many ways to skin a cat and the point I'm trying to make in this answer is that streaming is the solution. – MrCode Dec 03 '12 at 09:22
-1. Your code is basically stream_copy_to_stream() but less efficient. – Ja͢ck Dec 03 '12 at 09:38
@Jack how is it less efficient? I don't claim this is my code, as pointed out in the answer it's copied from the manual and used to illustrate that streaming will greatly increase performance. The different methods of streaming or the HTTP library employed will be micro-optimisation. – MrCode Dec 03 '12 at 09:52
Even if the loop is almost as efficient than PHP's internal code, a) it's not binary safe and b) it can be written in just three lines of code. – Ja͢ck Dec 03 '12 at 09:57
2

That is micro-optimisation, OP can do as much of that as he wants to shave a few nanoseconds but the point of this question is to address what is actually slow. Also less lines != better always. – MrCode Dec 03 '12 at 10:05
Binary safety is not micro-optimisation! At least change to fread() instead of fgets(). – Ja͢ck Dec 03 '12 at 10:49
fgets() is binary safe, what makes you think it isn't? – MrCode Dec 03 '12 at 10:52
Because fgets() reads to the next newline, skipping all characters after having read 128 bytes. – Ja͢ck Dec 03 '12 at 10:57
Let's put it another way; for example, a text document with each line longer than 128 bytes. Your current answer would chop off the last remainder bytes of each line. The results are worse for binary files. – Ja͢ck Dec 03 '12 at 23:35

score 3 · Answer 2 · edited May 23 '17 at 12:17

Instead of downloading the whole file before outputting it, consider streaming it out like this:

$in = fopen($url, 'rb', false, $context);
$out = fopen('php://output', 'wb');

header('Content-Type: video/mpeg');
stream_copy_to_stream($in, $out);

If you're daring, you could even try (but that's definitely experimental):

header('Content-Type: video/mpeg');
copy($url, 'php://output');

Another option is using internal redirects and making your web server proxy the request for you. That would free up PHP to do something else. See also my post regarding X-Sendfile and friends.

score 2 · Answer 3 · answered Dec 03 '12 at 09:17

2

As explained by @MrCode, first downloading the file to your server, then passing it on to the client will of course incur a doubled download time. If you want to pass the file on to the client directly, use readfile.

Alternatively, think about if you can't simply redirect the client to the file URL using a header("Location: $url") so the client can get the file directly from the source.

answered Dec 03 '12 at 09:17

deceze

510,633
85
743
889

can you change the User Agent of the request when using `readfile`? – Lucas Dec 04 '12 at 04:46
It also supports a `$context` resource, so yes. – deceze Dec 04 '12 at 07:49

How to speed up file_get_contents?

3 Answers3

Linked