Fatal error: Out of memory, but I do have plenty of memory (PHP)

Question

Since my question is getting longer and longer, I decide to re-write the whole question to make it better and shorter.

I run my website on dedicated server with 8GB memory. I am fully aware that I need to raise the memory limit on php.ini setting. I have set it from 128M to 256M and to -1. Still the problem is persistence.

Fatal error: Out of memory (allocated 786432) (tried to allocate 24576 bytes) in D:\www\football\views\main.php on line 81

The out of memory does not make sense because it said only 786432 bytes is allocated and it needed 24576 bytes more.

786432 bytes is only 768 kilobytes and is fairly small.

Hints

The error occurs on a very random line. It does not always error on line number 81.
At peak time, Apache only takes around 500mb of memory. I still have 6GB to spare.
There is no infinite loop.
The script takes 1,042,424 bytes. Getting this number from echo memory_get_peak_usage();
The resultset from MySQL is small (at most 12 of rows, purely text, no blob data)
(Important) If I restart Apache once every two days, the error is gone. It usually happens when Apache is running more than 2 days.
I have included the profiling the script and you can get it here.
This dedicated server is purely used to run only one website. This website is a high traffic website with average of 1,000 visitors every minute. At peak time, there will be 1,700 to 2,000 visitors accessing at the same time.

Server Spec

OS: Windows 2008 R2 64-Bit
CPU: Intel Core i5 - 4 cores
RAM: 8 GB
Apache 2.2
PHP 5.3.1
Storage: 2 x 1 TB hard drives
Bandwidth: 10 TB per month

Solution

I have finally tuned up and fixed the problem and I would like to share it here what I have done to improve:

favicon.ico was missing which mess up with my route engine. Although my route engine is very small, but by including favicon.ico, it helps reduce memory usage by not running my route engine. Most of part of my website has it and I forgot to put it for this new section.
Limit MaxRequestPerChild helps. In my other dedicated server, I have my MaxRequestPerChild limited. For this server, I set it to 0. I always thought that each script is isolated. Lets say if my script takes 800kb to run. Upon its completion, Apache or PHP should free 800kb memory. It seem like it doesn't work this way. Limited MaxRequestPerChild does help to prevent memory leak by creating new process after limited MaxRequestPerChild and the old process is dying. This is my new setting.
```
ThreadsPerChild      1500
MaxRequestsPerChild  10000 
```
ob_flush(); does reduce slightly more memory. It does not help much but every bit of optimization helps.
I have used xdebug which I have never used before as suggested by people who attempt to answer this question. I have to say it is great tool and I have optimized a few stuffs to make it run slightly faster.
I have disable a few unnecessary Apache module. I am trying to disable it one by one and leave it a few days test to ensure it works perfectly before I disable another one. I do have all unnecessary PHP extension disable now.
Most of my script in this server used traditional way (no template, no database layer, pure PHP, HTML, and legacy mysql_* function). To be honest, it runs very fast and used extremely small memory. However, maintenance the script is not very easy as the website is getting longer. I have tried to convert some parts of the website into proper framework (my own tiny framework). The reason that I used my own framework because it is tiny (3kb for the whole framework and include only what I need).
Switching to IIS7.5 solving this problem completely.

I smells infinite loop. :) What version of PHP? And how bout that code on line 81. — Jared Drake, Aug 18 '12 at 03:25
There is no infinite loop because 70% of the time, the scripts works, and around 30% of the time, it said it out of memory. I triple-check the code as well. One more thing is that I check the memory usage of the script with memory_get_peak_usage(). It consumes only 800kb per script. — invisal, Aug 18 '12 at 03:28
Just because it works 70% of the time doesn't mean that some input or variable won't make the train come of the tracks and kill the villagers. I am fairly positive that train is an infinite loop. :D — Jared Drake, Aug 18 '12 at 03:34
Can you do a `free -m` on the server itself and verify you actually have 800k of free memory to allocate. — Leigh, Aug 18 '12 at 04:12
When you say, "it works perfectly fine with local server", did you hit that local server with, say, 1 million page views? (Got that number from your 200,000 views/day * 5 days. So, not entirely from thin air.) — Mike Sherrill 'Cat Recall', Aug 25 '12 at 13:54
@Catcall: good question. I have not tested with 1,000,000 page views with local server. What I do not understand is that Apache only used around 400mb. There are around 6GB of memory left and the error said that I do not have 24576 bytes extra memory to allocate. My memory_limit is -1 which I believe it means there is no restriction. — invisal, Aug 25 '12 at 14:02
I remember having a similar problem, it was because I used a loop to insert some data into the database using doctrine, and for every iteration, he opened a cursor in the database ! — Anas, Aug 25 '12 at 14:02
@invisal: On the one hand, if you can reproduce the problem locally, it's a lot easier to troubleshoot. On the other hand, it can be hard to faithfully reproduce the execution environment even when you start with a dedicated server. (Hardware might come into play.) — Mike Sherrill 'Cat Recall', Aug 25 '12 at 14:07
Let's think a little out-of-the-box. Do you have any memory limit on apache processes? Could there be any `ulimit` in effect (maybe for the www-user or the owner of the PHP script, e.g. in .profile)? Have you tried running the script in CLI as soon as it stops working in the Apache module? Do you use any caching mechanism PHP modules? Do you use suhosin or similar hardening modules? — PhilMasteG, Aug 25 '12 at 14:16
@Catcall: The problem is that the error appeared so random that I cannot narrow down the cause. Only restart server once every 2 days can ensure that the problem will never happen. In peak time, there will be around 1,700 users connection at the same time. Can it be problem? — invisal, Aug 25 '12 at 14:17
@PhilippGrassl: There is no `ulimit` in my Apache Configuration. This dedicated server is dedicated to run only this website alone with no other site. I do not have any cache mechanism. I only use memcached for any expensive query that frequently called. (I have updated list of extension in my post. You can check.) — invisal, Aug 25 '12 at 14:33
What is the result of this mysql query: `EXPLAIN your_query_here`? — Jocelyn, Aug 25 '12 at 14:45
@Jocelyn: if you go to my live server link, you would probably see what the result return. It returns name of 2 teams, their logo image path, and their goals. The result will be grouped by date in PHP. (I know that this question is very headache and long +1 to every comment as reward for pay attention on this question) — invisal, Aug 25 '12 at 14:49
It is not what I am asking. I don't want to see the data returned by your query. I am asking the result of `EXPLAIN your_query_here`. Only someone with direct access to your mysql server can know that. — Jocelyn, Aug 25 '12 at 14:51
I have updated what you have requested on my post. You can check. — invisal, Aug 25 '12 at 14:56
It'd really help if you could share the code up to, say, line 85. — Bugs, Aug 25 '12 at 15:22
@Bugs: I have updated more code till line 86. Anyway, the error is very random. Sometimes it errors on different line. — invisal, Aug 25 '12 at 15:33
@invisal: by up to 85 I mean from 1 to 85. The 7 lines you posted tell very little. I can't see if it's in a loop, if it's executed conditionally, what happens before it, etc. As you can see it's not a straightforward issue, so the more you let us know the better. — Bugs, Aug 25 '12 at 16:22
Please post more of your code. It would also be interesting how much data is in your tables. If restarting the server helps for a while it could be a memory leak indeed but it does not necessary need to be PHPs fault. Maybe you're adding a lot of data to the sessions of your users? This could explain why it's working locally - there simply aren't that many / if any users. — konqi, Aug 27 '12 at 09:13
It is unlikely to be the line that the error mentions that is the actual cause of the problem, it will just be the last bit of code that pushed too far (hence why it changes). The things that I would be looking for before line 81 would $_SESSION requests, image generation, complicated or clever file include systems... Or it could be something else enitrely more server based. One thing to double check would be that your db connections are being closed after you're finished with them, especially on a highload site, you might even try persistent connections. What table type is your db table? — Pebbl, Aug 27 '12 at 09:28
I would also say as last resort, run away from Windows platforms, and migrate it to the same Linux (debian server) - this by itself might solve memory leaks (we all know what Windows is worth). Also, your EXPLAIN post is not the results of EXPLAIN. Run EXPLAIN ANALYZE and post the results - it should NOT include any normal result data in the response. — Carmageddon, Aug 27 '12 at 11:34

score 33 · Answer 1 · edited May 23 '17 at 11:54

I ran accross the same kind of problem with the server dying when trying to use the swap. This is because mod_php does not free memory ever. So Apache processes keep growing either reaching apache or PHP's memory limit or, if there's no limit, crashing the server.

Restarting apache makes it to spawn new fresh slim processes but as they run PHP scripts over time, they grow until problems arise.

The solution is to make apache to kill processes after a certain number of queries served so it will create new ones ( There are some questions related to that) reducing the MaxRequestsPerChild configuration option to, let's say 100 (Defaults to 1000).

Of course this may reduce server performances as it takes ressources to kill and spawn new processes but at least it keeps the site working. You might be tempted to raise the number of running processes to keep performances high, be sure PHP (or apache) memory limit x max number of processes do not get over your server's physical ram.

Here's my experience, hope it helps.

i have this error and i am using xampp right now..would i still get this kind of problem if i use server?from your answer i think the reason why i got error is my ram didnt process all but if a server with a higher ram would take care of this right?please correct me if i am wrong — Brownman Revival, May 30 '15 at 03:58
MaxRequestsPerChild 10000 defaults to 10000 it actually looks like http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild — blamb, Sep 11 '18 at 21:45

score 15 · Answer 2 · answered Aug 28 '12 at 16:17

For starters, memory_get_peak_usage() will not be helpful here. It will only return the amount of memory which was allocated, and that is the same number which caused the error.

memory_get_usage will return the active amount of memory which is being allocated when it is called.

ini_set('memory_limit', '256M'); will set the maximum allowance of PHP's footprint on your systems Memory. If you are getting OOM at 768K, upping it will not fix the problem.

There is no indication as to what version of PHP you are using, but I would suggest an upgrade immediately. There are several bugs where Zend's Memory Manager fails to deallocate memory, which would lead you exactly to the same problem.

Are both your local server and your production server running the same version of OS, the same long bit and the same version of PHP? The answer will be no.

If it is unrelated to the windows malloc() issue, being it is a sub domain and probably within a VirtualHost, and allocating only 768k, it almost sounds like an OS issue.

Run tasklist from the command prompt when you access your script. Do you see an additional Apache thread, or Memory usage across the processes spike?

One last idea is, run flush() and/or ob_flush(); after each loop for the table row/column. This should clear your buffer and save you some memory in the event this is where the issue is occurring.

If your files are Zend encoded - that could be the problem. Zend encoded files don't release memory like they should (Even if you unset variables). I've had scripts that would consistently run out of memory on large files - but only if the script was Zend encoded (using an unencoded version worked perfectly ever time) — msEmmaMays, Aug 28 '12 at 18:39
Adding ZendOpcache to PHP 5.4.13 w/ php-fpm and fcgi caused the memory limit error for me on select pages. Thanks for the Zend tip. — Kenneth Benjamin, May 29 '15 at 14:29

score 8 · Answer 3 · answered Aug 27 '12 at 08:55

8

I would start by upgrading PHP to 5.4+ as it's up to 50% faster for some applications. They fixed a large number of memory leaks. Please see becnhamrks: http://news.php.net/php.internals/57760

answered Aug 27 '12 at 08:55

maxwell2022

2,818
5
41
60

1

Upgrading to PHP 5.4+ would be my last resort if everything fails. Thanks. – invisal Aug 27 '12 at 10:00

score 7 · Answer 4 · answered Aug 29 '12 at 09:34

7

Please note that the error is Out of memory and is not Allowed memory size [..] exhausted.

So the memory leak is elsewhere on the system. It's possible that mysql server use a lot of system memory after this heavy query, leaving apache/php without it physical and swap.

This should explain the error always on the same line (and/or in the same script).

answered Aug 29 '12 at 09:34

Luca Rainone

16,138
2
38
52

In my case mysql was out of memory - and it was running on separate host. – brablc Mar 27 '16 at 23:41

score 6 · Answer 5 · answered Aug 27 '12 at 09:10

6

Install xdebug and enable profiler trigger. Generate a profiler file, then post the cachegrind file if you still would not be able to tell the source of the problem.

EDIT: profiler file of the page where the memory leak happens of course!

answered Aug 27 '12 at 09:10

Carmageddon

2,627
4
36
56

1

To be honest, I have never used xdebug, but I will give it a try and I will give you feedback later. – invisal Aug 27 '12 at 09:59
it could have been nice of you, if you posted step-by-step how to "do that process" from start to end. – T.Todua Jan 09 '19 at 08:38

score 5 · Answer 6 · answered Oct 24 '17 at 10:33

Fatal error: Out of memory (allocated SOLVED
i had similar problem, for months no solution. finally i was checking in one of the apache folders i.e(\apache\conf\extra) i came across this file that controls the memory allocation of apache. the file name is httpd-mpm in that file you are to increase the MaxMemFree which is set 2048 to something higher, i took mine to 10000 for the first MaxMemFree (IfModule !mpm_netware_module) then made the second one MaxMemFree to 5000 IfModule mpm_netware_module.

These solved my problem. hope it helps

Xeoncross · Answer 7 · 2012-08-29T19:38:09.453

4

I would guess that you either haven't edited the right php.ini or you haven't restarted PHP and/or the webserver.

Create a phpinfo.php page in your docroot with the contents <?php phpinfo(); to make sure you are changing the correct php.ini. In addition to the location of the php.ini file the webserver is using, it will also state the maximum script memory allowed.

Next, I would add some stack traces to your page so you can see the chain of events that led to this. The following function will catch fatal errors and provide more information about what happened.

register_shutdown_function(function()
{
    if($error = error_get_last())
    {
        // Should actually log this instead of printing out...
        var_dump($error);
        var_dump(debug_backtrace());
    }
});

Personally, Nginx + PHP-FPM is what I have used for years since I left slow ol' Apache.

edited Aug 29 '12 at 19:38

answered Aug 29 '12 at 19:32

Xeoncross

55,620
80
262
364

1

First of all, I fully aware that I need to restart Apache for my PHP setting to take effect. I have included cachegrind.out generated by xDebug. My website is a high-traffic website with average of 1,000 visitors every minute. – invisal Aug 30 '12 at 05:44
1

@invisal, I don't doubt that you are a very capable dev - but sometimes simple things get overlooked. Did you check that you are sure you are editing the correct `php.ini`? – Xeoncross Aug 30 '12 at 16:09
1

I have double check with phpinfo(). I got the correct setting now. I am working now on minimizing memory usage. (hope it will solve problem) – invisal Aug 30 '12 at 16:14

score 3 · Answer 8 · answered Aug 29 '12 at 14:07

3

Could be an issue with MySQL and the number of open connections, hence why it sorts itself out when you restart every few days. Are they auto closing on script shutdown?

answered Aug 29 '12 at 14:07

Matt Humphrey

1,554
1
12
30

1

I only restart Apache to solve problem. The script does not call MySQL often, because mostly they are cached with `memcached`. – invisal Aug 29 '12 at 14:09

score 3 · Answer 9 · answered Sep 01 '12 at 06:07

3

Hey I also got the same problem on my server. I just changed the following things :

change php.ini to...

memory_limit = 128M

and add to httpd.conf

RLimitMEM 1073741824 2147483648

and restart apache & i removed the error :

answered Sep 01 '12 at 06:07

Rohitashv Singhal

4,517
13
57
105

score 2 · Answer 10 · answered Aug 31 '12 at 12:57

Just to recap (I'm adding this answer quite a distance from the original question):

PHP is unable to allocate what appears to be a small amount of memory
the current memory usage at the time the error occrs + the requested amount is less than the memory limit currently in force
the system has 6Gb available for use by PHP when this occurs
since the problem is resolved by restarting apache - it's apache which is preventing the memory from beig available to PHP

If these are all valid then the only possible explanation is that the 6Gb is very fragmented - which I think is a little unlikely. You didn't say how PHP is invoked from Apache - mod_php? fpm? Fcgi?

I would start by examining each of the above predicates - particularly the free memory one. How do you know that there's 6Gb free when the error occurs? A more likely cause is that there's a memory leak occurring which you're not spotting.

You've not provided any details of how apache is configured; I'd also have a look at reducing MaxRequestsPerChild and MaxMemFree. (I'm not very familiar with worker apache where this will apply per thread - really you need a limit per process). If you provided the core setting from the apache config then maybe we could make further suggestions.

Unless you are using Ajax extensively, make sure your keepalive time is 2 or less.

Ja͢ck · Answer 11 · 2012-08-27T10:03:24.777

1

The following two facts definitely point to memory leaks:

The error appears at different lines in your code,
The error reports a relatively small memory allocation.

I would first single out PDO, disabling all the other extensions and let it run overnight using something like Siege / Apache Bench (ab). You could also try running it using the cli interface (just make sure you keep the same memory limits).

You could use the memory_get_peak_usage() function at the end of your script to see how much memory PHP thinks it has been using.

From your comment that's 800 kB, which is okay; definitely not the gigantic amount of memory that would cause an out-of-memory ;-)

Lastly, though I wouldn't recommend upgrading to 5.4 at this point, upgrading to the latest 5.3.x is probably worth it due to multiple vulnerabilities and leaks that have been addressed since 5.3.1

edited Aug 27 '12 at 10:03

answered Aug 27 '12 at 09:17

Ja͢ck

170,779
38
263
309

1

Thanks for paying attention to my question. One upvote as a reward. I have used `memory_get_peak_usauge()` and it used around 800kb each time. I have tweaked different configurations and I will update what I have observed later. – invisal Aug 27 '12 at 09:59
@invisal forgot to ask you; when you get the errors, is apache still at just 500MB? i wonder if you have preforked, whether there's another memory limit you can configure :) of course, having a max requests on your php could also help to limit the memory – Ja͢ck Sep 01 '12 at 13:23

kapitanluffy · Answer 12 · 2012-08-30T07:41:40.390

this happened to me a few days ago. I did a fresh installation and it still happened. as far as everyone sees and based on your server specs. most likely it is an infinite loop. it could be not on the PHP code itself but on the requests made to Apache.

lets say when you access this url http://localhost/mysite/page_with_multiple_requests

Check your Apache's access log if it receives multiple requests. trace that request and check out the code that might cause a 'bottleneck' to the system (mine's exec() when using sendmail). The bottleneck im talking about doesn't need to be an 'infinite loop'. It could be a function that takes sometime to finish. or maybe some of php's 'program execution functions'

You might need to check ajax requests too (the ones that execute when the page loads). if that ajax request redirects to the same url

e.g. httpx://localhost/mysite/page_with_multiple_requests

it would 'redo' the requests all over again

it would help if you post the random lines or the code itself where the script ends maybe there is a 'loop' code somewhere there. imho php won't just call random lines for nothing.

http://blog.piratelufi.com/2012/08/browser-sending-multiple-requests-at-once/

score 1 · Answer 13 · answered Sep 01 '12 at 11:02

Most of the time you get such a error, problem is in code. I am not trying to say that you are writing code that is bad, I am trying to say that you need to carefully observe what's there in that is using this much amount of memory.

Always remember "Garbage collection in PHP is pretty bad", it's not like Java, any other such language. there is a way to enforce garbage collection through gc_collect_cycle, but, in my personal opinion, that won't solve your problem. PHP free all memory used for executing a page, once request-response cycle is complete, so you may run into memory issues, if your script is long running, like a background script(Gearman etc), because, memory isn't freed till script is running.

If above is not the case with your scr,pt, and as you said there is no code that requires such a huge amount of memory, then problem is most definitely in code it self, and upgrading to any version of PHP won't solve the problem. I was facing with one of my Gearman scripts once, and there was a problem with one of my loop where I was appending one variable to one of my array, the variable itself was very heavy (approx 110KB of data). So I would suggest, do a careful inspection of your code.

Ravish

score 1 · Answer 14 · answered Sep 01 '12 at 13:56

I had a similar problem with PHP:

1) Check your error logs. Eliminate EVERY error before continuing. 2) Consider modifying your apache configuration to eliminate unused modules - this will reduce the footprint needed by PHP - here's a great link for this - it's specific to Wordpress but should still be very useful http://thethemefoundry.com/blog/optimize-apache-wordpress/

To give you an idea of the kind of bug I found, I had some code that was trying to post content to Facebook, Facebook then modified their API so this broke, I also used a 'content expirator' which basically meant that it kept retrying to post this content to Facebook and leaving loads of objects lying in memory.

Thanks for pointing out. I am on the process of tweaking my Apache configuration. I need a few days to monitor each configuration changed to make sure it perfectly is fine. Thanks. — invisal, Sep 01 '12 at 17:26

score 1 · Answer 15 · answered Sep 01 '12 at 14:09

From the profiler output file I've noticed a couple of things which I did not like/trust much and would look into these:

Apart from not knowing the meaning of the output numbers to detect an anomaly or how PHP scripts work..., Is this not a problem? Having an include to the same main.ph file, which looks like a recursive thing?

2121 fl=D:\www\football\views\main.php
2122 fn=include::D:\www\football\views\main.php

Noticed the file D:\www\football\views\main.php is making use of some string functions several times, I guess it is calling these functions on the data returned by your query:

strlen
substr
strtotime

If, like in the C language, these functions require strings be null terminated or some other end of string terminator to avoid memory problems, I would look at the strings returned by your query.

Can you post the url of your website?

http://football.khmerload.com/ I got the url from your earlier edits of your question. — Only You, Sep 01 '12 at 14:58
I use strlen, substr, and strtotime several time because I need to convert English date into my language date. `views\main.php` is included only once. — invisal, Sep 01 '12 at 17:24

score 1 · Answer 16 · answered Sep 26 '13 at 08:51

1

This is a known bug in PHP v 5.2 for Windows, it is present at least to version 5.2.3: https://bugs.php.net/bug.php?id=41615

None of the suggested fixes have helped for us, we're going to have to update PHP.

answered Sep 26 '13 at 08:51

Anonymous User

1
1

score 1 · Answer 17 · edited Mar 19 '15 at 11:56

Try to run php over fcgid, this may help:

These are the classic errors you will see when running PHP as an Apache module. We struggled with these errors for months. Switching to using PHP via mod_fcgid (as James recommends) will fix all of these problems. Be sure you have the latest Visual C++ Redistributable package installed:

http://support.microsoft.com/kb/2019667

Also, I recommend switching to the 64-bit version of MySQL. No real reason to run the 32-bit version anymore.

Source: Apache 2.4.6.0 crash due to a problem in php5ts.dll 5.5.1.0

score 0 · Answer 18 · answered Aug 27 '12 at 21:54

0

I would say the server is running out of physical/swap memory, so PHP can't allocate enough memory.

Can you paste the output of free here?

answered Aug 27 '12 at 21:54

Edson Medina

9,862
3
40
51

score 0 · Answer 19 · answered Oct 24 '17 at 10:15

For my case, this error was triggered because of a huge select query (hundreds of thousands of returned results).

It arose immediately after adding millions of records in my Database to test the scalability of WordPress, so it was the only probable reason for me.

score 0 · Answer 20 · answered Sep 23 '20 at 00:45

I know it's an old thread, but here's my experience getting it resolved.

My server is a hosted service running Apache.

My script crashed with out of memory at 6Mb, when my limit Was 256Mb - crazy, yeah?

It is being called synchronously via an http callback, from javascript running on my client, and crashed after around 550 calls. After much time wasted with incompetent "Escalated Support" guys, my script now magically runs.

They said all they did was to reset php.ini, but I checked the differences:

No changes there that I can see that could have a bearing on an Out of Memory error.

I suspect a memory leak in the web server which my "Escalated Support" guy is hiding under the guise of resetting the php.ini. And, really, I'm not a conspiracy theorist.

Fatal error: Out of memory, but I do have plenty of memory (PHP)

Hints

Server Spec

Solution

20 Answers20

Linked