PHP memory exhausted when running through thousands of records

Question

I'm running the following code over a set of 5,000 results. It's failing due to the memory being exhausted.

foreach ($data as $key => $report) {
  $data[$key]['data'] = unserialize($report['serialized_values']);
}

I know I can up the memory limit, but I'd like to run this without a problem instead. I'm not going to be able to keep upping the memory forever.

EDIT

The $data is in this format:

[1] => Array
    (
        [0] => 127654619178790249
        [report_id] => 127654619178790249
        [1] => 1
        [user_id] => 1
        [2] => 2010-12-31 19:43:24
        [sent_on] => 2010-12-31 19:43:24
        [3] => 
        [fax_trans_id] => 
        [4] => 1234567890
        [fax_to_nums] => 1234567890
        [5] => ' long html string here',
        [html_content] => 'long html string here',
        [6] => 'serialization_string_here',
        [serialized_values] => 'serialization_string_here',
        [7] => 70
        [id] => 70
    )

I realize this is an old question, but for anyone encountering something remotely similar to this, one of the first question you should ask yourself is, "if I remove all the code inside the foreach loop, does it still run out of memory?" Basically making sure you have actually narrowed the problem down to the smallest thing that will reproduce it. Similar things to try would be to remove as much code as possible before the loop and see if this still happens. — still_dreaming_1, Aug 30 '18 at 19:32

score 10 · Accepted Answer · answered Jan 01 '11 at 02:31

Beyond the problems of for and foreach, you need to re-architect your solution. You're hitting memory limits because you're legitimately using too much memory. Each time you unserialize the contents of the database column and store it in an array

$data[$key]['data']

PHP needs to set aside a chunk of memory to store that data so it can be accessed later. When your array gets too big you're out of memory. In plain english, you're telling PHP

Take all 5000 rows of data and store them in memory, I'm going to do something with them later.

You need to think of a different way to approach your problem. The below items are two quick thoughts on the problem.

You could not store the items in memory and just take whatever actions you wanted to in the loop, allowing php to discard the items as need be

foreach ($data as $key => $report) {
    $object = unserialize($report['serialized_values']);        
    //do stuff with $object here
}

You could also only store the information you need from the unserialized object, rather than storing the entire object

foreach ($data as $key => $report) {
    $object             = unserialize($report['serialized_values']);        
    $data               = array();
    $data['foo']        = $object->foo;
    $data[$key]['data'] = $data;
}

Long story short: you're hitting memory limits because you're actually using too much memory. There's no magic solution here. Storing serialized data and attempting to load it all in a single program is a memory intensive approach, irrespective of language/platform.

Fantastic explanation. 99% of the time it's a coding error when this happens, so it's nice (I guess?) for it to actually tell me the truth that I'm running out of memory. I ended up refactoring the code so it didn't process all of the 5000 rows at once. Thanks again for your help Alan! — James Skidmore, Jan 26 '11 at 01:56

score 3 · Answer 2 · answered Jan 01 '11 at 02:04

3

A foreach will load all 5,000 results into memory. See the numerous complaints in the docs. Use a for loop and access each result as you need it.

answered Jan 01 '11 at 02:04

moinudin

134,091
45
190
216

Good thought, but it's still throwing a memory error with `for` instead of `foreach`. Hmm... – James Skidmore Jan 01 '11 at 02:12
@James and if you remove the loop? It might not be the culprit. – moinudin Jan 01 '11 at 02:14
Also, would it make sense to `unset` the values in `$report` once you've `unserialize` d them? In other words, free the memory used by individual entries in `$data` after you process. Do this *inside* the loop. – moinudin Jan 01 '11 at 02:16
If I remove the loop altogether (and thus not process any of the data), the error disappears. – James Skidmore Jan 01 '11 at 02:16
Unsetting them after use doesn't work either, even if it's in a `for` loop. – James Skidmore Jan 01 '11 at 02:27
I don't quite get the point of 'A foreach will load all 5,000 results into memory' The records have been within $data in memory before and for each loop round there is only one result assigned and copied to $result. The other copies will be grabbed by garbage collection, won't they? – velop Feb 02 '14 at 17:20

score 1 · Answer 3 · answered May 05 '14 at 09:05

1

I think these bugs are not closed yet:

"When unserializing inside a loop the same serialized object, the total memory consumption increases every couple of iterations"

answered May 05 '14 at 09:05

chris

1,245
1
10
22

score 1 · Answer 4 · answered Jan 01 '11 at 02:27

What is $data and where are you getting it from? If it's a file, can't you fgets() one line at a time to parse, and if it's a database, can't you process one record at a time (at the expense of the MySQL waiting to close the result set). I think you should reconsider loading the entire of $data into memory at once and then looping over it.

score 0 · Answer 5 · answered Jan 01 '11 at 02:14

0

try this way

foreach ($data as $key => &$report) {
}

This will assign reference instead of copying the value.

answered Jan 01 '11 at 02:14

Fivell

11,829
3
61
99

Thanks for the response Fivell, but it still is throwing the error. – James Skidmore Jan 01 '11 at 02:15
what is structure of data elements ? – Fivell Jan 01 '11 at 02:16
That will actually consume more memory since PHP must now maintain that it's a variable reference. Before (without the &) PHP will update the pointers and if it's overwritten, only will it then create a new object and point to it, otherwise it'll point to the original. It doesn't actually copy it the first time to save memory. – Tim Green Jan 01 '11 at 02:25
James, unserializing produces additional data strucutres in memory , so if you have such problem than maybe it is time to start refactoring ... – Fivell Jan 01 '11 at 02:30
Also, Tim, you're right - references don't resolve such problems – Fivell Jan 01 '11 at 02:32

score 0 · Answer 6 · answered Jan 01 '11 at 02:21

This is actually the reason why many sites divide result in pages.

Suppose I have 5000 results (say users, to simplify) and I have a page that should display all those 5000 results. I would divide those 5000 results into 500 per page so that one page 1 displays 1 - 500, page 2 displays 501 - 1000, page 3 displays 1001 - 1500 and so on. In this way, memory is saved.

If you really need to display all 5000 results in one page, you really need to increase the memory limit. Or use for loop instead.

score 0 · Answer 7 · answered Jan 01 '11 at 02:22

0

I don't know for sure but you might use:

gzip($data set) to compress data to safe memory and deflate it on the fly.
limit (data)set.
create a cache like system. evict least recently used(LRU) data from cache if using too much memory.

answered Jan 01 '11 at 02:22

Alfred

60,935
33
147
186

PHP memory exhausted when running through thousands of records

7 Answers7

Linked