Codeigniter Allowed memory size exhausted while processing large files

Question

I'm posting this in case someone else is looking for the same solution, seeing as I just wasted two days on this bullshit.

I have a cron job that updates the database using a very large file once a day, using the following code:

if (($handle = fopen(dirname(__FILE__) . '/uncompressed', "r")) !== FALSE) 
{
    while (($data = fgets($handle)) !== FALSE) 
    {
        $thisline = json_decode($data, true);
        $this->regen($thisline);
    }
    fclose($handle);
}

This is in a Codeigniter controller that's only used for cron jobs. The $this->regen function runs through a bunch of different checks and stores the right information from the line in the database. The file itself is over 300MB of JSONs separated by newlines.

The problem: it would only process about 20,000 lines before the whole thing ran out of memory.

That's pretty cool, but... this should probably be closed ASAP since it isn't a "question". — Scott Arciszewski, Mar 22 '14 at 20:56
Yeah, I guess so. I don't have that ability though so I'll have to wait on someone else to do it. — , Mar 24 '14 at 00:42
@Cryan You could instead of using "the solution" create an answer for this, then mark that as answered ;) — Steve P, Mar 25 '14 at 15:25

score 14 · Accepted Answer · answered Mar 25 '14 at 15:38

I spent hours troubleshooting this and got nothing obvious. I'm using fgets, I have $query->free_result() in the right places. It didn't help. So then I started checking a loop of about 100 lines, and watched the output of memory_get_usage(). I finally narrowed it down to the Codeigniter Active Record class - every call to the class caused the memory usage to increase by a tiny amount.

Then I found this thread on Ellislabs and I got the answer. CI Active Record saves queries so that if you want to, you can build a query in multiple functions. (I am not even going to go into how dumb it is to have that switched on by default.)

Go to /config/database.php and add

$db['default']['save_queries'] = FALSE;

to the end of the file. Then make sure you build and execute queries using Active Record in a single function. If you need to switch it off just for one case, use

$this->db->save_queries = FALSE;

in the constructor or wherever you need to put it.

While I didn't find this answer until I knew what to search for, at work there's been a server that handled a whole lot of API calls and ran out of RAM just about immediately, bouncing against its limit and getting its processes killed... this turned out to be almost the entire problem. There's a lot about CI that makes me shake my head, but this one cost us enough time and hair-pulling that it's probably right at the top. — kungphu, Mar 03 '16 at 06:32

Codeigniter Allowed memory size exhausted while processing large files

1 Answers1

Linked