0

So after encountering the memory limits I copied the following code.

It worked great. My only issue now is in the context of processing a csv file, the structure limits based on the size of chunk, but this would mean it could cut a row mid way through.

What could I do to ensure when a chunk is made, that it ends up to the /n.

Community
  • 1
  • 1
Pete
  • 683
  • 1
  • 8
  • 17
  • As you are processing the lines in the chunk, when you encounter the last line, just hold on to the contents of the last line and add it to the front of the next chunk as you move to process that. When there are no more chunks, just process the last line. – Aaron D Sep 28 '14 at 04:13

2 Answers2

0

You can first try setting the memory limit with :

ini_set('memory_limit', '32MB');

Then, to read a line at once :

$handle = fopen("inputfile.csv", "r");
if ($handle) {
    while (($line = fgets($handle)) !== false) {
        // process the line read.
    }
} else {
    // error opening the file.
} 
fclose($handle);

Based on your code example, this would give :

function file_get_contents_chunked($file,$chunk_size,$callback)
{
    $handle = fopen($file, "r");
    $i = 0;
    if ($handle) {
        while (($line = fgets($handle)) !== false) {
           call_user_func_array($callback,array($line,&$handle,$i));
           $i++;
        }
    } else {
        return false;
    } 
    fclose($handle);
    return true;
}

Note that $chunk_size parameter is not needed anymore... so you can remove it if you want.

You can also use this function to read a sav file line by line :

fgetcsv(file,length,separator,enclosure);

Example 1

<?php
$file = fopen("contacts.csv","r");
print_r(fgetcsv($file));
fclose($file);
?>

The CSV file:

Kai Jim, Refsnes, Stavanger, Norway
Hege, Refsnes, Stavanger, Norway

The output of the code above will be:

Array
(
[0] => Kai Jim
[1] => Refsnes
[2] => Stavanger
[3] => Norway
)
Graben
  • 202
  • 3
  • 8
0

I'm interested to know what others do.

I changed the limit to based on a certain amount of lines to read instead of the size limit. Instead of using fread, i used fgets to get the whole line. Once again, this is all derived from the code linked in the question.

<?php
function file_get_contents_chunked($file,$chunk_size,$callback)
{
    try
    {
        $handle = fopen($file, "r");
        $i = 0;
        $x = 0;

        $chunk = array();
        while (!feof($handle)) {
            while ($row = fgets($handle)) {
                // can parse further $row by usingstr_getcsv
                $x ++;
                $chunk[] = $row;
                if ($x == $chunk_size) {          
                    call_user_func_array($callback, array($chunk, &$handle, $i));
                    unset($chunk);
                    $x = 0;
                }         
            }
        }
        fclose($handle);
    }
    catch(Exception $e)
    {
         trigger_error("file_get_contents_chunked::" . $e->getMessage(),E_USER_NOTICE);
         return false;
    }
    return true;
}

?>

//Fixed for what I actually intended, limit by x amount of lines

Pete
  • 683
  • 1
  • 8
  • 17
  • 1
    its funny... we ended up with the same answer pretty much at the same time (4 mins gap)... except the while ! feof... which is not needed as specified in the fgets php manual... I thought to add it but then looked it up to make sure if its needed or not. Lets look at what others will come up with ! – Graben Sep 28 '14 at 04:45
  • I'm still reviewing my changes as its not doing exactly what I thought it to do. I was hoping to specify to read x amount of lines, but it doesn't seem to be doing that at the moment. – Pete Sep 28 '14 at 04:51
  • I'm curious as to why you would limit the amount of lines as they are processed line per line so all you need to ensure is that each line is not too big for available memory... In the end i'm pretty sure the op wants the whole file processed no ? – Graben Sep 28 '14 at 15:00