What should be the convenient memory limit to php read 1Gb file?

Question

I have a 1Gb file1 that must be read. I choose php to filter and change some lines and than create another file2 with these changes. The code is good. If I read 50M file it works good. The code generate a file2 whit all changes, as expected. But when I try to run the 1Gb file, the file2 is not created and I get a error message from browser like this:

The connection to localhost was interrupted.
Check your Internet connection
Check any cables and reboot any routers, modems, or other network devices you may be using.
Allow Chrome to access the network in your firewall or antivirus settings.
If it is already listed as a program allowed to access the network, try removing it from the list and adding it again.
If you use a proxy server...
Check your proxy settings or contact your network administrator to make sure the proxy server is working. If you don't believe you should be using a proxy server: Go to the Chrome menu > Settings > Show advanced settings... > Change proxy settings... > LAN Settings and deselect "Use a proxy server for your LAN".

If I return and run the small file it works good again. I already sit the php memeory to 2040M ini_set('memory_limit', '2048M') but I dont know if it is enough or f it is possible.

So, how should be a convenient memory for this issue?

NOTE: Server is apache, win7, i7 8cores 64bits, 16G RAM. I think the code is not important but someone ask to the see it:

                       ini_set('memory_limit', '2048M');#Set the memory limit
                              $new_dbName="au_site";
                              $patern[0]="/di_site/";
                              $replace[0]=$new_dbName;
                              #xdebug_break();
                              $dirBase=dirname(__FILE__);
                              $dir2 = new DirectoryIterator(dirname(__FILE__));
                              #xdebug_break();
                              foreach ($dir2 as $fileinfo) {
                                     if (!$fileinfo->isDot() && $fileinfo->isFile()) {
                                          $str = $fileinfo->getFilename();
                                          if (preg_match("/\.sql/i", $str)) {
                                               #xdebug_break();
                                               $i=1;
                                               if(!($handle= fopen("$str", "r"))){

                                                die("Cannot open the file");
                                               }
                                               else{
                                                     while (!feof($handle)) {   
                                                             #xdebug_break();
                                                             $line=trim(fgets($handle), "\t\n\r\0\x0B");
                                                             $firstChar =  substr($line, 0,1) ;
                                                             $ord = ord($firstChar);
                                                             if(ord($firstChar)<>45){
                                                               if (preg_match("/di_site/", $line)) {
                                                                xdebug_break();
                                                                   $chn=preg_replace($patern, $replace, $line); 
                                                                   $line=$chn;
                                                               }
                                                               #echo $line."<br>";
                                                               $sql.=$line."\n";
                                                             }                                                            
                                                      }
                                                      xdebug_break();
                                                      $newDBsql=$dirBase."/".$new_dbName.".sql";
                                                      if(!$handle =  fopen($newDBsql,"w")){
                                                           die("Can not open the file");
                                                      }
                                                      else{
                                                       fwrite($handle, $sql);
                                                       fclose($handle);
                                                      }

                                               }
                                            } 
                                     }
                               }

This might be helpful: http://stackoverflow.com/questions/162176/reading-very-large-files-in-php the suggestion is to use [fget](http://php.net/fgets) reading the file line-by-line. — Mehdi, Jun 18 '15 at 02:28
The file is too large. The browser is timing out. I think the default is 30 seconds — ksealey, Jun 18 '15 at 03:03
Hi Dragon, i think the code is not important. It is working good for small files — IgorAlves, Jun 18 '15 at 03:22
`$sql.=$line."\n";` -- that's certainly an issue when have you have many lines. — Ja͢ck, Jun 18 '15 at 03:23
ksealey, I am not displaying in the browse. Only creating a file. — IgorAlves, Jun 18 '15 at 03:23
Jack. but it works for small files.. "\n" is to make new line on file, not to display on browser — IgorAlves, Jun 18 '15 at 03:25
"a few MB"? Do you need to read/write the file in a single pass, or can you just parse/generate it sequentially? Because the filesize on disk has nothing to do with the volume of bytes you need to have in memory to do the job you need to do. — Mike 'Pomax' Kamermans, Jun 18 '15 at 03:26
I dont want to read and write. I am reading, saving changes in a variable, and than at the end, creating a new folder with the stored information. — IgorAlves, Jun 18 '15 at 03:32

Ja͢ck · Accepted Answer · 2015-06-18T04:26:05.650

Instead of building up the whole file contents you're going to write (which takes a lot of memory), consider writing a stream filter instead.

A stream filter operates on a single buffered read from the underlying stream, typically around 8kB of data. The following example code defines such a filter, it splits each bucket into separate lines and calls your code to make changes to it.

<?php

class myfilter extends \php_user_filter
{
  private $buffer; // internal buffer to create data buckets with

  private $pattern = ['/di_site/'];
  private $replace = ['au_site'];

  function filter($in, $out, &$consumed, $closing)
  {
    while ($bucket = stream_bucket_make_writeable($in)) {
      $parts = preg_split('/(\n|\r\n)/', $bucket->data, -1, PREG_SPLIT_DELIM_CAPTURE);
      $buffer = '';
      // each line spans two array elements
      for ($i = 0, $n = count($parts); $i + 1 < $n; $i += 2) {
        $line = $parts[$i] . $parts[$i + 1];
        $buffer .= $this->treat_line($line);
        $consumed += strlen($line);
      }
      stream_bucket_append($out, stream_bucket_new($this->stream, $buffer));
    }
    return PSFS_PASS_ON;
  }

  /** THIS IS YOUR CODE **/
  function treat_line($line)
  {
    $line = trim($line, "\t\n\r\0\x0B");
    $firstChar =  substr($line, 0,1) ;
    if (ord($firstChar)<>45) {
      if (preg_match("/di_site/", $line)) {
        $line = preg_replace($this->pattern, $this->replace, $line);
      }
    }
    return $line . "\n";
  }

  function onCreate()
  {
    $this->buffer = fopen('php://memory', 'r+');
  }

  function onClose()
  {
    fclose($this->buffer);
  }
}

stream_filter_register("myfilter", "myfilter");

// open input and attach filter
$in = fopen(__FILE__, 'r');
stream_filter_prepend($in, 'myfilter');
// open output stream and start copying
$out = fopen('php://stdout', 'w');
stream_copy_to_stream($in, $out);

fclose($out);
fclose($in);

What should be the convenient memory limit to php read 1Gb file?

1 Answers1