How to read a large file line by line?

Question

I want to read a file line by line, but without completely loading it in memory.

My file is too large to open in memory, and if try to do so I always get out of memory errors.

The file size is 1 GB.

Answer 1

You can use the fgets() function to read the file line by line:

$handle = fopen("inputfile.txt", "r");
if ($handle) {
    while (($line = fgets($handle)) !== false) {
        // process the line read.
    }

    fclose($handle);
}

Answer 2

if ($file = fopen("file.txt", "r")) {
    while(!feof($file)) {
        $line = fgets($file);
        # do same stuff with the $line
    }
    fclose($file);
}

Answer 3

You can use an object oriented interface class for a file - SplFileObject http://php.net/manual/en/splfileobject.fgets.php (PHP 5 >= 5.1.0)

<?php

$file = new SplFileObject("file.txt");

// Loop until we reach the end of the file.
while (!$file->eof()) {
    // Echo one line from the file.
    echo $file->fgets();
}

// Unset the file to call __destruct(), closing the file handle.
$file = null;

Answer 4

If you want to use foreach instead of while when opening a big file, you probably want to encapsulate the while loop inside a Generator to avoid loading the whole file into memory:

/**
 * @return Generator
 */
$fileData = function() {
    $file = fopen(__DIR__ . '/file.txt', 'r');

    if (!$file) {
        return; // die() is a bad practice, better to use return
    }    
    while (($line = fgets($file)) !== false) {
        yield $line;
    }

    fclose($file);
};

Use it like this:

foreach ($fileData() as $line) {
    // $line contains current line
}

This way you can process individual file lines inside the foreach().

Note: Generators require >= PHP 5.5

Answer 5

There is a file() function that returns an array of the lines contained in the file.

foreach(file('myfile.txt') as $line) {
   echo $line. "\n";
}

Answer 6

The obvious answer wasn't there in all the responses.
PHP has a neat streaming delimiter parser available made for exactly that purpose.

$fp = fopen("/path/to/the/file", "r");
while (($line = stream_get_line($fp, 1024 * 1024, "\n")) !== false) {
  echo $line;
}
fclose($fp);

Answer 7

Use buffering techniques to read the file.

$filename = "test.txt";
$source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
while (!feof($source_file)) {
    $buffer = fread($source_file, 4096);  // use a buffer of 4KB
    $buffer = str_replace($old,$new,$buffer);
    ///
}

Answer 8

foreach (new SplFileObject(__FILE__) as $line) {
    echo $line;
}

Answer 9

One of the popular solutions to this question will have issues with the new line character. It can be fixed pretty easy with a simple str_replace.

$handle = fopen("some_file.txt", "r");
if ($handle) {
    while (($line = fgets($handle)) !== false) {
        $line = str_replace("\n", "", $line);
    }
    fclose($handle);
}

Answer 10

This how I manage with very big file (tested with up to 100G). And it's faster than fgets()

$block =1024*1024;//1MB or counld be any higher than HDD block_size*2
if ($fh = fopen("file.txt", "r")) { 
    $left='';
    while (!feof($fh)) {// read the file
       $temp = fread($fh, $block);  
       $fgetslines = explode("\n",$temp);
       $fgetslines[0]=$left.$fgetslines[0];
       if(!feof($fh) )$left = array_pop($lines);           
       foreach ($fgetslines as $k => $line) {
           //do smth with $line
        }
     }
}
fclose($fh);

Answer 11

SplFileObject is useful when it comes to dealing with large files.

function parse_file($filename)
{
    try {
        $file = new SplFileObject($filename);
    } catch (LogicException $exception) {
        die('SplFileObject : '.$exception->getMessage());
    }
    while ($file->valid()) {
        $line = $file->fgets();
        //do something with $line
    }

    //don't forget to free the file handle.
    $file = null;
}

Answer 12

Be careful with the 'while(!feof ... fgets()' stuff, fgets can get an error (returnfing false) and loop forever without reaching the end of file. codaddict was closest to being correct but when your 'while fgets' loop ends, check feof; if not true, then you had an error.

Answer 13

<?php
echo '<meta charset="utf-8">';

$k= 1;
$f= 1;
$fp = fopen("texttranslate.txt", "r");
while(!feof($fp)) {
    $contents = '';
    for($i=1;$i<=1500;$i++){
        echo $k.' -- '. fgets($fp) .'<br>';$k++;
        $contents .= fgets($fp);
    }
    echo '<hr>';
    file_put_contents('Split/new_file_'.$f.'.txt', $contents);$f++;
}
?>

Answer 14

You can use the fgets() function in combination with a loop:

$filename = 'path/to/your/file.txt';
$handle = fopen($filename, 'r');

if ($handle) {
    while (($line = fgets($handle)) !== false) {
        // Process the current line
        echo $line;

        // You can perform any desired operations on the line here
        // For example, you can parse and extract data from each line

        // Proceed to the next line
    }

    fclose($handle);
}

Answer 15

Function to Read with array return

function read_file($filename = ''){
    $buffer = array();
    $source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
    while (!feof($source_file)) {
        $buffer[] = fread($source_file, 4096);  // use a buffer of 4KB
    }
    return $buffer;
}