36

I've been bumping into a problem. I have a log on a Linux box in which is written the output from several running processes. This file can get really big sometimes and I need to read the last line from that file.

The problem is this action will be called via an AJAX request pretty often and when the file size of that log gets over 5-6MB it's rather not good for the server. So I'm thinking I have to read the last line but not to read the whole file and pass through it or load it in RAM because that would just load to death my box.

Is there any optimization for this operation so that it run smooth and not harm the server or kill Apache?

Other option that I have is to exec('tail -n 1 /path/to/log') but it doesn't sound so good.

Later edit: I DO NOT want to put the file in RAM because it might get huge. fopen() is not an option.

Lightness Races in Orbit
  • 378,754
  • 76
  • 643
  • 1,055
Bogdan Constantinescu
  • 5,296
  • 4
  • 39
  • 50
  • http://stackoverflow.com/questions/15025875/what-is-the-best-way-in-php-to-read-last-lines-from-a-file – zloctb Oct 28 '15 at 04:01

13 Answers13

57

This should work:

$line = '';

$f = fopen('data.txt', 'r');
$cursor = -1;

fseek($f, $cursor, SEEK_END);
$char = fgetc($f);

/**
 * Trim trailing newline chars of the file
 */
while ($char === "\n" || $char === "\r") {
    fseek($f, $cursor--, SEEK_END);
    $char = fgetc($f);
}

/**
 * Read until the start of file or first newline char
 */
while ($char !== false && $char !== "\n" && $char !== "\r") {
    /**
     * Prepend the new char
     */
    $line = $char . $line;
    fseek($f, $cursor--, SEEK_END);
    $char = fgetc($f);
}

fclose($f);

echo $line;

Note that this solution will repeat the last character of the line unless your file ends in a newline. If your file does not end in a newline, you can change both instances of $cursor-- to --$cursor.

Paulina Khew
  • 397
  • 4
  • 13
Ionuț G. Stan
  • 176,118
  • 18
  • 189
  • 202
  • Ionut, thank you, but read carefully next time. I don't want to read the file because it's large and I don't want it in RAM. – Bogdan Constantinescu Oct 02 '09 at 15:41
  • 8
    What do you mean you don't want to read the file? I'm not reading the whole file in memory. I just open a kind of a pointer to it, then seek it char by char. This is the most efficient way to work with large files. – Ionuț G. Stan Oct 02 '09 at 15:43
  • 22
    `fopen()` does not act like `file_get_contents()` – Ionuț G. Stan Oct 02 '09 at 15:45
  • 1
    fopen does not load the file in memory, it just creates a file descriptor (a pointer). – Wadih M. Oct 02 '09 at 17:48
  • it's really a good code thank you Ionuț G. Stan i modified your code a little and made it a function for reuseability you will find it as a new answer – Abdalla Mohamed Aly Ibrahim Jun 30 '14 at 05:06
  • 3
    I ended up with the last character repeated. When my last line was ``, This code put `>` into `$line`. To fix the behavior, I changed both instances of `$cursor--` to `--$cursor`. – rg89 Jan 05 '17 at 18:29
  • 1
    I confirm rg89's observation. If file does not end with a newline, then the code will output last character twice. Example 1: `123\n456` - Output: 4566 (incorrect). Example 2: `123\n456\n` - Output: 456 (works as expected). Excellent solution btw.. – Kristoffer Bohmann Jun 30 '17 at 15:50
  • An improvement can be done by using fread() to get a string block and then apply whatever search function you like – Jonas Nov 08 '18 at 22:10
  • Don't forget to `fclose($f);` your file pointer in the end. – Daan Feb 28 '21 at 08:08
19

Use fseek. You seek to the last position and seek it backward (use ftell to tell the current position) until you find a "\n".


$fp = fopen(".....");
fseek($fp, -1, SEEK_END); 
$pos = ftell($fp);
$LastLine = "";
// Loop backword util "\n" is found.
while((($C = fgetc($fp)) != "\n") && ($pos > 0)) {
    $LastLine = $C.$LastLine;
    fseek($fp, $pos--);
}
fclose($fp);

NOTE: I've not tested. You may need some adjustment.

UPDATE: Thanks Syntax Error for pointing out about empty file.

:-D

UPDATE2: Fixxed another Syntax Error, missing semicolon at $LastLine = ""

Daan
  • 7,685
  • 5
  • 43
  • 52
NawaMan
  • 25,129
  • 10
  • 51
  • 77
6

You're looking for the fseek function. There are working examples of how to read the last line of a file in the comments section there.

slikts
  • 8,020
  • 1
  • 27
  • 47
6

this the code of Ionuț G. Stan

i modified your code a little and made it a function for reuseability

function read_last_line ($file_path){



$line = '';

$f = fopen($file_path, 'r');
$cursor = -1;

fseek($f, $cursor, SEEK_END);
$char = fgetc($f);

/**
* Trim trailing newline chars of the file
*/
while ($char === "\n" || $char === "\r") {
    fseek($f, $cursor--, SEEK_END);
    $char = fgetc($f);
}

/**
* Read until the start of file or first newline char
*/
while ($char !== false && $char !== "\n" && $char !== "\r") {
    /**
     * Prepend the new char
     */
    $line = $char . $line;
    fseek($f, $cursor--, SEEK_END);
    $char = fgetc($f);
}

return $line;
}

echo read_last_line('log.txt');

you will get that last line

Abdalla Mohamed Aly Ibrahim
  • 3,743
  • 1
  • 26
  • 29
4

If you know the upper bound of line length you could do something like this.

$maxLength = 1024;
$fp = fopen('somefile.txt', 'r');
fseek($fp, -$maxLength , SEEK_END); 
$fewLines = explode("\n", fgets($fp, $maxLength));
$lastLine = $fewLines[count($fewLines) - 1];

In response to the edit: fopen just acquires a handle to the file (i.e. make sure it exists, process has permission, lets os know a process is using the file, etc...). In this example only 1024 characters from the file will be read into memory.

Lawrence Barsanti
  • 31,929
  • 10
  • 46
  • 68
4
function readlastline() 
{ 
       $fp = @fopen("/dosmnt/LOGFILE.DAT", "r"); 
       $pos = -1; 
       $t = " "; 
       while ($t != "\n") { 
             fseek($fp, $pos, SEEK_END); 
             $t = fgetc($fp); 
             $pos = $pos - 1; 
       } 
       $t = fgets($fp); 
       fclose($fp); 
       return $t; 
}

Source: http://forums.devshed.com/php-development-5/php-quick-way-to-read-last-line-156010.html

Daniel A. White
  • 187,200
  • 47
  • 362
  • 445
  • If the file is being written to constantly, you may want to store the current length of the file at the start before seeking backwards from that position. That way you're guaranteed to find a complete line even if more lines are added while you're seeking. If performance is a consideration, you can grab chunks instead of seeking byte-by-byte. – David Harkness Mar 24 '11 at 01:11
  • How about a do...while so that you don't need to initialize $t = " "; That's a tiny bit sloppy. ;) – OCDev Dec 26 '14 at 17:50
2

Your problem looks similar to this one

The best approach to avoid loading the whole file into memory seems to be:

$file = escapeshellarg($file); // for the security concious (should be everyone!)
$line = `tail -n 1 $file`;
Community
  • 1
  • 1
James Goodwin
  • 7,360
  • 5
  • 29
  • 41
0

Would it be possible to optimize this from the other side? If so, just let the logging application always log the line to a file while truncating it (i.e. > instead of >>)

Some optimization might be achieved by "guessing" though, just open the file and with the average log line width you could guess where the last line would be. Jump to that position with fseek and find the last line.

Wolph
  • 78,177
  • 11
  • 137
  • 148
  • I don't have control over the application, unfortunately. And the second problem is that the log file must exist for each and every operation, so to log only one line it'd be a messy problem. – Bogdan Constantinescu Oct 02 '09 at 15:16
0

This is my solution with only one loop

        $line = '';
        $f = fopen($file_path, 'r');
        $cursor = 0 ;
        do  {
            fseek($f, $cursor--, SEEK_END);
            $char = fgetc($f);
            $line = $char.$line;
        } while (
                $cursor > -1 || (
                 ord($char) !== 10 &&
                 ord($char) !== 13
                )
        );
caiofior
  • 429
  • 4
  • 17
0

Here is a compilation of the answers here wrapped into a function which can specify how many lines should be returned.

function getLastLines($path, $totalLines) {
  $lines = array();

  $fp = fopen($path, 'r');
  fseek($fp, -1, SEEK_END);
  $pos = ftell($fp);
  $lastLine = "";

  // Loop backword until we have our lines or we reach the start
  while($pos > 0 && count($lines) < $totalLines) {

    $C = fgetc($fp);
    if($C == "\n") {
      // skip empty lines
      if(trim($lastLine) != "") {
        $lines[] = $lastLine;
      }
      $lastLine = '';
    } else {
      $lastLine = $C.$lastLine;
    }
    fseek($fp, $pos--);
  }

  $lines = array_reverse($lines);

  return $lines;
}
DynamicDan
  • 425
  • 4
  • 12
0

Untested code from the comments of http://php.net/manual/en/function.fseek.php

jim at lfchosting dot com 05-Nov-2003 02:03
Here is a function that returns the last line of a file.  This should be quicker than reading the whole file till you get to the last line.  If you want to speed it up a bit, you can set the $pos = some number that is just greater than the line length.  The files I was dealing with were various lengths, so this worked for me. 

<?php 
function readlastline($file) 
{ 
        $fp = @fopen($file, "r"); 
        $pos = -1; 
        $t = " "; 
        while ($t != "\n") { 
              fseek($fp, $pos, SEEK_END); 
              $t = fgetc($fp); 
              $pos = $pos - 1; 
        } 
        $t = fgets($fp); 
        fclose($fp); 
        return $t; 
} 
?>
Syntax Error
  • 4,475
  • 2
  • 22
  • 33
0

This function will let you read last line or (optionally) entire file line-by-line from end, by passing $initial_pos which will tell from where to start reading a file from end (negative integer).

function file_read_last_line($file, $initial_pos = -1) {
  $fp = is_string($file) ? fopen($file, 'r') : $file;
  $pos = $initial_pos;
  $line = '';
  do {
    fseek($fp, $pos, SEEK_END);
    $char = fgetc($fp);
    if ($char === false) {
      if ($pos === $initial_pos) return false;
      break;
    }
    $pos = $pos - 1;
    if ($char === "\r" || $char === "\n") continue;
    $line = $char . $line;
  } while ($char !== "\n");
  if (is_string($file)) fclose($file);
  return $line;
}

Then, to read last line:

$last_line = file_read_last_line('/path/to/file');

To read entire file line-by-line from end:

$fp = fopen('/path/to/file', 'r');
$pos = -1;
while (($line = file_read_last_line($fp, $pos)) !== false) {
  $pos += -(strlen($line) + 1);
  echo 'LINE: ' . $line . "\n";
}
fclose($fp);
artnikpro
  • 5,487
  • 4
  • 38
  • 40
  • Looks a lot like this earlier posted answer to the same question https://stackoverflow.com/a/37748401/2943403 except that answer is cleaner because it writes all of the loop conditions in the `while()` versus your snippet that has `break` and `continue` littering the body of the loop. I haven't actually tested either answer for correctness -- just an observation. – mickmackusa Jun 07 '20 at 07:46
0

Traverse backward a file chunk by chunk, stops after find a new line, and returns everything after the last "\n".

function get_last_line($file) {
    if (!is_file($file)) return false;
    $fileSize   = filesize($file);  
    $bufferSize = 1024; // <------------------ Change buffer size here.
    $bufferSize = ($fileSize > $bufferSize) ? $bufferSize : $fileSize;

    $fp = fopen($file, 'r');
    
    $position = $fileSize - $bufferSize;
    $data = "";
    while (true) {
        fseek($fp, $position);
        $chunk  = fread($fp, $bufferSize);
        $data   = $chunk.$data;
        $pos    = strrchr($data, "\n");
        
        if ($pos !== false) return substr($pos, 1);
        if ($position <= 0) break;
        $position -= $bufferSize;
        if ($position <=0) $position = 0;
    }
    
    // whole file is 'the last line' :)
    return $data;
}

You can define the length of the chunk your self.

Smaller chunk = smaller memory usage, more iteration.

Bigger chunk = bigger memory usage, less iteration.

Donovan P
  • 591
  • 5
  • 9