php read large text file log

Question

I have a text log file, about 600 MB.

I want to read it using php and display the data on a html page, but I only need the last 18 lines that were added each time I run the script.

Since its a large file, I can't read it all in then flip the array as I would have hoped. Is their another way?

Can you execute external commands? Calling Linux's/Unix's `tail` through `exec()` might be the easiest way. — Pekka, Jan 27 '11 at 21:38
I have root, but ideally its something I want to roll out to my users too...would it be a security issue to use the unix way? — David19801, Jan 27 '11 at 21:41

Michael Irigoyen · Answer 1 · 2014-06-30T12:25:25.193

2

Loading that size file into memory would probably not be a good idea. This should get you around that.

$file = escapeshellarg($file);
$line = 'tail -n 18 '.$file;
system($line);

edited Jun 30 '14 at 12:25

answered Jan 27 '11 at 21:39

Michael Irigoyen

22,513
17
89
131

What permissions would I need to do this? – David19801 Jan 27 '11 at 21:42
normal permission should suffice. Maybe you need to chown the file. – Alfred Jan 28 '11 at 00:02

score 2 · Answer 2 · answered Jan 27 '11 at 21:41

2

Use fopen, filesize and fseek to open the file and start reading it only near the end of the file.

Comments on the fseek manual page include full code to read the last X lines of a large file.

answered Jan 27 '11 at 21:41

Dan Grossman

51,866
10
112
101

score 1 · Answer 3 · answered Jan 27 '11 at 21:43

1

you can stream it backwards with

$file = popen("tac $filename",'r');

while ($line = fgets($file)) {
  echo $line;
}

answered Jan 27 '11 at 21:43

score 0 · Answer 4 · edited May 11 '18 at 07:32

0

For the record, had the same problem and tried every solution here.

Turns out Dagon's popen "tac $filename" way is the fastest and the one with the lowest memory and CPU loads.

Tested with a 2Gb log-file reading 500, 1000 and 2000 lines each time. Smooth. Thank you.

edited May 11 '18 at 07:32

Muhammad Hassaan

7,296
6
30
50

answered Feb 26 '13 at 16:06

gomman

133
2
4
14

score 0 · Answer 5 · answered Jan 27 '11 at 21:42

0

The best way to do this is use fread and fgets to read line by line, this is extreamly fast as only one line is read at one time and not the while file:

Example of usage would be:

$handle = fopen("/logs/log.txt", "r")
if ($handle)
{
    fseek($handle,-18,SEEK_END); //Seek to the end minus 18 lines
    while (!feof($handle))
    {
        echo fgets($handle, 4096); //Make sure your line is less that 4096, otherwise update
        $line++;
    }
    fclose($handle);
}

answered Jan 27 '11 at 21:42

RobertPitt

56,863
21
114
161

yea I misread the line where he stated the last, I thought he wanted the first 18. – RobertPitt Jan 27 '11 at 21:47
this is high memory consuming ... Allowed memory size of 134217728 bytes exhausted – jacktrade Feb 28 '12 at 11:59

php read large text file log

5 Answers5

Linked