Starting reading file from last lines PHP

Question

I have no luck when the subject is reading text files. I have a small script to read a log file (real time updated) but I want to send some data to DB.

And the problem is, if I don't stat reading from the end of the files, I will get duplicated entries in database. Which can't happen!

// Keep alive
for (;;)
    {
    $handle = fopen("data.log", "r");
    if (!$handle) die("Open error - data.log");
    while (!feof($handle))
        {
        $line = fgets($handle, 4096);
        // If match with, I output the result
        if (strpos($line, ':gshop_trade:') > 0)
            {
            if (!preg_match('/([\d-: ]+)\s*.*\sformatlog:gshop_trade:userid=(\d+):(.*)item_id=(\d+):expire=(\d+):item_count=(\d+):cash_need=(\d+):cash_left=(\d+).*$/', $line, $data))
                {
                echo "Parsing error on line: {$line}";
                }
            // show the data
            }
        }
    sleep(5);
    }

This script is working, but as I mentioned above, I need to send the data to BD. But also, I need to leave script running, with this current code, the script match the wanted string, and instead of wait for new entries on data.log he starting reading the whole file again.

I see this question here and I tested but doesn't work. I'll start the script when I start the service that generates "data.log" but to prevent duplicate entries in database, I need to read the last lines.

How can I do that?

Maybe you can empty the file and every time just read the new content? — Aleksei Maide, Feb 08 '18 at 14:04
@Andreas Yes. Sorry for not answer. The problem is I need to follow a pattern, which looks more simple to me in that way (the only is to understand the `$strPos` and I dont want to read reverse, I want to start reading from the end and keep going (will print only new entries on console) — webmasterdro, Feb 08 '18 at 23:37
What do you mean by that? *I want to start reading from the end and keep going*. Keep going where? If you start at the end, there is nowhere to go. Let's say your file has three lines with values "abc,def,ghi" what line do you first read, then what line after that? — Andreas, Feb 09 '18 at 04:40
@Andreas Keep going I mean - waiting for new results. I'm reading a file that is real time updated every second, I want to save some data of this file in DB, but I can't read all the lines, I need to starting reading from the END of the file (the last line). All I'll have to do, is star the reader at the same moment of the services that generate the logs, so what happen before is alreayd in database and what happen next, will be add. Not dificult. — webmasterdro, Feb 09 '18 at 11:18
But you can't keep a PHP file running for ever (?). At some point the browser will time out. I suggest you keep a second file of what has been read and compare with the data log file and only read the part that is new. Do you want code for that? Could that solve your problem? — Andreas, Feb 09 '18 at 11:26
I don't understand the browser timeout. The infinity loop it's a bad idea in PHP (I heard that) so I'll go to 1 minute crontab, but eitherway, I'll post in database and I'll get the data from there in my site, not directly from the script. — webmasterdro, Feb 09 '18 at 12:24

Andreas · Answer 1 · 2018-02-09T11:39:36.173

Maybe you can use file_get_contents, explode and read the array backwards?

$arr = explode(PHP_EOL, file_get_contents("data.log")); // or file("data.log");
$arr = array_reverse($arr);

foreach($arr as $line){
    // do stuff here in reverse order
}

From comments above I suggest this method to only use the new data in your code.
It will read your log and a text file with what has been read last time.
Remove what was read last time and use the new data in the code.

$logfile = file_get_contents("data.log");
$ReadData = file_get_contents("readdata.txt");

$newdata = str_replace($ReadData, "", $logfile); // this is what is new since last run.
file_put_contents("readdata.txt", $logfile); // save what has been read.

$arr = explode(PHP_EOL, $newdata);

foreach($arr as $line){
    // do your stuff here with the new data.
}

?>
<!DOCTYPE html>
<html>
    <head>
        <meta http-equiv="refresh" content="5"> <!-- This will run the page every five seconds.
    </head>
</html>

`explode()` with `file_get_contents()` is the same as just `file()`. — Syscall, Feb 08 '18 at 14:09

score 1 · Accepted Answer · answered Feb 08 '18 at 14:09

1

Keep a track of the file offset from the previous reading using ftell() and keeping that result in a variable, and jump to that offset in the file when you re-open it for the next reading using fseek()

$lastPos = 0;
for (;;)
    {
    $handle = fopen("data.log", "r");
    if (!$handle) die("Open error - data.log");
    fseek($handle, $lastPos);    // <--- jump to last read position
    while (!feof($handle))
        {
        $line = fgets($handle, 4096);
        $lastPos = ftell($handle);    // <--- maintain last read position
        // If match with, I output the result
        if (strpos($line, ':gshop_trade:') > 0)
            {
            if (!preg_match('/([\d-: ]+)\s*.*\sformatlog:gshop_trade:userid=(\d+):(.*)item_id=(\d+):expire=(\d+):item_count=(\d+):cash_need=(\d+):cash_left=(\d+).*$/', $line, $data))
                {
                echo "Parsing error on line: {$line}";
                }
            // show the data
            }
        }
    sleep(5);
    }

answered Feb 08 '18 at 14:09

Mark Baker

209,507
32
346
385

Hello Mark. Thank you. It's almost there, the only problem is, seems that $lastPos needs to be the last line, because if for some reason I need to (re)start the script (again) will read all the lines (and will duplicate the entries in database). It's possible to make lastPos dynamic? – webmasterdro Feb 08 '18 at 14:16
Well you can always move `$lastPos = ftell($handle);` to ___after___ your data processing in case of failure processing the line.... and you can always save it to persistent storage (file, database, whatever) rather than maintain it purely within code – Mark Baker Feb 08 '18 at 15:09
I tried many ways to move $lastPos, but I failed. My weal point with PHP is reading text files, i never pratice this...=/ – webmasterdro Feb 08 '18 at 15:15
Mark, I tried again but or don't print anything on the console or print with the all lines, I can't figured out how to make, I added `FSEEK_END` on fseek, but still not printing. Do you have any idea? Reading file is my weak point with PHP. – webmasterdro Feb 09 '18 at 00:00
How do you mean make `$lastPos` dynamic? `fseek()` is what allows you to jump to a position in the file; `ftell()` is what tells you where in the file you currently are, and that can be saved to `$lastPos`. – Mark Baker Feb 09 '18 at 08:58
I mean, I need to start from the last line (the end of the file. I try to change "r" to "a" and added FSEEK_END, but for some reason, is not printing the data in console (which means that will not work on DB). – webmasterdro Feb 09 '18 at 11:19
You mean where I initialised `$lastPos = 0;` you want to set that to EOF; then initialise it using `$lastPos = filesize("data.log");` – Mark Baker Feb 09 '18 at 11:45
Awesome Mark. Working perfect now! Thank you so much for your help! – webmasterdro Feb 09 '18 at 12:26

Starting reading file from last lines PHP

2 Answers2