How to get last 1000 lines from a large file then remove them?

Question

Solved on 2 steps:

Get the lines: https://stackoverflow.com/a/51350572/8524395
Remove the lines after getting them: https://stackoverflow.com/a/51377052/8524395

I have a large file, I want to take 1000 lines from the end of this file, then remove them.

I am currently using this:

function deleteLineInFile($file,$string)
{
    $i=0;
    $array=array();

    $read = fopen($file, "r") or die("can't open the file");
    while(!feof($read)) {
        $array[$i] = fgets($read);  
        ++$i;
    }
    fclose($read);

    $write = fopen($file, "w") or die("can't open the file");
    foreach($array as $a) {
        if(!strstr($a,$string)) fwrite($write,$a);
    }
    fclose($write);
}
$goods = '';
$file = file("../products/".$PidFileName);
for ($i = max(0, count($file)-1001); $i < count($file); $i++) {
    $goods = $goods.$file[$i] . '<br />';
    deleteLineInFile("../products/".$PidFileName, $file[$i]);
}

I want to save the lines which I got in $goods

However, it times out because of the file size.

Please check this answer: https://stackoverflow.com/questions/6451232/reading-large-files-from-end. — Yousuf Khan, Jul 15 '18 at 16:50
Do you need to run it from the browser? You won't have a timeout from the command line. — user3783243, Jul 15 '18 at 16:51
I would recommend using a DB in that case. You're probably going to run out of memory with this approach, and deal with the filesystem locks. — user3783243, Jul 15 '18 at 16:53
@user3783243 - I want to take the purchased goods (1000 purchased good) from a file. What do you mean? :/ — Mario, Jul 15 '18 at 16:56
I've solved it temporarily by setting the max purchased goods to 400 - but no solutions for that? :/ — Mario, Jul 15 '18 at 17:10
Do you want to save the original input file after x lines have been removed? — wp78de, Jul 15 '18 at 18:51
Possible duplicate of [Tailing Log File and Write results to new file](https://stackoverflow.com/questions/16892270/tailing-log-file-and-write-results-to-new-file) — wp78de, Jul 15 '18 at 23:54
Relevant reading, I think: [What is the best way in PHP to read last lines from a file?](https://stackoverflow.com/q/15025875/2943403) — mickmackusa, Jul 16 '18 at 00:36

MrSmile · Accepted Answer · 2018-07-17T08:30:47.213

2

If you want to get N lines from EOF, you can use SPLFileObject (added in PHP 5.1):

$num_to_cut = 1000; // must be an integer and not a string
$new_file = new SplFileObject("limited_test.txt", "w");
$old_file = new SplFileObject('test.txt');

// here we get count of lines: go to EOF and get line number
$old_file->seek($old_file->getSize());
$linesTotal = $old_file->key()+1;

// and write data to new file
foreach( new LimitIterator($old_file, $linesTotal-$num_to_cut) as $line) {
    $new_file->fwrite($line);
}

edited Jul 17 '18 at 08:30

answered Jul 15 '18 at 17:29

MrSmile

1,217
2
12
20

Thank you very much! but is it possible to save into a database instead? – Mario Jul 17 '18 at 07:49
I think `$line` is what I should save into the database during the foreach loop, am I right? – Mario Jul 17 '18 at 07:51
@MatrixCow08, yes! If you want to save to DB immediately during foreach loop. Therefore, you even needn't open new file – MrSmile Jul 17 '18 at 08:10
But it still get 1467 lines and not the lines I entered in $num_to_cut ... I made sure that the number I entered is an integer and not a string but same. – Mario Jul 17 '18 at 08:13
@MatrixCow08, how many lines do you have already and how many do you want to cut? – MrSmile Jul 17 '18 at 08:15
2152 lines, I want to cut 1000 – Mario Jul 17 '18 at 08:15
@MatrixCow08, oh, maybe you want to get last 1000 lines? then it should be `new LimitIterator($old_file, $linesTotal-$num_to_cut)`, and you'll get last 1000 lines! – MrSmile Jul 17 '18 at 08:18
I was about to do that already. Thank you very much :) I am trying now. – Mario Jul 17 '18 at 08:19
It worked great now! But should I use `DeleteLineInFile` function to remove the lines or it may cause the timeout again? – Mario Jul 17 '18 at 08:23

Mario · Answer 2 · 2018-07-17T09:02:46.290

To remove the lines after getting them from a LARGE file:

The best way to do that is to use sed | But if you don't have access to use the exec() function then this is a function that you can use.

function replace_file($path, $string, $replace)
{
    set_time_limit(0);

    if (is_file($path) === true)
    {
        $file = fopen($path, 'r');
        $temp = tempnam('./', 'tmp');

        if (is_resource($file) === true)
        {
            while (feof($file) === false)
            {
                file_put_contents($temp, str_replace($string, $replace, fgets($file)), FILE_APPEND);
            }

            fclose($file);
        }

        unlink($path);
    }

    return rename($temp, $path);
}

Source of the function: https://stackoverflow.com/a/2159135/8524395

To remove the line use it like that:

replace_file('myfile.txt', 'RemoveThisPlease', '');

If you used MrSmile's answer to get the lines, then replace "RemoveThisPlease" with $line

How to get last 1000 lines from a large file then remove them?

Solved on 2 steps:

2 Answers2

To remove the lines after getting them from a LARGE file: