Extract and filter a range of lines from the input using Perl

Question

I'm quite new to Perl and I have some problems in skipping lines using a foreach loop. I want to copy some lines of a text file to a new one.

When the first words of a line are FIRST ITERATION, skip two more lines and print everything following until the end of the file or an empty line is encountered.

I've tried to find out a similar post but nobody talks about working with text files.

This is the form I thought of

use 5.010;
use strict;
use warnings;

open( INPUT, "xxx.txt" ) or die("Could not open log file.");
open( OUT, ">>yyy.txt" );

foreach my $line (<INPUT>) {

    if ( $line =~ m/^FIRST ITERATION/ ) {

        # print OUT
    }
}

close(OUT);
close(INFO);

I tried using next and $line++ but my program prints only the line that begins with FIRST ITERATION.

I may try to use a for loop but I don't know how many lines my file may have, nor do I know how many lines there are between "First Iteration" and the next empty line.

Answer 1

The simplest way is to process the file a line at a time and keep a state flag which is set to 1 if the current line is begins with FIRST ITERATION and 0 if it is blank, otherwise it is incremented if it is already positive so that it provides a count of the line number within the current block

This solution expects the path to the input file as a parameter on the command line and prints its output to STDOUT, so you will need to redirect the output to the file on the command line as necessary

Note that the regex pattern /\S/ checks whether there is a non-blank character anywhere in the current line, so not /\S/ is true if the line is empty or all blank characters

use strict;
use warnings;

my $lines = 0;

while ( <> ) {

    if ( /^FIRST ITERATION/ ) {
        $lines = 1;
    }
    elsif ( not /\S/ ) {
        $lines = 0;
    }
    elsif ( $lines > 0 ) {
        ++$lines;
    }

    print if $lines > 3;
}

This can be simplified substantially by using Perl's built-in range operator, which keeps its own internal state and returns the number of times it has been evaluated. So the above may be written

use strict;
use warnings;

while ( <> ) {
    my $s = /^FIRST ITERATION/ ... not /\S/;
    print if $s and $s > 3;
}

And the last can be rewritten as a one-line command line program like this

$ perl -ne '$s = /^FIRST ITERATION/ ... not /\S/; print if $s and $s > 3' myfile.txt

Answer 2

Use additional counter, that will say on which condition print line. Something like this:

$skipCounter = 3;

And in foreach:

if ($skipCounter == 2) {
    // print OUT
}
if ( $line =~ m/^FIRST ITERATION/) {
    $skipCounter = 0;
}

$skipCounter++;

Answer 3

Advice: Use STDIN and STDOUT instead of files, this will allowes you to change them without modifying script

Code:

#!/usr/bin/perl
use 5.010;
use strict;
use warnings;


open(INPUT, "xxx.txt" ) or die "Could not open log file: $!.";
open(OUT, ">yyy.txt") or die "Could not open output file: $!";
while( my $line = <INPUT> )
{
    if ( $line =~ m/^FIRST ITERATION/) {
        <INPUT>; # skip line
        <INPUT>; # skip line
        while( $line = <INPUT>) # print till empty line
        {
            last if $line eq "\n";
            print OUT $line;
        }
    };
};
close (OUT);
close (INPUT);

Answer 4

You're on the right track. What you need to use is the flip-flop operator (which is basically the range operator) ... It will toggle for you between two matches, so you get everything in between. After that, it's a matter of keeping track of the lines you want to skip.

So basically we are checking for FIRST ITERATION and for an empty line, and grab everything in between those. $skip is used to remember how many lines were skipped. It starts at 0 and gets incremented for the first two lines after we start being in the flip-flop if block. In the else case, where we are after the flip-flop, it gets reset to 0 so we can start over with the next block.

Since you know how to open and write files, I'll skip that.

use strict;
use warnings;

my $skip = 0;
while (<DATA>) {
    if (/^FIRST ITERATION$/ .. /^$/) {
        next if $skip++ <= 2;
        print $_;
    } else {
        $skip = 0;
    }
}
__DATA__
FIRST ITERATION
skip1
skip2
foo
bar
baz

don't print this

The output of this is:

foo
bar
baz

---

To stick with your own code, here's a very verbose solution that uses a foreach and no flip-flop. It does the same thing, just with a lot more words.

my $skip = 0;   # skip lines
my $match = 0;  # keep track of if we're in between the borders
foreach my $line (<DATA>) {
    if ( $line =~ m/^FIRST ITERATION/ ) {
        $match = 1; # we are inside the match
        next;
    }
    if ($line =~ m/^$/) {
        $match = 0; # we are done matching
        next;
    }
    if ($match) {
        $skip++;     # count skip-lines
        if ($skip <= 2) {
            next;    # ... and skip the first two
        }
        print $line; # this is the content we want  
    }
}

Answer 5

Using paragraph mode (which returns blocks separated by blank lines rather than lines):

local $/ = "";  # Paragraph mode.

while (<>) {
    s/\n\K\n+//;  # Get rid of trailing empty lines.
    print /^FIRST ITERATION\n[^\n]*\n[^\n]*\n(.*)/ms;
}

Answer 6

Using the flip-flop operator:

while (<>) {
    if (my $line_num = /^FIRST ITERATION$/ .. /^$/) {
        print if $line_num > 3 && $line_num !~ /E0/;
    }
}

$line_num !~ /E0/ is true when the flip-flop is flopping (i.e. for the first empty line after FIRST ITERATION). This is checked to avoid printing the blank line.