Sort Multiple line according to fields

Question

I have a record here with multiple lines, what I what to do is sort them according to type and the 6digits number in the line HEADER1.

Here is the record:

HEADER1|TYPE1|123456|JOHN SMITH
INFO|M|34|SINGLE
INFO|SGT
STATUS|KIA
MSG|NONE
HEADER1|TYPE3|654123|DANICA CLYNE
INFO|F|20|SINGLE
STATUS|MIA
MSG|HELP
MSG1||
HEADER1|TYPE2|987456|NIDALEE LANE
INFO|F|26|MARRIED
STATUS|INJURED
MSG|NONE
HEADER1|TYPE1|123456|JOHN CONNOR
INFO|M|34|SINGLE
STATUS|KIA
MSG|NONE
HEADER1|TYPE4|123789|CAITLYN MIST
INFO|F|19|SINGLE
INFO|||
STATUS|NONE
MSG|NONE
HEADER1|TYPE2|987456|NIDALEE CROSS
INFO|F|26|MARRIED
STATUS|INJURED
MSG|NONE

The Output should be like this: it sorted the line that matched to the rule

HEADER1|TYPE1|123456|JOHN SMITH
INFO|M|34|SINGLE
INFO|SGT
STATUS|KIA
MSG|NONE
HEADER1|TYPE1|123456|JOHN CONNOR
INFO|M|34|SINGLE
STATUS|KIA
MSG|NONE
HEADER1|TYPE2|987456|NIDALEE LANE
INFO|F|26|MARRIED
STATUS|INJURED
MSG|NONE
HEADER1|TYPE2|987456|NIDALEE CROSS
INFO|F|26|MARRIED
STATUS|INJURED
MSG|NONE
HEADER1|TYPE3|654123|DANICA CLYNE
INFO|F|20|SINGLE
STATUS|MIA
MSG|HELP
MSG1||
HEADER1|TYPE4|123789|CAITLYN MIST
INFO|F|19|SINGLE
INFO|||
STATUS|NONE
MSG|NONE

Do the numbers correspond to the type? ex. Type 1 always has the numbers 123456, or do you want them sorted by type and then by number? — hmatt1, Jul 10 '13 at 01:52

DVK · Answer 1 · 2013-07-10T02:53:41.337

If you don't care about performance, and every "record" consists of 4 lines:

# Assume STDIN since the question didn't say anything
my $line_index = 0;
my (@records, @record);
# Slurp in all records into array of quadruplets
while (<>) {
    if (0 == $line_index) {
        push @records, [];
    };
    $records[-1]->[$line_index] = $_; # -1 lets you access last element of array.
    $line_index++;
    $line_index = 0 if $line_index == 4; # better done via "%" 
}

# Sort the array. Since we sort by type+id, 
# we can simply sort the first strings alphabetically.
my @records_sorted = sort { $a->[0] cmp $b->[0] } @records;

foreach my $record (@records_sorted) {
    print join("", @$record); # Newlines never stripped, no need to append
}

If you're more adventurous, use List::MoreUtils::natatime:

use List::MoreUtils q/natatime/;
my @lines = File::Slurp::read_file("my_file.txt");
my $it = natatime 4, @lines;
my (@records, @record);
while ((@record) = $it->()) {
    push @records, \@record;
}
my @records_sorted = sort { $a->[0] cmp $b->[0] } @records;
foreach my $record (@records_sorted) {
    print join("", @$record);
}

Another option for creating @records from @lines is List::Gen:

use List::Gen qw/by/; 
foreach my $record (by 4 => @lines) {
    push @records, $record;
}

Please note that the above code assumes that all the #s are 6-digit. If that's not the case, you need to modify the code a bit:

use List::Gen qw/by/;
my @lines = File::Slurp::read_file("my_file.txt");
my @records;
foreach my $record (by 4 => @lines) {
    my @sort_by = split(m#/#, $record->[0]);
    push @records, [ $record, \@sort_by ];
}
my @records_sorted = sort { 
                             $a->[1]->[1] cmp $b->[1]->[1] 
                          || $a->[1]->[2] <=> $b->[1]->[1]
                     } @records;
foreach my $record (@records_sorted) {
    print join("", @{$record->[0]}); 
}

UPDATE: Since the OP decided that input file may have ANY # of lines per record, here's the updated code:

my (@records, @record);
# Slurp in all records into array of quadruplets
while (<>) {
    if (/HEADER1/) {
        my @sort_by = split(m#/#);            
        push @records, [[], \@sort_by];
    };
    push @{ $records[-1]->[0] }, $_;
}
my @records_sorted = sort { 
                             $a->[1]->[1] cmp $b->[1]->[1] 
                          || $a->[1]->[2] <=> $b->[1]->[1]
                     } @records;
foreach my $record (@records_sorted) {
    print join("", @{$record->[0]}); 
}

Please note - this is NOT the most elegant or idiomatic possible Perl code. I tried to make it more newbie friendly instead. — DVK, Jul 10 '13 at 02:29
Thanks DVK, it doesnt have to be always 4 lines, what should I adjust for that? — Soncire, Jul 10 '13 at 02:40
@Soncire - depends on what your file looks like. I answered based on your question. If your file format is different, better post as a new question since it's not quite fair to me to change the input file after I posted a comprehensive answer :) — DVK, Jul 10 '13 at 02:45

hmatt1 · Accepted Answer · 2013-07-10T03:13:13.273

2

Here's my solution.

#!/bin/perl

use warnings;
use strict;

# Read in the file
open(my $fh, '<', "./record.txt") or DIE $!;
my @lines = <$fh>;
my @records;

# populate @records with each element having 4 lines
for ( my $index = 0; $index < scalar @lines; $index+=4 ) {
    push @records, join("", ($lines[$index], $lines[$index+1], $lines[$index+2], $lines[$index+3]));
}

# sort by type and then by numbers
@records =  map { $_->[0] }
            sort { $a->[1] cmp $b->[1] || $a->[2] cmp $b->[2] }
            map { [ $_ , (split('\|', $_))[1], (split('\|', $_))[2] ] }
            @records;

print "@records";

Here's an updated version, same idea:

#!/bin/perl

use warnings;
use strict;


open(my $fh, '<', "./record.txt") or DIE $!;
my @lines = <$fh>;
my $temp = join ("", @lines);
my @records = split("HEADER1", "$temp");
my @new_records;

for my $rec (@records){
    push @new_records, "HEADER1" . $rec;
}
shift @new_records;



@records =  map { $_->[0] }
            sort { $a->[1] cmp $b->[1] || $a->[2] cmp $b->[2] }
            map { [ $_ , (split('\|', $_))[1], (split('\|', $_))[2] ] }
            @new_records;



print "@records";

edited Jul 10 '13 at 03:13

answered Jul 10 '13 at 02:34

hmatt1

4,939
3
30
51

what if the INFO is two lines or the MSG? like this HEADER1|TYPE1|123456|JOHN SMITH INFO|M|34|SINGLE INFO|UNEMPLOTED STATUS|KIA MSG|NONE MSG1|NONE – Soncire Jul 10 '13 at 02:38
You would just have to change how you read it in (read in every 2 lines instead of 4 in the loop), but the sorting will still be the same. Did you want to sort by the MSG too? – hmatt1 Jul 10 '13 at 02:42
no matt just by type & numbers, I just want to let you know that the record doesnt always compose of 4 lines – Soncire Jul 10 '13 at 02:45
Hey Matt your updated solution works :) I hope, I'll be able to understand how each line works! – Soncire Jul 10 '13 at 05:02
1

Warning - for a large file this may get excruciatingly slow :) – DVK Jul 10 '13 at 13:59

score 1 · Answer 3 · answered Jul 10 '13 at 05:31

Using List::MoreUtils 'apply' and setting the input_record_separator to 'HEADER', the code could be like below.

#!/usr/bin/perl
use strict;
use warnings;
use List::MoreUtils qw/ apply /;

my $fname =  'dup_data.txt';

open (my $input_fh, '<', $fname) or die "Unable to read '$fname' because $!";
open (my $OUTPUTA, ">", $fname .".reformat")
    or die "$0: could not write to '$fname.reformat'. $!";

{
    local $/ = "HEADER";

    print $OUTPUTA map{ "HEADER$_->[0]"}
                  sort {$a->[1] <=> $b->[1] || $a->[2] <=> $b->[2]}
                   map {[$_, /TYPE(\d+)\|(\d+)/]}
                  grep $_, apply {chomp} <$input_fh>;
}
close $input_fh or die $!;
close $OUTPUTA or die $!;

Sort Multiple line according to fields

3 Answers3