Split file by equal parts based on count

Question

I have file (node_list.txt) which is having list of nodes.

nod_1
nod_2
nod_3
nod_4
nod_5

I have list of host ip address(this count may vary) and need to devide the node_list into equal number of parts and send those splitted node files to each of the hosts. host_ip1 host_ip2 host_ip3

Dividing of nodes in the file are based on number of host_ip's available.

Here in my example I should get:

node_list_file_1.txt
nod_1
nod_2

node_list_file_2.txt
nod_3
nod_4

node_list_file_3.txt
nod_5

My code looks like this:

print Dumper(\@list_of_hosts);

my $node_file = "node_list.txt";
open(NODE_FILE, "< $node_file") or die "can't open $node_file: $!";
my $count;
$count += tr/\n/\n/ while sysread(NODE_FILE, $_, 2 ** 16);
print "COUNT:$count\n";

my $res = $count / scalar @list_of_ips;

In $res I am getting the count how much number of lines should go to each of the file. But how to put this into file.

Answer 1

This splits lines so that each file except the last receives the maximum equal number, whereby the last one gets the remainder. So with 10 lines to split over 3 files they'll go as 4-4-2.^†

use warnings;
use strict;
use feature 'say';
use autodie qw(open);

my @lines = <>;
my $num_files = 3;
my $lines_per_file = int @lines/$num_files;
$lines_per_file += 1  if @lines % $num_files;

my @chunks;
push @chunks, [ splice @lines, 0, $lines_per_file ] while @lines;

my @fhs_out = map { open my $fh, ">fout_$_.txt"; $fh } 1..$num_files;

for my $i (0..$#chunks) { 
    print {$fhs_out[$i]} $_ for @{$chunks[$i]};
};

Notes

• The <> reads all lines from files submitted at the command line

• If the number of files to write doesn't evenly divide the number of lines to split between them, we need one more line in each file (and the last one receives the remainder)

• The array with lines is successively splice-ed, in order to generate chunks of lines that will go into one file each, so it ends up emptied

• I open all needed output files and store filehandles into an array so to later conveniently write chunks of lines into their files. This is by no means necessary, as one can iterate over @chunks and open a file and write to it for each group ("chunk") of lines

• When writing to a filehandle that need be evaluated from an expression any more complex that just a basic scalar we must have that in a block, like { $fhs_out[$i] }. From print

  If you're storing handles in an array or hash, or in general whenever you're using any expression more complex than a bareword handle or a plain, unsubscripted scalar variable to retrieve it, you will have to use a block returning the filehandle value instead, [...]

  See this post for another way and more discussion.

---

^† If the distribution of lines must be 4-3-3 in this case, so split as evenly as possible, the code above need be modified like

my $lines_per_file = int @lines/$num_files;
my $extra = @lines % $num_files;

my @chunks;
push @chunks,
     [ splice @lines, 0, $lines_per_file + ( $extra-- > 0 ? 1 : 0 ) ] 
         while @lines;

The rest is the same.

Answer 2

my $num_buckets = 3;

my @lines = <>;

my $per_bucket = int( @lines / $num_buckets );
my $num_extras =      @lines % $num_buckets;

for my $bucket_num (0..$num_buckets-1) {
   my $num_lines = $per_bucket;
   if ($num_extras) {
      ++$num_lines;
      --$num_extras;
   }

   my $qfn = "node_list_file_${bucket_num}.txt";
   open(my $fh, '>', $qfn)
      or die("Can't create \"$qfn\": $!\n");

   $fh->print(splice(@lines, 0, $num_lines));
}

$per_bucket is the number of nodes per file.
$num_extras is how many files that have one extra node.

Note that the calculation of $num_lines can be condensed to the following (which I avoided for readability):

my $num_lines = $per_bucket + ( $num_extras-- > 0 );

---

The above loads the entire file into memory. The following is an alternative solution that doesn't:

my $num_buckets = 3;

my @fhs;
for my $bucket_num (1..$num_buckets) {
   my $qfn = "node_list_file_${bucket_num}.txt";
   open(my $fh, '>', $qfn)
      or die("Can't create \"$qfn\": $!\n");

   push @fhs, $fh;
}

$fhs[ ( $. - 1 ) % @fhs ]->print($_) while <>;

However, while it performs the requested task, the output isn't exactly as specified:

node_list_file_1.txt
--------------------
nod_1
nod_4

node_list_file_2.txt
--------------------
nod_2
nod_5

node_list_file_3.txt
--------------------
nod_3

Answer 3

Perhaps following code comply with your requirements

use strict;
use warnings;

use feature 'say';

use Data::Dumper;

my $debug = 1;                          # $debug = 1 -- debug mode

my $node_file = "node_list.txt";        # input filename

my @hosts = qw(host_ip1 host_ip2 host_ip3); # Hosts to distribute between

my $num_hosts = @hosts;                 # Number of hosts to distribute between

open(my $fh, "<", $node_file) 
        or die "can't open $node_file: $!";

my @nodes =  <$fh>;                     # read input lines into @nodes array

chomp @nodes;                           # trim newline from each element @nodes array

close $fh;

print Dumper(\@nodes) if $debug;        # print @nodes content in debug mode

my $count = @nodes;                     # count number nodes in @nodes array

print "COUNT: $count lines in the input file\n";

# How many lines store in out files
my $lines_in_file = int($count/$num_hosts + 0.5);

my $lines_out   = $lines_in_file;       # how many line to output per file
my $file_index  = 1;                    # index for output filenames
my $filename    = "node_list_file_${file_index}.txt";

# open OUT file
open(my $out, ">", $filename)
        or die "Couldn't open $filename";

foreach my $node_name (@nodes) {        # process each element of @nodes array
    say $out $node_name;                # store node in OUT file

    $lines_out--;                       # decrease number of left lines for output

    if( $lines_out == 0 ) {             # all lines per file stored
        close $out;                     # close file

        $lines_out = $lines_in_file;    # reinitialize number of lines for output

        $file_index++;                  # increase index for filename
        $filename = "node_list_file_${file_index}.txt";

        open($out, ">", $filename)      # open new OUT file
            or die "Couldn't open $filename";
    }
}

close $out;                             # close OUT file