Splitting large text file on every blank line

Question

I'm having a bit trouble of splitting a large text file into multiple smaller ones. Syntax of my text file is the following:

dasdas #42319 blaablaa 50 50
content content
more content
content conclusion

asdasd #92012 blaablaa 30 70
content again
more of it
content conclusion

asdasd #299 yadayada 60 40
content
content
contend done
...and so on

A typical information table in my file has anywhere between 10-40 rows.

I would like this file to be split in n smaller files, where n is the amount of content tables.

That is

dasdas #42319 blaablaa 50 50
content content
more content
content conclusion

would be its own separate file, (whateverN.txt)

and

asdasd #92012 blaablaa 30 70
content again
more of it
content conclusion

again a separate file whateverN+1.txt and so forth.

It seems like awk or Perl are nifty tools for this, but having never used them before the syntax is kinda baffling.

I found these two questions that are almost correspondent to my problem, but failed to modify the syntax to fit my needs:

Split text file into multiple files & How can I split a text file into multiple text files? (on Unix & Linux)

How should one modify the command line inputs, so that it solves my problem?

Answer 1

Setting RS to null tells awk to use one or more blank lines as the record separator. Then you can simply use NR to set the name of the file corresponding to each new record:

 awk -v RS= '{print > ("whatever-" NR ".txt")}' file.txt

RS: This is awk's input record separator. Its default value is a string containing a single newline character, which means that an input record consists of a single line of text. It can also be the null string, in which case records are separated by runs of blank lines, or a regexp, in which case records are separated by matches of the regexp in the input text.

$ cat file.txt
dasdas #42319 blaablaa 50 50
content content
more content
content conclusion

asdasd #92012 blaablaa 30 70
content again
more of it
content conclusion

asdasd #299 yadayada 60 40
content
content
contend done

$ awk -v RS= '{print > ("whatever-" NR ".txt")}' file.txt

$ ls whatever-*.txt
whatever-1.txt  whatever-2.txt  whatever-3.txt

$ cat whatever-1.txt 
dasdas #42319 blaablaa 50 50
content content
more content
content conclusion

$ cat whatever-2.txt 
asdasd #92012 blaablaa 30 70
content again
more of it
content conclusion

$ cat whatever-3.txt 
asdasd #299 yadayada 60 40
content
content
contend done
$ 

Answer 2

You could use the csplit command:

csplit \
    --quiet \
    --prefix=whatever \
    --suffix-format=%02d.txt \
    --suppress-matched \
    infile.txt /^$/ {*}

POSIX csplit only uses short options and doesn't know --suffix and --suppress-matched, so this requires GNU csplit.

This is what the options do:

• --quiet – suppress output of file sizes
• --prefix=whatever – use whatever instead fo the default xx filename prefix
• --suffix-format=%02d.txt – append .txt to the default two digit suffix
• --suppress-matched – don't include the lines matching the pattern on which the input is split
• /^$/ {*} – split on pattern "empty line" (/^$/) as often as possible ({*})

Answer 3

Perl has a useful feature called the input record separator. $/.

This is the 'marker' for separating records when reading a file.

So:

#!/usr/bin/env perl
use strict;
use warnings;

local $/ = "\n\n"; 
my $count = 0; 

while ( my $chunk = <> ) {
    open ( my $output, '>', "filename_".$count++ ) or die $!;
    print {$output} $chunk;
    close ( $output ); 
}

Just like that. The <> is the 'magic' filehandle, in that it reads piped data or from files specified on command line (opens them and reads them). This is similar to how sed or grep work.

This can be reduced to a one liner:

perl -00 -pe 'open ( $out, '>', "filename_".++$n ); select $out;'  yourfilename_here

Answer 4

You can use this awk,

awk 'BEGIN{file="content"++i".txt"} !NF{file="content"++i".txt";next} {print > file}' yourfile

(OR)

awk 'BEGIN{i++} !NF{++i;next} {print > "filename"i".txt"}' yourfile

More readable format:

BEGIN {
        file="content"++i".txt"
}
!NF {
        file="content"++i".txt";
        next
}
{
        print > file
}

Answer 5

In case you get "too many open files" error as follows...

awk: whatever-18.txt makes too many open files
 input record number 18, file file.txt
 source line number 1

You may need to close newly created file, before creating a new one, as follows.

awk -v RS= '{close("whatever-" i ".txt"); i++}{print > ("whatever-" i ".txt")}' file.txt

Answer 6

Since it's Friday and I'm feeling a bit helpful... :)

Try this. If the file is as small as you imply it's simplest to just read it all at once and work in memory.

use strict;
use warnings;

# slurp file
local $/ = undef;
open my $fh, '<', 'test.txt' or die $!;
my $text = <$fh>;
close $fh;

# split on double new line
my @chunks = split(/\n\n/, $text);

# make new files from chunks
my $count = 1;
for my $chunk (@chunks) {
    open my $ofh, '>', "whatever$count.txt" or die $!;
    print $ofh $chunk, "\n";
    close $ofh;
    $count++;
}

The perl docs can explain any individual commands you don't understand but at this point you should probably look into a tutorial as well.

Answer 7

awk -v RS="\n\n" '{for (i=1;i<=NR;i++); print > i-1}' file.txt

Sets record separator as blank line, prints each record as a separate file numbered 1, 2, 3, etc. Last file (only) ends in blank line.

Answer 8

Try this bash script also

#!/bin/bash
i=1
fileName="OutputFile_$i"
while read line ; do 
if [ "$line"  == ""  ] ; then
 ((++i))
 fileName="OutputFile_$i"
else
 echo $line >> "$fileName"
fi
done < InputFile.txt

Answer 9

You can also try split -p "^$"