How can I elegantly combine/concat files by section with python?

Question

Like many an unfortunate programmer soul before me, I am currently dealing with an archaic file format that refuses to die. I'm talking ~1970 format specification archaic. If it were solely up to me, we would throw out both the file format and any tool that ever knew how to handle it, and start from scratch. I can dream, but that unfortunately that won't resolve my issue.

The format: Pretty Loosely defined, as years of nonsensical revisions have destroyed almost all back compatibility it once had. Basically, the only constant is that there are section headings, with few rules about what comes before or after these lines. The headings are sequential (e.g. HEADING1, HEADING2, HEADING3,...), but not numbered and are not required (e.g HEADING1, HEADING3, HEADING7). Thankfully, all possible heading permutations are known. Here's a fake example:

# Bunch of comments

SHOES # First heading
# bunch text and numbers here

HATS # Second heading
# bunch of text here

SUNGLASSES # Third heading
...

My problem: I need to concatenate multiple of these files by these section headings. I have a perl script that does this quite nicely:

while(my $l=<>) {

    if($l=~/^SHOES/i) { $r=\$shoes; name($r);}
    elsif($l=~/^HATS/i) { $r=\$hats; name($r);}
    elsif($l=~/^SUNGLASSES/i) { $r=\$sung; name($r);}
    elsif($l=~/^DRESS/i || $l=~/^SKIRT/i ) { $r=\$dress; name($r);}
    ...
    ...
    elsif($l=~/^END/i) { $r=\$end; name($r);}
    else {
        $$r .= $l;
    }
    print STDERR "Finished processing $ARGV\n" if eof;
}

As you can see, with the perl script I basically just change where a reference points to when I get to a certain pattern match, and concatenate each line of the file to its respective string until I get to the next pattern match. These are then printed out later as one big concated file.

I would and could stick with perl, but my needs are becoming more complex every day and I would really like to see how this problem can be solved elegantly with python (can it?). As of right now my method in python is basically to load the entire file as a string, search for the heading locations, then split up the string based on the heading indices and concat the strings. This requires a lot of regex, if-statements and variables for something that seems so simple in another language.

It seems that this really boils down to a fundamental language issue. I found a very nice SO discussion about python's "call-by-object" style as compared with that of other languages that are call-by-reference. How do I pass a variable by reference? Yet, I still can't think of an elegant way to do this in python. If anyone can help kick my brain in the right direction, it would be greatly appreciated.

Answer 1

That's not even elegant Perl.

my @headers = qw( shoes hats sunglasses dress );

my $header_pat = join "|", map quotemeta, @headers;
my $header_re = qr/$header_pat/i;

my ( $section, %sections );
while (<>) {
    if    (/($header_re)/) { name( $section = \$sections{$1     } ); }
    elsif (/skirt/i)       { name( $section = \$sections{'dress'} ); }
    else { $$section .= $_; }

    print STDERR "Finished processing $ARGV\n" if eof;
}

Or if you have many exceptions:

my @headers = qw( shoes hats sunglasses dress );
my %aliases = ( 'skirt' => 'dress' );

my $header_pat = join "|", map quotemeta, @headers, keys(%aliases);
my $header_re = qr/$header_pat/i;

my ( $section, %sections );
while (<>) {
    if (/($header_re)/) {
       name( $section = \$sections{ $aliases{$1} // $1 } );
    } else {
       $$section .= $_;
    }

    print STDERR "Finished processing $ARGV\n" if eof;
}

Using a hash saves the countless my declarations you didn't show.

You could also do $header_name = $1; name(\$sections{$header_name}); and $sections{$header_name} .= $_ for a bit more readability.

Answer 2

I'm not sure if I understand your whole problem, but this seems to do everything you need:

import sys

headers = [None, 'SHOES', 'HATS', 'SUNGLASSES']
sections = [[] for header in headers]

for arg in sys.argv[1:]:
    section_index = 0
    with open(arg) as f:
        for line in f:
            if line.startswith(headers[section_index + 1]):
                section_index = section_index + 1
            else:
                sections[section_index].append(line)

Obviously you could change this to read or mmap the whole file, then re.search or just buf.find for the next header. Something like this (untested pseudocode):

import sys

headers = [None, 'SHOES', 'HATS', 'SUNGLASSES']
sections = defaultdict(list)

for arg in sys.argv[1:]:
    with open(arg) as f:
        buf = f.read()
    section = None
    start = 0
    for header in headers[1:]:
        idx = buf.find('\n'+header, start)
        if idx != -1:
            sections[section].append(buf[start:idx])
            section = header
            start = buf.find('\n', idx+1)
            if start == -1:
                break
    else:
        sections[section].append(buf[start:])

And there are plenty of other alternatives, too.

But the point is, I can't see anywhere where you'd need to pass a variable by reference in any of those solutions, so I'm not sure where you're stumbling on whichever one you've chosen.

---

So, what if you want to treat two different headings as the same section?

Easy: create a dict mapping headers to sections. For example, for the second version:

headers_to_sections = {None: None, 'SHOES': 'SHOES', 'HATS': 'HATS',
                       'DRESSES': 'DRESSES', 'SKIRTS': 'DRESSES'}

Now, in the code that doessections[section], just do sections[headers_to_sections[section]].

For the first, just make this a mapping from strings to indices instead of strings to strings, or replace sections with a dict. Or just flatten the two collections by using a collections.OrderedDict.

Answer 3

Assuming you're reading from stdin, as in the perl script, this should do it:

import sys
import collections
headings = {'SHOES':'SHOES','HATS':'HATS','DRESS':'DRESS','SKIRT':'DRESS'} # etc...
sections = collections.defaultdict(str)
key = None
for line in sys.stdin:
    sline = line.strip()
    if sline not in headings:
        sections[headings.get(key)].append(sline)
    else:
        key = sline

You'll end up with a dictionary where like this:

{
    None: <all lines as a single string before any heading>
    'HATS' : <all lines as a single string below HATS heading and before next heading> ],
    etc...
}

The headings list does not have to be defined in the some order as the headings appear in the input.

Answer 4

My deepest sympathies!

Here's some code (please excuse minor syntax errors)

  def foundSectionHeader(l, secHdrs):
    for s in secHdrs:
      if s in l:
        return True
    return False

  def main():
    fileList = ['file1.txt', 'file2.txt', ...]
    sectionHeaders = ['SHOES', 'HATS', ...]
    sectionContents = dict()
    for section in sectionHeaders:
      sectionContents[section] = []
    for file in fileList:
      fp = open(file)
      lines = fp.readlines()
      idx = 0
      while idx < len(lines):
        sec = foundSectionHeader(lines[idx]):
        if sec:
          idx += 1
          while not foundSectionHeader(lines[idx], sectionHeaders):
            sectionContents[sec].append(lines[idx])
            idx += 1

This assumes that you don't have content lines which look like "SHOES"/"HATS" etc.