Extracting data from a text file and writing it to csv or flat file

Question

I'm doing a project that involves creating a rdbms of US federal code in a certain format. I've obtained the whole code form official source which is not structured well. I have managed to scrape the US Code in the below format into text files using some code on GITHUB.

Can this be done using a Python script to write this to some csv or flat file in the below format?

I'm new to Python but I'm told that this can easily be done using Python.

End output would be a flat file or a csv file with the below schema:

Example:

**Title | Text | Chapter | text | Section | Text | Section text**


1     |  GENERAL PROVISIONS  |  1 | RULES OF CONSTRUCTION | 2 | "County" as including "parish", and so forth | The word "county" includes a parish, or any other equivalent subdivision of a State or Territory of the United States.

Input would be a text file with data that looks like below.

Sample data:

-CITE-
    1 USC Sec. 2                                                01/15/2013

-EXPCITE-
    TITLE 1 - GENERAL PROVISIONS
    CHAPTER 1 - RULES OF CONSTRUCTION

-HEAD-
    Sec. 2. "County" as including "parish", and so forth

-STATUTE-
      The word "county" includes a parish, or any other equivalent
    subdivision of a State or Territory of the United States.

-SOURCE-
    (July 30, 1947, ch. 388, 61 Stat. 633.)

-End-



-CITE-
    1 USC Sec. 3                                                01/15/2013

-EXPCITE-
    TITLE 1 - GENERAL PROVISIONS
    CHAPTER 1 - RULES OF CONSTRUCTION

-HEAD-
    Sec. 3. "Vessel" as including all means of water transportation

-STATUTE-
      The word "vessel" includes every description of watercraft or
    other artificial contrivance used, or capable of being used, as a
    means of transportation on water.

-SOURCE-
    (July 30, 1947, ch. 388, 61 Stat. 633.)

-End-

Answer 1

If you wanted to use a robust parser like pyparsing rather than regexes, the following should work for you:

import csv, re
from pyparsing import Empty, FollowedBy, Group, LineEnd, Literal, \
                      OneOrMore, Optional, Regex, SkipTo, Word
from pyparsing import alphanums, alphas, nums

def section(header, other):
    return Literal('-'+header+'-').suppress() + other

def tc(header, next_item):
    # <header> <number> - <name>
    begin = Literal(header).suppress()
    number = Word(nums)\
             .setResultsName('number')\
             .setParseAction(compress_whitespace)
    dash = Literal('-').suppress()
    name = SkipTo(Literal(next_item))\
           .setResultsName('name')\
           .setParseAction(compress_whitespace)
    return begin + number + dash + name

def compress_whitespace(s, loc, toks):
    return [re.sub(r'\s+', ' ', tok).strip() for tok in toks]

def parse(data):
    # should match anything that looks like a header
    header = Regex(re.compile(r'-[A-Z0-9]+-'))

    # -CITE- (ignore)
    citation = SkipTo('-EXPCITE-').suppress()
    cite_section = section('CITE', citation)

        # -EXPCITE- (parse)
    # grab title number, title name, chapter number, chapter name
    title = Group(tc('TITLE', 'CHAPTER'))\
            .setResultsName('title')
    chapter = Group(tc('CHAPTER', '-HEAD-'))\
              .setResultsName('chapter')
    expcite_section = section('EXPCITE', title + chapter)

    # -HEAD- (parse)
    # two possible forms of section number:
    # > Sec. 1. <head_text>
    # > CHAPTER 1 - <head_text>
    sec_number1 = Literal("Sec.").suppress() \
                  + Regex(r'\d+\w?.')\
                    .setResultsName('section')\
                    .setParseAction(lambda s, loc, toks: toks[0][:-1])
    sec_number2 = Literal("CHAPTER").suppress() \
                  + Word(nums)\
                    .setResultsName('section') \
                  + Literal("-")
    sec_number = sec_number1 | sec_number2
    head_text = SkipTo(header)\
                .setResultsName('head')\
                .setParseAction(compress_whitespace)
    head = sec_number + head_text
    head_section = section('HEAD', head)

    # -STATUTE- (parse)
    statute = SkipTo(header)\
              .setResultsName('statute')\
              .setParseAction(compress_whitespace)
    statute_section = section('STATUTE', statute)

    # -End- (ignore)
    end_section = SkipTo('-End-', include=True)

    # do parsing
    parser = OneOrMore(Group(cite_section \
                             + expcite_section \
                             + head_section \
                             + Optional(statute_section) \
                             + end_section))
    result = parser.parseString(data)

    return result

def write_to_csv(parsed_data, filename):
    with open(filename, 'w') as f:
        writer = csv.writer(f, lineterminator='\n')
        for item in parsed_data:
            if 'statute' not in item:
                continue
            row = [item['title']['number'],
                   item['title']['name'],
                   item['chapter']['number'],
                   item['chapter']['name'],
                   item['section'],
                   item['head'],
                   item['statute']]
            writer.writerow(row)



# your data is assumed to be in <source.txt>
with open('source.txt', 'r') as f:
    data = f.read()
result = parse(data)
write_to_csv(result, 'output.txt')

Output: see http://pastie.org/8654063.

This is certainly more verbose than using regexes, but it's also more maintainable and extensible in my opinion. (Granted, this comes with the overhead of learning how to do basic manipulations in pyparsing, which isn't necessarily trivial.)

---

In response to your request - I have updated the parser to accomodate all the text that appears in the file you linked me. It should now be more robust against unusual line breaks / punctuation.

As you requested, the citations that have an enumeration of sections (and lack a -STATUTE- section) are no longer included in the output.

Answer 2

1.loop over files lines

with open('workfile', 'r') as f:
    for line in f:
        ...

2.use python re to match one of ['CITE', 'EXPCITE', 'HEAD'...]

3.based on the line matched at 2, also use python re to match line content, consider having those matchers ready in some dict

d = {'EXPCITE': re.compile(pattern)}
# and then latter
m = d['EXPCITE'].match(string)
# get the relevant group, for exmaple
print m.group(0)

4.write to csv output file

with open('out.csv', 'w') as csvfile:
    writer = csv.writer(csvfile, delimiter='|')
    writer.writerow([...])

also, consider implementing a state machine to switch between point 2 to 3 above, see Python state-machine design using this technique you can switch between looking for a tag as described in point 2 to matching the content of the tag as described in point 3

Good luck!