Pythonic way of processing a file between two previously known strings

Question

I process log files with python. Let´s say that I have a log file that contains a line which is START and a line that is END, like below:

START
one line
two line
...
n line
END

What I do want is to be able to store the content between the START and END lines for further processing.

I do the following in Python:

with open (file) as name_of_file:
    for line in name_of_file:
        if 'START' in line:  # We found the start_delimiter
            print(line)
            found_start = True
            for line in name_of_file:  # We now read until the end delimiter
                if 'END' in line:  # We exit here as we have the info
                    found_end=True
                    break
                else:

                    if not (line.isspace()): # We do not want to add to the data empty strings, so we ensure the line is not empty
                        data.append(line.replace(',','').strip().split())  # We store information in a list called data we do not want ','' or spaces
if(found_start and found_end):
    relevant_data=data

And then I process the relevant_data.

Looks to far complicated for the purity of Python, and hence my question: is there a more Pythonic way of doing this?

Thanks!

Mad Physicist · Answer 1 · 2019-01-22T22:00:35.667

You are right that there is something not OK with having a nested loop over the same iterator. File objects are already iterators, and you can use that to your advantage. For example, to find the first line with a START in it:

line = next(l for l in name_of_file if 'START' in l)

This will raise a StopIteration if there is no such line. It also sets the file pointer to the beginning of the first line you care about.

Getting the last line without anything that comes after it is a bit more complicated because it's difficult to set external state in a generator expression. Instead, you can make a simple generator:

def interesting_lines(file):
    if not next((line for line in file if 'START' in line), None):
        return
    for line in file:
        if 'END' in line:
            break
        line = line.strip()
        if not line:
            continue
        yield line.replace(',', '').split()

The generator will yield nothing if you don't have a START, but it will yield all the lines until the end if there is no END, so it differs a little from your implementation. You would use the generator to replace your loop entirely:

with open(name_of_file) as file:
    data = list(interesting_lines(file))

if data:
    ... # process data

Wrapping the generator in list immediately processes it, so the lines persist even after you close the file. The iterator can be used repeatedly because at the end of your call, the file pointer will be just past the END line:

with open(name_of_file) as file:
    for data in iter(lambda: list(interesting_lines(file)), []):
        # Process another data set.

The relatively lesser known form of iter converts any callable object that accepts no arguments into an iterator. The end is reached when the callable returns the sentinel value, in this case an empty list.

abcdaire · Accepted Answer · 2019-01-22T22:22:05.843

2

To perform that, you can use iter(callable, sentinel) discussed in this post , that will read until a sentinel value is reached, in your case 'END' (after applying .strip()).

with open(filename) as file:
    start_token = next(l for l in file if l.strip()=='START') # Used to read until the start token
    result = [line.replace(',', '').split() for line in iter(lambda x=file: next(x).strip(), 'END') if line]

edited Jan 22 '19 at 22:22

answered Jan 22 '19 at 22:03

abcdaire

1,528
1
10
21

1

The sentinel is an interesting idea. – Demi-Lune Jan 22 '19 at 23:45
1

This is a much nicer answer than mine. I wish it was accepted. – Mad Physicist Jan 23 '19 at 09:12

Demi-Lune · Answer 3 · 2019-01-22T22:02:46.367

0

This is a mission for regular expressions re, for example:

import re
lines = """ not this line
START
this line
this line too
END
not this one
"""
search_obj = re.search( r'START(.*)END', lines, re.S)
search_obj.groups(1)
# ('\n    this line\n    this line too\n    ',)

The re.S is necessary for spanning multiple lines.

edited Jan 22 '19 at 22:02

answered Jan 22 '19 at 21:49

Demi-Lune

1,868
2
15
26

Are you suggesting it's worth loading an enormous log file in its entirety to apply this regex? – Mad Physicist Jan 22 '19 at 21:50
I presumed you could. How big is the log typically? – Demi-Lune Jan 22 '19 at 21:52
Good point. Megabytes probably, but some of mine reach Gigabytes at times. – Mad Physicist Jan 22 '19 at 21:54
Yes, files are very big sometimes, and that is why I filter them by parts, just between the START and END lines that interest me at a certain point. I am giving a try to what Mad Physicist suggest, it is certainly an improvement over what I have and more Pythonic. – Gonzalo Murillo Jan 22 '19 at 22:03
For Mb, loading the file entirely then using re will be easiest (and faster probably). For Gb (assuming the matched portion is not Gb in size), your yield solution will save memory. – Demi-Lune Jan 22 '19 at 22:06

Pythonic way of processing a file between two previously known strings

3 Answers3

Linked