Python reading text file every second line skipped

Question

I am processing a shell script in Python. My first step is to comb through the file and save only the important lines in a list (of strings). However, I have isolated a problem where every second line is ignored. Why is the second, fourth, etc. line skipped in the following code?

f = open("sample_failing_file.txt", encoding="ISO-8859-1")
readfile = f.read()
filelines = readfile.split("\n")

def remove_irrelevant_lines(filecontent: list[str]) -> list[str]:
    for line in filecontent:
        if drop_line_appropriate(line):
            filecontent.remove(line)
    return filecontent

def drop_line_appropriate(line: str) -> bool:
    if line.startswith("#"):
        return True
    # some more conditions, omitted here
    return False

filelines = remove_irrelevant_lines(filelines)
f.close()

When I run this code, I can see filecontent is complete. However, when I look at line, I can see e.g. some line 3 is never read. Here is a simplified version of the shell script, on which my Python script fails (sample_failing_file.txt)

#!/bin/sh
#
# some line 1
#
# some line 2
# some line 3

You should not [remove while iterating](https://stackoverflow.com/questions/1207406). Use a list comprehension instead: `return [line for line in filecontent if not drop_line_appropriate(line)]` — Jorge Luis, Mar 07 '23 at 12:27

score 0 · Accepted Answer · answered Mar 07 '23 at 12:35

As was pointed out in the comments, you shouldn't try to remove elements from a list while iterating over it. Additionally, when removing lines, don't want to use list.remove(), since that causes it to search for the line, which will make it run vastly slower than it should.

The following should fix your problem and also run vastly faster:

def remove_irrelevant_lines(filecontent: list[str]) -> list[str]:
    return [line for line in filecontent if not drop_line_appropriate(line)]

This creates and returns a new list, filtering out the lines indicated by drop_line_appropriate.

Python reading text file every second line skipped

1 Answers1