End-line characters from lines read from text file, using Python

Question

When reading lines from a text file using python, the end-line character often needs to be truncated before processing the text, as in the following example:

f = open("myFile.txt", "r")
for line in f:
    line = line[:-1]
    # do something with line

Is there an elegant way or idiom for retrieving text lines without the end-line character?

Does this answer your question? [How to read a file without newlines?](https://stackoverflow.com/questions/12330522/how-to-read-a-file-without-newlines) — mkrieger1, Jul 06 '21 at 14:45

score 53 · Accepted Answer · answered Dec 04 '08 at 08:04

53

The idiomatic way to do this in Python is to use rstrip('\n'):

for line in open('myfile.txt'):  # opened in text-mode; all EOLs are converted to '\n'
    line = line.rstrip('\n')
    process(line)

Each of the other alternatives has a gotcha:

file('...').read().splitlines() has to load the whole file in memory at once.
line = line[:-1] will fail if the last line has no EOL.

answered Dec 04 '08 at 08:04

efotinis

14,565
6
31
36

16

HTTP and other protocols specify '\r\n' for line endings, so you should use line.rstrip('\r\n') for robustness. – too much php Dec 10 '08 at 01:39
Thanks for your help! I needed to open a text file and I was amazed to see that the \n - thing is even in Python as it is in Perl, C and so many other languages. I'll bookmark this and never forget it. – bjd2385 Nov 01 '14 at 20:32

score 17 · Answer 2 · answered Dec 04 '08 at 04:13

17

Simple. Use splitlines()

L = open("myFile.txt", "r").read().splitlines();
for line in L: 
    process(line) # this 'line' will not have '\n' character at the end

answered Dec 04 '08 at 04:13

Vijay Dev

26,966
21
76
96

5

But do note this loads the entire file into memory first, which may render it unsuitable for some situations. – Matthew Trevor Dec 04 '08 at 07:26

score 6 · Answer 3 · answered Dec 04 '08 at 04:10

6

What's wrong with your code? I find it to be quite elegant and simple. The only problem is that if the file doesn't end in a newline, the last line returned won't have a '\n' as the last character, and therefore doing line = line[:-1] would incorrectly strip off the last character of the line.

The most elegant way to solve this problem would be to define a generator which took the lines of the file and removed the last character from each line only if that character is a newline:

def strip_trailing_newlines(file):
    for line in file:
        if line[-1] == '\n':
            yield line[:-1]
        else:
            yield line

f = open("myFile.txt", "r")
for line in strip_trailing_newlines(f):
    # do something with line

answered Dec 04 '08 at 04:10

Adam Rosenfield

390,455
97
512
589

3

Mac files using '\r', windows uses '\r\n', it starts to get chunky. Much better to use str.rstrip() – too much php Dec 04 '08 at 04:57
3

If the file is opened in text mode, the platform's native line endings are automatically converted to a single '\n' as they are read in. And only _really old_ Mac OSs use plain '\r'. You can't use rstrip() if you want to retain trailing spaces and tabs. – Adam Rosenfield Dec 04 '08 at 06:53
Good idea, with the generator. Would be handy in a reusable library. I would combine your solution with efonitis' solution (to save the if:else:). Without the reusable library at hand, I would prefer efotinis' solution (using line.rstrip('\n')). – pythonquick Dec 06 '08 at 16:12
+1; that's what I use. Could you please replace your `if`/`else` with `rstrip('\n')`? – max Apr 15 '12 at 02:21

score 5 · Answer 4 · answered Jul 13 '09 at 09:39

Long time ago, there was Dear, clean, old, BASIC code that could run on 16 kb core machines: like that:

if (not open(1,"file.txt")) error "Could not open 'file.txt' for reading"
while(not eof(1)) 
  line input #1 a$
  print a$
wend
close

Now, to read a file line by line, with far better hardware and software (Python), we must reinvent the wheel:

def line_input (file):
    for line in file:
        if line[-1] == '\n':
            yield line[:-1]
        else:
            yield line

f = open("myFile.txt", "r")
for line_input(f):
    # do something with line

I am induced to think that something has gone the wrong way somewhere...

While, considering that python is our best option for a entry level interpreted language, I agree on this comment, it could be convenient to notice that 16kb BASIC with a WHILE sentence were never common. — arivero, Oct 17 '12 at 16:43

score 3 · Answer 5 · answered Aug 08 '11 at 07:29

What do you thing about this approach?

with open(filename) as data:
    datalines = (line.rstrip('\r\n') for line in data)
    for line in datalines:
        ...do something awesome...

Generator expression avoids loading whole file into memory and with ensures closing the file

score 2 · Answer 6 · answered Dec 04 '08 at 04:18

2

You may also consider using line.rstrip() to remove the whitespaces at the end of your line.

answered Dec 04 '08 at 04:18

Martin Cote

28,864
15
75
99

I use rstrip() as well, but you have to keep in mind it also takes out trailing spaces and tabs – too much php Dec 04 '08 at 05:02
As efotinis has shown, if you specify the chars argument, you can specfy what to strip. From the documentation: """rstrip([chars]) The chars argument is a string specifying the set of characters to be removed. If omitted or None, the chars argument defaults to removing whitespace.""" – monkut Dec 05 '08 at 15:26

End-line characters from lines read from text file, using Python

6 Answers6

Linked