Python read file from command line and strip "\n\r" with very large files

Question

I am learning python for the first time and I've just learned that readlines() is incredibly slow and taxing on memory. This would be fine, but as I am programming for a data structures class with up to 10^6 inputs, I believe that runtime is very important.

This is what I have so far that works. I did not strip the '\r' yet.

def generateListOfPoints(stuff):
    List = open(stuff).readlines()

    a = []

    for i in range(len(List)):
        a.append(List[i].rstrip('\n').split(","))

    return a

This is what I tried to do with a for loop (which I heard was better), but all I'm getting is errors and I don't know what is going on.

def generateListOfPoints(stuff):

    a = []
    with open(stuff) as f:
        for line in f:
            a.append(stuff.rstrip('\n').rstrip('\r').split(","))
    return a

`a.append(line.rstrip('\n').lstrip('\r').split(",")` is what you need. — Ashwini Chaudhary, Apr 19 '15 at 06:21

Saksham Varma · Answer 1 · 2015-04-19T06:21:23.180

4

Replace stuff with line. stuff is simply the filepath, the actual content is in line -- the variable used for iterating over the generator f

a.append(line.rstrip('\n').split(","))

You might like to store the list formed after using split on line, as a tuple instead, such that a would be a list of tuples, where each tuple would correspond to a line in the file. You can do that using:

a.append(tuple(line.rstrip('\n').split(",")))

edited Apr 19 '15 at 06:21

answered Apr 19 '15 at 04:57

Saksham Varma

2,122
13
15

This is **wrong** because as Python will split the line on `\n` and due to that the `\r` will end up at the start of the next line(unless they are using old Mac OS which will consider `\r` as newline with universal newline support). – Ashwini Chaudhary Apr 19 '15 at 06:13
Copying @krock's answer is not going to help as their solution is also incorrect. – Ashwini Chaudhary Apr 19 '15 at 06:20
No, this is not correct either. Now you're simply using OP's first code. – Ashwini Chaudhary Apr 19 '15 at 06:23
I did not see '\r' ending up at the start of the next line, on my python which is on Ubuntu, so I don't see what has to be fixed. – Saksham Varma Apr 19 '15 at 06:28
I am not sure how you are testing it, try something like this: https://gist.github.com/ashwch/8090cff47f69ff0a3ad4 – Ashwini Chaudhary Apr 19 '15 at 06:42

score 1 · Answer 2 · edited May 23 '17 at 10:10

Make sure to name your variables so they make sense. Naming something stuff is convenient but obviously leads to errors. The example below renames this to filename and fixes appending line to the list instead of the filename.

Also, the rstrip function takes a set of characters to strip, so you can strip both \r and \n in one function call. So you would have:

def generateListOfPoints(filename):
    a = []
    with open(filename) as f:
        for line in f:
            a.append(line.rstrip('\r\n').split(","))
    return a

This will create a list of lists. If you want to flatten out the inner list in your solution, you will want to use extend instead of append.

http://stackoverflow.com/questions/29725941/python-read-file-from-command-line-and-strip-n-r-with-very-large-files#comment47586816_29725976 — Ashwini Chaudhary, Apr 19 '15 at 06:15

score 1 · Answer 3 · answered Apr 19 '15 at 05:40

I suggest you experiment using the command line interpreter. This makes it easy to learn how rstring and split work. Assuming you start using the line variable as suggested, You probably are not appending to the a list what you want. Also you can strip both \n and \r with one call to rstrip.

python
>>> a = []
>>> line = "this,is,a,test\n\r"
>>> line.rstrip('\n\r')
'this,is,a,test'
>>> line.rstrip('\n\r').split(',')
['this', 'is', 'a', 'test']
>>> a.append(line.rstrip('\n\r').split(','))
>>> a
[['this', 'is', 'a', 'test']]

Python read file from command line and strip "\n\r" with very large files

3 Answers3