How to iterate over the file in python

Question

I have a text file with some hexadecimal numbers and i am trying to convert it to decimal. I could successfully convert it, but it seems before the loop exist it reads some unwanted character and so i am getting the following error.

Traceback (most recent call last):
  File "convert.py", line 7, in <module>
    print >>g, int(x.rstrip(),16)
ValueError: invalid literal for int() with base 16: ''

My code is as follows

f=open('test.txt','r')
g=open('test1.txt','w')
#for line in enumerate(f):  
while True:
    x=f.readline()
    if x is None: break
    print >>g, int(x.rstrip(),16)

Each hexadecimal number comes in a new line for input

Well, take the debugger and figure out what the value of 'x' is causing the problem. Perhaps you have a file with a BOM? — , Apr 20 '11 at 16:20
@RestRisiko: If there was a BOM, the error message would show that. No, it just tried to feed the empty string `''` to `int`. — , Apr 20 '11 at 16:21

joaquin · Accepted Answer · 2011-04-20T17:04:15.573

78

The traceback indicates that probably you have an empty line at the end of the file. You can fix it like this:

f = open('test.txt','r')
g = open('test1.txt','w') 
while True:
    x = f.readline()
    x = x.rstrip()
    if not x: break
    print >> g, int(x, 16)

On the other hand it would be better to use for x in f instead of readline. Do not forget to close your files or better to use with that close them for you:

with open('test.txt','r') as f:
    with open('test1.txt','w') as g: 
        for x in f:
            x = x.rstrip()
            if not x: continue
            print >> g, int(x, 16)

edited Apr 20 '11 at 17:04

answered Apr 20 '11 at 16:20

joaquin

82,968
29
138
152

24

The indentation-heaviness of the latter can be reduced in more recent versions: `with open('test.txt','r') as f, open('test1.txt','w') as g` – Apr 20 '11 at 16:32
5

@delnan, Nice new thing! When indentation is not a problem I still prefer the two-lines form. I read it better... – joaquin Apr 20 '11 at 16:38
1

oops! why the downvote? downvotes are not useful without explanation especially when the OP already accepted the answer. – joaquin Apr 20 '11 at 16:42
1

The first program gives the same error of we have a blank line at the end – user567879 Apr 20 '11 at 16:44
I dont understand. If you have a blank line in any place in your text, x becomes '', so 'if not x' prevents execution of print. Note that I changed your 'if x is None' for 'if not x' – joaquin Apr 20 '11 at 16:51
4

This is a great example of the "cleanness" of Python idiom -- no need for EOF, readline(), writeline() or any guts -- just concept: iterate over lines in one file, transform, and export to another file. Very nice! (+1) – Assad Ebrahim Feb 11 '14 at 18:58

score 14 · Answer 2 · answered Apr 20 '11 at 16:20

Just use for x in f: ..., this gives you line after line, is much shorter and readable (partly because it automatically stops when the file ends) and also saves you the rstrip call because the trailing newline is already stipped.

The error is caused by the exit condition, which can never be true: Even if the file is exhausted, readline will return an empty string, not None. Also note that you could still run into trouble with empty lines, e.g. at the end of the file. Adding if line.strip() == "": continue makes the code ignore blank lines, which is propably a good thing anyway.

This does NOT strip the trailing newlines: `python -c 'with open("file.txt") as f: print(repr([l[-1] for l in f]))'` returns many instances of `\n` on Python 2.7.12 and 3.4.5 — JamesTheAwesomeDude, May 02 '17 at 04:02

score 7 · Answer 3 · edited Oct 08 '19 at 11:33

7

with open('test.txt', 'r') as inf, open('test1.txt', 'w') as outf:
    for line in inf:
        line = line.strip()
        if line:
            try:
                outf.write(str(int(line, 16)))
                outf.write('\n')
            except ValueError:
                print("Could not parse '{0}'".format(line))

edited Oct 08 '19 at 11:33

taras

6,566
10
39
50

answered Apr 20 '11 at 16:40

Hugh Bothwell

55,315
8
84
99

Hugh - does outf.write(int(line, 16)) work for you? My code is the exact same as yours except that I'm trying to write a dictionary d. I get a ValueError. See below: Traceback (most recent call last): File "./test.py", line 22, in outf.write(d) ValueError: I/O operation on closed file – blehman Aug 18 '13 at 06:22

bukzor · Answer 4 · 2011-04-23T17:25:53.903

You should learn about EAFP vs LBYL.

from sys import stdin, stdout
def main(infile=stdin, outfile=stdout):
    if isinstance(infile, basestring):
        infile=open(infile,'r')
    if isinstance(outfile, basestring):
        outfile=open(outfile,'w')
    for lineno, line in enumerate(infile, 1):
        line = line.strip()
         try:
             print >>outfile, int(line,16)
         except ValueError:
             return "Bad value at line %i: %r" % (lineno, line)

if __name__ == "__main__":
    from sys import argv, exit
    exit(main(*argv[1:]))

score 1 · Answer 5 · answered Apr 20 '11 at 16:21

1

This is probably because an empty line at the end of your input file.

Try this:

for x in f:
    try:
        print int(x.strip(),16)
    except ValueError:
        print "Invalid input:", x

answered Apr 20 '11 at 16:21

Rumple Stiltskin

9,597
1
20
25

How to iterate over the file in python

5 Answers5

Linked