Strange error with text file editing in Python

Question

I'm using a text file to store the weight of a neural network that I'm making, but I'm having serious trouble editing the weights stored in this text field. Essentially, I am making a file with a very regular format: Word + \t + Weight + \n, I then use the follow code to run through this text file and grab the parts:

with open(Neuron_File, 'r+') as Original_Neurons:
            for Neuron in Original_Neurons:
                Word_Stem = re.sub(r'^([a-z-]*)([\t]?)([0-9.]*)(\n)$', r'\1', Neuron)
                Weight = float(re.sub(r'^([a-z-]*)([\t]?)([0-9.]*)(\n)$', r'\3', Neuron))

Which is working, however I would then like to be able to change the value of Weight, and write it back to the same text file in the same place. I have managed to successfully create a new file that is modified in the way that I would like, however I am having a strange problem with writing it back to the original file. I am using the below code for it:

def Replace(New_File, Old_File):
    for line in open(New_File):
        open(Old_File, 'w').write(str(line))

But for some reason this function simply breaks at a certain point in the file. The first 80% transfers fine, but then it cuts the file off at a seemingly random point in the middle of a line. Any ideas? I know there are other questions that are on similar topics, but none of them seem applicable to my situation, and I can't find any mention of another error like the one I'm getting.

Problem is navigable, but my primary interest is in what the origin of this error was. I've never seen anything like it and it intrigued me, as I had no idea what was going on, hoping someone on here would have more of an idea.

Is there a reason you're not just using `shutil.copyfile`? ( http://docs.python.org/library/shutil.html#shutil.copyfile ) Or am I completely not understanding what you are trying to do... — mgilson, Jun 04 '12 at 18:51

score 1 · Accepted Answer · answered Jun 04 '12 at 18:57

1

with open('input.txt') as in_file:
    with open('output.txt', 'w') as out_file:
        for line in in_file.readlines():
            word, weight = line.split()[:2]
            out_file.write('%s\t%s' % (word, float(weight) * 2))

with-block automaticaly closes opened files

answered Jun 04 '12 at 18:57

Aleksei astynax Pirogov

2,483
15
19

There's no reason to `strip` the string before `split`. `split` without arguments will take care of removing whitespace at the start and end for you. – mgilson Jun 04 '12 at 18:58
Thanks, @mgilson. `strip()` not needed. – Aleksei astynax Pirogov Jun 04 '12 at 19:00

score 0 · Answer 2 · edited May 23 '17 at 11:56

0

You need to close the file handle for the file you're writing.

def Replace(New_File, Old_File):
    Old_File_Handle = open(Old_File, 'w')
    for line in open(New_File):
        Old_File_Handle.write(str(line))
    Old_File_Handle.close()

Alternately, use the with statement.

edited May 23 '17 at 11:56

Community

1
1

answered Jun 04 '12 at 18:51

Nick ODell

15,465
3
32
66

Is there a reason you can think of that it would lead to an error like this? I actually use with in an earlier iteration of code, but I figured this error was strange enough that I wanted to understand it. – Slater Victoroff Jun 04 '12 at 18:58
If a python program ends and you haven't flushed the output to a file, the output may be cut off. – Nick ODell Jun 04 '12 at 19:03
Error persists while using with, and alternatively closing the file. Will run multiple times, cutting off more from the file each time. – Slater Victoroff Jun 04 '12 at 19:13

score 0 · Answer 3 · answered Jun 04 '12 at 19:02

0

Alternatively, you can just use something like shelve to handle this for you.

answered Jun 04 '12 at 19:02

jncraton

9,022
3
34
49

score 0 · Answer 4 · answered Jun 04 '12 at 19:24

0

def Replace(New_File, Old_File):
    for line in open(New_File):
        # next line REWRITES outfile EVERY TIME!!!
        open(Old_File, 'w').write(str(line))

Result: Old_File will contain ONLY LAST line

Correct implementation:

def Replace(New_File, Old_File):
    # open files
    inp = open(New_File)
    out = open(Old_File, 'w')
    for line in inp:
        out.write(line)
    # close files
    out.close()
    inp.close()

answered Jun 04 '12 at 19:24

Aleksei astynax Pirogov

2,483
15
19

Not actually true. Python implicitly knows what you mean when you open a file, and keeps it open rather than rewriting it every time. Also this wouldn't explain anything about the error. – Slater Victoroff Jun 04 '12 at 19:29
@Slater Tyranus, `open(fn, 'w').write(x)` inside for-loop reopens file for write at EVERY iteration! Previous instance of file-object will be collected by GC. – Aleksei astynax Pirogov Jun 05 '12 at 01:44
try it. Just try it. Python erases the file context when it it opened for writing, and so if what you were saying is true and python didn't have handling for this it would only write out the last line, which is simply not the case. The previous instances are in fact collected by the garbage collector, but these instances are not the file itself, only pointers which python can identify as being equivalent, and so the script runs correctly. – Slater Victoroff Jun 06 '12 at 13:58
@SlaterTyranus, `for l in open('in.txt'): open('out.txt', 'w').write(l)`, in.txt='1\n2\n3' (3 lines) -> out.txt='3' - last line only! – Aleksei astynax Pirogov Jun 06 '12 at 15:49

Strange error with text file editing in Python

4 Answers4