Is it possible to modify lines in a file in-place?

Question

Is it possible to parse a file line by line, and edit a line in-place while going through the lines?

Answer 1

Is it possible to parse a file line by line, and edit a line in-place while going through the lines?

It can be simulated using a backup file as stdlib's fileinput module does.

Here's an example script that removes lines that do not satisfy some_condition from files given on the command line or stdin:

#!/usr/bin/env python
# grep_some_condition.py
import fileinput

for line in fileinput.input(inplace=True, backup='.bak'):
    if some_condition(line):
        print line, # this goes to the current file

Example:

$ python grep_some_condition.py first_file.txt second_file.txt

On completion first_file.txt and second_file.txt files will contain only lines that satisfy some_condition() predicate.

Answer 2

fileinput module has very ugly API, I find beautiful module for this task - in_place, example for Python 3:

import in_place

with in_place.InPlace('data.txt') as file:
    for line in file:
        line = line.replace('test', 'testZ')
        file.write(line)
    file.close()

main difference from fileinput:

• Instead of hijacking sys.stdout, a new filehandle is returned for writing.
• The filehandle supports all of the standard I/O methods, not just readline().

Important Notes:

1. This solution deletes every line in the file if you don't re-write it with the file.write() line.
2. Also, if the process is interrupted, you lose any line in the file that has not already been re-written.

Answer 3

No. You cannot safely write to a file you are also reading, as any changes you make to the file could overwrite content you have not read yet. To do it safely you'd have to read the file into a buffer, updating any lines as required, and then re-write the file.

If you're replacing byte-for-byte the content in the file (i.e. if the text you are replacing is the same length as the new string you are replacing it with), then you can get away with it, but it's a hornets nest, so I'd save yourself the hassle and just read the full file, replace content in memory (or via a temporary file), and write it out again.

Answer 4

If you only intend to perform localized changes that do not change the length of the part of the file that is modified (e.g. changing all characters to lower case), then you can actually overwrite the old contents of the file dynamically.

To do that, you can use random file access with the seek() method of a file object.

Alternatively, you may be able to use an mmap object to treat the whole file as a mutable string. Keep in mind that mmap objects may impose a maximum file-size limit in the 2-4 GB range on a 32-bit CPU, depending on your operating system and its configuration.

Answer 5

You have to back up by the size of the line in characters. Assuming you used readline, then you can get the length of the line and back up using:

file.seek(offset[, whence])

Set whence to SEEK_CUR, set offset to -length.

See Python Docs or look at the manpage for seek.