Consistent reading and writing a file in python

Question

I'm a Python beginner and facing the following : I have a script periodically reading a settings file and doing something according to those settings . I have another script triggered by some UI that writes the settings file with user input values. I use the ConfigParser module to both read and write the file.

I am wondering if this scenario is capable of leading into an inconsistent state (like in middle of reading the settings file, the other script begins writing). I am unaware if there are any mechanism behind the scene to automatically protect against this situations.

If such inconsistencies are possible, what could I use to synchronize both scripts and mantain the integrity of the operations ?

You can get a lock on the file. See http://stackoverflow.com/questions/489861/locking-a-file-in-python — Waleed Khan, Oct 16 '13 at 18:23

Maxim Egorushkin · Answer 1 · 2013-10-16T18:40:53.867

1

I'm a Python beginner and facing the following : I have a script periodically reading a settings file and doing something according to those settings . I have another script triggered by some UI that writes the settings file with user input values.

There may be a race condition when the reader reads while the writer writes to the file, so that the reader may read the file while it is incomplete.

You can protect from this race by locking the file while reading and writing (see Linux flock() or Python lockfile module), so that the reader never observes the file incomplete.

Or, better, you can first write into a temporary file and when done rename it to the final name atomically. This way the reader and writer never block:

def write_config(config, filename):
    tmp_filename = filename + "~"
    with open(tmp_filename, 'wb') as file:
        config.write(file)
    os.rename(tmp_filename, filename)

When the writer uses the above method no changes are required to the reader.

edited Oct 16 '13 at 18:40

answered Oct 16 '13 at 18:24

Maxim Egorushkin

131,725
17
180
271

Very interesting idea,though one more should then be done, deleting the temp file aftewr renaming the file – klaus johan Oct 16 '13 at 19:01
Oh, and is it possible for the rename to throw an exception? – klaus johan Oct 16 '13 at 19:02
@klausjohan It is possible, but may be very unlikely, since you already created a temporary file in that directory, which means that you had enough permissions to do so and there was enough disk space to create the file. However, there may occur a rare error, like the filesystem was just unmounted, or there is a directory with such a name already. In any case, it does not overwrite the old version of the file. See the list of [`rename()`](http://linux.die.net/man/2/rename) errors. – Maxim Egorushkin Oct 16 '13 at 19:13

score 0 · Answer 2 · answered Oct 16 '13 at 18:22

When you write the config file write it to a temporary file first. When it's done, rename it to the correct name. The rename operation (os.rename) is normally implemented as an atomic operation on Unix systems, Linux and Windows, too, I think, so there will be no risk of the other process trying to read the config while the writing has not been finished yet.

miku · Answer 3 · 2013-10-16T18:41:12.903

There are al least two ways to address this issue (assuming you are on a unix-ish system):

If you want to write, write to a temporary file first, then do something unix can do atomically, especially rename the temporary file into place.
Lock the file during any operation, e.g. with the help of this filelock module.

Personally, I like the first option because it utilizes the OS, although some systems have had problems with the atomicity: On how rename is broken in Mac OS X - another limitation: the rename system call can not rename files across devices.

Consistent reading and writing a file in python

3 Answers3

Linked