2

So I have a script that will get fired by incrond upon when new file events are detected in a watch dir. In the script I have this function which will read and save stuff to conf file. At the moment, I try not to allow more than one instance of the script to run to avoid race condition and end up corrupting the saved file.

And right now I'm trying to figure out how to make this function write to conf file safely when I have multiple instances of the script try to write to the same file. I read about file locking with fcntl module the other day but it's little over my head and I wasn't sure how it will work

def save_task_details(resp, filepath):
    conf = ConfigParser.SafeConfigParser()
    conf.read(CFG_PATH)

    filehash = get_filehash(filepath)
    if not conf.has_section(filehash):
        conf.add_section(filehash)

    conf.set(filehash, "filename", os.path.basename(filepath))
    conf.set(filehash, "id", resp.id)

    # ...

    with open(CFG_PATH, "wb") as configfile:
        conf.write(configfile)
Flint
  • 436
  • 1
  • 4
  • 15
  • 1
    possible duplicate of [Locking a file in Python](http://stackoverflow.com/questions/489861/locking-a-file-in-python) – ebarr Jun 20 '14 at 06:51

2 Answers2

1

File locking seems to be rather problematic, and databases inappropriate for writing a config file so, if you are not adverse to 3rd party modules and you are using *nix, you might consider using a posix_ipc named semaphore to coordinate writing to your config file. Very little additional code is required:

import posix_ipc as ipc

def save_task_details(resp, filepath):

    # ...

    with ipc.Semaphore('/config_semaphore', flags=ipc.O_CREAT, initial_value=1), open(CFG_PATH, "wb") as configfile:
           conf.write(configfile)
mhawke
  • 84,695
  • 9
  • 117
  • 138
0

Writing to single file from multiple processes is always something tricky and is better to avoid.

There are few options

File locking

No standard library for this. Possible problems with stale lock and file not being accessible

Writing to file based database

Good case is sqlite, others might be possible too. You may experience your script being blocked for a moment, if another script is working on shared data, but this is in fact something you wanted.

Using existing service

Like database, but also key-value stores like Redis or etcd could be the solution.

Running your own service

Write such a service is not so difficult (I would go for using zmq for interprocess communication, either in plain (no other packages around) or e.g. as part of zerorpc, which allows exposing services very easily.

Jan Vlcinsky
  • 42,725
  • 12
  • 101
  • 98