In search of reliable python process synchronization techniques (Linux non-portable)

Question

[I am having an extremely difficult time to implement a thread-/process-safe solution to acquire a file lock using python 3 on Linux (I do not care about portable solutions as the program I am working on makes extensive use of Linux-kernel-exclusive-containerization technologies).]

After reading http://apenwarr.ca/log/?m=201012#13 , I decided to use fcntl.lockf() to lock a file for process-exclusive access and wrote the following function:

import contextlib as Contextlib
import errno as Errno
import fcntl as Fcntl
import os as Os


@Contextlib.contextmanager
def exclusiveOpen(filename,
                  mode):
  try:
    fileDescriptor = Os.open(filename,
                             Os.O_WRONLY | Os.O_CREAT)
  except OSError as e:
    if not e.errno == Errno.EEXIST:
      raise

  try:
    Fcntl.lockf(fileDescriptor,
                Fcntl.LOCK_EX)
    fileObject = Os.fdopen(fileDescriptor,
                           mode)

    try:
      yield fileObject
    finally:
      fileObject.flush()
      Os.fdatasync(fileDescriptor)
  finally:
    Os.close(fileDescriptor)

Apart from that I am certain, that it is incorrect (why doesn't it block in Fcntl.lockf(fileDescriptor, Fcntl.LOCK_EX)?), the part which makes me feel uneasy the most, is where the fileDescriptor is acquired - if the file is non existent, it is created ... but what is going on, if two processes execute this part simultaneously? Isn't there a chance of a race condition, where both threads attempt to create the file? And if so, how could one possibly prevent that - certainly not with another lock file (?) because it would have to be created in the same manner (?!?!) I'm lost. Any help is greatly appreciated.

UPDATE: Posted another approach to the underlying problem. The problem I see in this approach, is that a procedure name must not equal the name of an existent UNIX domain socket (possibly created by another program) - am I correct with this?

score 1 · Accepted Answer · answered Apr 26 '16 at 13:59

AFAIK, there is no possible race condition when creating a file. If two (or more) processes or threads try to simultaneously create the same file by open with the O_CREAT flag and not the O_EXCL one, the file will be created once, and all callers will get an open file descriptor on the same file - exactely what you need. I assume that you are using C-Python on Linux, and that your Python code will end using the underlying C open function.

So when your code executes the lockf function, you have a opened file descriptor on an existing file, so the underlying lockf call guarantees that only one process can hold the lock at a time, provided the underlying file system supports locking.

I tried it on a Unix (FreeBSD) system and it works as expected.

@MCH Well I could not reproduce a race condition, but I just started 2 processes, open the same file in both (but using `O_RDWR` instead of `O_RDONLY`) and take the lock in first. When I try to take the lock in second one, it is blocked until first process releases the lock or closes the file. And once it has acquired the lock, it blocks any other process that would try to take it. — Serge Ballesta, Apr 26 '16 at 14:37

score -1 · Answer 2 · edited May 23 '17 at 12:07

-1

Based on https://stackoverflow.com/a/7758075/5449837 , I ended up with going with abstract UNIX sockets instead of lock files to synchronize processes, (if somebody posts a better answer I will gladly accept it).

import contextlib as Contextlib
import os as Os
import socket as Socket
import time as Time


@Contextlib.contextmanager
def locked(name,
           timeout):
  try:
    lock = Socket.socket(Socket.AF_UNIX, 
                         Socket.SOCK_DGRAM)

    while True:
      try:
        lock.bind('\0' + name)
        break
      except:
        if timeout:
          Time.sleep(1)
          timeout = timeout - 1 
        else:
          raise

    yield
  except:
    raise
  finally:
    lock.close()


# usage:
with locked("procedureName", 5):
  pass # perform actions on shared resource

To the down-voters: Care to explain why you think this is bad?

edited May 23 '17 at 12:07

Community

1
1

answered Apr 26 '16 at 00:47

MCH

415
3
11

unrelated: 1- please, do not rename stdlib modules. Also, [follow pep-8 naming conventions even for your own modules (in the public code)](https://www.python.org/dev/peps/pep-0008/#package-and-module-names) 2- don't put questions into the answer (it is not a discussion forum). If you have a question; [ask it as a question instead](http://stackoverflow.com/questions/ask) – jfs Apr 26 '16 at 07:59
Thanks (will fix that) :) - Out of curiosity why is no 1? – MCH Apr 26 '16 at 13:11
why? Because readability counts. If I see `Socket`; it looks like some custom class. If I see `socket` I assume it is a stdlib module. No need to create friction without necessity – jfs Apr 26 '16 at 13:23
So there are no underlying technical reasons ... To me it is more readable if I can name my variable 'socket' instead of 'socket_' because I imported socket as Socket, for example – MCH Apr 26 '16 at 13:26

In search of reliable python process synchronization techniques (Linux non-portable)

2 Answers2