18

I have read enough posts on stackoverflow regarding the difference between flock/lockf/fcntl but I am unable to answer the below observation:

>>> import fcntl
>>> a = open('/tmp/locktest', 'w')
>>> b = open('/tmp/locktest', 'w')
>>> fcntl.lockf(a, fcntl.LOCK_EX | fcntl.LOCK_NB)
>>> fcntl.lockf(a, fcntl.LOCK_EX | fcntl.LOCK_NB)
>>> fcntl.lockf(b, fcntl.LOCK_EX | fcntl.LOCK_NB)
>>>
>>> a.close()
>>> b.close()

>>> a = open('/tmp/locktest', 'w')
>>> b = open('/tmp/locktest', 'w')
>>> fcntl.flock(a, fcntl.LOCK_EX | fcntl.LOCK_NB)
>>> fcntl.flock(a, fcntl.LOCK_EX | fcntl.LOCK_NB)
>>> fcntl.flock(b, fcntl.LOCK_EX | fcntl.LOCK_NB)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 35] Resource temporarily unavailable

Why is the behaviour different in the two cases? I know the obvious answer that these are two different locking mechanisms. I am looking for:

  1. What actually lockf() or flock() does to file (inode/fd)?
  2. As per the demo, are we allowed taking the same lock recursively?

I understand the basics of fds and stuff so I would prefer to have a technical answer with more insights to operating system level details.

OSX 10.9.3, Python: 2.7.5

Gabriel Staples
  • 36,492
  • 15
  • 194
  • 265
Jatin Kumar
  • 2,635
  • 9
  • 36
  • 46
  • 1
    I've taken the liberty to edit the question to actually include a question. Please review and edit if that's not what you're asking. Thanks. – NPE Feb 12 '15 at 05:59
  • @NPE: It was part of my first sentence though its better to have it explicitly put there. Thanks :) – Jatin Kumar Feb 12 '15 at 06:50

1 Answers1

14

A nice article about this: On the Brokenness of File Locking

In short:

  • POSIX locks:

    lockf() most of the time implemented as just an interface of fcntl()

    fcntl() locks are bound to processes, not file descriptors. If a process has multiple open file descriptors for a particular file, any one of these file descriptors used for acquiring locking will RESET the lock.

  • BSD lock:

    flock() locks are bound to file descriptors, not processes.

Furthermore

A good analysis with tests: Advisory File Locking – My take on POSIX and BSD locks

Excerpt of Summary:

  • fcntl and flock style locks are completely orthogonal to each other. Any system providing both(Linux does) will be treating locks obtained through each one of them independently.
  • Both POSIX and BSD locks are automatically released when the process exits or is aborted.
  • Both POSIX and BSD locks are preserved across execve calls except when the process has set FD_CLOEXEC flag forcing the file descriptor to be closed, and not inherited by the new process.
Community
  • 1
  • 1
pppk520
  • 517
  • 4
  • 15
  • 5
    A fair assessment. It should be emphasized that **POSIX locks silently fail under multithreading scenarios** and should (*arguably*) be considered harmful for use in modern software stacks – [GIL](https://wiki.python.org/moin/GlobalInterpreterLock) notwithstanding. More critically, it should be emphasized that **both POSIX and BSD locks on group- or world-readable files pose an extreme security risk.** Why? Because *any* user with read permissions for the file to be locked can permanently deadlock your Python application by preemptively acquiring that lock first and then never releasing it. – Cecil Curry Mar 30 '17 at 05:09
  • 5
    **tl;dr:** **(A)** in-memory locks (e.g., [`multiprocessing` module](https://docs.python.org/3/library/multiprocessing.html#synchronization-between-processes) synchronization primitives) are strongly preferable to both BSD- and POSIX-style locks, **(B)** BSD-style `flock` locks are strongly preferable to POSIX-style `fcntl` locks, and **(C)** neither BSD- nor POSIX-style locks should be used unless permissions for the file(s) to be locked are strictly constrained to at most `0600`. – Cecil Curry Mar 30 '17 at 05:16
  • The only thing I would add to this answer is to note that you're using the `fcntl.LOCK_NB` flag. If you remove this flag, then your second case will block instead of raising an exception. – Droid Coder Aug 11 '19 at 09:17
  • Also, if you run another python session, in another window, and use lockf() on the file and leave the file open, then you'll find the exception occurs in the first call to `fcntl.lockf()`, in your second session (if we're back to using `fcntl.LOCK_NB`). This confirms that the state of `fcntl.lockf()` process-global, while the state of `fcntl.flock()` is scoped by the file descriptor. – Droid Coder Aug 11 '19 at 09:21
  • 1
    Regarding @CecilCurry's link, notice that Python multiprocessing module is *only* applicable to synchronize *Python subprocesses spawned by a common parent*, and if one of the process crashes [this leaks semaphores and shared memory segments](https://docs.python.org/3/library/multiprocessing.html#multiprocessing-start-methods) and [corrupts Queue objects](https://docs.python.org/3/library/multiprocessing.html#pipes-and-queues) and [pipes](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.connection.Connection.recv_bytes_into). – Stéphane Gourichon Mar 03 '21 at 21:07