7

What steps need to be taken to ensure that "full" lines are always correctly appended to the end of a file if multiple of the following (example) program are running concurrently.

#!/usr/bin/env python
import random
passwd_text=open("passwd.txt","a+")
u=("jsmith:x:1001:1000:Joe Smith,Room 1007,(234)555-8917,(234)555-0077,jsmith@rosettacode.org:/home/jsmith:/bin/sh",
  "jdoe:x:1002:1000:Jane Doe,Room 1004,(234)555-8914,(234)555-0044,jdoe@rosettacode.org:/home/jdoe:/bin/sh",
  "xyz:x:1003:1000:X Yz,Room 1003,(234)555-8913,(234)555-0033,xyz@rosettacode.org:/home/xyz:/bin/sh")
for i in range(random.randint(1,2)):
  print >> passwd_text, random.choice(u)
passwd_text.close()

And: Can an "all or nothing" append be guaranteed (on linux/unix) even if the the disk becomes full, or "ulimit -f" has been set?

(Note similar question: How do you append to a file?)

Community
  • 1
  • 1
NevilleDNZ
  • 1,269
  • 12
  • 31
  • I don't know how you could append data if the disk is full... can you add more water to a glass that's overflowing? Beyond that, use file locks. `flock()` is a good place to start. – Marc B Sep 26 '11 at 21:43
  • @Marc B - re: "disk full" @... good point, I see the "write" system call "Returns the number of bytes that were written. If value is negative, then the system call returned an error.". I expect the "umlimit -f" will have the same effect on "write". So an atomic append, may ½ append, but the full append is not guaranteed. (I also do not recall any ioctl that guarantees "all or nothing" writes on a file, even if the "record" is small) – NevilleDNZ Sep 26 '11 at 22:33
  • Aren't writes atomic? Your lines may be interleaved with other lines written by other processes, but the lines themselves should be intact, as long as you end up using a single `write` syscall (which `os.write` should guarantee, even if other kinds of writes might be buffered). If you have to write several lines that need to stay together, build them up into a single string first. – Tom Zych Sep 26 '11 at 22:53
  • re: "Aren't writes atomic?" ... c.f. Linux NFS Overview, FAQ and HOWTO Documents => http://nfs.sourceforge.net/#faq_a9 ... it seems it depends on the OS and on the file system. ¢ That's why Oracle pays such good dividends! :-) ¢ – NevilleDNZ Sep 26 '11 at 22:58

2 Answers2

10

I think the discussion of this "bug" in python's normal open function suggests that you don't get the POSIX atomic guarantee, but you do if you use

with io.open('myfile', 'a') as f:
    f.write('stuff')

http://docs.python.org/2/library/io.html#io.open

if the operating system implements its write sys call correctly...

http://bugs.python.org/issue15723

hwjp
  • 15,359
  • 7
  • 71
  • 70
  • So you're saying there is never a reason to use fctrl.flock as long as I use io.open ? – David Doria Apr 24 '14 at 14:03
  • I don't know exactly what you're doing, so I can't give you a cast-iron guarantee. But it looks to me like that you will get a POSIX atomic write guarantee if you use `io.open` and the `'a'` append mode. But you should read the bug report and associated links to make sure it applies for your use case. Here be dragons. – hwjp Apr 25 '14 at 17:18
  • I just have multiple scripts running that may at any time want to update something in a file. This is a case where I would have traditionally thought "you need a file lock", but sometimes Python is smarter than I'd expect :) – David Doria Apr 25 '14 at 17:29
  • I'd like to stress that although reportedly, when done correctly, under the right circumstances, Windows can do atomic appends, but Python's `open()` or `io.open()` don't give you that guarantee. – Almar Sep 10 '18 at 21:17
3

You have to lock the file in order to ensure that nobody else is writing to it at the same time. See File Locking and lockfile or posixfile for more details.

UPDATE: And you cannot write more data into the file if the disk is full. I am not sure about Python's implementation of output re-direction, but write system call can write less bytes than requested.

  • 1
    Note in-particular the "man" page extract dealing specifically with O_APPEND: If the O_APPEND flag of the file status flags is set, the file offset shall be set to the end of the file prior to each write and no intervening file modification operation shall occur between changing the file offset and the write operation. – NevilleDNZ Oct 06 '11 at 04:38
  • cf this discussion - it looks like python's implementation doesn't exactly match the O_APPEND expected behaviour: http://bugs.python.org/issue15723 – hwjp Nov 05 '12 at 12:23