0

The question is about API design. The scenario is, I have some bytes of KNOWN length in memory and I want to design a Pythonic API that flushes them to AWS S3 with few RPCs. By Pythonic, I mean if there is already an accepted API doing this, I want to copy it.

One way is to do something like io.create(filename, content). This can be translated into a single HTTP request using S3 XML API.

The traditional open, write, and close pattern is a streaming API. open doesn't accept a length argument so the created stream doesn't know it should buffer all writes into a single RPC.

Of course the API can look like:

with open(filename, buffering="UNLIMITED") as f:
   f.write(content)

But buffering doesn't really support "UNLIMITED" constant.

So what can I do? File a PEP? Thanks.

Yey
  • 554
  • 3
  • 15
  • 1
    why do you think you need atomic creation? are you running into resource collisions? or just trying to stay ahead of a curve that you imagine must be out there? (IE are you actually being impacted by not having atomic file creation?) – Joran Beasley Mar 12 '14 at 20:51
  • 1
    possible duplicate of [atomic writing to file with Python](http://stackoverflow.com/questions/2333872/atomic-writing-to-file-with-python) – jfs Mar 12 '14 at 20:55
  • 1
    I don't believe atomic creation would work in this case, since the purpose is to limit RPC on NFS shares. I looked into this some time ago, as I was writing metadata files onto Isilon NFS shares. I never came up with a good solution, so I ended up using io.BytesIO and io.StringIO and then writing out the file object. – JamCon Mar 12 '14 at 21:08

1 Answers1

0

NFS, at least in version 2 and 3, isn't terribly good at doing locking or atomicity.

The best bet, again at least in NFS 2 and 3, is to create your file with a temporary name, and then use os.rename().

Perhaps some progress has been made on this in NFS 4; I don't know.

dstromberg
  • 6,954
  • 1
  • 26
  • 27