Are writes to Amazon S3 atomic (all-or-nothing)?

Question

I have a large number of files that I am reading and writing to S3. I am just wondering if I need to code for the case where a file is "half written" e.g. the S3 PUT / Write only "half" worked.

Or are writes to S3 all-or-nothing?

I know there is a read-write eventual consistency issue which (I think) is largely a separate issue.

score 27 · Answer 1 · answered Oct 21 '14 at 19:08

27

See S3 PUT documentation:

Amazon S3 never adds partial objects; if you receive a success response, Amazon S3 added the entire object to the bucket.

answered Oct 21 '14 at 19:08

Mystic

980
1
9
17

4

For me, this is the answer to the question. – jckuester Jun 27 '19 at 08:39

Julio Faerman · Accepted Answer · 2016-02-22T21:57:59.123

21

For all regions except US Standard (us-east-1) you get read-after-write-consistency. This means that if you get an HTTP 200 OK for your PUT, you can read the object right away.

If your request is dropped in the middle, you would not get and HTTP 200 and your object would not be written at all.

UPDATE: All regions now support read-after-write consistency (thanks @jeff-loughridge): https://aws.amazon.com/about-aws/whats-new/2015/08/amazon-s3-introduces-new-usability-enhancements/

edited Feb 22 '16 at 21:57

answered May 20 '14 at 16:24

Julio Faerman

13,228
9
57
75

http://aws.amazon.com/s3/faqs/ "Amazon S3 buckets in the US West (Oregon), US West (Northern California), EU (Ireland), Asia Pacific (Singapore), Asia Pacific (Tokyo), Asia Pacific (Sydney), South America (Sao Paulo), and GovCloud (US) regions provide read-after-write consistency for PUTS of new objects and eventual consistency for overwrite PUTS and DELETES. Amazon S3 buckets in the US Standard region provide eventual consistency."" – Julio Faerman Oct 21 '14 at 19:20
2

All regions now support read-after-write consistency. See S3 FAQ. – Jeff Loughridge Feb 22 '16 at 14:45
Thanks for the link above. I found this ticket after getting 0.25% failure rate with a "If (not HEAD) then PUT; GET" workflow. Your link above lists the following limitation: "the caveat is that if you make a HEAD or GET request to the key name (to find if the object exists) before creating the object, Amazon S3 provides eventual consistency for read-after-write." So now I know my design won't work, but at least I'm not crazy. – Robert Calhoun May 31 '17 at 01:01
@RobertCalhoun you may want to check Netflix s3mper project and EMRFS consistent view – Julio Faerman Jun 05 '17 at 23:37
It's easy to code something in S3 that works as desired, so long as one is aware of the exceptions! – Robert Calhoun Jun 16 '17 at 17:12

score 6 · Answer 3 · answered Aug 07 '21 at 21:26

From the docs:

Updates to a single key are atomic. For example, if you PUT to an existing key from one thread and perform a GET on the same key from a second thread concurrently, you will get either the old data or the new data, but never partial or corrupt data.

This answer is somewhat similar to the existing ones, but it stresses on the fact that not only there is no risk of leaving a partially-written object behind, but also that a reader will never be put at risk of seeing (reading) a partially-written object.

Are writes to Amazon S3 atomic (all-or-nothing)?

3 Answers3

Linked