Amazon S3 avoid overwriting objects with the same name

Question

If I upload a file to S3 with the filename identical to a filename of an object in the bucket it overwrites it. What options exists to avoid overwriting files with identical filenames? I enabled versioning in my bucket thinking it will solve the problem but objects are still overwritten.

Answer 1

My comment from above doesn't work. I thought the WRITE ACL would apply to objects as well, but it only works on buckets.

Since you enabled versioning, your objects aren't overwritten. But if you don't specify the version in your GET request or URL, the latest version will be taken. This means when you put and object into S3 you need to save the versionID the response tells you in order to retrieve the very first object.

See Amazon S3 ACL for read-only and write-once access for more.

Answer 2

Here is my suggestion if you are using a DB to store the key of every file on your s3 bucket.

Generate a random key. Try insert/update the key your DB, in a field with a UNIQUE constraint that allows a null entry. If it fails the key has been used, repeat until you get a unique key.

Then put your file on s3 with your key that you know is unique.

Answer 3

You can also configure an IAM user with limited permissions. Writes are still writes (i.e., updates), but using an IAM user is a best practice anyway.

The owner (i.e., your "long-term access key and secret key") always has full control unless you go completely out of your way to disable it.

Answer 4

There is a solution to this in standard HTTP headers that prevent an action from being taken if certain conditions are not satisfied. For example, if you add this to an HTTP PUT request:

If-None-Match: *

The server is asked to not execute your action if the resource already exists. An explanation from MDN:

For other methods, and in particular for PUT, If-None-Match used with the * value can be used to save a file not known to exist, guaranteeing that another upload didn't happen before, losing the data of the previous put; this problem is a variation of the lost update problem.

It turns out that S3 will pay attention to this header and will correctly return an HTTP "412 Precondition Failed" status if a resource already exists and you try to replace it.

So, how to implement this with the normal AWS SDK? There is the concept of middleware which allows you to fiddle with the HTTP request before it is sent. Here's how I have handled this in the JS SDK v3:

// PreventObjectOverwriteMiddleware.js
export default (next, context) => async (args) => {
  args.request.headers['if-none-match'] = '*';
  return await next(args);
}

// Then, in my other code...
import PreventObjectOverwriteMiddlware from 'PreventObjectOverwriteMiddleware.js';
this.client = new S3Client(/* config */);
this.client.middlewareStack.add(PreventObjectOverwriteMiddleware, {
  step: 'build'
});

Now though, you said PHP. I haven't written any PHP in a long time, so this might need some modification, but based on the documentation I think this should get you started in the right direction:

use Aws\Middleware;
use Psr\Http\Message\RequestInterface;

// Create a command so that we can access the handler list
$command = $s3Client->getCommand('PutObject', [
    'Key'    => 'test',
    'Bucket' => 'mybucket'
]);
    
// Apply a custom middleware named "add-header" to the "build" lifecycle step
$command->getHandlerList()->appendBuild(
    Middleware::mapRequest(function (RequestInterface $request) {
        // Return a new request with the added header
        return $request->withHeader('If-None-Match', '*');
    }),
    'add-header'
);

Answer 5

S3 Object Lock could help. You can prevent delete or updates using this feature (write-once-read-many (WORM) model).

https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-overview.html