36

I'm currently working with Amazon S3 and I am writing a program that uses the modified dates. I'm looking for a way to edit the modified dates.

I could loop trough all the files and save them as they are, but this sounds like a bad solution.

In PHP there is this function touch().

Does anyone know a solution, or has the same problem?

Ron van der Heijden
  • 14,803
  • 7
  • 58
  • 82

6 Answers6

44

In response to @Daniel Golden's comment on @tkotisis answer. It looks like at least the AWS CLI tools do not let you copy an item on to itself. You can however 'force' a copy by updating the metadata.

$ aws s3 cp --metadata '{"touched":"now"}' s3://path/to/object s3://path/to/object

This recreates the object (downloads to the caller and reuploads it) replacing its content, owner and metadata. This will also trigger any attached Lambda events.

Jason
  • 9,408
  • 5
  • 36
  • 36
  • 7
    This results in an error: `fatal error: An error occurred (404) when calling the HeadObject operation: Key "index.html" does not exist`. – Chad Johnson Mar 06 '20 at 08:34
  • This won't work nicely if the S3 bucket versioning is enabled. It would create a duplicate copy... – Jaeyoung Chun Mar 28 '22 at 00:17
  • This doesn't work if the goal is to CREATE the object (instead of just touching an existing object). – Zhang18 Jan 19 '23 at 05:26
18

You can achieve the same through a copy object request, specifying the CopySource to be same as the target key.

In essence, this will issue a PUT Object - COPY request to S3 with the corresponding source and target bucket/key.

tkotisis
  • 3,442
  • 25
  • 30
  • And do you know what exactly this does? Does it fire a GET and a PUT request for each file? – Ron van der Heijden Nov 20 '12 at 07:50
  • Edited my answer to include this info. – tkotisis Nov 20 '12 at 08:08
  • 9
    On running this command: `s3cmd cp s3://path/to/file s3://path/to/file` where the two paths are the same, I get this error: `ERROR: S3 error: 400 (InvalidRequest): This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata, storage class, website redirect location or encryption attributes.` – Daniel Golden Aug 25 '14 at 15:48
  • Quite different environment, but in luigi + python + spark you do that like this: fs.put_string("", path) Where the fs is an S3Client (is available through eg. S3Target). Basically you can put empty string to S3. – pkopac Jul 28 '15 at 11:28
  • Using aws client java iibrary, I got Daniel's error too. To workaround, I did a copy to a new key, delete old key etc, etc – Bohemian May 17 '16 at 07:02
  • 4
    @DanielGolden Have you set the `x-amz-metadata-directive` request header to `REPLACE`, as noted in the docs? – tkotisis May 26 '16 at 09:20
  • 3
    This worked - thanks. `aws s3 cp s3://path/to/file s3://path/to/file --metadata-directive REPLACE` – Chirag Sejpal Oct 30 '19 at 19:06
6

I find myself performing the copy trick pretty often when testing, to the extent that I've added a handy function to my .bashrc:

s3-touch() {
  aws s3 cp \
    --metadata 'touched=touched' \
    --recursive --exclude="*" \
    --include="$2" \
    "${@:3}" \
    "$1" "$1"
}

Example usage:

# will do a dryrun on a copy operation
s3-touch s3://bucket/prefix/ "20200311*" --dryrun

# the real thing, creating events for all objects
# in s3://bucket/prefix/ that start with 20200311
s3-touch s3://bucket/prefix/ "20200311*"

I'm doing this mainly for the S3-events that I want to trigger.

g-io
  • 99
  • 1
  • 4
6

Here is another whay to upload a null (or o Byte) file to S3. I verified this works You can also use the S3 API to upload a file with no body, like so:

aws s3api put-object --bucket "myBucketName" --key "dir-1/my_null_file"

Normally you would specify a --body blob, but its option and will just add the key as expected. See more on S3 API put-object

The version of AWS CLI tested with is: aws-cli/2.0.4 Python/3.7.5 Windows/10 botocore/2.0.0dev8

Here's how I did it in PHP (even works in outdated 5.4, had to go way back):

// Init an S3Client
$awsConfig = $app->config('aws');
$aws       = Aws::factory($awsConfig);
$s3Bucket  = $app->config('S3_Bucket');
$s3Client  = $aws->get('s3');

// Set null/empty file.
$result = $s3Client->putObject([
    'Bucket' => $s3Bucket,
    'Key' => "dir-1/my_null_file",
    'Body' => '',
    'ServerSideEncryption' => 'AES256',
]);
b01
  • 4,076
  • 2
  • 30
  • 30
  • 1
    Finally, an answer that worked for me! Thank you! – akki Oct 12 '21 at 20:01
  • 1
    This doesn't actually answer the title question. `touch` should not change the contents of the file. It should only update the modification time if the file exists whereas this will truncate an existing file. – Andrew Pickin Oct 03 '22 at 15:24
  • Marked down as this doesn't answer the actual stated problem. – SamStephens Mar 14 '23 at 04:10
1

Following @g-io answer that simplified my day, here is another version of the same that makes it easy to touch a single file

s3-touch-single() {
  aws s3 cp \
    --metadata 'touched=touched' \
    "${@:3}" \
    "$1" "$1"
}

for example, looping an array of files we need to touch:

paths=("mydir/image.png" "mydir2/image2.png")
for i in "${paths[@]}"; do s3-touch-single "s3://my-bucket/$i"; done
Ben Yitzhaki
  • 1,376
  • 16
  • 31
  • Cool is there a way to do this with the java api ? – peterk Feb 17 '21 at 06:21
  • Q: Should the 4th line read "${@:2}" or is the second parameter for something I've missed? – Stormcloud Mar 05 '21 at 17:07
  • 1
    @Stormcloud honestly i can't remember why and it does look redundant when looking at the script now – Ben Yitzhaki Mar 07 '21 at 11:49
  • @peterk you could use the java sdk in a similar way (just using java instead of bash). it's probably using the same APIs at the backend for both. – Ben Yitzhaki Mar 07 '21 at 11:51
  • @BenYitzhaki Interesting I tried doing that that is changing a field in the metadata, then copying an object to itself with the new metadata. This does change the metadata but does NOT change the summary.getLastModified() value on subsequent access. – peterk Mar 08 '21 at 11:07
0

check out https://github.com/emdgroup/awscli-s3touch

It's a plugin to the AWS CLI that adds a touch command.

Usage:

aws s3 touch my-bucket --prefix myfolder/

It works by reading the events attached to the bucket and simulating them client side.

monken
  • 116
  • 1
  • 4
  • 1
    Reading the source code, that plugin is just triggering the events that should be triggered once the PUT operation occurs, without actually touching the files. Misleading repo name – Ben Yitzhaki Jun 09 '20 at 06:38