getting "NoSuchKey: The specified key does not exist." when trying to copyObject on a folder that I can prove exists

Question

To bottom line what I am trying to accomplish, I am trying to copy a folder in order to add tags to that folder, according to the docs evidently it's impossible to add (or edit tags) on folders and objects that already exist. In my app I need to delete folders full of attachments after a certain status of a record is set. So when the status changes in my app I need to update the tag in S3 so that the S3 management rules can kick in after 30 days and delete the folder.

I have read the javascript related post here, but I don't believe my issue is related

I am able to successfully make a folder using s3.putObject

await s3.putObject({
  Bucket: process.env.BUCKET,
  Key: '611d308b6ef5e4066e047824/'
}).promise()

I am able to confirm that the folder was created using s3.listObjectsV2

const data = await s3.listObjectsV2({
  Bucket: process.env.BUCKET,
  Delimiter: '/',
  Prefix: '611d308b6ef5e4066e047824' // <-- no slash
}).promise()

console.log(data)
// {
//    IsTruncated: false,
//    Contents: [ [Object] ],
//    Name: '...', // <-- bucket name
//    Prefix: '611d308b6ef5e4066e047824/',
//    Delimiter: '/',
//    MaxKeys: 1000,
//    CommonPrefixes: [],
//    KeyCount: 1
// }

I am also able to confirm the files are uploaded (im using multer/multer-s3 in express for this), but using s3.listObjectsV2 they are visible under Contents

const data = await s3.listObjectsV2({
  Bucket: process.env.BUCKET,
  Delimiter: '/',
  Prefix: '611d308b6ef5e4066e047824/' // <-- with a slash
}).promise()

console.log(data)

// {
//   IsTruncated: false,
//   Contents: [
//     {
//       Key: '611d308b6ef5e4066e047824/p19179.jpg',
//       LastModified: '2021-09-02T21:06:09.000Z',
//       ETag: '\"53e6ca3e139f171d486527bd7bcd16d4\"',
//       Size: 10154,
//       StorageClass: 'STANDARD'
//     }
//   ],
//   Name: '...', // <-- bucket name
//   Prefix: '611d308b6ef5e4066e047824/',
//   Delimiter: '/',
//   MaxKeys: 10,
//   CommonPrefixes: [],
//   KeyCount: 1,
//   StartAfter: '611d308b6ef5e4066e047824/0'
// }

The error appears when I try to run s3.copyObject, im adding tags here because Multer and Multer-S3 do not support adding tags, so it has to be accomplished via s3.copyObject after the fact

const data = await s3.copyObject({
  Key: '611d308b6ef5e4066e047824/',
  // Key: encodeURI('611d308b6ef5e4066e047824/'), <-- i have also tried this unsuccessfully
  // Key: '611d308b6ef5e4066e047824', <-- and this
  Tagging: 'myKey=myVal',
  Metadata: {
    myKey: 'myVal'
  },
  CopySource: `${process.env.BUCKET}/611d308b6ef5e4066e047824/`,
  // CopySource: `${process.env.BUCKET}/611d308b6ef5e4066e047824`, // <-- this does not work either
  Bucket: process.env.BUCKET,
  MetadataDirective: 'REPLACE',
  TaggingDirective: 'REPLACE'
}).promise()

below is my error stack from my application

  app:lib:aws:s3:folders:updateOptions FATAL The specified key does not exist. +0ms
  app:lib:aws:s3:folders:updateOptions FATAL NoSuchKey: The specified key does not exist.
  app:lib:aws:s3:folders:updateOptions     at Request.extractError (/Users/aronlilland/Documents/dev/controlair/fileManager/node_modules/aws-sdk/lib/services/s3.js:714:35)
  app:lib:aws:s3:folders:updateOptions     at Request.callListeners (/Users/aronlilland/Documents/dev/controlair/fileManager/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
  app:lib:aws:s3:folders:updateOptions     at Request.emit (/Users/aronlilland/Documents/dev/controlair/fileManager/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
  app:lib:aws:s3:folders:updateOptions     at Request.emit (/Users/aronlilland/Documents/dev/controlair/fileManager/node_modules/aws-sdk/lib/request.js:688:14)
  app:lib:aws:s3:folders:updateOptions     at Request.transition (/Users/aronlilland/Documents/dev/controlair/fileManager/node_modules/aws-sdk/lib/request.js:22:10)
  app:lib:aws:s3:folders:updateOptions     at AcceptorStateMachine.runTo (/Users/aronlilland/Documents/dev/controlair/fileManager/node_modules/aws-sdk/lib/state_machine.js:14:12)
  app:lib:aws:s3:folders:updateOptions     at /Users/aronlilland/Documents/dev/controlair/fileManager/node_modules/aws-sdk/lib/state_machine.js:26:10
  app:lib:aws:s3:folders:updateOptions     at Request.<anonymous> (/Users/aronlilland/Documents/dev/controlair/fileManager/node_modules/aws-sdk/lib/request.js:38:9)
  app:lib:aws:s3:folders:updateOptions     at Request.<anonymous> (/Users/aronlilland/Documents/dev/controlair/fileManager/node_modules/aws-sdk/lib/request.js:690:12)
  app:lib:aws:s3:folders:updateOptions     at Request.callListeners (/Users/aronlilland/Documents/dev/controlair/fileManager/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
  app:lib:aws:s3:folders:updateOptions     at Request.emit (/Users/aronlilland/Documents/dev/controlair/fileManager/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
  app:lib:aws:s3:folders:updateOptions     at Request.emit (/Users/aronlilland/Documents/dev/controlair/fileManager/node_modules/aws-sdk/lib/request.js:688:14)
  app:lib:aws:s3:folders:updateOptions     at Request.transition (/Users/aronlilland/Documents/dev/controlair/fileManager/node_modules/aws-sdk/lib/request.js:22:10)
  app:lib:aws:s3:folders:updateOptions     at AcceptorStateMachine.runTo (/Users/aronlilland/Documents/dev/controlair/fileManager/node_modules/aws-sdk/lib/state_machine.js:14:12)
  app:lib:aws:s3:folders:updateOptions     at /Users/aronlilland/Documents/dev/controlair/fileManager/node_modules/aws-sdk/lib/state_machine.js:26:10
  app:lib:aws:s3:folders:updateOptions     at Request.<anonymous> (/Users/aronlilland/Documents/dev/controlair/fileManager/node_modules/aws-sdk/lib/request.js:38:9) +0ms

So this error is incorrect, it does exist - whats more, I am able to successfully run s3.copyObject on some other test records, but the information above is representative of my folder names with live data, I have no idea what to do next ‍♂️

Sorry, but what are you actually wanting to accomplish? Are you trying to copy a 'folder'? If so, _why?_ — John Rotenstein, Sep 02 '21 at 23:50
trying to copy folder in order to add tags, according to the docs evidently its impossible to add (or edit tags) on items that already exist. In my app I need to delete attachments after a certain status of a record is set. So when the status changes I need to update the tag so the S3 management rules can kick in after 30 days and delete the folder. — alilland, Sep 02 '21 at 23:52
Does this answer your question? [How to copy/move all objects in Amazon S3 from one prefix to other using the AWS SDK for Node.js](https://stackoverflow.com/questions/30959251/how-to-copy-move-all-objects-in-amazon-s3-from-one-prefix-to-other-using-the-aws) — jscott, Sep 03 '21 at 00:37
Generally speaking, S3 doesn't have folders, it has objects, and APIs to list objects that start with a common prefix. You can't copy a "folder", you can copy a file, then another file, and so on. And rather than copying files, why not just use `putObjectTagging` from the AWS SDK — Anon Coward, Sep 03 '21 at 00:38
@jscott unfortunately im not looking to move/change prefix. @AnonCoward `putObjectTagging` looks intriguing, I will have to give it a shot tomorrow — alilland, Sep 03 '21 at 01:34

score 0 · Answer 1 · answered Sep 03 '21 at 04:26

0

Amazon S3 folders don't actually exist. When using the Create folder button in the S3 Management Console, it actually creates a zero-length object that 'forces' the folder to appear.

It is not possible to "copy a folder". Well, you can copy the zero-length object, but it doesn't copy anything inside the folder since objects don't actually exist inside a hierarchy.

If your goal is to add or edit tags on an object, there is no need to copy anything. You can add tags with:

AWS CLI: aws s3api put-object-tagging
Python boto3: put_object_tagging

answered Sep 03 '21 at 04:26

John Rotenstein

241,921
22
380
470

this works for updating objects, I need the zero-length object to be deleted along with the contents by the S3 management lifecycle rules, I dont think this will be the final solution, if i try to call `putObjectTagging` on the "folder" it returns the same error as before that the key cannot be found – alilland Sep 03 '21 at 16:33
I recommend that you **do not create the zero-length 'folder' object**. There is no need to create' folders' in Amazon S3. You can simply put objects in any location. Is there a particular reason you need to create the zero-length object? – John Rotenstein Sep 03 '21 at 21:32

getting "NoSuchKey: The specified key does not exist." when trying to copyObject on a folder that I can prove exists

1 Answers1