Undelete folders from AWS S3

Question

I have a S3 bucket with versioning enabled. It is possible to undelete files, but how can I undelete folders?

I know, S3 does not have folders... but how can I undelete common prefixes? Is there a possibility to undelete files recursively?

Answer 1

I created this simple bash script to restore all the files in an S3 folder I deleted:

#!/bin/bash
    
recoverfiles=$(aws s3api list-object-versions --bucket MyBucketName  --prefix TheDeletedFolder/ --query "DeleteMarkers[?IsLatest && starts_with(LastModified,'yyyy-mm-dd')].{Key:Key,VersionId:VersionId}")
for row in  $(echo "${recoverfiles}" | jq -c '.[]'); do
    key=$(echo "${row}" | jq -r '.Key'  )
    versionId=$(echo "${row}" | jq -r '.VersionId'  )
    echo aws s3api delete-object --bucket MyBucketName --key $key --version-id $versionId
done

yyyy-mm-dd = the date the folder was deleted

Answer 2

I found a satisfying solution here, which is described in more details here.

To sum up, there is no out-of-the-box tool for this, but a simple bash script wraps the AWS tool "s3api" to achieve the recursive undelete.

The solution worked for me. The only drawback I found is, that Amazon seems to throttle the restore operations after about 30.000 files.

Answer 3

You cannot undelete a common prefix. You would need to undelete one object at a time. When an object appears, any associated folder will also reappear.

Undeleting can be accomplished in two ways:

• Delete the Delete Marker that will reverse the deletion, or
• Copy a previous version of the object to itself, which will make the newest version newer than the Delete Marker, so it will reappear. (I hope you understood that!)

Answer 4

If a folder and its contents are deleted you can recover them using the below script inspired by a previous answer

The script is applicable to an S3 bucket where versioning is enabeled before hand. It uses the delete marker tag to restore files in an S3 prefix.

#!/bin/bash
# Inspired by https://www.dmuth.org/how-to-undelete-files-in-amazon-s3/

# This script can be used to undelete objects from an S3 bucket.
# When run, it will print out a list of AWS commands to undelete files, which you
# can then pipe into Bash.
#

#
# You will need the AWS CLI tool from https://aws.amazon.com/cli/ in order to run this script.
#
# Note that you must have the following permissions via IAM:
#
# Bucket permissions:
#
#   s3:ListBucket
#   s3:ListBucketVersions
#
# File permissions:
#
#   s3:PutObject
#   s3:GetObject
#   s3:DeleteObject
#   s3:DeleteObjectVersion
#
# If you want to do this in a "quick and dirty manner", you could just grant s3:* to
# the account, but I don't really recommend that.
#
                
# profile = company
# bucket = company-s3-bucket
# prefix = directory1/directory2/directory3/lastdirectory/
# pattern = (.*)

# USAGE
# bash undelete.sh  > recover_files.txt  | bash

read -p "Enter your aws  profile: " PROFILE
read -p "Enter your S3 bucket name: " BUCKET
read -p "Enter your S3 directory/prefix to be recovered from, leave empty for to recover all of the S3 bucket: " PREFIX
read -p "Enter the file pattern looking to recover, leave empty for all: " PATTERN

# Make sure Profile and Bucket are entered
[[ -z "$PROFILE" ]] && { echo "Profile is empty" ; exit 1; }
[[ -z "$BUCKET" ]] && { echo "Bucket is empty" ; exit 1; }

# Fill PATTERN to match all if empty
PATTERN=${PATTERN:-(.*)}

# Errors are fatal
set -e


if [ "$PREFIX" = "" ]; 

# To recover all of the S3 bucket
then

    aws --profile ${PROFILE} --output text s3api list-object-versions --bucket ${BUCKET} \
            | grep -i $PATTERN \
            | grep -E "^DELETEMARKERS" \
            | awk -v PROFILE=$PROFILE -v BUCKET=$BUCKET -v PREFIX=$PREFIX  \
            -F "[\t]+" '{ print "aws --profile " PROFILE " s3api delete-object --bucket " BUCKET "--key \""$3"\" --version-id "$5";"}' 


# To recover a directory
else

    aws --profile ${PROFILE} --output text s3api list-object-versions --bucket ${BUCKET} --prefix ${PREFIX} \
            | grep -E $PATTERN \
            | grep -E "^DELETEMARKERS" \
            | awk -v PROFILE=$PROFILE -v BUCKET=$BUCKET -v PREFIX=$PREFIX  \
            -F "[\t]+" '{ print "aws --profile " PROFILE " s3api delete-object --bucket " BUCKET "--key \""$3"\" --version-id "$5";"}' 
fi

Answer 5

If you enabled the version for the bucket, Use the below command to retrieve the files or folder.

For folder with respective: -

echo '#!/bin/bash' > undeleteScript.sh && aws --output text s3api list-object-versions --bucket bucketname --prefix path/to/retrieve| grep -E "^DELETEMARKERS" | awk '{FS = "[\t]+"; print "aws s3api delete-object --bucket buckername --key \42"$3"\42 --version-id "$5";"}' >> undeleteScript.sh && . undeleteScript.sh; rm -f undeleteScript.sh;

For Files in the Bucket: -

echo '#!/bin/bash' > undeleteScript.sh && aws --output text s3api list-object-versions --bucket bucketname --prefix | grep -E "^DELETEMARKERS" | awk '{FS = "[\t]+"; print "aws s3api delete-object --bucket buckername --key \42"$3"\42 --version-id "$5";"}' >> undeleteScript.sh && . undeleteScript.sh; rm -f undeleteScript.sh;

Answer 6

I wrote an optimize version to recover in batch of 1000 files:

#!/bin/bash

if [[ "$#" -lt 3 ]]; then
    echo "Run: $0 <BUCKET> <PREFIX> <PROFILE>"
    exit 1
fi

BUCKET=$1
PREFIX=$2
PROFILE=$3

echo "Restore bucket PREFIX - ${BUCKET}/${PREFIX}"

while true; do
  result="$(aws --profile ${PROFILE} s3api list-object-versions --max-items 1000 --bucket ${BUCKET} --prefix ${PREFIX} --query '{Objects: DeleteMarkers[0:999].{Key:Key,VersionId:VersionId}}')"
  
  if [ "$(echo $result|jq '.Objects')" == "null" ]; then
    echo "No more files to undelete."
    break
  fi
  
  echo "Restoring:"
  aws --profile ${PROFILE} --no-cli-pager s3api delete-objects --bucket ${BUCKET} --delete "$result" |grep Key
done