0

I have created an S3 bucket without public access in order to act as a shared folder with a directory of an EC2 instance.

I have assigned a role with a policy to the EC2 instance in order to be able to synchronise data with each other.

{
"Version": "2012-10-17",
"Statement": [
    {
        "Effect": "Allow",
        "Action": [
            "s3:PutObject",
            "s3:GetObject",
            "s3:DeleteObject"
        ],
        "Resource": "arn:aws:s3:::my-bucket-with-images/*"
    },
    {
        "Effect": "Allow",
        "Action": "s3:ListBucket",
        "Resource": "arn:aws:s3:::my-bucket-with-images"
    }
]

}

And, I am able to sync data between S3 bucket and EC2 and viceversa with the following command:

aws s3 sync s3://my-bucket-with-images /var/www/images

The problem is that I don't want a manual synchronisation, I want it to automatically detect new changes in S3 and apply them to the EC2 instance directory.

I know there are several threads on this forum about using a cron to run this command every so often but I am concerned about the CPU/MEM consumption and running the command too often without having to synchronise anything most of the time.

My question is, do you know of and/or is there a cleaner alternative even using other services if it was necessary?

user1911
  • 680
  • 1
  • 14
  • 36
  • 1
    *"I am concerned about the CPU/MEM"* - why? Have you performed any performance measurements? By far the most expensive part of the command is copying data, which will not hapern if there aren't any changes. Only listing S3 and listing your local file system takes a few network requests and is done. How much object are you planning on keeping in sync? 10s, 1000s, millions? – luk2302 Oct 07 '22 at 07:37
  • 1
    Why are you wanting the data locally on the EC2 disk volume? What are you actually wanting to achieve? Are you needing to use software that expects data to be on a locally-attached disk? Have you considered using s3fs to 'mount' the S3 volume instead of copying the data? Please tell us more about your actual requirements. – John Rotenstein Oct 07 '22 at 10:25

2 Answers2

0

SOLUTION:

Finally I found the best solution, S3FS-FUSE. A tool that allows you to have a folder as a shared volume with an S3 bucket instead of synchronising manually or with a cron. https://github.com/s3fs-fuse/s3fs-fuse

user1911
  • 680
  • 1
  • 14
  • 36
0

S3 notifications can help you there as well https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html. Its easy to subscribe and you can run the command then

Swapnil
  • 897
  • 7
  • 15