How to reduce AWS S3 data transfer costs?

Question

I have a AWS S3 bucket to store a combination of public and private files for a laravel app as follows:

AWS Bucket and folders:

myapp
    |____private
    |____public

In my blade files the images are displayed as follows:

<img src="http://myapp.com/files/10"/>

This is my route:

Route::get('files/{id}', 'FileController@show');

This is my controller method to show and upload files:

public function show(Request $request, $id)
{
    $file = \App\File::findOrFail($id);

    $user = auth()->check() ? auth()->user() : new User;

    $this->authorizeForUser($user, 'show', $file);

    return Image::make(Storage::disk('s3')->get($file->path))->response();

} 


public function store(FileRequest $request)
{

    $file = $request->file('file');
    $fileType = strtolower($request->input('file_type'));
    $filePath = $fileType . '/' . Str::random(40) . '.' .  $file->getClientOriginalExtension();
   
    $user = auth()->user();

    DB::beginTransaction();

    try
    {
        $this->authorizeForUser($user, 'store', \App\File::class);

        $newFile = \App\File::create([
               "name"          => $file->getClientOriginalName(),
               "path"          => $filePath,
               "size"          => $file->getSize(),
               "mime_type"     => $file->getMimeType(),
               "type"          => $fileType, 
               "user_id"       => $user->id
        ]);

        $image = Image::make($file->getRealPath())->resize(360, 180);
        
        Storage::disk('s3')->put($filePath, $image->stream());
       
        $content = "<textarea data-type=\"application/json\">{\"ok\": true, \"message\": \"Success\", \"path\": \"" . $newFile->path . "\",  \"id\": \"" . $newFile->id . "\",  \"name\": \"" . $newFile->name . "\" }</textarea>";

        DB::commit();

    } catch (\Exception $e)
    {
        DB::rollback();

        $error =  'There was a problem uploading your file';
        $content = "<textarea data-type=\"application/json\">{\"ok\": false, \"message\": \"" . $error . "\" }</textarea>";

    }

    return response($content);

}

This is the file policy where authenticated users can access both public and private files whereas guest users can only access public files:

public function show(User $user, File $file)
{
     if ($file->type == 'public' || (!is_null($user->getKey()) && $file->type == 'private'))
          return true;
     }
}

I want to add cache support because even small number of app users produce hundreds and sometimes thouthands of GET requests on AWS S3. How can I cache files to reduce S3 costs where the bucket is private i.e. can only be accessed via an IAM user?

I've read some posts that recommend using cloudfront but isn't that just adding another cost and how can I implement cloudfront with the controller method I have above and where the bucket is private at root level?

Would using laravels file cache be a suitable option?

I'd be interested to know what techniques I can use with S3 and laravel, preferably with some steps and code for guidance would really help.

PS. I've already looked at the following post. Laravel AWS S3 Storage image cache

@Parsifal where would i store the files, on cloudflare? Plus there no outof the box laravel api to use cloudflare as storage? — adam78, Aug 12 '20 at 13:24
CloudFlare is a CDN: it lives outside your application, and caches the files that have been delivered. When the user first retrieves the file, it comes from your site. When that user retrieves the file again, it comes from the Cloudflare cache. — Parsifal, Aug 12 '20 at 17:41
In your case, it might not be appropriate because you're authorizing all image requests. You can certainly make them cacheable on a per-user basis by changing the filename to include a user token, but that only helps you if the same user loads the image multiple times (and in that case browser-based caching would do the job). — Parsifal, Aug 12 '20 at 17:42

score 1 · Answer 1 · answered Aug 12 '20 at 12:16

1

The easiest option is to use CloudFront and if your problem is the cost, AWS S3 doesn't charge Data Transfer when the request goes to CloudFront.

It's very to set up CloudFront with S3, and you can keep the Bucket private, you'll just need to create OAI.

Check these links below:

https://aws.amazon.com/pt/premiumsupport/knowledge-center/cloudfront-https-requests-s3/

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3.html

As the CloudFront is a public service, you'll need to change your current authentication to another format (check this link cloudfront-authorization-at-edge) or you can serve private content using Signed URLs or Signed Cookies (check this another link PrivateContent); and you can use your existing IAM user to generate these Signed URLs/Signed Cookies.

answered Aug 12 '20 at 12:16

Raul Barreto

979
5
9

1

S3 doesn't charge for data transfer to CloudFront, but CloudFront charges for data transfer to the Internet. However, it is slightly less expensive ($0.085 in us-east-1 for CloudFront, versus $0.09 for S3). – Parsifal Aug 12 '20 at 12:53
Yes, but it's a little less expensive but when you start to hit cache in CloudFront it will reduce a lot of your costs. Besides this, you will also have a lower latency with CloudFront. – Raul Barreto Aug 12 '20 at 13:00
2

It will reduce your costs by $0.005 per GB. That's not "a lot". – Parsifal Aug 12 '20 at 13:01
@Raul in my controller I have `Image::make(Storage::disk('s3')->get($file->path))->response()` to get the file from S3. If I'm using cloudfront how would I get the file? – adam78 Aug 13 '20 at 06:27
@adam78 I don't know what language you're using but if Storage::disk('s3') means to have like an s3 file system, you'll have to change the request to an HTTP(S) request. – Raul Barreto Aug 13 '20 at 12:06
@RaulBarreto I'm using laravel. You mean getting it from the cloud front url something like this: `response()->file($filePath);` where `$filePath` is something like: `https://[your-distribution].cloudfront.net/your-asset.png` – adam78 Aug 17 '20 at 08:07

How to reduce AWS S3 data transfer costs?

1 Answers1