5

We're currently using Capifony and the ec2-capify plugin to do rolling deployments of our code to a set of instances behind an ELB. We also use CloudFront to manage static assets, which we version with a query string (e.g. ?v1 or ?v2).

We've stumbled across a rare issue regarding updating the asset version. If the current version is v1, and we do a rolling deployment of v2 one server at a time then the following can happen for requests on the v2 box:

  1. CloudFront is asked for v2 and misses.
  2. CloudFront goes to the ELB and asks for the asset.
  3. The ELB chooses a server and one of two things happens: Cloudfront hits one of the new deployed servers (serving v2) OR it hits the old servers (v1).
  4. Either way, Cloudfront stores the content as v2. In the case where it hit a v1 server, the content is served incorrectly.

Our current solution to this is we have to do another deploy with a new asset version.

Is there a way to force Cloudfront through the ELB to only hit one of our updated (v2) servers, and to ignore the v1's?

Or am I missing an alternate solution which would resolve the problem?

edhgoose
  • 887
  • 8
  • 26

3 Answers3

1

I think proper deployment strategy in your case would be to first deploy instances that are able to serve both v1 and v2 assets (but still serving v1), and then doing another rolling deployment to switch to v2.

Also there is 'sticky sessions' available on ELB (http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/US_StickySessions.html), but I don't see how this could be used here - per-viewer cookies will eliminate benefit of CloudFront caching

Alex Z
  • 1,867
  • 1
  • 29
  • 27
1

The approach we opted for was to broadly abandon our existing asset deployment pipeline. In the "old" way, we opted for an asset.css?v=<version> model, where CloudFront was pointed at an origin which was served by multiple instances.

The way we resolved it was to move to a hash name asset model and S3 based origin. This means that instead of asset.css?v=<version> we have asset-<hash-of-contents>.css which are synced to an S3 bucket. The bucket progressively adds newer and newer versions, but old versions are always available if we decided to go back or if something like an email links to it (common problem with images).

The sync to S3 script runs before we deploy to our webservers that contain the HTML that references the assets, so CloudFront is always able to serve the latest asset.

Here's a sample script:

#!/usr/bin/env bash

set -e # Fail on any error
set -x

if [ "$#" -ne "1" ]
then
    echo "Usage: call with the name of the environment you're deploying to"
    exit 1
fi

CDN_ENVIRONMENT=$1

S3_BUCKET="s3://static-bucket-name-of-your-choice/${CDN_ENVIRONMENT}/"

echo "Generating assets

... do the asset generation here ...

echo "Copying to S3"

# Now do the actual copying of the web dir. We use size-only because otherwise all files are newer, and all get copied.
aws s3 sync --exclude "some-folder-to-exclude/*" --acl public-read --size-only ./web/ ${S3_BUCKET}

echo "Copy to S3 complete"
edhgoose
  • 887
  • 8
  • 26
0

When Cloudfront gets a 404 from your origin (presumably because it has yet to receive the new build), it caches that 404 for 5 minutes. You can alter that behavior by creating a new "Custom Error Response" for your distribution. The custom response lets you set a very low TTL, so that Cloudfront will passing through to your ELB until it finds the new file. Downside of this is that Cloudfront will effectively no longer cache 404s - your ELB will need to handle that load (which is hopefully small!)

http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/HTTPStatusCodes.html#HTTPStatusCodes-no-custom-error-pages

Adam Rabung
  • 5,232
  • 2
  • 27
  • 42