14

I am attempting to figure out a good way to push out a new commit to a group of EC2 server instances behind a ELB (load balancer). Each instance is running Nginx and PHP-FPM

I would like to perform the following workflow, but I am unsure of a good way to push out a new version to all instances behind the load balancer.

  • Dev is done on a local machine
  • Once changes are ready, I perform a "git push origin master" to push the changes to BitBucket (where I host all my git repos)
  • After being pushed to bitbucket, I would like to have the new version pushed out to all EC2 instances simultaneously.
  • I would like to do this without having to SSH in to each instance (obviously).

Is there a way to configure the remote servers to accept a remote push? Is there a better way to do this?

Jonathan Coe
  • 1,485
  • 4
  • 18
  • 36
  • Am I correct in assuming you have scale up/scale down scripts active? – thatidiotguy Dec 13 '12 at 21:49
  • Not currently. At the moment, I am really just using the load balancer for fail over protection and to simplify https configuration. In the future I would like to introduce automatic scaling to the system, but currently it is not a major requirement. – Jonathan Coe Dec 13 '12 at 21:53
  • I ask because with scale up/down you specify which AMI to load when your servers reach a predetermined level of stress. So let's say you made the rule, always have three servers. If you change the scripts to boot an instance from a new AMI (the one containing your new code), you could then remove old machines one by one and new machines would come up with your new code. This is a simple example of how the scale up/down can phase in new versions of your servers in a production environment. – thatidiotguy Dec 13 '12 at 21:54
  • @thatidiotguy hmm... that's actually a pretty interesting way to approach it - thanks! – Jonathan Coe Dec 13 '12 at 21:56

6 Answers6

10

Yes, I do this all of the time (with the same application stack, actually).

  1. Use a base AMI from a trusted source, such as the default "Amazon Linux" ones, or roll your own.

  2. As part of the launch configuration, use the "user data" field to bootstrap a provisioning process on boot. This can be as simple as a shell script that runs yum install nginx php-fpm -y and copies files down from a S3 bucket or do a pull from your repo. The Amazon-authored AMI's also include support for cloud-init scripts if you need a bit more flexibility. If you need even greater power, you can use a change management and orchestration tool like Puppet, Chef, or Salt (my personal favorite).

  3. As far as updating code on existing instances: there are two schools of thought:

    • Make full use of the cloud and just spin up an entirely new fleet of instances that grab the new code at boot. Then you flip the load balancer to point at the new fleet. It's instantaneous and gives you a really quick way to revert to the old fleet if something goes wrong. Hours (or days) later, you then spin down the old instances.
    • You can use a tool like Fabric or Capistrano to do a parallel "push" deployment to all the instances at once. This is generally just re-executing the same script that the servers ran at boot. Salt and Puppet's MCollective also provide similar functionality that mesh with their basic "pull" provisioning.
BenMorel
  • 34,448
  • 50
  • 182
  • 322
jamieb
  • 9,847
  • 14
  • 48
  • 63
  • 2
    Can you please elaborate more on the fleet part? I am a newbie. Does it mean subnets? Or do you mean, keep a record of new servers, deregister old instances and register new instances? – MavWolverine Oct 17 '13 at 15:58
2

Option one

  1. Push it to one machine.
  2. Have a git hook created on it http://git-scm.com/book/en/Customizing-Git-Git-Hooks.
  3. Make hook run pull on other machines.

Only problem , you'll have to maintain list of machines to run update on.

Another option

Have cron job to pull from your bitbucket account. on a regular base.

E_p
  • 3,136
  • 16
  • 28
2

The tool for this job is Capistrano.

I use an awesome gem called capistrano-ec2group in order to map capistrano roles with EC2 security groups.

This means that you only need to apply an EC2 security group (eg. app-web or app-db) to your instances in order for capistrano to know what to deploy to them.

This means you do not have to maintain a list of server IPs in your app.

The change to your workflow would be that instead of focussing on automating the deploy on pushing to bitbucket, you would push and then execute

cap deploy

If you really don't want to do to steps, make an alias :D

alias shipit=git push origin master && cap deploy
Chris Aitchison
  • 4,656
  • 1
  • 27
  • 43
2

This solution builds on E_p's idea. E_p says the problem is you'd need to maintain a server list somewhere in order to tell each server to pull the new update. If it was me, I'd just use tags in ec2 to help identify a group of servers (like "Role=WebServer" for example). That way you can just use the ec2 command line interface to list the instances and run the pull command on each of them.

for i in \
    `ec2din --filter "tag-value=WebServer" --region us-east-1 \
    | grep "running" \
    | cut -f17`\
; do ssh $i "cd /var/www/html && git pull origin"; done

Note: I've tested the code that fetches the ip addresses of all tagged instances and connects to them via ssh, but not the specific git pull command.

You need the amazon cli tools installed wherever you want this to run, as well as the ssh keys installed for the servers you're trying to update. Not sure what bitbucket's capabilities are but I'm guessing this code won't be able to run there. You'll either need to do as E_p suggests and push your updates to a separate management instance, and include this code in your post-commit hook, OR if you want to save the headache you could do as I've done and just install the CLI tools on your local machine and run it manually when you want to deploy the updates.

Credit to AdamK for his response to another question which made it easy to extract the ip address from the ec2din output and iterate over the results: How can I kill all my EC2 instances from the command line?

EC2 CLI Tools Reference: http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/Welcome.html

Community
  • 1
  • 1
Tom McQuarrie
  • 1,087
  • 9
  • 16
  • sounds like a great idea - are those commands still working? I have not yet worked with the cli tools and want to make sure it's not me causing the problem ;-) – Jörn Berkefeld Aug 26 '14 at 15:52
  • Yep should still work fine. aws documentation still references the "ec2-describe-instances" command (ec2din is the short version). http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DescribeInstances.html. If you are having problems with the command above, try breaking it down into its component parts and see where the issues lie. Its possible the ec2din output has changed slightly so "cut -f17" might have to change to something else to extract the ip address column. – Tom McQuarrie Aug 26 '14 at 21:20
  • 1
    Actually after running some tests, for some reason now I have to specify the region when running the ec2din function. I've updated the example above accordingly. – Tom McQuarrie Aug 26 '14 at 23:58
  • 1
    a slight improvement: run the above code from a dedicated EC2 that's only running during updates. This EC2 then will be the sole entity that can SSH into your production EC2s. That reduces the chance of anyone hacking into them. – Jörn Berkefeld Aug 27 '14 at 11:41
1

Your best bet might be to actually use AMI's for deployments.

Personally, I typically have a staging instance where I can pull any repo changes into. Once I have confirmed it is operating the way I want, I create an AMI from that instance.

For deployment, I use an autoscaling group behind the load balancer (doesn't need to be dynamically scaling or anything). In a simple set up where you have a fixed number of servers in the autoscale group (for example 10 instances). I would change the AMI associated with the autoscale group to the new AMI, then start terminating a few instances at a time in the autoscale group. So, say I have 10 instances and I terminate two to take it down to 8 instances. The autoscale group is configured to have a minimum of 10 instances, so it will automatically start up two new instances with the new AMI. You can then keep removing instances at whatever rate makes sense for your level of load, so as to not impact the performance of your fleet.

You can obviously do this manually, even without an autoscale group by directly adding/removing instances from the ELB as well.

If you are looking to make this fully automated (i.e. continuous deployment), then you might want to look at using a build system such as Jenkins, which would allow for a commit to kick off a build and then run the necessary AWS commands to make AMI's and deploy them.

Mike Brant
  • 70,514
  • 10
  • 99
  • 103
  • 1
    This isn't really a recommended way of doing it because even a minor code change would require creating a new AMI. Change management becomes a bit of a nightmare. The recommended strategy is to use a base AMI and then use the "user data" field to bootstrap a provisioning process. – jamieb Dec 14 '12 at 20:54
  • @jamieb I guess that really depends on your deployment approach and the amount of time you want to spend in the bootstrap process. If you are deploying changes frequently, I would agree that this process would become a bit cumbersome if you were doing everything manually (i.e. not automating with a build system). However if you deploy more infrequently, than the manual time to do this approach is not really all that much. – Mike Brant Dec 14 '12 at 21:04
  • 1
    @jamieb Now if you have a large codebase that would take a while to deploy, such that your server can get running, or you have a lot of dependency on order in bootstrap process such that your cloudinit (or similar) scripts become overcomplex or fragile, then deploying a ready to go AMI might make more sense to you. This may also be more important during upscaling events, where the time it takes to get your instance responding to ELB health checks could be important depending on your traffic profile. – Mike Brant Dec 14 '12 at 21:07
  • drop/setup new machines each time you deploy? Github/Etsy/other deploy 10-20 times a day. So do you advice to drop/setup hundred servers 10 times a day? What about deployment rollback? – Anatoly Mar 06 '13 at 20:45
  • @mikhailov With that sort of deployment workload, you would probably need to look at a more robust deployment system (i.e. not have a build system create AMI's at every code commit). You can certainly still use AMI's as the basis for deployment, you would just need the deployment system to be able to rollback to previous (stable) AMI's. – Mike Brant Mar 06 '13 at 21:51
  • @MikeBrant rollback to previous system state by drop/setup servers (via AMI or not)? Do you really think it's a good idea? – Anatoly Mar 06 '13 at 22:25
  • @mikhailov I would think there are a lot of factors to consider. In some cases it might be faster to start up a new instance based on a stable AMI than it would be to remove an instance from a load balancer, clean up any temp files (local filesystem caches, etc.), deploy a new build, then bring the updated instance back into the load balancer. You also might need to maintain a minimum number of active servers in the load balancer during any deploy/rollback process. This is easy to achieve using autoscaling and removing instances. I think each application certainly would have different needs. – Mike Brant Mar 06 '13 at 23:24
  • @MikeBrant If we talk about **typical tasks**, all of them have **typical solutions**, not easy often, but proven enough. I don't hear about deployment that requires any servers reboot/stop/create unless pretty major changes. But it works well for you, there is no point to change it – Anatoly Mar 06 '13 at 23:35
0

I am looking for a solution to the same problem. I came across this post and thought it was an interesting approach

https://gist.github.com/Nilpo/8ed5e44be00d6cf21f22#pc

Go to "Pushing Changes To Multiple Servers"

Basically the idea is to create another remote call it "production" or whatever you want, and then add multiple urls (The ip's of all of the servers) to that remote. This can be done by editing .git/config

Then you can run git push production <branch> and it should push out to all of the urls listed under "production"

One requirement to this approach is that the repo on the servers need to be bare repos and you will need to have a post-receive hook to update the working tree.

Here is an example of how to do that: Setting up post-receive hook for bare repo

Shawn Northrop
  • 5,826
  • 6
  • 42
  • 80