9

As part of our blue-green deployment strategy we are snapshoting the prod RDS instance and then restoring this snapshot into a new instance applying db migrations after it and linking the newly Green application to it.

Our RDS instance has a 100 GB space, but our DB uses only 10 MB at the moment.

Taking a snapshot takes roughly < 2 minutes

Restoring from the Snapshot takes 25 minutes!

25 minutes for the restore is too long considering users are forced to stay in read only mode for all this period and that our DB size is less than 10 mb at the moment.

I am wondering if this restore time is the usual time for Amazon RDS or if we are doing something wrong.

  • Amazon RDS Postgres.
  • Multi AZ: Yes
  • Instance Class: Medium
  • General Purpose (SSD)
  • IOPS: disabled.
user1990009
  • 855
  • 1
  • 7
  • 16
  • 2
    What instance class are you using? SSD storage? Is it Multi-AZ? – Matt Houser Feb 18 '16 at 20:00
  • This isn't going to be particularly great or scalable if you're expecting your database to grow. Have you considered any alternative solutions, like promoting read replicas to master or AWS database migration service? – Anthony Neace Feb 18 '16 at 22:33
  • @MattHouser: I updated the post with answers to your questions – user1990009 Feb 19 '16 at 00:30

1 Answers1

10

After some experimentation we were able to reduce the restoring time from 25 minutes to 5 minutes. This was due to the fact, that RDS first tries to restores the snapshot. (In our case this took 5 minutes). And afterwards it applied the Multi Az change to the new instance. (this was taking like 20 minutes)

Previously we were waiting for the DB to finish the MULTI AZ change, and status="available" to continue with our Deployment, but after contacting AWS, they have confirmed that is safe to start using the new instance even when the instance is being modified to apply the MULTI AZ change. So we continue our deployment process as soon as the restored instance status change from "creating" to "modifying"

This solution as correctly said, might not scale very well but at the moment this is not a concern as we are not expecting this DB to grow significantly.

We consider this approach to be very safe, as any DB schema changes wont affect the live DB and we can safely test the whole GREEN stack before switching to PROD. The only caveat here is that the application need to be in read-only mode, so as not to loose information between the blue and green environments

user1990009
  • 855
  • 1
  • 7
  • 16