3

I got the following setup with AWS code deploy:

enter image description here

However it takes about 7 minutes from the moment I push a git branch remotely to the moment the deploy is completed. The time is broken down like so:

  • bitbucket deploy: 45 seconds

enter image description here

  • provisioning a new EC2 instance: 3:40 minutes

enter image description here

  • installing application on replacement instance: 40 seconds

enter image description here

  • 2 minutes rerouting traffic from old instance to new one *this one is extremely long, if it were done manually it would literally take 10 seconds)

  • 17 seconds terminating second instance

enter image description here

All together it's around 7 minutes.

How can I optimize this? I'm looking to shrink it to below 1 minute.

chicks
  • 1,911
  • 1
  • 13
  • 29
abbood
  • 473
  • 4
  • 13

3 Answers3

2

bitbucket deploy: 45 seconds

This seems correct. Not sure of your repo size, but if it is hosted outside of your region, that's not out of the ordinary.

bitbucket deploy: 45 seconds

Again, seems correct. Not much room for optimization here.

provisioning a new ec2 instance: 3:40 minutes

What size of instance are you provisioning? Are you provisioning one or more instances? The obvious solution would be to use in place deployment vs blue-green, but I think the extra time is well worth it. If this is for production/staging, I think you are fine. If this is in a testing environment, and you are aiming for quick feed back, you could consider provisioning new instances after a successful blue-green so that, if there is no AMI change, they are ready to go for installation immediately.

installing application on replacement instance: 40 seconds

Nothing to optimize here

2 minutes rerouting traffic from old instance to new one *this one is extremely long, if it were done manually it would literally take 10 seconds)

My guess is that step is performing extra steps that you do not perform when you manually reroute traffic. It's possible that CodeDeploy is performing extra checks to make sure that connections are draining, new connections are established before old ones are closed, and other test/checks. This is probably your best area to optimize, but not at the sacrifice of checks and balances.

17 seconds terminating second instance

Nothing to optimize here.

Summary: I don't think it is feasible or worthwhile to attempt to optimize your deployment further. The amount of effort it would take and sacrifices you would need to make in your test and checks far out ways the potential minute or two you would save in my opinion. I also don't think it is possible to achieve a <1 minute deployment. That would average ~15-20 seconds a step, which would require a lot of sacrifices in your deployment strategy.

If your goal is for quick iterations for QA, code testing, integration testing, etc., and you absolutely want sub 1 minute deployments, I recommend creating a Cloud Formation stack with your exact deployment except skipping the blue/green deployment model for an in place deployment model. This way, you can quickly see the results of your code. You should then later run your full suite blue/green deployment after your testing in either another testing environment or your staging environment.

Preston Martin
  • 3,288
  • 4
  • 18
  • 39
2

got this answer from aws support

Hello,

Andrew from AWS Support here.

I understand that you would like to know if there is a way to speed up Code Deploy as currently it takes 7 minutes to complete. Please correct me if needed.

After examining the link and the Code Deploy environment I can see that the step which involves rerouting traffic from the old instance to the new instance can potentially be tuned to try and decrease the overall deployment time. In addition, using a different deployment configuration may also help to decrease the deployment time.

To try and reduce the time taken to perform a blue/green deployment edit the health check on the Elastic Load Balancer (ELB). Reduce the health check interval from 30 seconds to 5 seconds and reduce the response timeout from 20 seconds to the default of 5 seconds [1]. This should decrease the time it takes the ELB to check whether an instance is healthy and ready to receive traffic.

Another setting that may also help decrease the time it takes to deploy is the deployment configuration, set the deployment configuration from the currently used "one at a time" to "all at once". CodeDeploy "all at once" will at once attempt to deploy to as many instances as possible, the application status will be shown as succeeded if the application is deployed successfully to one or more of the instances. This configuration will also reroute all the traffic in the placement group to all instances at once and will succeed if the traffic is successfully routed to at least once instance [2].

To reduce this deployment from 7 to 1 minute will be high unlikely as EC2 launches are managed by CodeDeploy, there may be ways to improve to EC2 launch times manually but this is not the case when using CodeDeploy.

I hope you find this information useful, if you have any other questions or need clarification on anything above please let me know, I'd be happy to help

Take care,

References: [1] Configure Health Checks for Your Classic Load Balancer - Health Check Configuration - https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-healthchecks.html#health-check-configuration

[2] Working with Deployment Configurations in AWS CodeDeploy - Predefined Deployment Configurations for an EC2/On-Premises Compute Platform - https://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-configurations.html#deployment-configurations-predefined

Best regards,

Andrew v. Amazon Web Services

abbood
  • 473
  • 4
  • 13
0

Depending on the value of a faster deployment in your situation, some options are:

  • Maintain spare capacity in your ECS cluster (either larger instance types, or more smaller ones) to allow for side-by-side deployment of new versions of the service along with the old, so that new tasks can be started without having to provision new instances. This obviously has a dollar cost, but again, depending on the value of rapid deployments to your organization, and the amount of deployments you expect to do per day, it might be worthwhile.

  • Use multiple sets of target groups for optimal Blue-Green deployments. This still requires extra cluster capacity, and extra work in your pipeline, but you can ensure that once tested, the replacement set of tasks all go live at once (and are fully tested).

Dan1701
  • 201
  • 1
  • 3