Perform blue green deployment on AWS ECS using CodePipeline and CodeDeploy

In today’s fast-paced software development landscape, deploying changes to production while ensuring minimal downtime and risk is paramount. There are many strategies which can be used to perform a successful minimal downtime deployment. One of the strategies to achieve this is by using blue-green deployments. Blue-green deployments are a deployment strategy where you have two identical environments, one is the production environment (blue) and the other is the staging environment (green). When you want to deploy a new version of your application, you deploy it to the green environment. Once the deployment is successful and the application is tested, you switch the traffic from the blue environment to the green environment. That was a very high level of what a blue green deployment is. This strategy helps in reducing the risk of deploying changes to production and also helps in minimizing downtime.

In this post, I will explain how to perform blue-green deployments on ECS using CodePipeline and CodeDeploy. Leveraging AWS CodePipeline, a continuous integration and delivery service, in conjunction with AWS CodeDeploy, a fully managed deploy service, we’ll walk you through the step-by-step process of automating the deployment of your containerized applications with confidence and efficiency.

The GitHub repo for this post can be found Here. If you want to follow along, the repo can be cloned and the code files can be used to stand up your own infrastructure.

Pre Requisites

Before I start the walkthrough, there are some some pre-requisites which are good to have if you want to follow along or want to try this on your own:

• Basic AWS knowledge
• Github account to follow along with Github actions
• An AWS account
• AWS CLI installed and configured
• Basic Terraform knowledge
• Terraform Cloud free account. You can sign up here. Its not mandatory but I have used this for state management in the Terraform scripts. You can use S3 as well.

With that out of the way, lets dive into the details.

What is Blue-Green Deployment?

Blue-green deployment is a deployment strategy that helps in reducing the risk of deploying changes to production and also helps in minimizing downtime. This is a very useful strategy when you want to deploy changes to production without any downtime. In this strategy you have two identical environments, one is the production environment (blue) and the other is the staging environment (green). While one environment serves live production traffic, the other remains inactive. The deployment happens to green environment first and then traffic gradually shifts from blue. This ensures there is no downtime and the new version of the application is tested thoroughly before switching the traffic. Also it helps in rolling back to the previous version of the application easily.
Some of the key benefits of blue-green deployments are:

• Reduced risk: Blue-green deployments help in reducing the risk of deploying changes to production. Since the new version of the application is deployed to the staging environment, it can be tested thoroughly before switching the traffic.
• Minimal downtime: Blue-green deployments help in minimizing downtime. Since the new version of the application is deployed to the staging environment, the traffic can be switched to the staging environment once the deployment is successful.
• Rollback: Blue-green deployments help in rolling back to the previous version of the application easily. If the new version of the application has issues, the traffic can be switched back to the production environment.
• Testing: Blue-green deployments help in testing the new version of the application thoroughly before switching the traffic. Since the new version of the application is deployed to the staging environment, it can be tested thoroughly before switching the traffic.

What is AWS CodePipeline and CodeDeploy?

AWS CodePipeline and Codedeploy are two integral services provided by Amazon Web Services (AWS) that enable automated software delivery and continuous integration/continuous deployment (CI/CD) pipelines. AWS CodePipeline is a continuous integration and continuous delivery service that automates the build, test, and deploy phases of your release process every time there is a code change, based on the release model you define. AWS CodeDeploy is a fully managed deployment service that automates software deployments to a variety of compute services such as Amazon EC2, AWS Fargate, AWS Lambda, and your on-premises servers. CodeDeploy provides a capability where you can perform blue-green deployments on ECS by controlling the traffic shifting between the two environments. We will cover more about this in the upcoming sections.

Understanding the sample application

First let’s understand the sample application which we will be deploying using blue-green deployments. Below image shows what the sample application does.

The sample application is a simple html page hosted on nginx server. The application is containerized using Docker and the Docker image is pushed to Amazon Elastic Container Registry (ECR). The application is deployed to Amazon Elastic Container Service (ECS) using Fargate. The application is first deployed to the blue environment. When any new deployment is to be released, a new green environment is created and the new version of the application is deployed to the green environment. At that point traffic is gradually switched from the blue environment to the green environment. So user is served traffic from both of the environments and ensured that the new green environment is stable. Gradually the traffic is increased to the green environment and once the green environment is stable, the traffic is switched completely to the green environment. The blue environment is then deleted. Overall these are the steps which are followed to deploy the application using blue-green deployments:

• The application is containerized using Docker and the Docker image is pushed to Amazon Elastic Container Registry (ECR).
• The application is deployed to Amazon Elastic Container Service (ECS) using Fargate.
• The application is deployed to the blue environment.
• When any new deployment is to be released, a new green environment is created and the new version of the application is deployed to the green environment.
• At that point traffic is gradually switched from the blue environment to the green environment. CodeDeploy helps in controlling the traffic shifting between the two environments.
• User is served traffic from both of the environments and ensured that the new green environment is stable.
• Once the green environment is stable, the traffic is switched completely to the green environment. The blue environment is then deleted.

Overall Tech infrastructure

Now lets understand the overall tech infrastructure which we will be using to deploy the application using blue-green deployments. The infrastructure can be classified into two categories:

• Application Infrastructure: This is the infrastructure which is used for hosting and running the sample application. This will be the actual infrastructure where the application will be deployed and accessed
• CI/CD Infrastructure: This is the infrastructure which is used for automating the deployment of the application. This infrastructure will be used for setting up the CI/CD pipeline which will automate the deployment of the application and control the blue-green deployment.
  Lets go through each of the infrastructure in detail.

Application Infrastructure

Below image shows the application infrastructure which will be used for hosting and running the sample application.

Networking

This includes of the networking needed for the application. Some of the components which are part of the architecture are:

• VPC: The Virtual Private Cloud (VPC) is the networking layer for the application. The VPC is used to isolate the application from other resources in the AWS cloud.
• Subnets: The subnets are used to divide the VPC into multiple networks. The subnets are used to deploy the application in multiple availability zones.
• Internet Gateway: The Internet Gateway is used to provide internet access to the application. The Internet Gateway is attached to the VPC.
  There are other components like Route tables, Security groups etc which are also deployed as part of the networking.

App Load Balancer
This is a very important component of the stack. The load balancer provides the endpoint through which the app can be accessed. It routes the traffic to the app which is running as Fargate tasks in ECS. Most important function of this load balancer is to route the traffic between the blue and green environments. When the switchover is happening between blue and green environments, this load balancer handles and balances the traffic flow to both of the environments. Based on the control from Codedeploy, it balances different percent of traffic to the two environments as two different target groups.

ECR for Docker images
To run the sample app on ECS as containers, we will need a image repo from which the images will be pulled. Here I am using ECR as that repo. It stores the versions of the app docker image which are used to run the app on ECS. The images are built using the Dockerfile and pushed to ECR.

ECS Cluster
The ECS cluster is used to run the sample application as containers. The ECS cluster is used to manage the Fargate tasks which run the sample application. The cluster is deployed in a private subnet for security and the tasks running are only exposed via load balancers.

ECS Service and Fargate Tasks
These are the actual app components which are running on ECS. An ECS service is deployed with some specific definitions of the task, which is used to spin up needed number of Fargate Tasks on ECS. The tasks pull image from the ECR and run the app. The ECS service is used to manage the tasks and ensure that the desired number of tasks are running. This service is exposed via load balancer which provides an endpoint for the application.

IAM Roles
For the app to run as Fargate task, we will need different IAM roles to provide proper access to the task. These roles are deployed and associated with the respective components during deployment. These are the roles we create as part of this:

• Task Execution Role: This role is used by the ECS service to run the Fargate tasks. The role provides the necessary permissions to the tasks to pull the Docker image from ECR, write logs to CloudWatch, etc.
• Task Role: This role is used by the Fargate tasks to access other AWS services. The role provides the necessary permissions to the tasks to access other AWS services like S3, DynamoDB, etc.

CI/CD Infrastructure

Below image shows the CI/CD infrastructure which will be used for automating the deployment of the application.

CodeCommit
This is the code repository for the app. This is where the code is pushed after changes are made. The code is then picked up by the CI/CD pipeline for building and deploying the app.

CodeBuild
This is the build service which is used to build the Docker image of the app. The Docker image is built using the Dockerfile which is part of the code repo and pushed to ECR. The build details are defined in the buildspec file which is also part of the repo. The build is triggered when the code is pushed to the CodeCommit repository. This gets triggered as part of the CodePipeline.

CodeDeploy
This is the deployment service which is used to deploy the app to ECS. CodeDeploy is used to control the blue-green deployment of the app. It controls the traffic shifting between the blue and green environments. When new changes are detected by the pipeline, that triggers a new build and then a new CodeDeploy deploy. CodeDeploy is configured for blue green deployment, which makes it deploy in a controlled manner and control the traffic shift gradually from blue to green. The deployment details are defined in the appspec file which is part of the repo. The deployment is triggered when the build is successful. This gets triggered as part of the CodePipeline. We will see this in action in the upcoming sections.

CodePipeline
This is the CI/CD pipeline service which is used to orchestrate the whole deployment process. This pipeline handles the step by step process of triggering each stage and triggering all of the above components tro complete the deployment. The stages involved are:

• Source: This is the source stage where the code is pulled from the CodeCommit repository.
• Build: This is the build stage where the Docker image is built using CodeBuild. This stage triggers a new build on Codebuild
• Deploy: This is the deploy stage where the app is deployed to ECS using CodeDeploy. This stage triggers a new deployment on CodeDeploy.
  A new change pushed to the repo triggers the pipeline which then triggers the build and deploy stages. The pipeline is configured to run the stages in sequence and the deployment is automated.

IAM Roles
To support the pipeline runs and its stages, different IAM roles are created to provide the needed access. Some of the roles which are created are:

• CodeBuild Role: This role is used by CodeBuild to build the Docker image of the app. The role provides the necessary permissions to CodeBuild to pull the code from CodeCommit, build the Docker image, push the Docker image to ECR, etc.
• CodeDeploy Role: This role is used by CodeDeploy to deploy the app to ECS. The role provides the necessary permissions to CodeDeploy to deploy the app to ECS, control the traffic shifting between the blue and green environments, etc.
• CodePipeline Role: This role is used by CodePipeline to orchestrate the whole deployment process. The role provides the necessary permissions to CodePipeline to trigger the build and deploy stages, etc.

That sums up all of the tech infrastructure detailing the components involved in the deployment. Now lets move on to the actual deployment steps.

Deploy the infrastructure

Now lets move on to deploying all of the components to AWS. Below image shows the steps we will follow to deploy each part

The whole deployment is handled using Terraform with Terraform cloud as the state storage. The sequence doesn’t exactly follow the diagram above but the Terraform code is divided into modules which get deployed together. I will cover each of those below. Lets first understand the folder structure in my repo.

Folder Structure

If you are following using my Github repo, then you will need to understand the folder structure of the repo. Below image shows the folder structure of the repo.

• app-sample: This folder contains the sample code and files for the sample application. This includes the Dockerfile, appspec file, buildspec file, etc.

• infra: This folder contains all the Terraform modules for different components of the infrastructure. This includes the modules for VPC, ECS, CodePipeline, etc. It contains sub folders for the modules

  • app_infra: This folder contains the module which deploys the application infrastructure.
  • ecs: This folder contains the module to deploy the ECS cluster
  • networking: This folder contains the module to deploy the networking components like VPC, subnets, etc.
  • pipeline: This folder contains the module which deploys the CI/CD infrastructure. This deploys the CodePipeline, CodeBuild, CodeDeploy, etc.
  • security: This folder contains the module which deploys the security components like IAM roles, security groups, etc.
• main.tf: This is the main Terraform file which deploys all the modules. This file is used to deploy the whole infrastructure. The modules are defined in it and get deployed based on dependencies between them

Initial Setups

Before we can start the deploy steps, there are few initial setups which need to be taken care of if you want to follow along

• Setup a free Terraform cloud account and create a workspace. You can sign up here. This is not mandatory but I am using it for this post
• Create an AWS IAM user which will be used for all the deployments. Get the access key and secret key for this user. The use should have proper access
• Update the keys as environment variables on Terraform cloud workspace. This is needed to authenticate the Terraform cloud with AWS.
  
  
• Install Terraform on local machine
• Clone my repo and navigate to the infra folder. This is where all the Terraform code is present.
• If you are using Terraform cloud, make sure to follow steps from Here to setup the local repo for use with Terraform cloud

App Infrastructure Deployment

Lets first see some of the Terraform resources which are involved as part of the app infrastructure deployment.

• Networking: This includes the VPC, subnets, internet gateway, route tables, etc. The networking module contains all network related components and outputs the details like VPC ID and subnet ids.
  
  
• Security: This module deploys IAM related components like IAM roles, policies, etc. The security module contains all security related components and outputs the IAM role ARNs. Below is an example
  
  
  
  
• ECS: This module deploys the ECS cluster. It also outputs the cluster details like id and name for use by the other modules.
  
  
  
  
• App Infra: This module deploys the actual app components like the ECS task definition for the app, ECS service for load balancing, etc. Below is an example of the task definition. In a way this is the part where the app gets deployed to ECS via the service.
  
  

• App Files: Now this doesnt actually get deployed as part of the Terraform modules but this app code files are pushed to the Codecommit repo separately. The code files contains these parts:

  • index.html and indexgreen.html: These are the html files which are served by the app. The index.html is the blue version and indexgreen.html is the green version. We will switch them when we are simulating a deployment change
  • Dockerfile: This is the Dockerfile which is used to build the Docker image of the app. The Dockerfile is used by CodeBuild to build the Docker image.
  • appspec.yml: This is the appspec file which is used by CodeDeploy to deploy the app to ECS. The appspec file contains the deployment details like the ECS task definition, etc.
    
    
  • buildspec.yml: This is the buildspec file which is used by CodeBuild to build the Docker image of the app. The buildspec file contains the build details like the Docker build command, etc. This file also defines the output artifacts which are produced by the Codebuild. These are the updated spec files with task and image details
    
    
  • taskdef.json: This is the ECS task definition file which is used to define the task definition for the app. This file in the repo is a template file which gets updated by the Codebuild with the image details.
    
    
  • nginx.conf: This is the nginx configuration file which is used by the app to serve the html files. The nginx.conf file is used by the Dockerfile to configure the nginx server.

CI/CD Infrastructure Deployment

Lets see what comprises of the pipeline infrastructure module. The Pipeline module contains all the components related to the CI/CD pipeline like CodePipeline, CodeBuild, CodeDeploy, etc. Below are all part of the pipeline module

• CodeBuild: This is the Codebuild resource in the module which deploys a Codebuild project for the container image builds.
  
  

• CodeDeploy: This is the CodeDeploy resource in the module which deploys a CodeDeploy application and deployment group for the app. This is the resource where we define the blue green deployment pattern for the app and how the blue to green traffic needs to be shifted. The config in CodeDeploy defines how the traffic will be shifted from blue to green environment. When a new deployment gets triggered via the Codepipeline, Codedeploy takes these steps to deploy:

  • Deploys the new changes to a new environment or rather creates a new task with the new definition
  • Starts shifting traffic to the new Task based on the config defined. Here I have selected to shift 10% of the traffic to the new task and then gradually increase it to 100% (CodeDeployDefault.ECSLinear10PercentEvery1Minutes)
  • Once it reaches 100% and no issues are seen, it deletes the old task
    
    
• Cloudwatch Metrics and Alarms: There are some Cloudwatch metrics and alarms defined which monitor the environment traffic shifts and unavailabilities
  
  
• CodePipeline: Finally this resource deploys the Codepipeline aggregating all of the above stages. This pipeline triggers when a new change is pushed to the Codecommit repo.
  
  

Finally all of these modules are deployed by the main tf file which is the entry point for the Terraform deployment. Below is the main.tf file which deploys all of the modules.

Deploy Infrastructure

Now lets deploy all of the above which we just went through. If you are using Terraform cloud, make sure the variables are set to the access keys for AWS. Now navigate to the ‘infra’ folder and initialize Terraform.

terraform init

This will initialize the Terraform cloud backend and install the modules.

Now lets run Terraform plan to view what all it will spin up on AWS

terraform plan

This will start a plan run on Terraform cloud where the details can be viewed. Below is a sample of the plan output.

If everything looks good, then apply the plan to deploy the infrastructure.

terraform apply --auto-approve

This will start an apply run on Terraform cloud. This will take a while to complete. Track the progress on the Terraform cloud console.

Once it finishes, check AWS for some of the resources created

CodeCommit Repo

CodePipeline

Now the pipeline shows failed and thats expected. Because we dont have any code in the repo for the build to happen. If you check on ECS, the service is also failing, since there is no image for it to pull

Now lets push the code to the repo and see the pipeline in action. Get the Codecommit repo url from the AWS console.

Before you can push, you will need to generate GIT credentials for the user you are using. Based on whether you are using HTTPS or SSH, create the credentials from IAM user page

To configure the credentials on your local machine, follow the on screen instructions on the AWS console. It will have instructions to set the credentials for different OS systems. Once the credentials are set, clone the repo to your local

git clone <repo_url>

Copy all the contents from my repo ‘app-sample’ folder to the cloned repo folder. Now commit and push the code to the Codecommit repo.

git add .
git commit -m "Initial commit"
git push

You can view the code on AWS console

The push triggers a new build on the Codepipeline. You can view it running on console

Let the pipeline succeed. This triggers a new Deployment on Codedeploy. You can view the deployment on the console

Lets check the ECS cluster. There should be the app service now running. The number of replicas provided in the task definition defines how many tasks are running. Here I specified 1 replica

Lets test if the app is accessible. Navigate to the load balancer which was deployed. Get the Load balancer DNS. This will be the app url. Navigate to the url in the browser. You should see the app running.

Blue Green deploy demo

Now lets see how it performs a blue-green deployment. Lets change the index html file to the one which says green in the folder. rename the index_green.html to index.html. Commit and push the changes to the Codecommit repo.

git add .
git commit -m "Change to green"
git push

This will trigger a new build on the pipeline

This triggers a new deploy on CodeDeploy. Since we have blue green enabled for Codedeploy, it spins up a second task on ECS instead of updating in place. If we check the service on ECS, we will see a new task now running

The new task is the green environment. Now Codedeploy will start switching traffic to this new task which was created. The load balancer handles balancing the traffic between the two tasks. If we check the deployment in progress on CodeDeploy, we will see it shows the % traffic which is shifted between two environments. Since we selected 10% increase every 1 minute, it will shift 10% after every minute. During that time if we hit the app url, you may randomly see the green page popping up.

Now wait for the full traffic to get shifted. You can view the progress on CodeDeploy console. During this time, both environments will randomly serve the app. Looks like we are closing towards the full shift.

You will now see more persisting green page on web page reload. Once the shift is complete, the Codedeploy deployment enters into a waiting phase. During this phase we can verify if the green environment is stable and give some time to evaluate.

Once the waiting period finishes and no issue seen, it proceeds and deletes the old Blue environment. The deployment is now complete. The full traffic is now served by the green environment. At anytime we feel the deployment didn’t go through well, we can click on the rollback button and start a rollback to the previous version. This gives a good control over the deployment and avoids downtime.

Ok now, we are happy with our deployment and we didnt rollback. So now we have the completed deployment and the new Green app live

That completes our blue green deployment example. Hope you were able to follow along and understand the steps. To avoid charges on your AWS account, make sure to delete all the resources which were created. You can do this by running the below command in the ‘infra’ folder.

terraform destroy --auto-approve

Conclusion

Throughout this guide, we’ve explored the fundamentals of blue-green deployment, delved into the intricacies of setting up your AWS environment, configuring CodePipeline and CodeBuild, defining ECS task definitions, and deploying your application with confidence. We’ve also discussed best practices, tips, and strategies for automating the deployment pipeline to achieve continuous delivery seamlessly. Hope I was able to provide you with a good understanding of how to perform blue-green deployments on ECS using CodePipeline and CodeBuild. If you have any questions or issues, feel free to reach out to me from the contact page.