CSC/ECE 517 Spring 2015/ch1a 4 RW

Blue-Green Deployment

This article is a part of a writing assignment<ref>https://docs.google.com/a/ncsu.edu/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc/edit</ref>

Blue-Green deployment is a technique used to reduce risk and delay in continuous integration<ref>http://martinfowler.com/bliki/BlueGreenDeployment.html</ref>. It uses an exact copy of production environment, where new changes can be deployed and production testing can be done. After running successful tests, we can just point router<ref>http://en.wikipedia.org/wiki/Router_%28computing%29</ref> to this copy and make it production. In case of any error in system after deployment, system can be rolled back by re-pointing router to old application instance. This process reduces the production downtime during migration and expedite any rollback operation<ref>http://en.wikipedia.org/wiki/Rollback_%28data_management%29</ref> needed.

Background

     Software applications are becoming more and more complex. Software development life cycle is trying to adapt to this new paradigm by implementing ways to release new features/updates as frequently as they can be conceptualized. Agile methodology<ref>http://en.wikipedia.org/wiki/Agile_software_development</ref> has reduced release-time of new piece of code. On an average, Amazon is pushing changes to production every 11.6 seconds<ref>https://www.youtube.com/watch?v=dxk8b9rSKOo</ref>. Google and Facebook <ref>http://www.infoq.com/presentations/Facebook-Release-Process</ref>are releasing frequent changes to production. With new updates and features going into production at such a remarkable frequency, a need was felt to automate production deployment.

    More and more software development houses are using Continuous Integration (CI) nowadays. In continuous integration, a new piece of software is easily integrated into existing application code, and tested to make it ready for release. With continuous integration becoming a norm, continuous delivery <ref>http://en.wikipedia.org/wiki/Continuous_delivery</ref> is becoming a necessity. As per Martin Fowler, "Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time". Continuous delivery demands ability to automate deployment on production and create environment that can support frequent deployments. One of the challenges of automated deployment is to take software from test environment to live production. The new update may include a change in database schema as well. Updating existing production database server without bringing application down, is an arduous and risky task. Many business domains expect a minimum downtime for such updates, while maintaining a need for continuous delivery.

Blue-Green deployment provides a solution for continuous delivery by using two nearly identical production environments. One environment, which is in production at the moment is called Blue and other environment is called Green.

     Green environment is used to deploy all the new updates and run tests. After successful testing, router can be configured to route all the new requests to Green system. In case of any error in Green system, instead of using usual roll back mechanism, we can just point router to Blue system. There are few database consistency issues involved in such methods. There is a time delay between the moment, Green copy is generated from Blue system and the moment when it is made live. Any change that took place on Blue system during this interval must be replicated on Green system. For database applications, this can achieved using modification scripts <ref>http://www.infoq.com/articles/db-versioning-scripts</ref>. Similarly, in case of error in Green system after making it live, we have to use modification scripts to bring consistency between Blue database an Green database. This process can not be completed without losing information, if new features include database schema changes. Since database schema of Blue system and Green system differs, transferring new database changes may take a lot of time and it requires use of a complex and rigorous design.

     Both these issues can be handled to a greater extent by making each system read-only during transition phase. Just before Blue environment is copied to Green, Blue can be made read only. Application can be up and running while certain/all write features are blocked. After Green environment is tested for production and made live, we can keep it read only for a fixed amount of time. If any error occurs during this phase, then for roll back, all we have to do is point router to Blue environment and debug Green environment for error.

Advantages

Two basic necessities of a smooth production deployment is:

• Nearly zero downtime
• In case of error, immediate roll back from new version to older version

By using read only constraint on database during transition, Blue-Green deployment can be used to achieve almost zero down time. In case of error, roll back from new version to older version is almost immediate.

Disadvantages

Database Migration

As explained before, we need two databases servers. If things are not planned properly, users may easily end up working against an old Green database. The database migration during transition should be kept a minimal as possible. System should be made read only for most of the operations right before transition starts. Load any remaining data changes from the blue to the green, that may have occurred during transition (which should be relatively small, and a quicker operation in theory). Though a perfect zero downtime cannot be achieved using this, but a near zero downtime can be easily achieved.
Overall, database changes should be rolled out incrementally. Many organizations, which deploy multiple times a day, don't roll out database changes frequently. Instead, they use the expand/contract pattern. The rule<ref>http://www.informit.com/articles/article.aspx?p=1833567</ref> is that you never change existing objects all at once. Instead, divide the changes into reversible steps:

• Before the release goes out, add new objects to the database that will be required by that new release.
• Release the new version of the app, which writes to the new objects, but reads from the old objects if necessary so as to migrate data "lazily" (data will migrate as it’s used). If one need to roll back at this point, it can be done without a need to roll back the database changes.
• Finally, once the new version of the app is stable and everything look perfect, apply the update script to finish migrating any old data and remove any old objects.

Financial Burden

Maintaining two identical production environments require a lot of space and computational resources. This requirement can put a considerable amount of financial burden on companies. In some cases, same machine can be used to host both the environments. Virtual machines can be used to maintain two separate identical environments as well. Not only system, but many applications have huge databases, and maintaining two copies is a huge financial commitment.

Examples

Blue Green deployment on Amazon AWS

Many cloud service providers <ref>http://searchcloudprovider.techtarget.com/definition/cloud-provider</ref> are releasing new products for web services hosting<ref>http://en.wikipedia.org/wiki/Web_hosting_service</ref>. Amazon web services <ref>http://aws.amazon.com/</ref> is one such product from Amazon <ref>http://www.amazon.com/Careers-Homepage/b?ie=UTF8&node=239364011</ref>. Amazon Elastic Compute Cloud (Amazon EC2) <ref>http://aws.amazon.com/ec2/</ref> is a web service that provides re-sizable compute capacity in the cloud. User can select to either host web services using single instance or multiple instances of EC2. In single instance hosting, each instance of AWS (Amazon Web Serivices) has a private and a public IP address assigned to it. Public ip address can be accessed from outside world, while private ip address is for internal access. During blue green deployment in AWS, for single EC2 instance, switching between instances is as simple as an API<ref>http://en.wikipedia.org/wiki/Application_programming_interface</ref> call. It's easy to replicate current production instance and run updates on that. After updates, all the new in-coming requests can be routed to this system by assigning public ip address to this new instance. Amazon web service provides an API call to assign public ip address to any instance. Older instance can still be accessed using private ip address. Same method cannot be used for multiple EC2 instances. For multiple instances, entire pool has to be changed with a new version of application, which can be achieved either by automating a series of API calls or using AutoScaling <ref>http://aws.amazon.com/autoscaling/</ref> groups

Variation

A variation on blue-green deployment, applicable when running a cluster of servers, is canary releasing<ref>http://martinfowler.com/bliki/CanaryRelease.html</ref>. With this pattern, rather than upgrading a whole cluster to the latest version all at once, you do it incrementally. For example, as described in an excellent talk by Facebook's release manager, Chuck Rossi, Facebook pushes new builds to production in three phases.

• First the build goes to A1—a small set of production boxes to which only employees are routed.
• If the A1 deployment looks good, the build goes to A2, a "couple of thousand" boxes to which only a small percentage of users are routed.
• A1 and A2 are like canaries in a coal mine—if a problem is discovered at these stages, the build goes no further. Only when no problems occur is the build promoted to A

An interesting extension of this technique is the cluster immune system. This system monitors business metrics as a new version is being rolled out through a canary releasing system. It automatically rolls back the deployment if any parameters exceed tolerance limits, emailing everyone who checked in since the last deployment so that they can fix the problem.

Conclusion

In the agile software development world, where continuous delivery is increasing becoming a necessity, Blue-Green deployment provides a solution with almost zero downtime. The performance provided by this methodology requires a huge financial commitment. Maintaining two separate production environment is a big investment. But in application domains, where continuous delivery and zero downtime is a necessity, this methodology should be used to achieve better results.

References

<references/>