CSC/ECE 517 Spring 2015/ch1a 4 RW

Blue-Green Deployment

Blue-Green deployment is a technique used to reduce risk and delay in continuous integration<ref>http://martinfowler.com/bliki/BlueGreenDeployment.html</ref>. It uses an exact copy of production environment, where new changes can be deployed and production testing can be done. After running successful tests, we can just point router to this copy and make it production. In case of any error in system after deployment, system can be rolled back by re-pointing router to old application instance. This process reduces the production downtime during migration and expedite any rollback operation needed.

Background

Software applications are becoming more and more complex. Software development life cycle is trying to adapt to this new paradigm by implementing ways to release new features/updates as frequently as they can be conceptualized. Agile methodology has reduced release-time of new piece of code. On an average, Amazon is pushing changes to production every 11.6 seconds<ref>https://www.youtube.com/watch?v=dxk8b9rSKOo</ref>. Google and Facebook <ref>http://www.infoq.com/presentations/Facebook-Release-Process</ref>are releasing frequent changes to production. With new updates and features going into production at such a remarkable frequency, a need was felt to automate production deployment. More and more software development houses are using Continuous Integration (CI) nowadays. In continuous integration, a new piece of software is easily integrated into existing application code, and tested to make it ready for release. With continuous integration becoming a norm, continuous delivery is becoming a necessity. As per Martin Fowler, "Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time". Continuous delivery demands ability to automate deployment on production and create environment that can support frequent deployments. One of the challenges of automated deployment is to take software from test environment to live production. The new update may include a change in database schema as well. Updating existing production database server without bringing application down, is an arduous and risky task. Many business domains expect a minimum downtime for such updates, while maintaining a need for continuous delivery.
Blue-Green deployment provides a solution for continuous delivery by using two nearly identical production environments. One environment, which is in production at the moment is called Blue and other environment is called Green.

Green environment is used to deploy all the new updates and run tests. After successful testing, router can be configured to route all the new requests to Green system. In case of any error in Green system, instead of using usual roll back mechanism, we can just point router to Blue system. There are few database consistency issues involved in such methods. There is a time delay between the moment, Green copy is generated from Blue system and the moment when it is made live. Any change that took place on Blue system during this interval must be replicated on Green system. For database applications, this can achieved using modification scripts. Similarly, in case of error in Green system after making it live, we have to use modification scripts to bring consistency between Blue database an Green database. This process can not be completed without losing information, if new features include database schema changes. Since database schema of Blue system and Green system differs, transferring new database changes may take a lot of time and it requires use of a complex and rigorous design.
Both these issues can be handled to a greater extent by making each system read-only during transition phase. Just before Blue environment is copied to Green, Blue can be made read only. Application can be up and running while certain/all write features are blocked. After Green environment is tested for production and made live, we can keep it read only for a fixed amount of time. If any error occurs during this phase, then for roll back, all we have to do is point router to Blue environment and debug Green environment for error.

Advantages

Two basic necessities of a smooth production deployment is:

• Nearly zero downtime
• In case of error, immediate roll back from new version to older version

By using read only constraint on database during transition, Blue-Green deployment can be used to achieve almost zero down time. In case of error, roll back from new version to older version is almost immediate.

Challenges

Redirection

Conceptually, blue-green deployments are simple and the benefits are significant (deploy often, never go down, roll back with ease). But without the right infrastructure, they can be tricky to implement. For example, to accomplish the critical fourth step of “flipping the switch”, you need a fast, reliable, and automated means of mapping and unmapping production URLs between the blue and green environments. A DNS-based approach fails on most or all of these criteria.
Cloud Foundry Router<ref>http://docs.pivotal.io/pivotalcf/devguide/deploy-apps/blue-green.html</ref> is one of the possible solutions for the mapping and unmapping service request with application instance. Following shows how to use CF Router to implement blue green deployment.

cf push Blue -n demo-time

Push the app. Router sends all the traffic to blue, the current working environment.

cf push Green -n demo-time-temp

Now, some changes has been made and push the updated application to green environment.

cf map-route Green example.com -n demo-time

Map the original URL to green environment. Router starts to load balance between blue and green.

cf unmap-route Blue example.com -n demo-time

Once green environment is ready, router unmaps the blue environment. Now all the URLs are directed to green.

Database Migration

Intuitively, you need to implement two databases on two seperate servers. You could very easily have green users working against a set of gold data in a green database. Sometime relatively close to when you would want to flip the router, you would run whatever ETL processes would be necessary to populate the green database with data from the blue database. Then, just before you actually flip the router, you would stop any further changes to the blue database data. Then load any remaining data changes from the blue to the green, that may have occurred since you initially transferred the data (which should be a relatively small, and much quicker operation in theory). While you wouldn’t have a “zero-downtime” scenario doing it this way, it could be very close to one. However, maintaining two identical production environment requires a lot of space and computational resource. This requirement can put a considerable amount of financial burden on companies. In some cases, same machine can be used to host both the environment. Virtual machines can be used to maintain two separate identical environments. Lots of applications have huge databases and keeping two copies of databases during application deployment is a big financial commitment.
Overall, database changes should be rolled out incrementally. Even organizations like Flickr, which deploy multiple times a day, don't roll out database changes that frequently. Instead, they use the expand/contract pattern. The rule<ref>http://www.informit.com/articles/article.aspx?p=1833567</ref> is that you never change existing objects all at once. Instead, divide the changes into reversible steps:

• Before the release goes out, add new objects to the database that will be required by that new release.
• Release the new version of the app, which writes to the new objects, but reads from the old objects if necessary so as to migrate data "lazily." If you need to roll back at this point, you can do so without having to roll back the database changes.
• Finally, once the new version of the app is stable and you're sure you won't need to roll back, apply the contract script to finish migrating any old data and remove any old objects.

Practical Issue

Though Blue-Green deployment offers a nearly zero downtime solution for many applications which require continuous delivery and no downtime. In domains such as Banking, Retail and Social Networking, application downtime is unacceptable. Blue-Green deployment offers an acceptable solution but with a Read-Only constraint on database.

Variation

A variation on blue-green deployment, applicable when running a cluster of servers, is canary releasing. With this pattern, rather than upgrading a whole cluster to the latest version all at once, you do it incrementally. For example, as described in an excellent talk by Facebook's release manager, Chuck Rossi, Facebook pushes new builds to production in three phases.

• First the build goes to A1—a small set of production boxes to which only employees are routed.
• If the A1 deployment looks good, the build goes to A2, a "couple of thousand" boxes to which only a small percentage of users are routed.
• A1 and A2 are like canaries in a coal mine—if a problem is discovered at these stages, the build goes no further. Only when no problems occur is the build promoted to A

An interesting extension of this technique is the cluster immune system. Developed by the engineers at IMVU, this system monitors business metrics as a new version is being rolled out through a canary releasing system. It automatically rolls back the deployment if any parameters exceed tolerance limits, emailing everyone who checked in since the last deployment so that they can fix the problem.

Conclusion

In the agile software development world, where continuous delivery is increasing becoming a necessity, Blue-Green deployment provides a solution with almost zero downtime. The performance provided by this methodology requires a huge financial commitment. Maintaining two separate production environment is a big investment. But in application domains, where continuous delivery and zero downtime is a necessity, this methodology should be used to achieve better results.

References

<references/>