CSC/ECE 517 Fall 2011/ch6 6f jd

Continous Integration

Introduction

Continuous Integration (CI) is a software development practice that is commonly used in Agile software development methodology. Agile is a software development method based on iterative and incremental development. In principles, it works with small team; it requires developers to deliver working code in short iterations; it uses Test Driven Development (TDD), which requires developers to develop tests before writing code. Finally all the codes must be reviewed and tested continuously as the development goes, and that is where CI fits in.

What is CI?

CI is used to ensure that all changes to a software project’s code are successfully built, tested, reported on, and rapidly made available to all parties after they are introduced. (1) It is aimed to deliver high quality software using the least amount of time, by applying quality control throughout the whole software development process. Without CI, there is no chance to succeed in delivering refined products in Agile’s short release cycles.

How does CI work?

The above diagram shows an overview of a simple CI environment, which will give us a good idea of how CI works. (2) In order to apply quality control throughout the project, CI requires members of the team integrate their work into the main code repository at least daily, which leads to multiple integrations per day. Each developer will first build and run all the tests locally to ensure all the changes are good, then commit the changes to the repository using some version control tools, such as Subversion. Once the CI sever detects the new commits, it will kick out an automated build that runs all the automated tests that could detect compile/integration errors as quickly as possible. If the automated build fails, the team will be notified and required to resolve the error immediately. In addition, at least once every 24 hours, a nightly build will be executed to ensure the quality of the code in the main repository.

CI Reporting Metrics

The team can also generate reports that indicate the overall health of the application on the CI environment. This application health can be measured by the CI reporting metrics. Some examples of these metrics are shown below:

1. Time To Reponses (TTR): the elapsed time to fix the most recent broken build
2. Time Between Failures (TBF): the elapsed time between consecutive breakages (non-zero errors) of the same build
3. Unit Test Count: the total number of unit tests on each automated build
4. Unit Test Passed Count: the total number of unit tests that are passed on each automated build
5. Unit Test Covered Lines: the count of the non-commented source lines scanned by the coverage tool that are exercised by statement-level automated unit testing on each automated build
6. Validation Test Count: the total number of validation tests on each automated build
7. Validation Test Passed Count: the total number of validation tests that are passed on each automated build
8. Error Count: the total number of errors occurs on each automated build
9. Warning Count: the total number of warnings occurs on each automated build
10. Code Based Size: the total lines of code of the overall organization

Why Implement CI?

Most of the major companies producing software for sale are in the business to make money. Companies have discovered that the best way to make money is decrease the number of defects going out to customers. Tons of research has been done on how to make quality software with fewer defects. The usual cost of fixing a defect slipping from one phase to next phase increases by ten times. So for example if a defect cost $60 to fix during the coding phase will cost $600 if found during validation testing. Moreover if a defect is shipped along with the product to the customers it can certain times cost millions of dollars. Additionally these small defects can have big impact on company’s reputation. Below are some examples which shows how small defects when slipped or goes out undetected can cost millions of dollars.

1. In 2007 a single faulty piece of embedded software, on a network card, sends out faulty data on the United States Customs and Border Protection network, bringing the entire system to a halt. Nobody is able to leave or enter the U.S. from the LA Airport for over eight hours. Over 17,000 planes were grounded for the duration of the outage resulting in millions of dollars in damage.
2. A forgotten error handling statement which caused the famous ping of death also known as blue screen in 1995. A lack of error handling in the IP fragmentation reassembly code makes it possible to crash many Windows, Macintosh, and Unix operating systems by sending a malformed “ping” packet from anywhere on the Internet.
3. In 2004, EDS software giant introduced a large, complex IT system to the U.K.’s Child Support Agency (CSA). At the exact same time, the Department for Work and Pensions (DWP) decided to restructure the entire agency. The restructure and the new software were completely incompatible, and irreversible errors were introduced as a result. With over 500 bugs still reported as open in the new system, the clash of the two events has crippled the CSA’s network. As a result the system somehow managed to overpay 1.9 million people, underpay another 700,000, had $7 billion in uncollected child support payments, a backlog of 239,000 cases, 36,000 new cases “stuck” in the system, and has cost the UK taxpayers over $1 billion to date.

The above examples show why testing phase is as important as coding phase. The more you test the better the software becomes as fewer and fewer defects are found. Below are some of the reasons why big companies are leaning more towards adopting agile and CI for software development.

1. Many developers work on a software through multiple agile teams. Though the focus of each agile team is a specific module of the software but it is important to make sure the overall integrity of the system is always correct.
2. One of the main benefits of agile project development is to have a shippable product at the end of each sprint. If testing is kept until the end than you can’t have a shippable product at the end of each sprint. You have to have some automated testing that can test this functionality on daily basis.
3. The other problem that used to occur by following traditional software development was that management did not know the exact status of the product which includes the number of bugs in the system until the validation phase. This can cause problem in releasing software on time. CI gives an option to generate various reports to solve these kinds of problems and to give management early indication if the product release can be done on time.
4. The main reason for using CI is to cut down on validation cost by finding the bugs early during coding phase.
5. CI infrastructure provides way to decrease Time to Response (TTR) for fixing defects.
6. Additionally some CI infrastructure provides valuable tools like build management and build acceleration. With build acceleration developers and QA do not have to wait long periods to get a completed builds. Build acceleration decreases the build time by performing non-dependent tasks in parallel resulting in increased productivity.

CI Setup and Reporting

There are lots of different ways continuous integration can be incorporated with a software product development. The level of automation varies by project to project. Some project might be able to achieve complete automation and continuous integration whereas some project can be restricted by the testing infrastructure. For example when making a software for say airplane controls there is only limited amount of automated testing that you can do to test. It requires some level of manual steps. Whereas for computer application like word you can automate the testing and can achieve 100% continuous integration.

There are many different kinds of software that are available out there for implementing continuous integration within a software project. Some of the software includes Electric Commander by Electric Cloud, CI server by Hudson, and Team Foundation Server, Lab Management, Test Manager by Microsoft. All of the software listed above can be used as the backbone for setting up the CI infrastructure. Additionally, different reporting services can also be added to achieve additional goals. Below are the usual steps that are performed once the CI infrastructure is incorporated.

1. Whenever the new code or functionality is checked into the repository an automated build process is started.
2. Once the product is build unit tests are executed to make sure all the unit tests are passing and that the new functionality has not broken the old functionality. Reports are generated for the results.
3. Once unit test are completed and passed automated smoke tests are performed. The application is usually deployed on a remote machine as a fresh copy and the system testing is performed. Reports are generated for the results.
4. Then additional reporting are performed on the code to see it the code has fulfilled the required criteria (For example, code coverage > 90%, code complexity, Dead code, Duplicate code, Coding standard enforcement)
5. Lastly, when all the 4 steps above are passed additional matrix is generated for management to see. For example TTR to fix a bug, Time between Failures (TBF), Error counts, and Warning reports.

If any of the above steps fail an automated email is generated and send to targeted user to notify that the new code submitted to the repository needs to be fixed. In this way teams can achieve that quick response instead of waiting to hear from validation team which can take days compared to minutes through CI.

References

• [1] Software Bugs Top Ten Software Bugs
• [2] Electric Cloud Electric Commander

Expand your knowledge

• Continuous Integration Wikipedia CI