CSC/ECE 517 Fall 2011/ch6 6f jd: Difference between revisions

From Expertiza_Wiki
Jump to navigation Jump to search
No edit summary
 
(60 intermediate revisions by 2 users not shown)
Line 1: Line 1:
<big>'''Testing frameworks for Ruby'''</big>
<big>'''Continous Integration'''</big>


__TOC__
__TOC__


==Introduction==
==Introduction==
Continuous Integration [http://en.wikipedia.org/wiki/Continuous_integration (CI)] is a software development practice that is commonly used in [http://en.wikipedia.org/wiki/Agile_software_development Agile software development] methodology. <ref name="Agile">"Manifesto for Agile Software Development" http://www.agilemanifesto.org/</ref>Agile is a software development method based on iterative and incremental development. In principle, it works with small team, requires developers to deliver working code in short iterations, and uses Test Driven Development [http://en.wikipedia.org/wiki/Test-driven_development (TDD)], which requires developers to develop tests before writing code. Also it requires all the codes to be reviewed and tested continuously as the development goes, which is where CI fits in.


A testing framework is used on a software project when it has grown large enough that manual debugging and system upkeep would be too time and resource intensive.  The framework provides a way to automate testing to provide for low cost maintenance. The framework itself contains a set of assumptions, concepts, and tools that allow for various types of testing.  Choosing the right framework depends on many factors associated with the system in development and the mindset of the developers and [http://en.wikipedia.org/wiki/Stakeholder stakeholders]. Every framework has a few basic features: A process for defining the format to express user expectations, a way to execute the tests, and a way to report the results of the test to the user  Examples are provided later to help give context to the definitions below.
==What is CI?==
CI is used to ensure that all changes to a software project’s code are successfully built, tested, reported on, and rapidly made available to all parties after they are introduced. <ref name="CIRef1">Enterprise Solution Providers, Inc.  "Why and How to build a Continuous Integration Environment for the .NET platform" http://www.espusa.com/whitepapers/continuous_integration_v1.0.pdf</ref> It is aimed to deliver high quality software using the least amount of time, by applying quality control throughout the whole software development process. Without CI, there is no chance for Agile to succeed in delivering refined products in short release cycles.


===Definitions===
==How does CI work?==
====Test Driven Development (TDD)====
[[File:ContinuousIntegration.jpg|center]]
Test-Driven Development (TDD) is the process of writing tests first and foremost.  These tests represent desired improvements and/or new functionality in the system.  Since these changes have yet to be implemented, the tests must fail when initially run.  Then code is written until these tests can be passed, assuring that the desired behavior is exhibited by the code.  Finally the code is rewritten to meet acceptable standards.<br>
[[File:tddSteps.jpg]]


====Behavior Driven Development(BDD)====
The above diagram shows an overview of a simple CI environment, which will give us a good idea of how CI works. <ref name="CIRef2">Thomas Jaspers.  "Continuous Integration – Overview" http://blog.codecentric.de/en/2009/11/</ref>.In order to apply quality control throughout the project, CI requires members of the team integrate their work into the main code repository at least daily, which leads to multiple integrations per day. Each developer will first build and run all the tests locally to ensure all the changes are good, then commit the changes to the repository using some version control tools, such as Subversion. Once the CI sever detects the new commits, it will kick out an automated build that runs all the automated tests that could detect compile/integration errors as quickly as possible. If the automated build fails, the team will be notified and required to resolve the error immediately. In addition, at least once every 24 hours, a nightly build will be executed to ensure the quality of the code in the main repository. <ref name="CIRef5">Laurie Williams.  "Scrum + Engineering Practices: Experiences of Three Microsoft Teams" http://collaboration.csc.ncsu.edu/laurie/Papers/ESEM11_SCRUM_Experience_CameraReady.pdf</ref>
Behavior-Driven Development [http://en.wikipedia.org/wiki/Behavior_Driven_Development (BDD)] is very similar to test driven development; it is a different approach to the same subject.  BDD is an approach to software development that combines Test-Driven Development, Domain Driven Design, and Acceptance Test-Driven Planning. With this type of development users are more focused on the behavior that is exhibited in the code and less on testing each method and class. The goal of BDD is to use terminology focused on the behavioral aspects of the system to allow a seamless transition between specification, design, and implementation of a system.  BDD has a much better alignment with business concerns.


====Mocking in Testing====
==Why Implement CI?==
Mocking provides users with a helpful way to write unit tests for objects.  These mock objects are used to imitate existing roles in a system.  Instead of calling these real objects, the mock objects are called instead. They will assert that the correct methods were called with the right parameters in the right order.  So instead of needing to create a database and fill it with rows, we can instead use a mock object that would supply the same information as the database. It is the high [http://en.wikipedia.org/wiki/Coupling_(computer_programming) coupling] between modules and tests that creates the need for a mocking framework.
Most of the major companies producing software for sale are in the business to make money. Companies have discovered that the best way to make money is decrease the number of defects going out to customers. Tons of research has been done on how to make quality software with fewer defects. The usual cost of fixing a defect slipping from one phase to next phase increases by ten times.  So for example if a defect cost $60 to fix during the coding phase will cost $600 if found during validation testing. Moreover if a defect is shipped along with the product to the customers it can certain times cost millions of dollars. Additionally these small defects can have big impact on company’s reputation. Below are some examples <ref name="Bugs">Top Ten Most Infamous Software Bugs Of All Time" http://able2know.org/topic/129489-1</ref>which shows how small defects when slipped or goes out undetected can cost millions of dollars.


====Benchmarking in Testing====
# In 2007 a single faulty piece of embedded software, on a network card, sends out faulty data on the United States Customs and Border Protection network, bringing the entire system to a halt. Nobody is able to leave or enter the U.S. from the LA Airport for over eight hours. Over 17,000 planes were grounded for the duration of the outage resulting in millions of dollars in damage.
Benchmark testing is used to determine things like [http://en.wikipedia.org/wiki/Throughput throughput] performance, CPU load, and memory utilization.  It is good practice to perform benchmark testing on any system going into a production environment. This allows users to determine current system efficiency and identify things such as performance bottlenecks throughout the code. This can be done by refining code and/or database configurations. Benchmark tests must be run inside a repeatable environment in order to yield results that can be accurately compared.
# A forgotten error handling statement which caused the famous ping of death also known as blue screen in 1995.  A lack of error handling in the IP fragmentation reassembly code makes it possible to crash many Windows, Macintosh, and Unix operating systems by sending a malformed “ping” packet from anywhere on the Internet.
# In 2004, EDS software giant introduced a large, complex IT system to the U.K.’s Child Support Agency (CSA). At the exact same time, the Department for Work and Pensions (DWP) decided to restructure the entire agency. The restructure and the new software were completely incompatible, and irreversible errors were introduced as a result. With over 500 bugs still reported as open in the new system, the clash of the two events has crippled the CSA’s network. As a result the system somehow managed to overpay 1.9 million people, underpay another 700,000, had $7 billion in uncollected child support payments, a backlog of 239,000 cases, 36,000 new cases “stuck” in the system, and has cost the UK taxpayers over $1 billion to date.


====Testing Framework====
The above examples show why testing phase is as important as coding phase. The more you test the better the software becomes as fewer and fewer defects are found. Below are some of the reasons why big companies are leaning more towards adopting agile and CI for software development.
A testing framework allows for automated software testing. It contains a set of assumptions, concepts, and tools that assist in this testing. The main advantage of using a testing framework is the lower cost for maintenance. The testing framework is responsible for :
# Many developers work on a software through multiple agile teams. Though the focus of each agile team is a specific module of the software but it is important to make sure the overall integrity of the system is always correct.
<ul>
# One of the main benefits of agile project development is to have a shippable product at the end of each sprint<ref name="Ready Product">Why Continuous Integration?" http://www.cavdar.net/2009/03/07/why-continuous-integration/</ref>. If testing is kept until the end than you can’t have a shippable product at the end of each sprint. You have to have some automated testing that can test this functionality on daily basis.
<li>defining the format in which to express expectations</li>
# The other problem that used to occur by following traditional software development was that management did not know the exact status of the product which includes the number of bugs in the system until the validation phase. This can cause problem in releasing software on time. CI gives an option to generate various reports to solve these kinds of problems and to give management early indication if the product release can be done on time.
<li>creating a mechanism to hook into or drive the application under test</li>
# The main reason for using CI is to cut down on validation cost by finding the bugs early during coding phase.
<li>executing the tests</li>
# CI infrastructure provides way to decrease Time to Response [http://en.wikipedia.org/wiki/Response_time_(technology) (TTR)] for fixing defects.
<li>reporting results</li>
# Additionally some CI infrastructure provides valuable tools like build management and build acceleration. With build acceleration developers and QA do not have to wait long periods to get a completed builds. Build acceleration decreases the build time by performing non-dependent tasks in parallel resulting in increased productivity.
</ul>


==Testing Framework Evolution==
==CI Setup and Reporting==
Test driven development (TDD) is linked with the concepts of [http://en.wikipedia.org/wiki/Extreme_Programming extreme programming] which began in 1999. It offered more than just simple validation of correctness; it also drove the design of the program. By focusing on the test cases first, one needed to imagine how the functionality would be used by clients.  Test-driven development ensured that all written code was covered by at least one test. This gave the programming team, as well as users and stakeholders, a greater level of confidence in the code.<br>
There are lots of different ways continuous integration can be incorporated with a software product development. The level of automation varies by project to project. Some project might be able to achieve complete automation and continuous integration whereas some project can be restricted by the testing infrastructure. For example when making a software for say airplane controls there is only limited amount of automated testing that you can do to test. It requires some level of manual steps. Whereas for computer application like word you can automate the testing and can achieve 100% continuous integration.
One major problem seen with TDD can be the mindset it puts users in. This mindset could be detrimental to the overall project.  Users needed to start thinking in terms of behavior specifications, not verification tests. This required two things:
 
<ul>
There are many different kinds of software that are available out there for implementing continuous integration within a software project. Some of the software includes Electric Commander<ref name="EC">Electric Commander" http://www.electriccloud.com/</ref> by [http://en.wikipedia.org/wiki/Electric_Cloud Electric Cloud], CI server by Hudson, and Team Foundation Server, [http://www.microsoft.com/visualstudio/en-us/scenarios/virtual-lab-management Lab Management], Test Manager by Microsoft.  All of the software listed above can be used as the backbone for setting up the CI infrastructure. Additionally, different reporting services can also be added to achieve additional goals. Below are the usual steps that are performed once the CI infrastructure is incorporated.  
<li>A language shift in traditional TDD</li>
<li>Driving your tests from a domain/user story perspective rather than technical perspective</li>
</ul>


===TDD vs BDD===
The following example shows how a typical test is setup for TDD.


<pre>
[TestFixture]
public class directory_tests
{
    [SetUp]
    public void SetUp()
    {
        // setup code for test
    }
    [Test]
    public void can_add_person_to_addressbook()
    {
        // test assertions
    }
}
</pre>


Now here is the same test written in the style of BDD.
[[File:EC.jpg|center]]
<pre>
[Concern("Address Book"), TestFixture]
public class when_searching_for_a_person_in_an_addressbook : ContextSpecification
{
    [Context]
    protected override void  Context()
    {
        base.Context();
        // setup code for test
    }
    [Observation, Test]
    public void addressbook_contains_person()
    {
        // test assertions
    }
}
</pre>


Our TestFixtures have now been grouped into concerns.  Setup has been replaced by Context and instead of Test, we now have Observations. Tests now become the acceptance criteria for the context allowing us to write tests from a [http://en.wikipedia.org/wiki/Use_case use case] perspective


==Ruby Testing Frameworks==
Compared to five years ago the number of testing frameworks available for Ruby has increased rapidly. Selecting a perfect testing framework for a particular project has become more difficult.
   
===Overview of testing framework===
Selection of testing frameworks for Ruby projects depend on many different factors, such as: which approach are we going to use TDD or BDD, is GUI testing included, is acceptance testing required, do we have to merge legacy Ruby test code with new testing framework? Altough GUI testing and acceptance testing are out of scope for this wiki, we have made an attempt to cover them in some detail at the end of this wiki page.
====Test:Unit====
Test-Unit is one of the oldest testing frameworks for Ruby testing. Test-Unit is basically used for performing test driven development similar to how Java’s JUnit is used. It’s in a way very similar to Java’s JUnit. Test-Unit framework, when initially released, was part of the ruby distribution package. The newer version since then has been converted to gems and has been provided to developers as a standalone package. Additionally, some of the features like notify, capybara, [http://en.wikipedia.org/wiki/GTK%2B GTK+],[http://en.wikipedia.org/wiki/Tk_%28framework%29 Tk], and [http://en.wikipedia.org/wiki/Fox_toolkit FOX Toolkit] have been split from the package and provided separately[http://test-unit.rubyforge.org/].


====RSpec====
# Whenever the new code or functionality is checked into the repository an automated build process is started.
RSpec is a Behavior-Driven Development (BDD) tool for Ruby programmers. As defined earlier, BDD is an approach to software development that combines Test-Driven Development, Domain Driven Design, and Acceptance Test-Driven Planning. RSpec helps you do the TDD part of that equation, focusing on the documentation and design aspects of TDD [http://rspec.info/]
# Once the product is build unit tests are executed to make sure all the unit tests are passing and that the new functionality has not broken the old functionality. Reports are generated for the results.
# Once unit test are completed and passed automated smoke tests are performed. The application is usually deployed on a remote machine as a fresh copy and the system testing is performed. Reports are generated for the results.
# Then additional reporting are performed on the code to see it the code has fulfilled the required criteria (For example, code coverage > 90%, code complexity, Dead code, Duplicate code, Coding standard enforcement)  
# Lastly, when all the 4 steps above are passed additional matrix is generated for management to see. For example TTR to fix a bug, Time between Failures (TBF), Error counts, and Warning reports.


RSpec is one of the most widely used testing frameworks for Ruby. The main difference between RSpec and Test-Unit is semantics. RSpec provides a way to write tests which are in a more human readable form. This reduces the expense for maintaining the test and also provides more easily understandable tests. Additionally when RSpec tests fail, it provides more information about what functionality failed, and why. This compared to Test-Unit tests, where you must put custom messages on all assertion statements to point out exactly what happened, is much simplier.
If any of the above steps fail an automated email is generated and send to targeted user to notify that the new code submitted to the repository needs to be fixed. In this way teams can achieve that quick response instead of waiting to hear from validation team which can take days compared to minutes through CI.


====Riot====
===CI Reporting Metrics===
Riot is another BDD Ruby unit testing framework. Riot testing framework is a mixture of both RSpec and Test-Unit or RSpec and Shoulda (which is based on Test-Unit). The main purpose of authoring the Riot testing framework was to come up with a Ruby testing framework which is fast in execution but is in a more human readable form. As in the case of other testing frameworks, Riot does not have variables described or initialized in its test cases. It has a setup block which returns a variable as default. This allows Riot to be fast during execution, as it does not have to initialize or do much of pre-test setup. One of the unique attributes of the Riot testing framework is that the tests only focus on small functionality.  
The team can also generate reports that indicate the overall health of the application on the CI environment. This application health can be measured by the CI reporting metrics. Some examples of these metrics are shown below:<ref name="CIRef6">Designing data warehouse for equipment management system" http://www.tandfonline.com/doi/abs/10.1080/00207540701222776?journalCode=tprs20#preview</ref>


====Shoulda====
#'''Time To Response (TTR):''' the elapsed time to fix the most recent broken build
#'''Time Between Failures (TBF):''' the elapsed time between consecutive breakages (non-zero errors) of the same build
#'''Unit Test Count:''' the total number of unit tests on each automated build
#'''Unit Test Passed Count:''' the total number of unit tests that are passed on each automated build
#'''Unit Test Covered Lines:''' the count of the non-commented source lines scanned by the coverage tool that are exercised by statement-level automated unit testing on each automated build
#'''Validation Test Count:''' the total number of validation tests on each automated build
#'''Validation Test Passed Count:''' the total number of validation tests that are passed on each automated build
#'''Error Count:''' the total number of errors occurs on each automated build
#'''Warning Count:''' the total number of warnings occurs on each automated build
#'''Code Base Size:''' the total lines of code of the overall organization <ref name="CIRef7">Code base size, complexity and language choice" http://ayende.com/blog/3070/code-base-size-complexity-and-language-choice</ref>


Due to the fact that Test-Unit was the first testing framework available for Ruby, there are many legacy Ruby projects where they have implemented test suites using Test-Unit. For such projects, adopting to a new testing framework while maintaining the old test suites becomes a nightmare. For the very same reason, the Shoulda test framework was developed. Shoulda testing framework is derived from Test-Unit. Shoulda testing framework is a BDD based testing framework that easily integrates with old legacy Test-Unit test suites. TDD can also be easily achieved by using Shoulda testing framework.
These reporting metrics provide a way to improve the transparency of the project. It helps the team to understand the build, design, and code quality of the overall application; it helps us understand the trend of development process, and eliminate defects before introduction; it also helps developers to understand their individual impact to the project, and improve personal performance.


===Testing Framework Details===
==Advantages and Disadvantages==
----------------------------------------------------------------------------------------------------------------------------------------
===Advantages===
Continuous Integration has the following advantages:<ref name="CIRef1"></ref><ref name="wiki">Wikipedia "Continuous integration" http://en.wikipedia.org/wiki/Continuous_integration</ref>
*Early warning of broken/incompatible code and conflicting changes
*Guarantees successfully compiled software
*Visible program reporting and problem tracking
*Easy to revert the code base to bug-free state
*Reduce development integration effort
*Immediate unit testing for all changes
*Constant availability of functioning code for demo or release purposes
*High impact environment upgrade with low maintenance
*Help developers to understand their individual impact to project, and improve their personal performance
===Disadvantages===
Continuous Integration has the following disadvantages:<ref name="CIRef1"></ref><ref name="wiki"></ref>
*Migration of internal development projects into a CI environment requires a lot of initial setup time and tight planning
*Well-developed test-suite is required for the automated build
*Costs for CI building machines
*Requires a good understanding of CI and discretion when setting up projects


{| class="wikitable sortable" style="font-size: 95%; text-align: center; width: auto;"
Assuming that the CI environment is configured the way it supposes to be, it doesn’t matter how experienced the developer is, once he/she understands how to build and commit code he/she will be always benefited from it. For example, they can find out immediately if the build is broken after they commit the code, and learn how to write high-quality code.
|-
! Testing Framework
! #Released Date|Released Date
! #Released Version|Released Version
! #Current Version|Current Version
! #Testing Supported|Testing Type Supported
! #Install Command|Install Command
! Documentation/Version Support
|-
! Test:Unit
| 03-20-2008
| 1.2.3
| 2.4.0
| TDD,Mocking
| gem install test-unit
| [http://rubyforge.org/projects/test-unit/ Test:Unit Site]
[http://rubyforge.org/frs/?group_id=5650 Versions]
|-
! Shoulda
| 04-26-2008
| 4.1
| 2.11.3
| BDD,TDD
| gem install shoulda
| [https://github.com/thoughtbot/shoulda Shoulda Site]
[https://github.com/thoughtbot/shoulda/downloads Versions]
|-
! RSpec
| 05-26-2008
| 1.1.4
| 2.6.0
| BDD,TDD, DDD (Domain Driven Design)
| gem install rspec
| [http://rspec.info/documentation/ RSpec Site]
[https://github.com/dchelimsky/rspec/downloads Versions]
|-
! Riot
| 10-19-2009
| 0.9.11
| 0.12.5
| BDD,TDD
| gem install riot
| [https://github.com/thumblemonks/riot Riot Site]
[https://github.com/thumblemonks/riot/downloads Versions]
|-
! Cucumber
| 05-20-2009
| 0.3.6
| 1.0.6
| Acceptance Testing
| gem install cucumber
| [http://cukes.info/ Cucumber Site]
[https://github.com/cucumber/cucumber/downloads Versions]
|}
 
The above table displays information about most of the popular testing frameworks. It includes when these testing frameworks were released and what was the released version. It also displays their current version available. The list also includes some of the new testing frameworks like Riot.
 
===Examples===
 
Above we described how each Ruby testing framework differs from each other. But to better understand each testing framework lets go through a single example with tests written in most of the above testing frameworks.   
 
 
'''TravelTime'''
 
The sample that we have used below is a simple travel time example. If you give it a distance between two cities it will calculate the time it will take you to travel from City A to City B. It assumes that you will be driving at 60 miles/Hr with minimal traffic on the road.
 
<pre>
class TravelTime
  attr_accessor :distance
 
  #Divide the distance by 60 and return the float
  def time_to_travel
    @time = @distance.fdiv(60)
  end
 
  #Format the float to two decimal and return the string
  def format_time_to_travel(time)
    @time = format("%3.2f",time)
  end
 
  def initialize(distance)
    @distance = distance
  end
end
</pre>
 
 
The code above has two functions and one initialize function. The initialize function takes in distance between two cities and stores it into a distance variable. The first function, time_to_travel, when called calculates the time it takes to travel by dividing the distance provided during initialize stage by 60 (for 60 miles/hr speed). This function returns a float value as time. The second function, format_time_to_travel, takes a provided float number and will format the float number up to two decimal points and return the string. This function will be used in cases where you get a large float number for time, like 2.36666667 hours, and you just want it to be 2.37. Then you can pass the float number to get 2.37 hrs. Now lets see how this function can be tested in different testing frameworks.
 
 
'''Test-Unit'''
 
The first test framework that we are going to use is Test-Unit.  The code for TravelTimeTest_TU class is shown below, which seems similar to Java JUnit test code. The class below has a setup function which is used for setting up pre-test data before running tests. In our case we have initialized three different instances of TravelTime class with distances 120, 150, and 135 miles. The class also has a teardown function which can be used to discard all the data that was created before or during test execution. The remaining three functions are the actual tests that will be ran once this test is executed.
 
<pre>
gem 'test-unit'
require 'test/unit'
require File.dirname(__FILE__) + '/travel_time'
 
class TimeTravelTest_TU < Test::Unit::TestCase
 
  # Called before every test method runs. Can be used
  # to set up fixture information.
  def setup
    @traveltime1 = TravelTime.new(120)
    @traveltime2 = TravelTime.new(150)
    @traveltime3 = TravelTime.new(135)
  end
 
  def teardown
    # Do nothing
  end
 
  def test_time_to_travel_integer
    time = @traveltime1.time_to_travel
    assert_equal 2.0, time
  end
 
  def test_time_to_travel_floatWithOneDecimal
    time = @traveltime2.time_to_travel
    assert_equal 2.5, time
  end
 
  def test_time_to_travel_floatWithTwoDecimal
    temp = @traveltime3.time_to_travel
    time = @traveltime3.format_time_to_travel(temp)
    assert_equal "2.25", time
  end
 
end
</pre>
 
The first test function is to test if the provided distance of 120 miles returns a travel time of 2 hrs. The second test function is to test if the provided distance of 150 miles comes back with 2.5 hours. And the last test function checks if the travel time is a large floating number, like 2.2444445 hours, and if it that result can correctly be formatted to 2.25 hours. We have used different instances of travel time class for different tests. The reason for this is to see if the performance suffers between test frameworks. The results for this test are shown below. It shows that three test as described above were ran and three assertions were executed. All the tests passed. Executing this test took 0.004 seconds which is fast compared to some of the other test frameworks described below.
 
<pre>
Finished in 0.004 seconds.
 
3 tests, 3 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed
</pre>
 
 
'''RSpec'''
 
The second testing framework that we implemented was RSpec. As described in section 3.1 above, RSpec test framework is a BDD test framework which gives you more information about your test when it fails. As you can see below it does not have setup or teardown functions as Test-Unit did for pretest data setup or discarding data. All the test scenarios for RSpec framework are included inside a '''describe''' block. The individual tests are included inside an '''it''' block.
 
<pre>
gem 'rspec'
require File.dirname(__FILE__) + '/travel_time'
#spec TravelTimeTest_RSpec.rb --format specdoc #Command to run from command line
 
describe TravelTime do
 
  it "should be 2.0 hours when distance is 120 miles" do
    @traveltime1 = TravelTime.new(120)
    time = @traveltime1.time_to_travel
    time.should be == 2.0
  end
 
  it "should be 2.5 hours when distance is 150 miles" do
    @traveltime1 = TravelTime.new(150)
    time = @traveltime1.time_to_travel
    time.should be == 2.5
  end
 
  it "should be 2.25 hours when distance is 135 miles" do
    @traveltime1 = TravelTime.new(135)
    temp = @traveltime1.time_to_travel
    time = @traveltime1.format_time_to_travel(temp)
    time.should be == "2.25"
  end
 
end
</pre>
 
As seen from above we have three '''it''' blocks which include our three test scenarios. All the '''it''' blocks have a string statement which describes the test case. In case a test fails, it will display this string to show what functionality is broken. So for example, if last test case fails it will show that TravelTime should be 2.25 hours when distance is 135 miles but the return value was 2.45. This is the reason RSpec is more popular over Test-unit as it describes what went wrong and where. The result produced after executing this test is shown below. It shows all the test cases passed and it also shows what test cases were run. One of the downsides of using RSpec framework is that it takes more time to execute than Test-Unit, which can be seen below.
 
<pre>
TravelTime
- should be 2.0 hours when distance is 120 miles
- should be 2.5 hours when distance is 150 miles
- should be 2.25 hours when distance is 135 miles
 
Finished in 0.249016 seconds
 
3 examples, 0 failures
</pre>
 
 
'''Riot'''
 
The third testing framework that we tested was Riot. Riot is a relatively new testing framework and not very popular as of today. But we selected Riot, as it has some interesting qualities. Riot testing framework gives you the flavor of both worlds, RSpec and Test-Unit. Riot test framework is not based on test-unit but gives you some of the functionality for test-unit and the BDD functionality of RSpec while reducing the execution time. As seen below, Riot tests are usually defined between a '''context''' block. This block can contain one or more setup and assert blocks. In our test we have created three context blocks to test each case individually.
 
<pre>
gem 'riot'
require 'riot'
require File.dirname(__FILE__) + '/travel_time'
 
context TravelTime do
  setup {TravelTime.new(120)}
  asserts("Travel time is 2.0 hours when distance is 120 miles"){
    topic.time_to_travel == 2.0
  }
end
 
context TravelTime do
  setup {TravelTime.new(150)}
  asserts("Travel time is 2.5 hours when distance is 150 miles"){
    topic.time_to_travel == 2.5
  }
  end
 
context TravelTime do
  setup {TravelTime.new(135)}
  asserts("Travel time is 2.25 hours when distance is 135 miles"){
    topic.format_time_to_travel(topic.time_to_travel) == "2.25"
  }
end
</pre>
 
As seen from above the three context blocks test the same three cases but it has some new setup blocks. The setup block in each case does not instantiate any variables like Test-Unit. The entire setup box returns a common single item topic. This topic is used in assert blocks to see if the condition is met. This enables riot to execute faster as it does not have to go generate variables like other testing frameworks. All the assert blocks have strings like RSpec for notification. From the results below you can see that Riot testing frameworks is executed in the same time as Test-Unit, but it also shows the tests that were executed like RSpec. Riot was by far the easiest of all the testing framework to write.
 
<pre>
TravelTime
  + asserts Travel time is 2.0 hours when distance is 120 miles
TravelTime
  + asserts Travel time is 2.5 hours when distance is 150 miles
TravelTime
  + asserts Travel time is 2.25 hours when distance is 135 miles
 
3 passes, 0 failures, 0 errors in 0.004 seconds
</pre>
 
 
'''Shoulda'''
 
The last testing framework that we tested was Shoulda. Shoulda is fairly similar to Riot but is directly derived from Test-Unit. We included this framework as it’s one of the popular frameworks used in real world environments. Shoulda is also a mix of RSpec and Test-Unit. It’s a BDD testing framework. Similar to Test-Unit it has a setup block which is used to instantiate any variables. It uses the '''should''' block like RSpec to describe test scenarios.
 
<pre>
gem 'shoulda'
gem 'shoulda-context'
gem 'test-unit'
require 'test/unit'
require File.dirname(__FILE__) + '/travel_time'
 
class TravelTimeTest_Shoulda < Test::Unit::TestCase
  context TravelTime do
    setup do
      @traveltime1 = TravelTime.new(120)
      @traveltime2 = TravelTime.new(150)
      @traveltime3 = TravelTime.new(135)
    end
 
    should "be 2.0 hours when distance is 120 miles" do
      time = @traveltime1.time_to_travel
      assert_equal 2.0,time
    end
 
    should "be 2.5 hours when distance is 150 miles" do
      time = @traveltime2.time_to_travel
      assert_equal 2.5,time
    end
 
    should "be 2.25 hours when distance is 135 miles" do
      temp = @traveltime3.time_to_travel
      time = @traveltime3.format_time_to_travel
      assert_equal "2.25",time
    end
 
  end
end
</pre>
 
As seen from the code above it has same three test cases and has three assert statements. Like RSpec, it also has string messages for display. The result generated from this test is shown below and as you can see the timing is quite similar to Test-Unit and Riot.
 
<pre>
Started
...F...
Finished in 0.004500 seconds.
 
3 tests, 3 assertions, 0 failures, 0 errors, 0 skips
</pre>
 
==Additional Testing Frameworks==
===UI Testing Framework===
User interface (UI) testing frameworks allow for the testing of systems within a graphical environment.  This could take the form of a custom GUI written for the system to use, or leverage programs that are already developed, such as a word processor or web browser.
====Win32-AutoGUI====
Win32-AutoGUI is a framework that allows for GUI application testing of Windows binaries and Windows applications.  It facilitates integration testing of these binaries using Ruby based tools like RSpec and Cucumber regardless of the language used to create the binaries[https://github.com/robertwahler/win32-autogui].
 
===Acceptance testing===
In addition to the unit testing frameworks discussed above, there are testing frameworks that work as acceptance testing frameworks for Ruby projects. They perform the same functionality as unit testing, but are more like documentation and business criteria driven testing. Some of these types of testing frameworks include Cucumber, which is described below.
 
====Cucumber====
Cucumber is designed to allow you to execute feature documentation written in plain text (often known as "stories") [http://www.rubyinside.com/cucumber-the-latest-in-ruby-testing-1342.html]. Cucumber testing framework is a basic testing framework which allows you to describe your test cases in plain English text. On the back end, you make a file with regular expressions which can parse this plain English text and perform the test case. Today cucumber is widely used in Ruby projects, especially with the RSpec testing framework.


==Conclusion==
==Conclusion==
 
In the end continuous integration infrastructure provides that platform for big companies to produce software that is cost effective and profitable with less defects. Combining it with the agile software development methodologies makes sense as it gives big companies options to have shippable product that is well tested at the end of each sprint. Moreover CI also solves the overall integration problem of big software projects were small teams are working individually on small components that includes open source third party software by testing overall system functionality daily. The above reasoning shows why big companies that are adopting agile as their main software development strategy more than often also adopts continuous integration as their system and component integration and testing infrastructure.


==References==
==References==
 
<references/>
* [1] Test-Unit Site [http://test-unit.rubyforge.org/ Unit testing framework]
* [2] RSpec Site [http://rspec.info/ RSpec Testing]
* [3] Win32-autogui Site [https://github.com/robertwahler/win32-autogui Win32-autogui on Github]
* [4] Mike Gunderloy [http://www.rubyinside.com/cucumber-the-latest-in-ruby-testing-1342.html Cucumber: The Latest in Ruby Testing]


==Expand your knowledge==
==Expand your knowledge==
 
* Continuous Integration Wikipedia [http://en.wikipedia.org/wiki/Continuous_integration CI]
* [1] Dan Noth [http://behaviour-driven.org/ Behavior Driven Development]
* Why Continuous Integration [http://www.espusa.com/whitepapers/continuous_integration_v1.0.pdf]
* [2] Dave Astels [http://blog.daveastels.com/files/BDD_Intro.pdf A new look at Test Driven Development]
* [3] Scott W Ambler[http://www.agiledata.org/essays/tdd.html Introduction to Test Driven Design (TDD)]
* [4] IBM [http://publib.boulder.ibm.com/infocenter/db2luw/v8/index.jsp?topic=/com.ibm.db2.udb.doc/admin/c0005059.htm Benchmark Testing]
* [5] IBM Developers Work [http://www.ibm.com/developerworks/library/j-mocktest/index.html Unit testing with mock objects]
* [6] Steve Freeman [http://www.mockobjects.com/ Mock Objects]

Latest revision as of 16:49, 22 November 2011

Continous Integration

Introduction

Continuous Integration (CI) is a software development practice that is commonly used in Agile software development methodology. <ref name="Agile">"Manifesto for Agile Software Development" http://www.agilemanifesto.org/</ref>Agile is a software development method based on iterative and incremental development. In principle, it works with small team, requires developers to deliver working code in short iterations, and uses Test Driven Development (TDD), which requires developers to develop tests before writing code. Also it requires all the codes to be reviewed and tested continuously as the development goes, which is where CI fits in.

What is CI?

CI is used to ensure that all changes to a software project’s code are successfully built, tested, reported on, and rapidly made available to all parties after they are introduced. <ref name="CIRef1">Enterprise Solution Providers, Inc. "Why and How to build a Continuous Integration Environment for the .NET platform" http://www.espusa.com/whitepapers/continuous_integration_v1.0.pdf</ref> It is aimed to deliver high quality software using the least amount of time, by applying quality control throughout the whole software development process. Without CI, there is no chance for Agile to succeed in delivering refined products in short release cycles.

How does CI work?

The above diagram shows an overview of a simple CI environment, which will give us a good idea of how CI works. <ref name="CIRef2">Thomas Jaspers. "Continuous Integration – Overview" http://blog.codecentric.de/en/2009/11/</ref>.In order to apply quality control throughout the project, CI requires members of the team integrate their work into the main code repository at least daily, which leads to multiple integrations per day. Each developer will first build and run all the tests locally to ensure all the changes are good, then commit the changes to the repository using some version control tools, such as Subversion. Once the CI sever detects the new commits, it will kick out an automated build that runs all the automated tests that could detect compile/integration errors as quickly as possible. If the automated build fails, the team will be notified and required to resolve the error immediately. In addition, at least once every 24 hours, a nightly build will be executed to ensure the quality of the code in the main repository. <ref name="CIRef5">Laurie Williams. "Scrum + Engineering Practices: Experiences of Three Microsoft Teams" http://collaboration.csc.ncsu.edu/laurie/Papers/ESEM11_SCRUM_Experience_CameraReady.pdf</ref>

Why Implement CI?

Most of the major companies producing software for sale are in the business to make money. Companies have discovered that the best way to make money is decrease the number of defects going out to customers. Tons of research has been done on how to make quality software with fewer defects. The usual cost of fixing a defect slipping from one phase to next phase increases by ten times. So for example if a defect cost $60 to fix during the coding phase will cost $600 if found during validation testing. Moreover if a defect is shipped along with the product to the customers it can certain times cost millions of dollars. Additionally these small defects can have big impact on company’s reputation. Below are some examples <ref name="Bugs">Top Ten Most Infamous Software Bugs Of All Time" http://able2know.org/topic/129489-1</ref>which shows how small defects when slipped or goes out undetected can cost millions of dollars.

  1. In 2007 a single faulty piece of embedded software, on a network card, sends out faulty data on the United States Customs and Border Protection network, bringing the entire system to a halt. Nobody is able to leave or enter the U.S. from the LA Airport for over eight hours. Over 17,000 planes were grounded for the duration of the outage resulting in millions of dollars in damage.
  2. A forgotten error handling statement which caused the famous ping of death also known as blue screen in 1995. A lack of error handling in the IP fragmentation reassembly code makes it possible to crash many Windows, Macintosh, and Unix operating systems by sending a malformed “ping” packet from anywhere on the Internet.
  3. In 2004, EDS software giant introduced a large, complex IT system to the U.K.’s Child Support Agency (CSA). At the exact same time, the Department for Work and Pensions (DWP) decided to restructure the entire agency. The restructure and the new software were completely incompatible, and irreversible errors were introduced as a result. With over 500 bugs still reported as open in the new system, the clash of the two events has crippled the CSA’s network. As a result the system somehow managed to overpay 1.9 million people, underpay another 700,000, had $7 billion in uncollected child support payments, a backlog of 239,000 cases, 36,000 new cases “stuck” in the system, and has cost the UK taxpayers over $1 billion to date.

The above examples show why testing phase is as important as coding phase. The more you test the better the software becomes as fewer and fewer defects are found. Below are some of the reasons why big companies are leaning more towards adopting agile and CI for software development.

  1. Many developers work on a software through multiple agile teams. Though the focus of each agile team is a specific module of the software but it is important to make sure the overall integrity of the system is always correct.
  2. One of the main benefits of agile project development is to have a shippable product at the end of each sprint<ref name="Ready Product">Why Continuous Integration?" http://www.cavdar.net/2009/03/07/why-continuous-integration/</ref>. If testing is kept until the end than you can’t have a shippable product at the end of each sprint. You have to have some automated testing that can test this functionality on daily basis.
  3. The other problem that used to occur by following traditional software development was that management did not know the exact status of the product which includes the number of bugs in the system until the validation phase. This can cause problem in releasing software on time. CI gives an option to generate various reports to solve these kinds of problems and to give management early indication if the product release can be done on time.
  4. The main reason for using CI is to cut down on validation cost by finding the bugs early during coding phase.
  5. CI infrastructure provides way to decrease Time to Response (TTR) for fixing defects.
  6. Additionally some CI infrastructure provides valuable tools like build management and build acceleration. With build acceleration developers and QA do not have to wait long periods to get a completed builds. Build acceleration decreases the build time by performing non-dependent tasks in parallel resulting in increased productivity.

CI Setup and Reporting

There are lots of different ways continuous integration can be incorporated with a software product development. The level of automation varies by project to project. Some project might be able to achieve complete automation and continuous integration whereas some project can be restricted by the testing infrastructure. For example when making a software for say airplane controls there is only limited amount of automated testing that you can do to test. It requires some level of manual steps. Whereas for computer application like word you can automate the testing and can achieve 100% continuous integration.

There are many different kinds of software that are available out there for implementing continuous integration within a software project. Some of the software includes Electric Commander<ref name="EC">Electric Commander" http://www.electriccloud.com/</ref> by Electric Cloud, CI server by Hudson, and Team Foundation Server, Lab Management, Test Manager by Microsoft. All of the software listed above can be used as the backbone for setting up the CI infrastructure. Additionally, different reporting services can also be added to achieve additional goals. Below are the usual steps that are performed once the CI infrastructure is incorporated.



  1. Whenever the new code or functionality is checked into the repository an automated build process is started.
  2. Once the product is build unit tests are executed to make sure all the unit tests are passing and that the new functionality has not broken the old functionality. Reports are generated for the results.
  3. Once unit test are completed and passed automated smoke tests are performed. The application is usually deployed on a remote machine as a fresh copy and the system testing is performed. Reports are generated for the results.
  4. Then additional reporting are performed on the code to see it the code has fulfilled the required criteria (For example, code coverage > 90%, code complexity, Dead code, Duplicate code, Coding standard enforcement)
  5. Lastly, when all the 4 steps above are passed additional matrix is generated for management to see. For example TTR to fix a bug, Time between Failures (TBF), Error counts, and Warning reports.

If any of the above steps fail an automated email is generated and send to targeted user to notify that the new code submitted to the repository needs to be fixed. In this way teams can achieve that quick response instead of waiting to hear from validation team which can take days compared to minutes through CI.

CI Reporting Metrics

The team can also generate reports that indicate the overall health of the application on the CI environment. This application health can be measured by the CI reporting metrics. Some examples of these metrics are shown below:<ref name="CIRef6">Designing data warehouse for equipment management system" http://www.tandfonline.com/doi/abs/10.1080/00207540701222776?journalCode=tprs20#preview</ref>

  1. Time To Response (TTR): the elapsed time to fix the most recent broken build
  2. Time Between Failures (TBF): the elapsed time between consecutive breakages (non-zero errors) of the same build
  3. Unit Test Count: the total number of unit tests on each automated build
  4. Unit Test Passed Count: the total number of unit tests that are passed on each automated build
  5. Unit Test Covered Lines: the count of the non-commented source lines scanned by the coverage tool that are exercised by statement-level automated unit testing on each automated build
  6. Validation Test Count: the total number of validation tests on each automated build
  7. Validation Test Passed Count: the total number of validation tests that are passed on each automated build
  8. Error Count: the total number of errors occurs on each automated build
  9. Warning Count: the total number of warnings occurs on each automated build
  10. Code Base Size: the total lines of code of the overall organization <ref name="CIRef7">Code base size, complexity and language choice" http://ayende.com/blog/3070/code-base-size-complexity-and-language-choice</ref>

These reporting metrics provide a way to improve the transparency of the project. It helps the team to understand the build, design, and code quality of the overall application; it helps us understand the trend of development process, and eliminate defects before introduction; it also helps developers to understand their individual impact to the project, and improve personal performance.

Advantages and Disadvantages

Advantages

Continuous Integration has the following advantages:<ref name="CIRef1"></ref><ref name="wiki">Wikipedia "Continuous integration" http://en.wikipedia.org/wiki/Continuous_integration</ref>

  • Early warning of broken/incompatible code and conflicting changes
  • Guarantees successfully compiled software
  • Visible program reporting and problem tracking
  • Easy to revert the code base to bug-free state
  • Reduce development integration effort
  • Immediate unit testing for all changes
  • Constant availability of functioning code for demo or release purposes
  • High impact environment upgrade with low maintenance
  • Help developers to understand their individual impact to project, and improve their personal performance

Disadvantages

Continuous Integration has the following disadvantages:<ref name="CIRef1"></ref><ref name="wiki"></ref>

  • Migration of internal development projects into a CI environment requires a lot of initial setup time and tight planning
  • Well-developed test-suite is required for the automated build
  • Costs for CI building machines
  • Requires a good understanding of CI and discretion when setting up projects

Assuming that the CI environment is configured the way it supposes to be, it doesn’t matter how experienced the developer is, once he/she understands how to build and commit code he/she will be always benefited from it. For example, they can find out immediately if the build is broken after they commit the code, and learn how to write high-quality code.

Conclusion

In the end continuous integration infrastructure provides that platform for big companies to produce software that is cost effective and profitable with less defects. Combining it with the agile software development methodologies makes sense as it gives big companies options to have shippable product that is well tested at the end of each sprint. Moreover CI also solves the overall integration problem of big software projects were small teams are working individually on small components that includes open source third party software by testing overall system functionality daily. The above reasoning shows why big companies that are adopting agile as their main software development strategy more than often also adopts continuous integration as their system and component integration and testing infrastructure.

References

<references/>

Expand your knowledge

  • Continuous Integration Wikipedia CI
  • Why Continuous Integration [1]
Retrieved from "https://wiki.expertiza.ncsu.edu/index.php?title=CSC/ECE_517_Fall_2011/ch6_6f_jd&oldid=56200"