CSC/ECE 517 Fall 2011/ch2 2e ad

Testing frameworks for Ruby

Introduction

A testing framework is used on a software project when it has grown large enough that manual debugging and system upkeep would be too time and resource intensive. The framework provides a way to automate testing to provide for low cost maintenance. The framework itself contains a set of assumptions, concepts, and tools that allow for various types of testing. Choosing the right framework depends on many factors associated with the system in development and the mindset of the developers and stakeholders. Every framework has a few basic features: A process for defining the format to express user expectations, a way to execute the tests, and a way to report the results of the test to the user Examples are provided later to help give context to the definitions below.

Definitions

Test Driven Development (TDD)

Test-Driven Development (TDD) is the process of writing tests first and foremost. These tests represent desired improvements and/or new functionality in the system. Since these changes have yet to be implemented, the tests must fail when initially run. Then code is written until these tests can be passed, assuring that the desired behavior is exhibited by the code. Finally the code is rewritten to meet acceptable standards.

Behavior Driven Development(BDD)

Behavior-Driven Development (BDD) is very similar to test driven development; it is a different approach to the same subject. BDD is an approach to software development that combines Test-Driven Development, Domain Driven Design, and Acceptance Test-Driven Planning. With this type of development users are more focused on the behavior that is exhibited in the code and less on testing each method and class. The goal of BDD is to use terminology focused on the behavioral aspects of the system to allow a seamless transition between specification, design, and implementation of a system. BDD has a much better alignment with business concerns.

Mocking in Testing

Mocking provides users with a helpful way to write unit tests for objects. These mock objects are used to imitate existing roles in a system. Instead of calling these real objects, the mock objects are called instead. They will assert that the correct methods were called with the right parameters in the right order. So instead of needing to create a database and fill it with rows, we can instead use a mock object that would supply the same information as the database. It is the high coupling between modules and tests that creates the need for a mocking framework.

Benchmarking in Testing

Benchmark testing is used to determine things like throughput performance, CPU load, and memory utilization. It is good practice to perform benchmark testing on any system going into a production environment. This allows users to determine current system efficiency and identify things such as performance bottlenecks throughout the code. This can be done by refining code and/or database configurations. Benchmark tests must be run inside a repeatable environment in order to yield results that can be accurately compared.

Testing Framework

A testing framework allows for automated software testing. It contains a set of assumptions, concepts, and tools that assist in this testing. The main advantage of using a testing framework is the lower cost for maintenance. The testing framework is responsible for :

• defining the format in which to express expectations
• creating a mechanism to hook into or drive the application under test
• executing the tests
• reporting results

Testing Framework Evolution

Test driven development (TDD) is linked with the concepts of extreme programming which began in 1999. It offered more than just simple validation of correctness; it also drove the design of the program. By focusing on the test cases first, one needed to imagine how the functionality would be used by clients. Test-driven development ensured that all written code was covered by at least one test. This gave the programming team, as well as users and stakeholders, a greater level of confidence in the code.
One major problem seen with TDD can be the mindset it puts users in. This mindset could be detrimental to the overall project. Users needed to start thinking in terms of behavior specifications, not verification tests. This required two things:

• A language shift in traditional TDD
• Driving your tests from a domain/user story perspective rather than technical perspective

TDD vs BDD

The following example shows how a typical test is setup for TDD.

[TestFixture]
public class directory_tests
{
    [SetUp]
    public void SetUp()
    {
        // setup code for test
    }
 
    [Test]
    public void can_add_person_to_addressbook()
    {
        // test assertions
    }
}

Now here is the same test written in the style of BDD.

[Concern("Address Book"), TestFixture]
public class when_searching_for_a_person_in_an_addressbook : ContextSpecification
{
    [Context]
    protected override void  Context()
    {
        base.Context();
 
        // setup code for test
    }
 
    [Observation, Test]
    public void addressbook_contains_person()
    {
        // test assertions
    }
}

Our TestFixtures have now been grouped into concerns. Setup has been replaced by Context and instead of Test, we now have Observations. Tests now become the acceptance criteria for the context allowing us to write tests from a use case perspective

Ruby Testing Frameworks

Compared to five years ago the number of testing frameworks available for Ruby has increased rapidly. Selecting a perfect testing framework for a particular project has become more difficult.

Overview of testing framework

Selection of testing frameworks for Ruby projects depend on many different factors, such as: which approach are we going to use TDD or BDD, is GUI testing included, is acceptance testing required, do we have to merge legacy Ruby test code with new testing framework? Altough GUI testing and acceptance testing are out of scope for this wiki, we have made an attempt to cover them in some detail at the end of this wiki page.

Test:Unit

Test-Unit is one of the oldest testing frameworks for Ruby testing. Test-Unit is basically used for performing test driven development similar to how Java’s JUnit is used. It’s in a way very similar to Java’s JUnit. Test-Unit framework, when initially released, was part of the ruby distribution package. The newer version since then has been converted to gems and has been provided to developers as a standalone package. Additionally, some of the features like notify, capybara, GTK+,Tk, and FOX Toolkit have been split from the package and provided separately[1].

RSpec

RSpec is a Behavior-Driven Development (BDD) tool for Ruby programmers. As defined earlier, BDD is an approach to software development that combines Test-Driven Development, Domain Driven Design, and Acceptance Test-Driven Planning. RSpec helps you do the TDD part of that equation, focusing on the documentation and design aspects of TDD [2]

RSpec is one of the most widely used testing frameworks for Ruby. The main difference between RSpec and Test-Unit is semantics. RSpec provides a way to write tests which are in a more human readable form. This reduces the expense for maintaining the test and also provides more easily understandable tests. Additionally when RSpec tests fail, it provides more information about what functionality failed, and why. This compared to Test-Unit tests, where you must put custom messages on all assertion statements to point out exactly what happened, is much simplier.

Riot

Riot is another BDD Ruby unit testing framework. Riot testing framework is a mixture of both RSpec and Test-Unit or RSpec and Shoulda (which is based on Test-Unit). The main purpose of authoring the Riot testing framework was to come up with a Ruby testing framework which is fast in execution but is in a more human readable form. As in the case of other testing frameworks, Riot does not have variables described or initialized in its test cases. It has a setup block which returns a variable as default. This allows Riot to be fast during execution, as it does not have to initialize or do much of pre-test setup. One of the unique attributes of the Riot testing framework is that the tests only focus on small functionality.

Shoulda

Due to the fact that Test-Unit was the first testing framework available for Ruby, there are many legacy Ruby projects where they have implemented test suites using Test-Unit. For such projects, adopting to a new testing framework while maintaining the old test suites becomes a nightmare. For the very same reason, the Shoulda test framework was developed. Shoulda testing framework is derived from Test-Unit. Shoulda testing framework is a BDD based testing framework that easily integrates with old legacy Test-Unit test suites. TDD can also be easily achieved by using Shoulda testing framework.

Testing Framework Details

---

Testing Framework	Released Date	Released Version	Current Version	Testing Type Supported	Install Command	Documentation/Version Support
Test:Unit	03-20-2008	1.2.3	2.4.0	TDD,Mocking	gem install test-unit	Test:Unit Site Versions
Shoulda	04-26-2008	4.1	2.11.3	BDD,TDD	gem install shoulda	Shoulda Site Versions
RSpec	05-26-2008	1.1.4	2.6.0	BDD,TDD, DDD (Domain Driven Design)	gem install rspec	RSpec Site Versions
Riot	10-19-2009	0.9.11	0.12.5	BDD,TDD	gem install riot	Riot Site Versions
Cucumber	05-20-2009	0.3.6	1.0.6	Acceptance Testing	gem install cucumber	Cucumber Site Versions

The above table displays information about most of the popular testing frameworks. It includes when these testing frameworks were released and what was the released version. It also displays their current version available. The list also includes some of the new testing frameworks like Riot.

Examples

Above we described how each Ruby testing framework differs from each other. But to better understand each testing framework lets go through a single example with tests written in most of the above testing frameworks.

TravelTime

The sample that we have used below is a simple travel time example. If you give it a distance between two cities it will calculate the time it will take you to travel from City A to City B. It assumes that you will be driving at 60 miles/Hr with minimal traffic on the road.

class TravelTime
   attr_accessor :distance
  
  #Divide the distance by 60 and return the float
  def time_to_travel
    @time = @distance.fdiv(60)
  end

  #Format the float to two decimal and return the string
  def format_time_to_travel(time)
    @time = format("%3.2f",time)
  end
  
  def initialize(distance)
    @distance = distance
  end
end

The code above has two functions and one initialize function. The initialize function takes in distance between two cities and stores it into a distance variable. The first function, time_to_travel, when called calculates the time it takes to travel by dividing the distance provided during initialize stage by 60 (for 60 miles/hr speed). This function returns a float value as time. The second function, format_time_to_travel, takes a provided float number and will format the float number up to two decimal points and return the string. This function will be used in cases where you get a large float number for time, like 2.36666667 hours, and you just want it to be 2.37. Then you can pass the float number to get 2.37 hrs. Now lets see how this function can be tested in different testing frameworks.

Test-Unit

The first test framework that we are going to use is Test-Unit. The code for TravelTimeTest_TU class is shown below, which seems similar to Java JUnit test code. The class below has a setup function which is used for setting up pre-test data before running tests. In our case we have initialized three different instances of TravelTime class with distances 120, 150, and 135 miles. The class also has a teardown function which can be used to discard all the data that was created before or during test execution. The remaining three functions are the actual tests that will be ran once this test is executed.

gem 'test-unit'
require 'test/unit'
require File.dirname(__FILE__) + '/travel_time'

class TimeTravelTest_TU < Test::Unit::TestCase

  # Called before every test method runs. Can be used
  # to set up fixture information.
  def setup
    @traveltime1 = TravelTime.new(120)
    @traveltime2 = TravelTime.new(150)
    @traveltime3 = TravelTime.new(135)
  end

  def teardown
    # Do nothing
  end

  def test_time_to_travel_integer
    time = @traveltime1.time_to_travel
    assert_equal 2.0, time
  end

  def test_time_to_travel_floatWithOneDecimal
    time = @traveltime2.time_to_travel
    assert_equal 2.5, time
  end

  def test_time_to_travel_floatWithTwoDecimal
    temp = @traveltime3.time_to_travel
    time = @traveltime3.format_time_to_travel(temp)
    assert_equal "2.25", time
  end

end

The first test function is to test if the provided distance of 120 miles returns a travel time of 2 hrs. The second test function is to test if the provided distance of 150 miles comes back with 2.5 hours. And the last test function checks if the travel time is a large floating number, like 2.2444445 hours, and if it that result can correctly be formatted to 2.25 hours. We have used different instances of travel time class for different tests. The reason for this is to see if the performance suffers between test frameworks. The results for this test are shown below. It shows that three test as described above were ran and three assertions were executed. All the tests passed. Executing this test took 0.004 seconds which is fast compared to some of the other test frameworks described below.

Finished in 0.004 seconds.

3 tests, 3 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed

RSpec

The second testing framework that we implemented was RSpec. As described in section 3.1 above, RSpec test framework is a BDD test framework which gives you more information about your test when it fails. As you can see below it does not have setup or teardown functions as Test-Unit did for pretest data setup or discarding data. All the test scenarios for RSpec framework are included inside a describe block. The individual tests are included inside an it block.

gem 'rspec'
require File.dirname(__FILE__) + '/travel_time'
#spec TravelTimeTest_RSpec.rb --format specdoc #Command to run from command line

describe TravelTime do

  it "should be 2.0 hours when distance is 120 miles" do
    @traveltime1 = TravelTime.new(120)
    time = @traveltime1.time_to_travel
    time.should be == 2.0
  end

  it "should be 2.5 hours when distance is 150 miles" do
    @traveltime1 = TravelTime.new(150)
    time = @traveltime1.time_to_travel
    time.should be == 2.5
  end

  it "should be 2.25 hours when distance is 135 miles" do
    @traveltime1 = TravelTime.new(135)
    temp = @traveltime1.time_to_travel
    time = @traveltime1.format_time_to_travel(temp)
    time.should be == "2.25"
  end

end

As seen from above we have three it blocks which include our three test scenarios. All the it blocks have a string statement which describes the test case. In case a test fails, it will display this string to show what functionality is broken. So for example, if last test case fails it will show that TravelTime should be 2.25 hours when distance is 135 miles but the return value was 2.45. This is the reason RSpec is more popular over Test-unit as it describes what went wrong and where. The result produced after executing this test is shown below. It shows all the test cases passed and it also shows what test cases were run. One of the downsides of using RSpec framework is that it takes more time to execute than Test-Unit, which can be seen below.

TravelTime
- should be 2.0 hours when distance is 120 miles
- should be 2.5 hours when distance is 150 miles
- should be 2.25 hours when distance is 135 miles

Finished in 0.249016 seconds

3 examples, 0 failures

Riot

The third testing framework that we tested was Riot. Riot is a relatively new testing framework and not very popular as of today. But we selected Riot, as it has some interesting qualities. Riot testing framework gives you the flavor of both worlds, RSpec and Test-Unit. Riot test framework is not based on test-unit but gives you some of the functionality for test-unit and the BDD functionality of RSpec while reducing the execution time. As seen below, Riot tests are usually defined between a context block. This block can contain one or more setup and assert blocks. In our test we have created three context blocks to test each case individually.

gem 'riot'
require 'riot'
require File.dirname(__FILE__) + '/travel_time'

context TravelTime do
  setup {TravelTime.new(120)}
  asserts("Travel time is 2.0 hours when distance is 120 miles"){
    topic.time_to_travel == 2.0
  }
end

context TravelTime do
  setup {TravelTime.new(150)}
  asserts("Travel time is 2.5 hours when distance is 150 miles"){
    topic.time_to_travel == 2.5
  }
  end

context TravelTime do
  setup {TravelTime.new(135)}
  asserts("Travel time is 2.25 hours when distance is 135 miles"){
    topic.format_time_to_travel(topic.time_to_travel) == "2.25"
  }
end

As seen from above the three context blocks test the same three cases but it has some new setup blocks. The setup block in each case does not instantiate any variables like Test-Unit. The entire setup box returns a common single item topic. This topic is used in assert blocks to see if the condition is met. This enables riot to execute faster as it does not have to go generate variables like other testing frameworks. All the assert blocks have strings like RSpec for notification. From the results below you can see that Riot testing frameworks is executed in the same time as Test-Unit, but it also shows the tests that were executed like RSpec. Riot was by far the easiest of all the testing framework to write.

TravelTime
  + asserts Travel time is 2.0 hours when distance is 120 miles
TravelTime
  + asserts Travel time is 2.5 hours when distance is 150 miles
TravelTime
  + asserts Travel time is 2.25 hours when distance is 135 miles

3 passes, 0 failures, 0 errors in 0.004 seconds

Shoulda

The last testing framework that we tested was Shoulda. Shoulda is fairly similar to Riot but is directly derived from Test-Unit. We included this framework as it’s one of the popular frameworks used in real world environments. Shoulda is also a mix of RSpec and Test-Unit. It’s a BDD testing framework. Similar to Test-Unit it has a setup block which is used to instantiate any variables. It uses the should block like RSpec to describe test scenarios.

gem 'shoulda'
gem 'shoulda-context'
gem 'test-unit'
require 'test/unit'
require File.dirname(__FILE__) + '/travel_time'

class TravelTimeTest_Shoulda < Test::Unit::TestCase
  context TravelTime do
    setup do
      @traveltime1 = TravelTime.new(120)
      @traveltime2 = TravelTime.new(150)
      @traveltime3 = TravelTime.new(135)
    end

    should "be 2.0 hours when distance is 120 miles" do
      time = @traveltime1.time_to_travel
      assert_equal 2.0,time
    end

    should "be 2.5 hours when distance is 150 miles" do
      time = @traveltime2.time_to_travel
      assert_equal 2.5,time
    end

    should "be 2.25 hours when distance is 135 miles" do
      temp = @traveltime3.time_to_travel
      time = @traveltime3.format_time_to_travel
      assert_equal "2.25",time
    end

  end
end

As seen from the code above it has same three test cases and has three assert statements. Like RSpec, it also has string messages for display. The result generated from this test is shown below and as you can see the timing is quite similar to Test-Unit and Riot.

 Started
 ...F...
 Finished in 0.004500 seconds.

 3 tests, 3 assertions, 0 failures, 0 errors, 0 skips

Additional Testing Frameworks

UI Testing Framework

User interface (UI) testing frameworks allow for the testing of systems within a graphical environment. This could take the form of a custom GUI written for the system to use, or leverage programs that are already developed, such as a word processor or web browser.

Win32-AutoGUI

Win32-AutoGUI is a framework that allows for GUI application testing of Windows binaries and Windows applications. It facilitates integration testing of these binaries using Ruby based tools like RSpec and Cucumber regardless of the language used to create the binaries[3].

Acceptance testing

In addition to the unit testing frameworks discussed above, there are testing frameworks that work as acceptance testing frameworks for Ruby projects. They perform the same functionality as unit testing, but are more like documentation and business criteria driven testing. Some of these types of testing frameworks include Cucumber, which is described below.

Cucumber

Cucumber is designed to allow you to execute feature documentation written in plain text (often known as "stories") [4]. Cucumber testing framework is a basic testing framework which allows you to describe your test cases in plain English text. On the back end, you make a file with regular expressions which can parse this plain English text and perform the test case. Today cucumber is widely used in Ruby projects, especially with the RSpec testing framework.

Conclusion

As shown from above ruby testing frameworks have evolved tremendously over time and today there are many different testing frameworks that are available for testing ruby. But selection of right testing framework depends on the requirements and needs of the project. Many different factors like use of testing framework (Unit testing, Acceptance testing etc), test execution time, and Quality of test failure results contribute in selection of the best testing framework.

References

• [1] Test-Unit Site Unit testing framework
• [2] RSpec Site RSpec Testing
• [3] Win32-autogui Site Win32-autogui on Github
• [4] Mike Gunderloy Cucumber: The Latest in Ruby Testing

Expand your knowledge

• [1] Dan Noth Behavior Driven Development
• [2] Dave Astels A new look at Test Driven Development
• [3] Scott W AmblerIntroduction to Test Driven Design (TDD)
• [4] IBM Benchmark Testing
• [5] IBM Developers Work Unit testing with mock objects
• [6] Steve Freeman Mock Objects