CSC/ECE 517 Fall 2010/ch1 1f vn

Unit-Testing Frameworks for Ruby

This wiki-page serves as a knowledge source for understanding the different Unit-Testing Frameworks available for Ruby.

Introduction

Unit testing is a method by which we can isolate and test a unit functionality of the program, typically individual methods during and long after the code is written. ^[1] It helps to identify errors in the program even without running the entire program. It also helps to do regressing testing to identify buggy code additions in the future. Unit testing frameworks provides us with constructs which simplifies the process of unit testing. Using a standard unit test framework helps other developers to add test cases easily. ^[2] This chapter walks through three different unit testing frameworks available for Ruby and explains how to use them with examples. The three commonly used unit testing frameworks for ruby are

1. Test::Unit
2. Shoulda
3. RSpec

Test::Unit

Now we shall consider Test::Unit framework

Background

Ruby comes with an in-built, ready to use unit testing framework called Test::Unit. It is a XUnit type framework and typically have a setup method for initialization, a teardown method for cleanup and the actual test methods itself. The tests themselves are bundled separately in a test class from the code it is testing.

Test Fixture

Test fixture represents the initial environment setup(eg. initialization data) and/or the expected outcome of the tests for that environment. This is typically done in the setup() and teardown() methods and it helps to separate test initialization and cleanup from the actual tests. It also helps to reuse the same fixture for more than one tests.^[3]

For example, consider a method prime_check(num) which takes an integer number as input and outputs whether it is prime number or not. In order to unit test this method we can create the following fixture containing a 2-dimensional array with a number and the expected output of whether it is prime or not.

 def setup
   @NUMBERS = [[3,true], [4,false], [7,true], [10,false]]    
 end

Assertions

The core part of test::unit framework is the ability to assert a statement of expected outcome. If an assert statement is correct then the test will proceed, otherwise the test will fail. This feature helps us to verify the method under test with different types of inputs and track the results. Test::unit provides a bunch of assert methods for this purpose:

assert( boolean, [message] )	True if boolean
assert_equal( expected, actual, [message] ) assert_not_equal( expected, actual, [message] )	True if expected == actual
assert_raise( Exception,... ) {block} assert_nothing_raised( Exception,...) {block}	True if the block raises (or doesn't) one of the listed exceptions.

For the full list of assertion methods provided by test::unit refer to test::unit assertions. ^[4]

Example

The test case class BinarySearchTest subclasses the Test::Unit::TestCase class and overrides setup and teardown methods. The test methods should start with 'test_' prefix. This helps in isolating the test methods from the helper methods if any. The Test::Unit::TestCase class takes care of making the test methods into tests, wrapping them into a suite and running the individual tests. The test results are collected into Test::Unit::TestResult object.

   require 'test/unit'
   require 'binarysearch'

   class BinarySearchTest < Test::Unit::TestCase

     def setup
       @input_array = [1,2,3,4,5]      #The test fixture is initialized        
     end
     
     def test_success_left_half
       assert_equal(binary_search(@input_array,1),true)   #tests if the element present in left half of the array is found
       assert_equal(binary_search(@input_array,2),true)    
     end

     def test_success_right_half
       assert_equal(binary_search(@input_array,5),true)   #tests if the element present in the right half of the array is found
       assert_equal(binary_search(@input_array,4),true)
     end

     def test_success_middle                             #tests if the element present in the middle of the array is found
       assert_equal(binary_search(@input_array,3),true)
     end

     def test_failure
       assert_equal(binary_search(@input_array,6),false)   #tests if an element not present in the array is not found
     end

     def teardown
       #nothing to do her
     end

   end

Here we have four test methods testing different logical paths of the binary search algorithm. Each test method can have one or more assert statements to test whether conditions are correct in each situation. To run the tests we simply have to run the file binary_search_test.rb and the output is as follows:

 Loaded suite binarysearch
 Started
 F...
 Finished in 0.372 seconds.
 
   1) Failure:
 test_failure(BinarySearchTest) [binarysearch.rb:25]:
 <true> expected but was
 <false>.
 
 4 tests, 6 assertions, 1 failures, 0 errors

The results show that the last test case test_failure, testing the negative scenario is failing. The reason is because the assert statement is expecting false when number 6, which not present in the array is passed. But the binary_search method is returning true.

Test Suite

Sometimes it is useful to combine a bunch of related test cases and run them as batch. Test::Unit provides a class called TestSuite for this purpose. The below example demonstrates how to bundle binary and sequential test case classes into a single search test suite.

  require 'test/unit/testsuite'
  require 'binary_search_test'
  require 'sequential_search_test'
 
  class Search_Tests
    def self.suite
      suite = Test::Unit::TestSuite.new
      suite << BinarySearchTest.suite
      suite << SequentialSearchTest.suite
      return suite
    end
  end
  Test::Unit::UI::Console::TestRunner.run(Search_Tests)

Shoulda

Background

One of the downsides of Test::Unit is we end up writing lots of code in order to test the actual code which is sometimes not easy to understand. Shoulda is a library that allows us to write better and more understandable tests for ruby application. Shoulda is not a testing framework by itself. It extends the Test::Unit framework with the idea of context. We can mix Test::Unit test cases with Shoulda test cases. Shoulda allows us to provide context to the tests so that we can group the tests according to a specific feature or scenario. ^[5]

Example

  require 'shoulda'
  require 'test/unit'
  require 'binarysearch'

  class BinarySearchTest < Test::Unit::TestCase
     
    context "Input array of size 5" do
     
      def setup
        @input_array = [1,2,3,4,5]
      end
     
      should "have the number in the left half of the array" do    #tests if the element present in left half of the array is found
        assert_equal(binary_search(@input_array,1),true)
        assert_equal(binary_search(@input_array,2),true)
      end

      should "have the number in the right half of the array" do   #tests if the element present in right half of the array is found
        assert_equal(binary_search(@input_array,5),true)
        assert_equal(binary_search(@input_array,4),true)
      end

      should "have the number in the middle of the array" do       #tests if the element present in middle of the array is found
        assert_equal(binary_search(@input_array,3),true)
      end

      should "not have the number in the array" do                #tests if the element not present in the array is not found
        assert_equal(binary_search(@input_array,6),false)
      end

      def teardown
        #nothing to do here
      end
    end
   
    context "Input array of size 1" do
    
      def setup
        @input_array = [1]
      end
     
      should "have the number in the array" do
        assert_equal(binary_search(@input_array,1),true)           #tests if an element is found in the single element array
      end

      should "not have the number in the array" do                 # tests if an element is not found in the single element array
        assert_equal(binary_search(@input_array,2),true)         
      end
     
      def teardown
        #nothing to do here
      end
    end
  end

Notice that we are still sub-classing the Test::Unit::TestCase class. In this example we have two contexts one for input array of size 5 and the other for input array of size 1. Each context has its own environment of setup/teardown methods. We can also create nested contexts - the outer setup gets run before the execution of each of the inner contexts. And the setup in the inner contexts gets run when running that context. Each should construct is converted into individual test methods and are run. If a test case fails we will get a better description of what that test case is doing from the should description.

RSpec

Now let us consider about RSpec Testing Framework in detail.

Background

Behaviour Driven Development (BDD) ^[6] is an Agile development process ^[7] that comprises aspects of Acceptance Test Driven Planning ^{[8] [9]}, Domain Driven Design ^{[10] [11]} and Test Driven Development (TDD). ^{[12] [13]} RSpec is a Behavioural Driven Development (BDD) tool aimed at Test Driven Development, originally created by Dave Astels and Steven Baker. However David Chelimsky ^[14] is really the gatekeeper of the RSpec project. ^[15]

Traditionally we use Unit Test frameworks like JUnit, NUnit or RUnit for writing Test cases. We spend a lot of time writing tests that test every unit of code in our software system. Instead we can shift our focus from Unit testing to Behaviour testing or Behaviour Driven Development (BDD) using RSpec. By focusing on the behaviour of the system it helps clarify in our minds what the system should actually be doing. It also helps us to perform more ‘useful’ tests. Useful tests, cover what the system should be doing and build in enough redundancy so that it should be easy to refactor our code without having to re-write every test.

RSpec is really two projects merged into one. The RSpec project pages describes these merged projects as:

1. application level behaviour described by a Story Framework
   
2. object level behaviour described by a Spec Framework
   
   

Dan North created rbehave ^[16] which is the Story Framework and David Chelimsky created the Spec Framework. By encompassing two frameworks RSpec equips a programmer with a thorough set of testing tools, allowing you to think about your software problem from a number of perspectives.

Prerequisites

The prerequisites are

1. Ruby 1.8.4 or later
2. RSpec Gem (latest)

To install Ruby, please visit official Ruby Website ^[17]
To install RSpec, open a command shell, go to /bin folder in Ruby directory and type

> gem install rspec

Terms & Definitions

Here are some terms which are used frequently while working with RSpec. ^[18]

1. subject code - The code whose behavior is specified using RSpec
2. expectation - The expected behavior of subject code is expressed using expectation (Similar to 'Assertions' statements used in Test::Unit or other tools in other languages)
3. code example - An executable example containing the subject code and the expectations (Similar to 'Test Method' terminology used elsewhere)
4. example group - A group of code examples (Similar to 'Test Case' terminology used elsewhere)
5. spec file - A file which contains one or more example groups

Example

Let us go through an example to be clear on the usage of RSpec.

  require 'binarysearch'
  
  describe BinarySearchTest do
    before(:all) do
      @input_array = [1, 2, 3, 4, 5] # The Input Array
    end

    after(:all) do
      # do nothing here
    end

    it "should be in the left-half of the array" do  # Test case for element to be present in left-half of given array
      bst = BinarySearch.new
      bst.should be_binary_search(1)
    end

    it "should be in the right-half of the array" do  # Test case for element to be present in right-half of given array
      bst = BinarySearch.new
      bst.should be_binary_search(5)
    end

    it "should be in the middle of the array" do  # Test case for element to be present in the middle of given array
      bst = BinarySearch.new
      bst.should be_binary_search(3)
    end

    it "should not be in the array"
  end

Here it is assumed that the method binary_search will return true/false based on whether the provided value exists in the array or not. It should produce the following output. The un-implemented tests are marked as Pending in the output.

  BinarySearchTest
  - should be in the left-half of the array
  - should be in the right-half of the array
  - should be in the middle of the array
  - should not be in the array (PENDING: Not Yet Implemented)

  Pending:
  BinaryTestSearch should not be in the array (Not Yet Implemented)
    Called from binarysearch.rb:27

  Finished in 0.006682 seconds

  4 examples, 0 failures, 1 pending

describe() method

The describe() method can take an arbitrary number of arguments and a block and returns a sub-class of Spec::Example::ExampleGroup. We generally use only one or two arguments which is used to describe the behavior. The first argument can be a reference to a Class or module or a string. The second argument is optional and should be a string when used.

it() method

Similar to the describe() method, the it() method takes a single String, an optional Hash and an optional block. The String expression within the it() should be such that it informs the behavior of the code within the block.

Expectations in RSpec

There are two methods available for checking expectations: should() and should_not(). Both the methods accept either an expression matcher or a Ruby expression using a specific subset of Ruby operators. An expression matcher is an objects that matches an expression.

Built-in Matchers

There are several matchers that can be used with should and should_not, which are divided into well-separated categories.

Equality

subject.should == ece517
subject.should === ece517
subject.should eql(subject)
subject.should equal(subject)

The == method is used to express equivalence and equal is used when you want the receiver and the argument to be the same object. Instead of using !=, you should use the should_not method!

Floating Point Calculations

piValue.should be_close(3.14, 0.001593)

Sometimes the values generated might be correct upto some fixed decimal positions, after that they may have slight variations. To avoid the test beings failed, we provide the (value, delta) to be_close method which passes the test if the obtained value lies within the range (value+delta).

Regular Expressions

resultExpression.should match(/this regular expression/)
resultExpression.should =~ /this regular expression/

This can be very useful when dealing with multiple-line expectations, instead of using the open file technique to compare contents.

Changes

lambda {
 User.create!(:role => "admin" )
}.should change{ User.admins.count }

OR

lambda {
 User.create!(:role => "admin" )
}.should change{ User.admins.count }.to(1)

OR

lambda {
 User.create!(:role => "admin" )
}.should change{ User.admins.count }.from(0).to(1)

This is really useful when working with database changes or changes to objects. The matcher is change(), which takes a block and accepts the from(), to() or by() modifiers. ^[18]

Errors

field = CricketGround.new(:players => 11)
lambda {
 field.remove(:players, 15)
}.should raise_error(NotEnoughPlayers,“attempted to remove more players than there is on cricket stadium”)

Useful when needed to check for Exceptions. The matcher is raise_error and takes an ExceptionObject and/or a String/Regexp.

Throw

speech = Speech.new(:seats => 100)
100.times { speech.register Person.new }
lambda {
 speech.register Person.new
}.should throw_symbol(:speech_full, 100)

When dealing with “errors that are not really exceptions”, you use catch and throw. Rspec can check if a throw has been called by using the throw_symbol matcher. It accepts 0,1 or 2 arguments. The first argument needs to be a Symbol and the second can be any Object that is thrown along.

Predicate Matchers

A Ruby predicate method is a method that ends with a “?” and returns a boolean value, like string.empty? or regexp.match? methods. Instead of writing:

a_string.empty?.should == true

We can write using RSpec:

a_string.should be_empty

When using a be_something matcher, RSpec removes the “be_”, appends a “?” and calls the resulting method in the receiver. A very common construct of this method is be_true, which checks if the receiver is true (any object except false or nil) or false (false or nil).

Check Ownership

Sometimes you will want to check something the object owns and not the object itself.

The have_something() method

security_access.has_key?(:id).should == true

is the same as

security_access.should have_key(:id)

RSpec uses method_missing to convert anything that begins with have_something to has_something? and performs the checking.

The have() method

field.players.select {|p| p.team == home_team }.length.should == 9

is the same as

home_team.should have(9).players_on(field)

As have() does not respond to players_on(), it delegates to the receiver (home_team). It encourages the home_team object to have useful methods like players_on.
You can get a NoMethodError if the players_on method doesn´t exist, you can get another NoMethodError if the result of the players_on method doesn´t respond to size() or length() and if the size of the collection doesn´t match the expected size, you will get a failed expectation. ^[19]

Checking Collections Themselves

Sometimes we create expectations about a collection itself and not about an owned collection. RSpec lets us use the have() method to express this as well, as in:

basket_collection.should have(10).items

items is just providing some meaning to the expectation.

Strings

Strings are not collections by definition but they respond to a lot of methods that collections do, like length() and size(). This allow us to use have() to expect a string of a specific length.

“apple”.should have(5).characters

characters is just providing meaning to the expectation as well.

Have() modifiers for precision

The have() method has some relatives that allow us to check for upper and lower conditions.

work.should have_exactly(8).hours
classroom.should have_at_most(100).people
bag.should have_at_least(5).items

Operator Expressions

There may be sometimes when you want to expect a value to be not an exact amount but something like greater than or less than. RSpec allows you to do this by using the regular operators from Ruby!

number.should == 3
number.should be >= 2
number.should be <= 4
number should be > 0

References

1. Unit Testing. en.wikipedia.org. Retrieved Sep 17, 2010.
2. Ruby Programming & Unit Testing. en.wikibooks.org. Retrieved Sep 17, 2010.
3. Ruby Test::Unit. ruby-doc.org. Retrieved Sep 17, 2010.
4. Ruby Assertions. ruby-doc.org. Retrieved Sep 17, 2010.
5. Shoulda Explained. pragprog.com. Retrieved Sep 17, 2010.
6. Behavior Driven Development. en.wikipedia.org. Retrieved Sep 17, 2010.
7. Agile Software Development. en.wikipedia.org. Retrieved Sep 17, 2010.
8. Proceedings of Extreme Programming and Agile Methods Conference Carmen Zannier, Hakan Erdogmus and Lowell Lindstrom. Extreme Programming and Agile Methods - XP/Agile Universe 2004. 4th Conference on Extreme Programming and Agile Methods, Calgary, Canada, August 15-18, 2004.
9. Acceptance Testing. en.wikipedia.org. Retrieved Sep 17, 2010.
10. Domain-driven design: tackling complexity in the heart of software. By Eric Evans. books.google.com. Retrieved Sep 17, 2010.
11. Domain Driven Design. en.wikipedia.org. Retrieved Sep 17, 2010.
12. Test-driven development: by example. By Kent Beck. books. google.com. Retrieved Sep 17, 2010.
13. Test Driven Development. en.wikipedia.org. Retrieved Sep 17, 2010.
14. David Chelimsky Blog. davidchelimsky.net. Retrieved Sep 17, 2010.
15. Understanding RSpec Stories - A Tutorial. emson.co.uk. Retrieved Sep 17, 2010.
16. rbehave. dannorth.net. Retrieved Sep 17, 2010.
17. Ruby Website. ruby-lang.org. Retrieved Sep 17, 2010.
18. RSpec Book. pragprog.com. Retrieved Sep 17, 2010.
19. RSpec - Expectations. wordpress.com. Retrieved Sep 17, 2010.