CSC/ECE 517 Fall 2010/ch1 1f vn
Unit-Testing Frameworks for Ruby
This wiki-page serves as a knowledge source for understanding the different Unit-Testing Frameworks available for Ruby.
Introduction
Unit testing is a method by which we can isolate and test a unit functionality of the program, typically individual methods during and long after the code is written. [1] It helps to identify errors in the program even without running the entire program. It also helps to do regressing testing to identify buggy code additions in the future. Unit testing frameworks provides us with constructs which simplifies the process of unit testing. Using a standard unit test framework helps other developers to add test cases easily. [2] This chapter walks through three different unit testing frameworks available for Ruby and explains how to use them with examples. The three commonly used unit testing frameworks for ruby are
- Test::Unit
- Shoulda
- RSpec
Test::Unit
Now we shall consider Test::Unit framework
Background
Ruby comes with an in-built, ready to use unit testing framework called Test::Unit. It is a XUnit type framework and typically have a setup method for initialization, a teardown method for cleanup and the actual test methods itself. The tests themselves are bundled separately in a test class from the code it is testing.
Test Fixture
Test fixture represents the initial environment setup(eg. initialization data) and/or the expected outcome of the tests for that environment. This is typically done in the setup() and teardown() methods and it helps to separate test initialization and cleanup from the actual tests. It also helps to reuse the same fixture for more than one tests.[3]
For example, consider a method prime_check(num) which takes an integer number as input and outputs whether it is prime number or not. In order to unit test this method we can create the following fixture containing a 2-dimensional array with a number and the expected output of whether it is prime or not.
def setup @NUMBERS = [[3,true], [4,false], [7,true], [10,false]] end
Assertions
The core part of test::unit framework is the ability to assert a statement of expected outcome. If an assert statement is correct then the test will proceed, otherwise the test will fail. This feature helps us to verify the method under test with different types of inputs and track the results. Test::unit provides a bunch of assert methods for this purpose:
| assert( boolean, [message] ) | True if boolean |
| assert_equal( expected, actual, [message] ) assert_not_equal( expected, actual, [message] ) |
True if expected == actual |
| assert_raise( Exception,... ) {block} assert_nothing_raised( Exception,...) {block} |
True if the block raises (or doesn't) one of the listed exceptions. |
For the full list of assertion methods provided by test::unit refer to test::unit assertions. [4]
Example
The test case class BinarySearchTest subclasses the Test::Unit::TestCase class and overrides setup and teardown methods. The test methods should start with 'test_' prefix. This helps in isolating the test methods from the helper methods if any. The Test::Unit::TestCase class takes care of making the test methods into tests, wrapping them into a suite and running the individual tests. The test results are collected into Test::Unit::TestResult object.
require 'test/unit'
require 'binarysearch'
class BinarySearchTest < Test::Unit::TestCase
def setup
@input_array = [1,2,3,4,5] #The test fixture is initialized
end
def test_success_left_half
assert_equal(binary_search(@input_array,1),true) #tests if the element present in left half of the array is found
assert_equal(binary_search(@input_array,2),true)
end
def test_success_right_half
assert_equal(binary_search(@input_array,5),true) #tests if the element present in the right half of the array is found
assert_equal(binary_search(@input_array,4),true)
end
def test_success_middle #tests if the element present in the middle of the array is found
assert_equal(binary_search(@input_array,3),true)
end
def test_failure
assert_equal(binary_search(@input_array,6),false) #tests if an element not present in the array is not found
end
def teardown
#nothing to do her
end
end
Here we have four test methods testing different logical paths of the binary search algorithm. Each test method can have one or more assert statements to test whether conditions are correct in each situation. To run the tests we simply have to run the file binary_search_test.rb and the output is as follows:
Loaded suite binarysearch Started F... Finished in 0.372 seconds. 1) Failure: test_failure(BinarySearchTest) [binarysearch.rb:25]: <true> expected but was <false>. 4 tests, 6 assertions, 1 failures, 0 errors
The results show that the last test case test_failure, testing the negative scenario is failing. The reason is because the assert statement is expecting false when number 6, which not present in the array is passed. But the binary_search method is returning true.
Test Suite
Sometimes it is useful to combine a bunch of related test cases and run them as batch. Test::Unit provides a class called TestSuite for this purpose. The below example demonstrates how to bundle binary and sequential test case classes into a single search test suite.
require 'test/unit/testsuite'
require 'binary_search_test'
require 'sequential_search_test'
class Search_Tests
def self.suite
suite = Test::Unit::TestSuite.new
suite << BinarySearchTest.suite
suite << SequentialSearchTest.suite
return suite
end
end
Test::Unit::UI::Console::TestRunner.run(Search_Tests)
Shoulda
Background
One of the downsides of Test::Unit is we end up writing lots of code in order to test the actual code which is sometimes not easy to understand. Shoulda is a library that allows us to write better and more understandable tests for ruby application. Shoulda is not a testing framework by itself. It extends the Test::Unit framework with the idea of context. We can mix Test::Unit test cases with Shoulda test cases. Shoulda allows us to provide context to the tests so that we can group the tests according to a specific feature or scenario. [5]
Example
require 'shoulda'
require 'test/unit'
require 'binarysearch'
class BinarySearchTest < Test::Unit::TestCase
context "Input array of size 5" do
def setup
@input_array = [1,2,3,4,5]
end
should "have the number in the left half of the array" do #tests if the element present in left half of the array is found
assert_equal(binary_search(@input_array,1),true)
assert_equal(binary_search(@input_array,2),true)
end
should "have the number in the right half of the array" do #tests if the element present in right half of the array is found
assert_equal(binary_search(@input_array,5),true)
assert_equal(binary_search(@input_array,4),true)
end
should "have the number in the middle of the array" do #tests if the element present in middle of the array is found
assert_equal(binary_search(@input_array,3),true)
end
should "not have the number in the array" do #tests if the element not present in the array is not found
assert_equal(binary_search(@input_array,6),false)
end
def teardown
#nothing to do here
end
end
context "Input array of size 1" do
def setup
@input_array = [1]
end
should "have the number in the array" do
assert_equal(binary_search(@input_array,1),true) #tests if an element is found in the single element array
end
should "not have the number in the array" do # tests if an element is not found in the single element array
assert_equal(binary_search(@input_array,2),true)
end
def teardown
#nothing to do here
end
end
end
Notice that we are still sub-classing the Test::Unit::TestCase class. In this example we have two contexts one for input array of size 5 and the other for input array of size 1. Each context has its own environment of setup/teardown methods. We can also create nested contexts - the outer setup gets run before the execution of each of the inner contexts. And the setup in the inner contexts gets run when running that context. Each should construct is converted into individual test methods and are run. If a test case fails we will get a better description of what that test case is doing from the should description.
RSpec
Now let us consider about RSpec Testing Framework in detail.
Background
Behaviour Driven Development (BDD) [6] is an Agile development process [7] that comprises aspects of Acceptance Test Driven Planning [8] [9], Domain Driven Design [10] [11] and Test Driven Development (TDD). [12] [13] RSpec is a Behavioural Driven Development (BDD) tool aimed at Test Driven Development, originally created by Dave Astels and Steven Baker. However David Chelimsky [14] is really the gatekeeper of the RSpec project. [15]
Traditionally we use Unit Test frameworks like JUnit, NUnit or RUnit for writing Test cases. We spend a lot of time writing tests that test every unit of code in our software system. Instead we can shift our focus from Unit testing to Behaviour testing or Behaviour Driven Development (BDD) using RSpec. By focusing on the behaviour of the system it helps clarify in our minds what the system should actually be doing. It also helps us to perform more ‘useful’ tests. Useful tests, cover what the system should be doing and build in enough redundancy so that it should be easy to refactor our code without having to re-write every test.
RSpec is really two projects merged into one. The RSpec project pages describes these merged projects as:
- application level behaviour described by a Story Framework
- object level behaviour described by a Spec Framework
Dan North created rbehave [16] which is the Story Framework and David Chelimsky created the Spec Framework. By encompassing two frameworks RSpec equips a programmer with a thorough set of testing tools, allowing you to think about your software problem from a number of perspectives.
Prerequisites
The prerequisites are
- Ruby 1.8.4 or later
- RSpec Gem (latest)
To install Ruby, please visit official Ruby Website [17]
To install RSpec, open a command shell, go to /bin folder in Ruby directory and type
> gem install rspec
Terms & Definitions
Here are some terms which are used frequently while working with RSpec. [18]
- subject code - The code whose behavior is specified using RSpec
- expectation - The expected behavior of subject code is expressed using expectation (Similar to 'Assertions' statements used in Test::Unit or other tools in other languages)
- code example - An executable example containing the subject code and the expectations (Similar to 'Test Method' terminology used elsewhere)
- example group - A group of code examples (Similar to 'Test Case' terminology used elsewhere)
- spec file - A file which contains one or more example groups
Example
Let us go through an example to be clear on the usage of RSpec.
require 'binarysearch'
describe BinarySearchTest do
before(:all) do
@input_array = [1, 2, 3, 4, 5] # The Input Array
end
after(:all) do
# do nothing here
end
it "should be in the left-half of the array" do # Test case for element to be present in left-half of given array
bst = BinarySearch.new
bst.should be_binary_search(1)
end
it "should be in the right-half of the array" do # Test case for element to be present in right-half of given array
bst = BinarySearch.new
bst.should be_binary_search(5)
end
it "should be in the middle of the array" do # Test case for element to be present in the middle of given array
bst = BinarySearch.new
bst.should be_binary_search(3)
end
it "should not be in the array"
end
Here it is assumed that the method binary_search will return true/false based on whether the provided value exists in the array or not. It should produce the following output. The un-implemented tests are marked as Pending in the output.
BinarySearchTest
- should be in the left-half of the array
- should be in the right-half of the array
- should be in the middle of the array
- should not be in the array (PENDING: Not Yet Implemented)
Pending:
BinaryTestSearch should not be in the array (Not Yet Implemented)
Called from binarysearch.rb:27
Finished in 0.006682 seconds
4 examples, 0 failures, 1 pending
describe() method
The describe() method can take an arbitrary number of arguments and a block and returns a sub-class of Spec::Example::ExampleGroup. We generally use only one or two arguments which is used to describe the behavior. The first argument can be a reference to a Class or module or a string. The second argument is optional and should be a string when used.
it() method
Similar to the describe() method, the it() method takes a single String, an optional Hash and an optional block. The String expression within the it() should be such that it informs the behavior of the code within the block.
Expectations in RSpec
There are two methods available for checking expectations: should() and should_not(). Both the methods accept either an expression matcher or a Ruby expression using a specific subset of Ruby operators. An expression matcher is an objects that matches an expression.
Built-in Matchers
There are several matchers that can be used with should and should_not, which are divided into well-separated categories.
Equality
subject.should == ece517 subject.should === ece517 subject.should eql(subject) subject.should equal(subject)
The == method is used to express equivalence and equal is used when you want the receiver and the argument to be the same object. Instead of using !=, you should use the should_not method!
Floating Point Calculations
piValue.should be_close(3.14, 0.001593)
Sometimes the values generated might be correct upto some fixed decimal positions, after that they may have slight variations. To avoid the test beings failed, we provide the (value, delta) to be_close method which passes the test if the obtained value lies within the range (value+delta).
Regular Expressions
resultExpression.should match(/this regular expression/) resultExpression.should =~ /this regular expression/
This can be very useful when dealing with multiple-line expectations, instead of using the open file technique to compare contents.
Changes
lambda {
User.create!(:role => "admin" )
}.should change{ User.admins.count }
OR
lambda {
User.create!(:role => "admin" )
}.should change{ User.admins.count }.to(1)
OR
lambda {
User.create!(:role => "admin" )
}.should change{ User.admins.count }.from(0).to(1)
This is really useful when working with database changes or changes to objects. The matcher is change(), which takes a block and accepts the from(), to() or by() modifiers. [18]
Errors
field = CricketGround.new(:players => 11)
lambda {
field.remove(:players, 15)
}.should raise_error(NotEnoughPlayers,“attempted to remove more players than there is on cricket stadium”)
Useful when needed to check for Exceptions. The matcher is raise_error and takes an ExceptionObject and/or a String/Regexp.
Throw
speech = Speech.new(:seats => 100)
100.times { speech.register Person.new }
lambda {
speech.register Person.new
}.should throw_symbol(:speech_full, 100)
When dealing with “errors that are not really exceptions”, you use catch and throw. Rspec can check if a throw has been called by using the throw_symbol matcher. It accepts 0,1 or 2 arguments. The first argument needs to be a Symbol and the second can be any Object that is thrown along.
Predicate Matchers
A Ruby predicate method is a method that ends with a “?” and returns a boolean value, like string.empty? or regexp.match? methods. Instead of writing:
a_string.empty?.should == true
We can write using RSpec:
a_string.should be_empty
When using a be_something matcher, RSpec removes the “be_”, appends a “?” and calls the resulting method in the receiver. A very common construct of this method is be_true, which checks if the receiver is true (any object except false or nil) or false (false or nil).
Check Ownership
Sometimes you will want to check something the object owns and not the object itself.
The have_something() method
security_access.has_key?(:id).should == true
is the same as
security_access.should have_key(:id)
RSpec uses method_missing to convert anything that begins with have_something to has_something? and performs the checking.
The have() method
field.players.select {|p| p.team == home_team }.length.should == 9
is the same as
home_team.should have(9).players_on(field)
As have() does not respond to players_on(), it delegates to the receiver (home_team). It encourages the home_team object to have useful methods like players_on.
You can get a NoMethodError if the players_on method doesn´t exist, you can get another NoMethodError if the result of the players_on method doesn´t respond to size() or length() and if the size of the collection doesn´t match the expected size, you will get a failed expectation. [19]
Checking Collections Themselves
Sometimes we create expectations about a collection itself and not about an owned collection. RSpec lets us use the have() method to express this as well, as in:
basket_collection.should have(10).items
items is just providing some meaning to the expectation.
Strings
Strings are not collections by definition but they respond to a lot of methods that collections do, like length() and size(). This allow us to use have() to expect a string of a specific length.
“apple”.should have(5).characters
characters is just providing meaning to the expectation as well.
Have() modifiers for precision
The have() method has some relatives that allow us to check for upper and lower conditions.
work.should have_exactly(8).hours classroom.should have_at_most(100).people bag.should have_at_least(5).items
Operator Expressions
There may be sometimes when you want to expect a value to be not an exact amount but something like greater than or less than. RSpec allows you to do this by using the regular operators from Ruby!
number.should == 3 number.should be >= 2 number.should be <= 4 number should be > 0
References
- Unit Testing. en.wikipedia.org. Retrieved Sep 17, 2010.
- Ruby Programming & Unit Testing. en.wikibooks.org. Retrieved Sep 17, 2010.
- Ruby Test::Unit. ruby-doc.org. Retrieved Sep 17, 2010.
- Ruby Assertions. ruby-doc.org. Retrieved Sep 17, 2010.
- Shoulda Explained. pragprog.com. Retrieved Sep 17, 2010.
- Behavior Driven Development. en.wikipedia.org. Retrieved Sep 17, 2010.
- Agile Software Development. en.wikipedia.org. Retrieved Sep 17, 2010.
- Proceedings of Extreme Programming and Agile Methods Conference Carmen Zannier, Hakan Erdogmus and Lowell Lindstrom. Extreme Programming and Agile Methods - XP/Agile Universe 2004. 4th Conference on Extreme Programming and Agile Methods, Calgary, Canada, August 15-18, 2004.
- Acceptance Testing. en.wikipedia.org. Retrieved Sep 17, 2010.
- Domain-driven design: tackling complexity in the heart of software. By Eric Evans. books.google.com. Retrieved Sep 17, 2010.
- Domain Driven Design. en.wikipedia.org. Retrieved Sep 17, 2010.
- Test-driven development: by example. By Kent Beck. books. google.com. Retrieved Sep 17, 2010.
- Test Driven Development. en.wikipedia.org. Retrieved Sep 17, 2010.
- David Chelimsky Blog. davidchelimsky.net. Retrieved Sep 17, 2010.
- Understanding RSpec Stories - A Tutorial. emson.co.uk. Retrieved Sep 17, 2010.
- rbehave. dannorth.net. Retrieved Sep 17, 2010.
- Ruby Website. ruby-lang.org. Retrieved Sep 17, 2010.
- RSpec Book. pragprog.com. Retrieved Sep 17, 2010.
- RSpec - Expectations. wordpress.com. Retrieved Sep 17, 2010.