CSC/ECE 517 Fall 2011/ch2 2e kt

Evolution of Testing Architectures

Across all of the testing frameworks that Ruby has, most of them fall into two categories: Test Driven Development and Behavior Driven Development. These two types of development styles helped to drive the evolution of the Ruby testing frameworks we will be discussing.

Test Driven Development

Test Driven Development (or TDD) is a development style that relies on writing the tests for a given program first, thereby establishing the functionality, then proceeding to write the code to fulfill that criteria. As the code is being written, the tests can be run and examined, thus providing status on the development of the features. Once all of the tests pass for a given function, the development is complete and any refactoring can be done before finalizing the code. TDD has the benefit of providing for a short development cycle and encouraging simple designs. However, there are some shortcomings which have lead to the creation of the Behavior Development Cycle.

Behavior Driven Development

Behavior Driver Development (or BDD) arose out of the fact that tests written in the TDD style are only clearly readable by the coders themselves. If a project manager without programming knowledge were to look at the tests as a description of the functionality to be provided, this would not very meaningful. BDD evolved though the idea that the tests could be written in such a way as to provide a set of project requirements and functions that is readable by the average person. This allows the tests to not only act as the testing framework for a project but also as an executable requirement list. This style still leverages the TDD development cycle benefits with the added readability for those outside of the development team.

Ruby Testing Frameworks

Some of the more popular Ruby testing frameworks include<ref>http://ruby-toolbox.com/categories/testing_frameworks.html</ref>:

• Test::Unit
• Mini::Test
• RSpec
• Shoulda
• Cucumber

Since this will be a comparison across the different frameworks listed, using a common class to test against will provide us with a baseline for our comparisons. As such, we will use the following class implementing a bank account:

 class Account
   @balance
   @name
 
   attr_accessor :balance
   attr_accessor :name
 
   def initialize(amount)
     @balance = amount
   end
 
   def deposit(amount)
     @balance += amount
   end
 
   def addinterest(rate)
     @balance *= (1 + rate)
   end
 
   def withdrawal(amount)
     @balance -= amount
   end
 end

Test::Unit

Overview

Test::Unit was one of the first widely adopted testing frameworks for Ruby, even being included as part of the standard library as of version 1.8. It is based on the SUnit Smalltalk framework <ref>Programming Ruby: The Pragmatic Programmers' Guide, Second Ed.</ref> and is very similar to testing frameworks in other languages such as JUnit (Java) and NUnit (.NET). The Test::Unit framework falls under the category of a Test Driven Design framework and relies on creating test cases, under which reside test methods. An individual test case consists of a class which inherits from Test::Unit::TestCase and contains test methods which, together, usually test a particular function or feature. Within each test case, only methods starting with the name "test" are automatically run, with the exception of the optional setup or teardown methods. These methods can be used to initialize and clean up any structures which may be used across the test methods. As of Ruby 1.9, Test::Unit has been replaced by MiniTest as the testing framework included in the standard library.

Functionality

Unit::Test integration into Eclipse via a built-in test runner

As Test::Unit is a Test Driven Development framework, it provides many of the low level testing constructs that you would expect to find in similar frameworks. It primarily relies on simple assertions to verify test cases. The functionality of Test::Unit is considered to be one of the more simple frameworks and, as such, one of the easier to learn.

assert	assert_nil	assert_not_nil	assert_equal	assert_not_equal
assert_in_delta	assert_raise	assert_nothing_raised	assert_instance_of	assert_kind_of
assert_respond_to	assert_match	assert_no_match	assert_same	assert_not_same
assert_operator	assert_throws	assert_send	flunk

Test::Unit allows for the creation of test suites, which are collections of test cases, allowing for a higher level grouping of tests. Such test suites can be produced by creating a file which requires 'test/unit' as well as any other files which contain test cases.

When a test is usually run, it is called via ruby against the file containing the test case. In some cases, we wish to modify the way the test results are output; this is where a part of Test::Unit called "test runners" come in. These test runners allow the output of the test cases to be modified by overriding the default output methods. In this way, test outputs can be easily modified to fit new testing environments or even integration into IDEs.

Code Example

In order to create a test case for the Account class mentioned earlier, one needs to create a new class which inherits from Test::Unit::TestCase and requires 'test/unit' as well as the class under test, in this case 'Account.rb'. In order to get the test results, one simply has to run the ruby interpreter against the test file. For this example, we also include a failing test in order to point out the information provided in the event of a failed test.

 require "test/unit"
 require_relative("../Account.rb")
 
 class AccountTest < Test::Unit::TestCase
 
   def test_balance
     a = Account.new(100)
     assert_equal(100, a.balance())
   end
 
   def test_deposit 
     a = Account.new(100)
     assert_equal(200, a.deposit(100))
   end
   
   def test_withdrawal
     a = Account.new(100)
     assert_equal(50, a.withdrawal(50))
   end
   
   def test_name
     a = Account.new(100)
     a.name = "Checking"
     assert_not_nil(a.name())
   end
 
   def test_interest
     a = Account.new(100)
     assert_equal(150, a.addinterest(0.5))
   end
 
   def test_fail
     a = Account.new(100)
     assert_equal(200, a.balance())
   end
 end

From this we will get the output:

 Loaded suite AccountTest-ut
 Started
 ..F...
 Finished in 0.000782 seconds.
 
   1) Failure:
 test_fail(AccountTest) [AccountTest-ut.rb:40]:
 <200> expected but was
 <100>.
 
 6 tests, 6 assertions, 1 failures, 0 errors, 0 skips

Here we can see that in total all 6 test methods were run with one of them failing. The failing test method is indicated by name with the code line, expected output and actual output.

Mini::Test

Overview

Mini::Test is one of the more recent additions to the Ruby Testing frameworks. Mini::Test was created to be a small, clean and fast replacement for the Test::Unit framework. As previously mentioned, MiniTest is included in the standard library as of Ruby 1.9. (If you are running Ruby 1.8, you can run gem install minitest to get Mini::Test. Alternately, you can install the test-unit gem if you have Ruby 1.9 and want to get the Test::Unit functionality.) MiniTest provides both a Test Driven Development framework and a Behavior Driven Development Framework. Test cases in MiniTest::Unit are built similarly to those in Test::Unit - you create test cases with test methods containing assertions along with optional setup and teardown methods . Mini/spec (the BDD component of MiniTest) borrowed concepts from existing BDD frameworks, such as RSpec and Shoulda. Test cases are created using expectations instead of assertions (see Code Example - mini/spec below for details).

Functionality

Most of the assertions in Mini::Test::TestCase are the same as those in its predecessor with some new ones added. The most notable difference is in the negative assertions. In Test::Unit where you have an assert_not_something method, Mini::Test gave them a streamlined name of a refute_something method. (assert_not_raise and assert_not_throws are no longer available.) Mini::Test::Unit provides the following assertions:

assert	assert_block	assert_empty	refute	refute_empty
assert_equal	assert_in_delta	assert_in_epsilon	refute_equal	refute_in_delta	refute_in_epsilon
assert_includes	assert_instance_of	assert_kind_of	refute_includes	refute_instance_of	refute_kind_of
assert_match	assert_nil	assert_operator	refute_match	refute_nil	refute_operator
assert_respond_to	assert_same	assert_output	refute_respond_to	refute_same
assert_raises	assert_send	assert_silent	assert_throws

While mini/spec contains the following expectations:

must_be	must_be_close_to	must_be_empty	wont_be	wont_be_close_to	wont_be_empty
must_be_instance_of	must_be_kind_of	must_be_nil	wont_be_instance_of	wont_be_kind_of	wont_be_nil
must_be_same_as	must_be_silent	must_be_within_delta	wont_be_same_as	wont_be_within_delta
must_be_within_epsilon	must_equal	must_include	wont_be_within_epsilon	wont_equal	wont_include
must_match	must_output	must_raise	wont_match
must_respond_to	must_send	must_throw	wont_respond_to

Additional Features

Aside from the API improvements, Mini::Test also provides some additional features such as test randomization. In unit testing, tests should run independent from each other (i.e. the outcome or resulting state(s) of one test should not affect another). By randomizing the order, Mini::Test prevents tests from becoming order-dependent. Should you need to repeat a particular order to test for such issues, Mini::Test provides the current seed as part of the output and gives you the option to run the test using this same seed.

Mini::Test also gives you the ability to skip tests that are not working correctly (for debug at a later time) as well as additional options for determining the performance of your test suite.

Code Example - Mini::Test:Unit

We follow the same steps to create a test case for our Account class that we did in Test::Unit Code Example , except that this class inherits from MiniTest:Unit::TestCase and requires 'minitest/autorun'. (Requiring minitest/unit does not cause the test methods to be automatically invoked when the test case is run, hence we use minitest/autorun which provides this functionality.) We also need to update the assertions to those provided with MiniTest - for this class the assert_not_nil was changed to refute_nil. The AccountTest test case is below.

require 'minitest/autorun'
require_relative 'account.rb'

class AccountTest < MiniTest::Unit::TestCase

  def setup
         @a = Account.new(100)
  end

  def test_deposit
    assert_equal(200, @a.deposit(100))
  end
 
  def test_withdrawal
    assert_equal(50, @a.withdrawal(50))
  end
 
  def test_name
    @a.name = "Checking"
    refute_nil(@a.name())
    assert_match(@a.name, "Checking")
  end

  def test_interest
    assert_in_delta(@a.addinterest(0.333), 130, 5)
  end

  def test_fail
    assert_equal(@a.balance(), 200)
  end
 
  def test_whatru
      assert_instance_of(Account, @a)
  end
     
end

Executing this test case gives the following results, showing that 6 tests were run with 1 faliure. Information is provided about the failure, including test name expected output and actual output. Note that the seed value of the random test run is also provided.

Loaded suite C:/Users/Tracy2/Desktop/NCSU/CSC 541/Workspace/Account/test_account2
Started
.F....
Finished in 0.001000 seconds.

  1) Failure:
test_fail(AccountTest) [C:/Users/Tracy2/Desktop/NCSU/CSC 541/Workspace/Account/test_account2.rb:30]:
Expected 100, not 200.

6 tests, 8 assertions, 1 failures, 0 errors, 0 skips

Test run options: --seed 13360

Code Example - mini/spec

 require 'minitest/autorun'
 require_relative 'account.rb'

 describe Account do

   before do
      @a = Account.new(100)
   end

   describe "deposit" do
     it "should add amount to balance" do
       @a.deposit(100).must_equal 200
     end
   end
 
   describe "withdraw" do
     it "should subtract amount from balance" do
       @a.withdrawal(50).must_equal 50
     end
   end
 
   describe "set name" do
     it "should set account name" do
       @a.name = "Checking"
       @a.name.wont_be_nil
       @a.name.must_match "Checking"
     end
   end
 
   describe "interest" do
     it "should add interest to balance" do
       @a.addinterest(0.333).must_be_within_delta(130, 5)
     end
   end

   describe "fail" do
     it "should fail" do
       @a.balance.must_equal 200
     end
   end

   describe "what are you" do
     it "should be an instance of Account" do
       @a.must_be_instance_of Account
     end
   end

 end

Gives the following results:

Loaded suite C:/Users/Tracy2/Desktop/NCSU/CSC 541/Workspace/Account/test_account3
Started
.F....
Finished in 0.003000 seconds.

  1) Failure:
test_0001_should_fail(AccountSpec::FailSpec) [C:/Users/Tracy2/Desktop/NCSU/CSC 541/Workspace/Account/test_account3.rb:39]:
Expected 200, not 100.

6 tests, 8 assertions, 1 failures, 0 errors, 0 skips

Test run options: --seed 21728

RSpec

Code Example

 require_relative("../Account.rb")
 
 describe "The Account" do  
   before(:each) do
     @a = Account.new(100)
   end
   
   it "should be created with a balance" do
     @a.balance.should == 100
   end
 
   it "should take a deposit" do
     @a.deposit(100).should == 200
   end
   
   it "should be capable of withdrawals" do
     @a.withdrawal(50).should == 50
   end
   
   it "should have a name" do
     @a.name = "Checking"
     @a.name.should == "Checking"
   end
 
   it "should calculate interest" do
     @a.addinterest(0.5).should == 150
   end
 
   it "should have a failure here as an example" do
     @a.balance.should == 200
   end
 
   it "should provide a bank statement" do
     pending "Not yet implemented"
   end
 end

Running the above code with the '--format doc' option we will get the output:

 The Account
   should be created with a balance
   should take a deposit
   should be capable of withdrawals
   should have a name
   should calculate interest
   should have a failure here as an example (FAILED - 1)
   should provide a bank statement (PENDING: Not yet implemented)
 
 Pending:
   The Account should provide a bank statement
     # Not yet implemented
     # ./AccountTest-rspec.rb:33
 
 Failures:
 
   1) The Account should have a failure here as an example
      Failure/Error: @a.balance.should == 200
        expected: 200
             got: 100 (using ==)
      # ./AccountTest-rspec.rb:30:in `block (2 levels) in <top (required)>'
 
 Finished in 0.00132 seconds
 7 examples, 1 failure, 1 pending
 
 Failed examples:
 
 rspec ./AccountTest-rspec.rb:29 # The Account should have a failure here as an example

Criteria

• IDE integration
• Test output detail
• Testing constructs available
• TDD or BDD
• Documentation
• Learning curve and ease of use

Framework Matrix

Framework	Website	Documentation	IDE Integration	Type	Ease of Use
Unit::Test		RubyDoc		TDD	Easy to learn; Use difficulty increases with project size
MiniTest::Unit	GitHub	RubyDoc		TDD/BDD
RSpec	http://rspec.info/			BDD	Slightly more difficult to learn; Easy to use
Shoulda
Cucumber

References

<references/>