CSC/ECE 517 Fall 2011/ch2 2e ps

Testing Frameworks For RUBY

Introduction to Testing Frameworks

A dynamically-typed language such as Ruby relies heavily on extensive testing to ensure all objects operate as they are expected to. Ruby has got a pool of testing frameworks.

This article goes through different testing frameworks available for Ruby and explains how to use them with examples.

The Frameworks discussed for Ruby are

 Test::Unit

 Shoulda

 RSpec

 Cucumber

 RIOT

1. The Concept of Unit Testing

1.1 Unit

A unit is the building block of a code. It can be a class, typically individual methods or lines within methods, an interface etc.

1.2 What does Unit Testing mean?

Unit testing is the process of testing a code by focusing on a small chunk (unit) of code at a time, checking whether it functions properly, to give the desired results.

1.3 Why Unit Testing?

1.3.1 Benefits of Unit Testing:

 Fast – It gives instant feedback on the accuracy of your code.

 Design – For testing, the code is designed as small modules facilitating the reusability of the units.

 Individuality – Each unit is designed independent of each other and tested individually making error identification easy.

 Efficiency - Bugs can be scanned in the probable bug prone code areas making debugging efficient.

 Understanding – Helps any reader, easily understand the methods’ functionality.

1.3.2 Limitations of Unit Testing:

 Tedious Job for Complex Codes – Implementing and running test cases for complex codes is time consuming. Moreover, coding a unit test is as tedious as coding the target program itself.

 Impact of Change in the Code Requirements - In an agile project that is accompanied with periodic changes in requirements will result in constant updating of test cases to match the latest developments.

2.Testing Techniques

2.1 Test-Driven Development(TDD):

Test Driven Development (TDD)is an advanced technique of using automated unit tests to drive the design of software. The result of using this practice is a comprehensive suite of unit tests that can be run at any time to keep track of the code development.

Here

 Test cases are written initially keeping in mind the final result.

 The initial test is conducted which eventually fails because of absence of a target code.

 The test suggests further steps to build/refactor an error free code and only those programs associated with the tests go into production.

 Tests are then re-conducted until satisfactory results are achieved.

Development Cycle of TTD

2.2 Behavior driven development (BDD):

Behavior driven development (BDD)is a software development technique that relies on the use of simple vocabulary. Tests are designed using natural language so that even a non-technical participant will be able to follow. BDD is an evolution in the thinking behind Test Driven Development but stresses extensively on the behavior aspect of the methods used in the code.

Eg: Tests result such as “the point cannot be created without code description” is more understandable than something like Test_category_ThrowsArgumentNullException .

3. Frameworks

A framework is a tool to support writing and running of a program. A Unit Test framework is a set of reusable libraries or classes for a software system. Code is written in these frameworks, they are capable of running and checking the test code by themselves.

Here, a unit test framework will evaluate the code based on a variety of inputs provided in the test case and check for the accuracy of results.

3.1 Testing Framework Parlance:

a. Test Case: It is a main class where test methods are coded. All the units to be tested are inherited to this class.

b. Assertions (Test::Unit::Assertions): An assertion is the main condition that when held true would result in successful culmination of the test. If the assertion fails then the test would generate an error message with pertinent information so that the programmer can make necessary changes.

c. Test Fixture: A test fixture is used to clean up methods for conducting more than one test, eliminating duplication of methods.

d. Test Method: It is a process to handle individual units referred for testing.

e. Test Runners: Test Runners GUI’s that are used to conduct testing and provide with the results. Test runners such as Test::Unit::UI::Console::Test Runner and GTK test runners are used.

f. Test Suite: It is a collection of tests.

We need a target code to conduct tests.

3.2 Example Problem Statement for Target Code:

To find whether a given point lies on the x-axis, y-axis or the origin.

Solution: We write a class to define the position (x-axis, y-axis or origin) of the point.

Code : Position of a Point

class Point 

attr_writer :xcoordinate 
attr_writer :ycoordinate

def initialize(xcoordinate,ycoordinate)
  @xcoordinate = xcoordinate
  @ycoordinate = ycoordinate
end
#-----------Location of point on X-axis ----------------#
def isOnXaxis?
   if @ycoordinate == 0
    return true   	
  end
end
#---------- Location of point on Y-axis ----------------#
 def isOnYaxis?
   if @xcoordinate == 0
     return true
  end
 end
#---------- Location of point on Origin-----------------#
 def isOnOrigin?
   if @xcoordinate == 0 && @ycoordinate == 0
   return true
  end
 end
end

4. Testing Frameworks

4.1 Test::Unit

4.1.1 Overview

a. It is based on the Test Driven Development Technique (TDD).

b. The Test::Unit framework integrates the following features:

1. Way of expressing individual tests.

2. Provides a framework for structuring the tests.

3. Has flexible ways of invoking the tests.

a. Installation

Test::Unit framework is pre-installed on all Ruby versions.

b. Algorithm

 We use a UI is used to run the Test code and display the gathered results. Here, we have used the web based UI.

 Structuring the tests: Inherit the required classes.

a. Here, “point” class is inherited to call the target units which have to be tested.

b. The “test/unit” class is inherited for methods required to conduct testing.

 Write the test methods.

 Test methods are prefixed with test. Eg: test_examp where examp is the unit being tested.

 In the methods write Assertions comparing them with expected results for the target units.

 Text fixtures are optional; they can be used to clean the methods.

 Run the tests as Test::Unit test class name.rb

 To run only a particular method we can its name as classname.rb --name method

 If there are bugs, refactor the code.

 Repeat testing until all the bugs are fixed.

Flowchart

Development Cycle of TTD

CODE Test::Unit

require "Point"
require "test/unit"
class TestCase_Point < Test::Unit::TestCase 
 def test_isOnXaxis
   assert(Point.new(3,0).isOnXaxis?)
   end
  def test_isOnYaxis
      assert(Point.new(0,4).isOnYaxis?)
 end
  def test_isOnOrigin
      assert(Point.new(3,4).isOnOrigin?)
 end
end

Result Test::Unit

Below are the test results that are obtained after running the above test cases.

Failed output

This is how the failed test cases are shown once we run the test cases,

Development Cycle of TTD

Passed output

Development Cycle of TTD

Storing Test Files

Test files can be stored in the same directory as the target file but eventually the directory gets bloated storing 2 files for the same target code. Instead we can store it in a separate test directory for better organization. Here, one must take care to clearly specify the path of the target code for inheritance purpose in the test file.

Eg: If point.rb is stored in the directory lib and test_point.rb is stored in the test directory then to inherit point.rb we specify the path. require “../lib/point”

---

4.2 Shoulda:

4.2.1 Overview

a. Shoulda is a library that allows us to write better and easily understandable test cases for the ruby applications compared to Test::Unit.

b. It allows us to use the "it"-blocks from RSpec to define nice error-messages if a test fails.

c. In other words it is improvised form of Test::Unit and Rspec. It works inside the Test::Unit — you can even mix Shoulda tests with regular Test::Unit test methods.

a. Installation

Shoulda is installed by running “gem install shoulda” in command prompt after setting the environment path as “…\Ruby\bin”

b. Algorithm

 Inherit all the classes involved in testing i.e. the target class and the shoulda library class. To achieve this add require "rubygems", require ”Point” and require "shoulda" to the test class.

 Then setup up a context “Point” (context is nothing but a region of code which deals with the functionality you are interested in.)

 Then define a method starting with should “method name” do and then describe the entire method.

 Make proper assertions to test the functionality in your method.

Code for shoulda

require "rubygems"
require “Point”
require "test/unit"
require "shoulda"
class TestCase_Point_Shoulda < Test::Unit::TestCase
 context "Point" do
  should "lie on the X-axis " do
     assert(Point.new(3,0).isOnXaxis?)
  end
  should "lie on the Y-axis " do
     assert(Point.new(0,4).isOnYaxis?)
  end
  should "lie on the Origin" do
   assert(Point.new(0,4).isOnOrigin?)
  end	
 end
end

Result

Below are the test results that are obtained after running the above test cases,

Failed Output

Development Cycle of TTD

Passed output

Development Cycle of TTD

---

4.3 RSpec

4.3.1 Overview

a. RSpec is Behavior Driven Development Unit test tool for ruby.

b. It allows us to write an initial code specifications (spec) document. The spec document talks about the program’s function.

c. Thereby, we can write the code based on the specifications (spec file).

d. The (spec) document is written in Domain Specific Language which makes it simple for the programmer to handle.

e. Better understanding of the test cases compared to shoulda as the cases and notifications are more descriptive.

f. In other words it is improvised form of Test::Unit and Rspec. It works inside the Test::Unit — you can even mix Shoulda tests with regular Test::Unit test methods.

a. Installation

RSpec is installed by running “gem install rspec” in command prompt after setti ng the environment path as “…\Ruby\bin”

b. Algorithm

 We start by describing what our application behaves like. So we write down a specifications file.

 Using RSpec's “describe” block we describe the basic functions of the application.

 We then write the behaviors/expectations defining what we expect our system to behave like (expectations are similar to assertions in Test::Unit framework).

 There are two methods available for checking expectations: should() and should_not(). Spec::Expectations - Object#should(matcher) and Object#should_not(matcher).

 The code returns list of all the methods to be implemented.

 Main code implementing all the specified methods is written.

 Then, a recheck is done for fulfillment of all the expectations.

Initial Requirement(spec)

In the below code describe() method returns an ExampleGroup class(similar to TestCase in Test::Unit). The it()method returns an instance of the ExampleGroup in which that example is run.

require "Point"
describe "The location of a point" do
 it "should check whether point lies on X-axis correctly"                        
 it " should check whether point lies on Y-axis correctly " 
 it " should check whether point lies on Origin correctly " 
end

The are two methods available to check the expectations and they are should and should_not. The result obtained by running the initial requirement doc by using $ spec rspec_test.rb is

****
Pending:
The location of a point should check whether point lies on X-axis correctly (Not Yet Implemented)
./02file.rb:2
The location of a point should check whether point lies on Y-axis correctly (Not Yet Implemented)
./02file.rb:3
The location of a point should check whether point lies on Origin correctly (Not Yet Implemented)
./02file.rb:5
Finished in 0.021001 seconds
4 examples, 0 failures, 4 pending

Hence it clearly says that the above methods are yet to be implemented. This helps us in developing the code more efficiently such that they pass the above tests Code block with expectations :

require “Point”
require "rubygems"
describe "The location of a point" do
 before(:each) do
   @point = Point.new(0,2)
 end
 it "should check whether point lies on X-axis correctly" do
   @point.isOnXaxis?.should be_true
 end
 it "should check whether point lies on Y-axis correctly" do
   @point.isOnYaxis?.should be_true
 end
 it "should check whether point lies Origin correctly" do
   @point.isOnOrigin?.should be_true
 end
end

Failed output

Below are the test results that are obtained after running the above test cases,

Development Cycle of TTD

4.Cucumber:

BDD focuses heavily on the behavior of the unit. Taking another step further towards making testing non-technical user friendly, cucumber addresses a lot of these issues. Here, basically the tests are written in plain English (or any other language). Cucumber will then parse the features described and generate test templates which the developer to execute.

• It is of GWT (Given, When,Then) pattern.
• Cucumber itself is written in Ruby.

Installing Cucumber

This can be installed by running “gem install cucumber” in command prompt after setting the environment path as “…\Ruby\bin”

Test Code:

The requirement(spec) file for the cucumber is given below,

Feature: Check the point location
 In order perform check
 As a user
 I want the two coordinates of a point
 Scenario: check the location on X-axis
  Given I have entered <xcoordinate>
  And   I have entered <ycoordinate>
  When  I give check
  Then  The result should be <output>

The code implementation based on the above Requirement for Cucumber is as shown below,

require “Point”
require "rubygems"
Before do
@xcoordinate=3
@ycoordinate=0
end
Given /^I have entered <input(\d+)>$/ do |arg1|
  @point = Point.new(@xcoordinate,@ycoordinate)
end
When /^I give check$/ do
 @result = @calc.isOnXaxis?
end
Then /^The result should be <output>$/ do
puts @result
end

5.Riot:

Crisp and more expressive test cases can be obtained using Riot. It is a very fast unit test framework compared to other unit test frameworks. Unlike shoulda it is not depended on test::unit Before running the assertions in the context the setup block is executed.

Installing Riot

This can be installed by running “gem install riot” in command prompt after setting the environment path as “…\Ruby\bin”

Test Code:

require “Point”
require "rubygems"
context "The point locater" do
  setup{Point.new(3,0)}
  asserts("Location on Xaxis") {topic.isOnXaxis?}.nil
end

Conclusion:

Test::unit is the traditionally used unit testing framework as it comes default by ruby but based on the readability of the test cases Rspec is preferred by majority of the people. Below is the plot showing the comparison of the times taken to execute the test cases in various unit test frameworks,

Development Cycle of TTD

Hence we can infer from the above tabulation that riot is faster compared to majority of the unit test Frameworks

See also

• Unit Testing Frameworks for Ruby
• Community Preferred Unit Testing Framework

External Links:

• Test::Unit
• Ruby-doc
• Shoulda
• RSpec
• RSpec Documentation
• Shoulda
• Cucumber
• Unit testing
• Test-driven development
• Riot

References:

R.Venkat Rajendran, White Paper on Unit Testing

Benchmark Conclusions