CSC/ECE 517 Fall 2012/ch1 1w18 as

Testing frameworks for Ruby

Software Testing

Quality assurance is one of the most important task in software development life cycle that is applied at each step of a software process. Testing is process of executing the program with the intent of finding errors or bugs in it. Software testing ensures that a software functions as expected and confers to the customer's requirements and specifications. An objective, rigorous and comprehensive testing validates and verifies the functionalities of software product but is not restricted to it. Not all software defects are functional. A software is also tested against some non-functional requirements such as scalability, usability, performance, and security etc.

The methodology of the test is governed by the chosen software development methodology. Different software development models will focus the test effort at different points in the development process. In traditional development models, most of the test execution occurs after the requirements have been defined and the coding process has been completed. Newer development models, such as Agile, often employ test-driven development and place an increased portion of the testing in the hands of the developer, before it reaches a formal team of testers.

The general aim of testing is to affirm the quality of software systems by systematically exercising the software in carefully controlled circumstances [1]. Testing cannot establish that a product functions properly under all circumstances but can only establish that it does not function properly under specific circumstances.

Testing Approaches

Testing approaches must have a logico-mathematical form, i.e., have one right answer. There are different approaches to software testing and are classified into different levels, depending on the stage of SDLC in which it is done. The different levels are unit testing, integration testing, system testing, system integration testing and performance testing.

Unit testing

Unit testing is a method by which individual units of source code are tested to determine if they are fit for use. It tests the basic unit of software, which is the smallest testable piece of software, and is often called “unit”, “module”, or “component”. For example, in procedural programming a unit could be an entire module but is more commonly an individual function or procedure or in object-oriented programming a unit is often an entire interface, such as a class, but could be an individual method. Unit tests are created by programmers during the development of the individual units.

Integration testing

Integration Testing is the phase in software testing in which individual software modules are combined and tested as a group.

It occurs after unit testing and before system testing. Integration testing takes as its input modules that have been unit tested, groups them in larger aggregates, applies tests defined in an integration test plan to those aggregates, and delivers as its output the integrated system ready for system testing.

In its simplest form, two units that have already been tested are combined into a component and the interface between them is tested. A component, in this sense, refers to an integrated aggregate of more than one unit. In a realistic scenario, many units are combined into components, which are in turn aggregated into even larger parts of the program. The idea is to test combinations of pieces and eventually expand the process to test modules with those of other groups. Eventually all the modules making up a process are tested together.

System Testing

System testing of software or hardware is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or logic. [1] As a rule, system testing takes, as its input, all of the "integrated" software components that have successfully passed integration testing and also the software system itself integrated with any applicable hardware system(s). The purpose of integration testing is to detect any inconsistencies between the software units that are integrated together (called assemblages) or between any of the assemblages and the hardware. System testing is a more limited type of testing; it seeks to detect defects both within the "inter-assemblages" and also within the system as a whole. System testing is performed on the entire system in the context of a Functional Requirement Specification(s) (FRS) and/or a System Requirement Specification (SRS).

System Integration Testing

In the context of software systems and software engineering, system integration testing (SIT) is a testing process that exercises a software system's coexistence with others. With multiple integrated systems, assuming that each have already passed system testing, SIT proceeds to test their required interactions. Following this, the deliverables are passed on to acceptance testing. SIT is part of the software testing life cycle for collaborative projects. Usually, a round of SIT precedes the user acceptance test (UAT) round. And software providers usually run a pre-SIT round before consumers run their SIT test cases.

Software performance testing

Performance testing is in general executed to determine how a system or sub-system performs in terms of responsiveness and stability under a particular workload. It can also serve to investigate, measure, validate or verify other quality attributes of the system, such as scalability, reliability and resource usage. Load testing is primarily concerned with testing that the system can continue to operate under a specific load, whether that be large quantities of data or a large number of users. This is generally referred to as software scalability. The related load testing activity of when performed as a non-functional activity is often referred to as endurance testing. Volume testing is a way to test software functions even when certain components (for example a file or database) increase radically in size. Stress testing is a way to test reliability under unexpected or rare workloads. Stability testing(often referred to as load or endurance testing) checks to see if the software can continuously function well in or above an acceptable period. There is little agreement on what the specific goals of performance testing are. The terms load testing, performance testing, reliability testing, and volume testing, are often used interchangeably.

Test Driven Development (TDD)

In Test Driven Development, tests are written even before the code. It defines a rigid set of rules for developing software. It is also called as Test first programming. The code is written only in order to make the test cases successful which are already written. There are different testing frameworks available in Ruby based on TDD e.g. MiniTest, Riot which we will explain in the following sections.

Behavior Driven Development (BDD)

Behaviour Driven Development (BDD) focuses on the languages and interactions used during the process of software development. It stresses on specifying behaviours that are easily understandable by people who are from non technical background. It allows non programmers to write test cases in a natural language. Also, BDD lets you look at the code implementation from a behavioral abstraction perspective. Ruby has BDD based testing frameworks like Capybara, MiniSpec, Cucumber, RSpec etc. for testing rails applications. We will be explaining some of these in the next sections.

Testing in Ruby

Ruby makes it simpler for the developers to write tests. Testing support was present into Ruby from the beginning. It provides a framework in its standard library for setting up organizing, and running tests called Test::Unit. There are many cases where a small bit of code needs to be run before and/or after each test. Test::Unit provides the setup and teardown member functions which are run before and after every test case.

Ruby on Rails also has testing support woven into it from the beginning. While creating the models and controllers, it produces the skeleton test code in the background. Only running these rails tests can also ensure that the code adheres to a desired functionality even after major code refactoring. These tests can even simulate some browser requests and allows us to test an application’s response without testing it through a browser. As Rails applications interact with databases heavily, it provides a test database for testing sample data.

In Ruby, Test::Unit provides a set of different assertions listed below with meaning of each assertion.

assert( boolean, [message] )	True if boolean
assert_equal( expected, actual, [message] ) assert_not_equal( expected, actual, [message] )	True if expected == actual
assert_match( pattern, string, [message] ) assert_no_match( pattern, string, [message] )	True if string =~ pattern
assert_nil( object, [message] ) assert_not_nil( object, [message] )	True if object == nil
assert_in_delta( expected_float, actual_float, delta, [message] )	True if (actual_float - expected_float).abs <= delta
assert_instance_of( class, object, [message] )	True if object.class == class
assert_kind_of( class, object, [message] )	True if object.kind_of?(class)
assert_same( expected, actual, [message]) assert_not_same( expected, actual, [message] )	True if actual.equal?( expected ).
assert_raise( Exception,... ) {block} assert_nothing_raised( Exception,...) {block}	True if the block raises (or doesn't) one of the listed exceptions.
assert_throws( expected_symbol, [message] ) {block} assert_nothing_thrown( [message] ) {block}	True if the block throws (or doesn't) the expected_symbol.
assert_respond_to( object, method, [message] )	True if the object can respond to the given method.
assert_send( send_array, [message] )	True if the method sent to the object with the given arguments return true.
assert_operator( object1, operator, object2, [message] )	Compares the two objects with the given operator, passes if true

There are different testing frameworks available in Ruby<ref name = rubytool />:

• Riot
• MiniTest
• MiniSpec
• Capybara
• Jasmine

Back to top

Riot

Overview and Functionality

This is a TDD based framework for testing Ruby applications. As opposed to some other popular testing frameworks in Ruby like Test::Unit, RSepc etc, Riot does not run the setup and teardown function before and after each test. This speeds up test execution quite a bit, but also changes how you write your tests. This speeds up test execution quite a bit, but also changes how you write your tests. In Riot, tests reside in contexts. Within these, a topic object is defined through a setup block. The actual assertions are then made with an asserts or denies block.

Running Riot

It is quite simple to run Riot tests. You can put the tests in any folder, but it is recommended to put it under the “test” folder. One can run the individual tests using the normal ruby command or the entire test suite. The normal ruby command would look like following:

 ruby test/units/array_empty_test.rb

In order to be able to use Riot framework, you also need to change the test_helper.rb file under the test directory. Add the following line into the test_helper.rb file.

 require 'riot'

Code Example

We can write a simple test to see if an array is empty

 context "An empty Array" do
   setup { Array.new }
   asserts("it is empty") { topic.empty? }
 end # An Array

As you can see, the setup block doesn’t use any instance variables to save the object under test — rather, the return value of the block is used as the topic. This object can then be accessed in the assertions using the topic attribute. Furthermore, at their very basic level, assertions need only return a boolean. When using asserts, true indicates a pass while false indicates a fail; subsequently, when using denies, true indicates a failure whereas false indicates success. We can also nest contexts within one another. the setup blocks are executed outside-in; as in, the parents’ setups are run before the current context allowing for a setup hierarchy. teardown blocks are run inside out; the current context’s teardowns are run before any of its parents’.

Back to top

MiniTest

Overview and Functionality

MiniTest is one of the more recent additions to the Ruby Testing frameworks. MiniTest was created to be a small, clean and fast replacement for the Test::Unit framework.

As previously mentioned, MiniTest is included in the standard library as of Ruby 1.9. (If you are running Ruby 1.8, you can run gem install minitest to get Mini::Test. Alternately, you can install the test-unit gem if you have Ruby 1.9 and want to get the Test::Unit functionality.) MiniTest provides both a Test Driven Development framework and a Behavior Driven Development Framework. Test cases in MiniTest::Unit are built similarly to those in Test::Unit - you create test cases with test methods containing assertions along with optional setup and teardown methods .

Because Mini::Test was based on familiar frameworks, using it is realtively intuitive. As creator, Ryan Davis says, "There is no magic". (View Ryan's presentation on Mini::Test at the Cascadia Ruby 2011 conference.) Mini::Test is simple and straightforward.

Most of the assertions in Mini::Test::TestCase are the same as those in its predecessor with some new ones added. The most notable difference is in the negative assertions. In Test::Unit where you have an assert_not_something method, Mini::Test gave them a streamlined name of a refute_something method. (assert_not_raise and assert_not_throws are no longer available.) Mini::Test::Unit provides the following assertions:

assert	assert_block	assert_empty	refute	refute_empty
assert_equal	assert_in_delta	assert_in_epsilon	refute_equal	refute_in_delta	refute_in_epsilon
assert_includes	assert_instance_of	assert_kind_of	refute_includes	refute_instance_of	refute_kind_of
assert_match	assert_nil	assert_operator	refute_match	refute_nil	refute_operator
assert_respond_to	assert_same	assert_output	refute_respond_to	refute_same
assert_raises	assert_send	assert_silent	assert_throws

Additional Features

Minitest supports both TDD and BDD. Apart from it, MiniTest also provides benchmarking.

minitest/unit is a small and incredibly fast unit testing framework. It provides a rich set of assertions to make your tests clean and readable. minitest/spec is a functionally complete spec engine. It hooks onto minitest/unit and seamlessly bridges test assertions over to spec expectations.

minitest/benchmark is an awesome way to assert the performance of your algorithms in a repeatable manner.

Aside from the API improvements, MiniTest also provides some additional features such as test randomization. In unit testing, tests should run independent from each other (i.e. the outcome or resulting state(s) of one test should not affect another). By randomizing the order, MiniTest prevents tests from becoming order-dependent. Should you need to repeat a particular order to test for such issues, MiniTest provides the current seed as part of the output and gives you the option to run the test using this same seed.

Mini::Test also gives you the ability to skip tests that are not working correctly (for debug at a later time) as well as additional options for determining the performance of your test suite.

Code Example - Mini::Test:Unit

We follow the same steps to create a test case that are done in Test::Unit, except that this class inherits from MiniTest:Unit::TestCase and requires 'minitest/autorun'. (Requiring minitest/unit does not cause the test methods to be automatically invoked when the test case is run, hence we use minitest/autorun which provides this functionality.) We also need to update the assertions to those provided with MiniTest - for this class the assert_not_nil was changed to refute_nil. Let us assume we have an Account class as follows:

 class Account
   @balance
   @name
 
   attr_accessor :balance
   attr_accessor :name
 
   def initialize(amount)
     @balance = amount
   end
 
   def deposit(amount)
     @balance += amount
   end
 
   def addinterest(rate)
     @balance *= (1 + rate)
   end
 
   def withdrawal(amount)
     @balance -= amount
   end
 end

The AccountTest test case is below.

require 'minitest/autorun'
require_relative 'account.rb'

class AccountTest < MiniTest::Unit::TestCase

  def setup
    @a = Account.new(100)
  end

  def test_deposit
    assert_equal(200, @a.deposit(100))
  end
 
  def test_withdrawal
    assert_equal(50, @a.withdrawal(50))
  end
 
  def test_name
    @a.name = "Checking"
    refute_nil(@a.name())
    assert_match("Checking", @a.name)
  end

  def test_interest
    assert_in_delta(130, @a.addinterest(0.333), 5)
  end

  def test_fail
    assert_equal(200, @a.balance())
  end
 
  def test_whatru
    assert_instance_of(Account, @a)
  end
     
end

Back to top

MiniSpec

Overview and Functionality

Minispec is one more BDD based framework for testing Ruby on Rails applications. It is the BDD component of MiniTest and borrows concepts from existing BDD frameworks such as RSpec and Shoulda.

MiniSpec is a small library built on top of Test::Unit to provide rspec syntax for systems where rspec may not be available (such as old versions of macruby, or when it's just not feasible to run rspec).

MiniSpec contains the following expectations:

must_be	must_be_close_to	must_be_empty	wont_be	wont_be_close_to	wont_be_empty
must_be_instance_of	must_be_kind_of	must_be_nil	wont_be_instance_of	wont_be_kind_of	wont_be_nil
must_be_same_as	must_be_silent	must_be_within_delta	wont_be_same_as	wont_be_within_delta
must_be_within_epsilon	must_equal	must_include	wont_be_within_epsilon	wont_equal	wont_include
must_match	must_output	must_raise	wont_match
must_respond_to	must_send	must_throw	wont_respond_to

Code Example - MiniSpec

Back to top

Capybara

Overview and Functionality

This is a framework used for doing integration testing for rails applications. It is based on Behaviour Driven Development testing. It provides more english like way to write the test cases during development. It helps you test Rails applications by simulating how a real user would interact with your application. It is agnostic about the driver running your tests and comes with Selenium (http://seleniumhq.org/) support built in. Capybara can be used with RSpec and Test::Unit and Cucumber. We will discuss steps of using Capybara with Test::Unit and Cucumber in our further discussion.

In order to be able to use this framework, you will need to add the following lines to the Gemfile of your project in rails.

 group :development, :test do
   gem 'rspec-rails'
   gem 'capybara'   < ---- Add this line. 
 end

Now you have to run “Bundle install” . After that, you will need to add following line to test_helper.rb file.

 require 'capybara/rails'

For writing BDD based test cases, you can use Capybara with Cucumber. The cucumber-rails gem comes with Capybara support built-in. If you are not using Rails, manually load the capybara/cucumber module by adding following line:

 require 'capybara/cucumber'

Code Example

You can write a simple test case for login by using Capybara DSL as:

 When /I sign in/ do
   within("#session") do
     fill_in 'Login', :with => 'user@example.com'
     fill_in 'Password', :with => 'password'
   end
   click_link 'Sign in'
 end

Jasmine

Code Example

Back to top

Conclusion

Testing has always been an integral part of the Ruby landscape. This culture of testing has given rise to a variety of different testing frameworks. In this article we have reviewed only a few of the most popular offerings<ref name = rubytool /> representing both the Test Driven Development and Behavior Driven Development styles. The matrix below provides a summary of the tools highlighted on this page. Should none of these frameworks meet the Ruby programmer's needs, there are currently a number of others with more being developed every day. See the Ruby Toolbox for a list of additional Ruby testing options.

Framework Matrix

Framework	Website	Documentation	IDE Integration	Type	Ease of Use
Riot	RubyDoc	RubyDoc	Eclipse, RubyMine	TDD	Easy to learn; Easy to use
MiniTest	GitHub	RubyDoc	Eclipse	TDD/BDD	Easy to learn; Easy to use
MiniSpec	http://rspec.info/	http://rspec.info/	Eclipse, RubyMine	BDD	Slightly more difficult to learn due to BDD topology; Easy to use
Capybara	GitHub	RubyDoc	Eclipse, RubyMine	BDD	Easy to learn; Easy to use
Jasmine	Cucumber	Cucumber Wiki	RubyMine	BDD	Steeper learning curve; Complexity makes use a bit more challenging

References

<references> <ref name = rubytool> http://ruby-toolbox.com/categories/testing_frameworks.html </ref> <ref name = progruby> Programming Ruby: The Pragmatic Programmers' Guide, Second Ed. </ref> </references>