CSC/ECE 517 Fall 2012/ch1 1w18 as

Testing frameworks for Ruby

Software Testing

Quality assurance is one of the most important task in software development life cycle that is applied at each step of a software process. Testing is process of executing the program with the intent of finding errors or bugs in it. Software testing ensures that a software functions as expected and confers to the customer's requirements and specifications. An objective, rigorous and comprehensive testing validates and verifies the functionalities of software product but is not restricted to it. Not all software defects are functional. A software is also tested against some non-functional requirements such as scalability, usability, performance, and security etc.

The methodology of the test is governed by the chosen software development methodology. Different software development models will focus the test effort at different points in the development process. In traditional development models, most of the test execution occurs after the requirements have been defined and the coding process has been completed. Newer development models, such as Agile, often employ test-driven development and place an increased portion of the testing in the hands of the developer, before it reaches a formal team of testers.

The general aim of testing is to affirm the quality of software systems by systematically exercising the software in carefully controlled circumstances <ref name = testing />. Testing cannot establish that a product functions properly under all circumstances but can only establish that it does not function properly under specific circumstances.

Testing Approaches

Testing approaches must have a logico-mathematical form, i.e., have one right answer. There are different approaches to software testing and are classified into different levels, depending on the stage of Software Development Life cycle (SDLC) in which it is done. The different levels are unit testing, integration testing, system testing, system integration testing and performance testing.

Unit testing

Unit testing is a method by which individual units of source code are tested to validate that they are functionally correct. It tests the basic unit of software, which is often called “unit”, “module”, or “component”. In a procedural programming, a unit could be an entire module but is more commonly an individual function or procedure. In an object-oriented programming, a unit is often an entire interface, such as a class, but could be an individual method. Unit tests are created by programmers during the development of the individual units.

Integration testing

Integration Testing is the phase in software testing in which individual unit tested software modules are combined and tested together.

It occurs after unit testing and before system testing. Integration testing takes as its input modules that have been unit tested, groups them in larger aggregates, applies tests defined in an integration test plan to those aggregates, and delivers as its output the integrated system ready for system testing.

In its simplest form, two units that have already been tested are combined into a component and the interface between them is tested. A component, in this sense, refers to an integrated aggregate of more than one unit. In a realistic scenario, many units are combined into components, which are in turn aggregated into even larger parts of the program. The idea is to test combinations of pieces and eventually expand the process to test modules with those of other groups. Eventually all the modules making up a process are tested together.

System Testing

System testing of software or hardware is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing should require no knowledge of the inner design of the code or logic. <ref name = system-testing /> Once the components of a software are integrated, the system as a whole needs to be rigorously tested to ensure that it meets the Quality Standards.Thus the System testing builds on the previous levels of testing namely unit testing and Integration Testing.

In the SDLC System Testing is the first level where the System is tested as a whole. The System is tested, in an environment that closely resembles the production environment where the application will be finally deployed, to verify if it meets the functional and technical requirements. The System Testing enables us to test, verify and validate both the Business requirements as well as the Application Architecture.

Software performance testing

Software performance testing is performed to determine if an application will meet the specified performance requirements. Performance testing can test load by simulating a heavy load, such as many users or large chunks of data going through the system. It can also perform stress tests to determine the upper limits of load that a system can endure. Performance tests monitor the attributes of a system such as throughput, system latency, or response time. These attributes can be monitored post-deployment to help troubleshoot or alert the team of performance delays that might occur. Performance testing can also serve to investigate, measure, validate or verify other quality attributes of the system, such as scalability, reliability and resource usage.

Test Driven Development (TDD)

In Test Driven Development or Test first programming, tests are written first even before the code. The code is written in a way that is just enough to pass those tests. Therefore, the first step in implementing any new functionality is to describe your expectations with failing test cases and only then write a code to implement the functionality and pass the described failure test. To write a test, the developer must clearly understand the feature's specification and requirements, it makes the developer focus on the requirements before writing the code, which is a subtle but important difference. There are different testing frameworks available in Ruby based on TDD e.g. MiniTest, Riot which we will explain in the following sections.<ref name = TDD />

Behavior-Driven Development (BDD)

The focus of Behavior-Driven Development (BDD) is on the languages and interactions used during the process of software development. It stresses on specifying behaviors that are easily understandable by people who are from non technical background. It allows non programmers to write test cases in a natural language. Behaviour-Driven Development encourages developers to think of the behavior of the component that they are developing, and of the other objects it interacts with. This allows the developers to focus on why the code should be created, rather than the technical details, and minimizes translation between the technical language in which the code is written and the domain language spoken by the" domain experts. Ruby has BDD based testing frameworks like Capybara, MiniSpec, Cucumber, RSpec etc. for testing rails applications. We will be explaining some of these in the next sections.<ref name = BDD />

Testing in Ruby

Ruby makes it simpler for the developers to write tests. Testing support was present into Ruby from the beginning. It provides a framework in its standard library for setting up organizing, and running tests called Test::Unit. There are many cases where a small bit of code needs to be run before and/or after each test. Test::Unit provides the setup and teardown member functions which are run before and after every test case.

Ruby on Rails also has testing support woven into it from the beginning. While creating the models and controllers, it produces the skeleton test code in the background. Only running these rails tests can also ensure that the code adheres to a desired functionality even after major code refactoring. These tests can even simulate some browser requests and allow us to test an application’s response without testing it through a browser. As Rails applications interact with databases heavily, it provides a test database for testing sample data.

In Ruby, Test::Unit provides a set of different assertions listed below with meaning of each assertion.

assert( boolean, [message] )	True if boolean
assert_equal( expected, actual, [message] ) assert_not_equal( expected, actual, [message] )	True if expected == actual
assert_match( pattern, string, [message] ) assert_no_match( pattern, string, [message] )	True if string =~ pattern
assert_nil( object, [message] ) assert_not_nil( object, [message] )	True if object == nil
assert_in_delta( expected_float, actual_float, delta, [message] )	True if (actual_float - expected_float).abs <= delta
assert_instance_of( class, object, [message] )	True if object.class == class
assert_kind_of( class, object, [message] )	True if object.kind_of?(class)
assert_same( expected, actual, [message]) assert_not_same( expected, actual, [message] )	True if actual.equal?( expected ).
assert_raise( Exception,... ) {block} assert_nothing_raised( Exception,...) {block}	True if the block raises (or doesn't) one of the listed exceptions.
assert_throws( expected_symbol, [message] ) {block} assert_nothing_thrown( [message] ) {block}	True if the block throws (or doesn't) the expected_symbol.
assert_respond_to( object, method, [message] )	True if the object can respond to the given method.
assert_send( send_array, [message] )	True if the method sent to the object with the given arguments return true.
assert_operator( object1, operator, object2, [message] )	Compares the two objects with the given operator, passes if true

There are different testing frameworks available in Ruby<ref name = rubytool />:

• Riot
• MiniTest
• MiniSpec
• Capybara
• Jasmine

Back to top

Riot

Overview and Functionality

This is a TDD based framework for testing Ruby applications. As opposed to some other popular testing frameworks in Ruby like Test::Unit, RSepc etc, Riot does not run the setup and teardown functions before and after each test. This speeds up test execution quite a bit, but also changes how you write your tests. In Riot, tests reside in contexts. Within these, a topic object is defined through a setup block. The actual assertions are then made with an asserts or denies block.<ref name = Riot-info />

Running Riot

It is quite simple to run Riot tests. You can put the tests in any folder, but it is recommended to put it under the “test” folder. One can run the individual tests using the normal ruby command or the entire test suite can also be run. The normal ruby command would look like following:

 ruby test/units/array_empty_test.rb

In order to be able to use Riot framework, you also need to change the test_helper.rb file under the test directory in your rails project. Add the following line into the test_helper.rb file.

 require 'riot'

Code Example

We can write a simple test to see if an array is empty

 context "An empty Array" do
   setup { Array.new }
   asserts("it is empty") { topic.empty? }
 end # An Array

As you can see, the setup block doesn’t use any instance variables to save the object under test — rather, the return value of the block is used as the topic. This object can then be accessed in the assertions using the topic attribute. Furthermore, at their very basic level, assertions need only return a boolean. When using asserts, true indicates a pass while false indicates a fail; subsequently, when using denies, true indicates a failure whereas false indicates success. We can also nest contexts within one another. the setup blocks are executed outside-in; as in, the parents’ setups are run before the current context allowing for a setup hierarchy. teardown blocks are run inside out; the current context’s teardowns are run before any of its parents’.

Back to top

MiniTest

Overview and Functionality

MiniTest is one of the more recent additions to the Ruby Testing frameworks.<ref name = previous/> MiniTest was created to be a small, clean and fast replacement for the Test::Unit framework.

MiniTest is included in the standard library as of Ruby 1.9. (If you are running Ruby 1.8, you can run gem install minitest to get Mini::Test. Alternately, you can install the test-unit gem if you have Ruby 1.9 and want to get the Test::Unit functionality.) MiniTest provides both a Test Driven Development framework and a Behavior Driven Development Framework. Test cases in MiniTest::Unit are built similarly to those in Test::Unit - you create test cases with test methods containing assertions along with optional setup and teardown methods .

Because Mini::Test was based on familiar frameworks, using it is relatively intuitive. As creator, Ryan Davis says, "There is no magic". (View Ryan's presentation on Mini::Test at the Cascadia Ruby 2011 conference.) Mini::Test is simple and straightforward.

Most of the assertions in Mini::Test::TestCase are the same as those in its predecessor with some new ones added. The most notable difference is in the negative assertions. In Test::Unit where you have an assert_not_something method, Mini::Test gave them a streamlined name of a refute_something method. (assert_not_raise and assert_not_throws are no longer available.) Mini::Test::Unit provides the following assertions:

assert	assert_block	assert_empty	refute	refute_empty
assert_equal	assert_in_delta	assert_in_epsilon	refute_equal	refute_in_delta	refute_in_epsilon
assert_includes	assert_instance_of	assert_kind_of	refute_includes	refute_instance_of	refute_kind_of
assert_match	assert_nil	assert_operator	refute_match	refute_nil	refute_operator
assert_respond_to	assert_same	assert_output	refute_respond_to	refute_same
assert_raises	assert_send	assert_silent	assert_throws

Additional Features

Minitest supports both TDD and BDD. Apart from it, MiniTest also provides benchmarking. minitest/unit is a small and incredibly fast unit testing framework. It provides a rich set of assertions to make your tests clean and readable. minitest/spec is a functionally complete spec engine. It hooks onto minitest/unit and seamlessly bridges test assertions over to spec expectations. minitest/benchmark is an awesome way to assert the performance of your algorithms in a repeatable manner.

Aside from the API improvements, MiniTest also provides some additional features such as test randomization. In unit testing, tests should run independent from each other (i.e. the outcome or resulting state(s) of one test should not affect another). By randomizing the order, MiniTest prevents tests from becoming order-dependent. Should you need to repeat a particular order to test for such issues, MiniTest provides the current seed as part of the output and gives you the option to run the test using this same seed.

Mini::Test also gives you the ability to skip tests that are not working correctly (for debug at a later time) as well as additional options for determining the performance of your test suite.<ref name = MiniTest-Info />

Code Example - Mini::Test:Unit

We follow the same steps to create a test case that are done in Test::Unit, except that this class inherits from MiniTest:Unit::TestCase and requires 'minitest/autorun'. (Requiring minitest/unit does not cause the test methods to be automatically invoked when the test case is run, hence we use minitest/autorun which provides this functionality.) We also need to update the assertions to those provided with MiniTest - for this class the assert_not_nil was changed to refute_nil. Let us assume we have an Account class as follows:

 class Account
   @balance
   @name
 
   attr_accessor :balance
   attr_accessor :name
 
   def initialize(amount)
     @balance = amount
   end
 
   def deposit(amount)
     @balance += amount
   end
   def withdrawal(amount)
     @balance -= amount
   end
 end

The AccountTest test case is below.

require 'minitest/autorun'
require_relative 'account.rb'

class AccountTest < MiniTest::Unit::TestCase

  def setup
    @a = Account.new(100)
  end

  def test_deposit
    assert_equal(200, @a.deposit(100))
  end
 
  def test_withdrawal
    assert_equal(50, @a.withdrawal(50))
  end
end

Back to top

MiniSpec

Overview and Functionality

Minispec is one more BDD based framework for testing Ruby on Rails applications. It is the BDD component of MiniTest and borrows concepts from existing BDD frameworks such as RSpec <ref name = Rspec /> and Shoulda <ref name = Shoulda />. MiniSpec is a small library built on top of Test::Unit to provide Rspec syntax for systems where Rspec may not be available (such as old versions of macruby, or when it's just not feasible to run Rspec).

MiniSpec contains the following expectations:

must_be	must_be_close_to	must_be_empty	wont_be	wont_be_close_to	wont_be_empty
must_be_instance_of	must_be_kind_of	must_be_nil	wont_be_instance_of	wont_be_kind_of	wont_be_nil
must_be_same_as	must_be_silent	must_be_within_delta	wont_be_same_as	wont_be_within_delta
must_be_within_epsilon	must_equal	must_include	wont_be_within_epsilon	wont_equal	wont_include
must_match	must_output	must_raise	wont_match
must_respond_to	must_send	must_throw	wont_respond_to

Code Example - MiniSpec

Let us assume we have some accounts files, and we are trying to read a file from this group of files. If the file is not of accounts, the test case would fail. We can write something like following <ref name = MiniSpec-example />:

 require 'minitest/spec'
 require 'minitest/autorun'
 require './account_file'
 describe AccountFile do
   describe "parse" do
     it "will return an instance of AccountFile" do
       AccountFile.parse('test_file.txt').must_be_kind_of(AccountFile)
     end
   end
 end

Back to top

Capybara

Overview and Functionality

This is a framework used for doing integration testing for rails applications. It is based on Behaviour Driven Development. It provides more english like way to write the test cases during development. It helps you test Rails applications by simulating how a real user would interact with your application. It is agnostic about the driver running your tests and comes with Selenium (http://seleniumhq.org/) support built in. Capybara can be used with RSpec and Test::Unit and Cucumber. We will discuss steps of using Capybara with Test::Unit and Cucumber in our further discussion. <ref name = Capybara-README />

In order to be able to use this framework, you will need to add the following lines to the Gemfile of your project in rails.

 group :development, :test do
   gem 'rspec-rails'
   gem 'capybara'   < ---- Add this line. 
 end

Now you have to run “Bundle install” . After that, you will need to add following line to test_helper.rb file.

 require 'capybara/rails'

For writing BDD based test cases, you can use Capybara with Cucumber. The cucumber-rails gem comes with Capybara support built-in. If you are not using Rails, manually load the capybara/cucumber module by adding following line:

 require 'capybara/cucumber'

Code Example

You can write a simple test case for login by using Capybara DSL as <ref name = capybara-example />:

 When /I sign in/ do
   within("#session") do
     fill_in 'Login', :with => 'user@example.com'
     fill_in 'Password', :with => 'password'
   end
   click_link 'Sign in'
 end

Jasmine

Overview

Jasmine is a Behavior Driven Development (BDD) based testing framework. It can be used to test the JavaScript code and can also be used in Rails framework. Jasmine ships as a Rubygem and leverages familiar rake tasks and generators to automate common workflows. Jasmine has a syntax very similar to RSepc. In order to be able to use Jasmine in Rails, you will need to add the following line in the Gemfile of your project.

 gem "jasmine"

Then update your gems, add example specs to your project, and run them:

 $ bundle install
 $ bundle exec rails generate jasmine:install

Better Rails 3/RSpec 2 support is on master, but has not yet been released.

Code Example

Consider a JavaScript code for addition of two numbers. We can write a test case for checking if the addition is returned correctly. We can write a test case in Jasmine as below:<ref name = jasmine-example />

 describe('JavaScript addition operator', function () {  
   it('adds two numbers together', function () {  
       expect(1 + 2).toEqual(3);  
   });   
 });
 

Back to top

Conclusion

Testing has always been an integral part of the Ruby landscape. This culture of testing has given rise to a variety of different testing frameworks. In this article we have reviewed only a few of the most popular offerings<ref name = rubytool /> representing both the Test Driven Development and Behavior Driven Development styles. The matrix below provides a summary of the tools highlighted on this page. Should none of these frameworks meet the Ruby programmer's needs, there are currently a number of others with more being developed every day. See the Ruby Toolbox for a list of additional Ruby testing options.

Framework Matrix

Framework	Website	Documentation	IDE Integration	Type	Ease of Use
Riot	RubyDoc	RubyDoc	Eclipse, RubyMine	TDD	Easy to learn; Easy to use
MiniTest	GitHub	RubyDoc	RubyMine	TDD/BDD	Easy to learn; Easy to use
MiniSpec	http://rspec.info/	http://rspec.info/	RubyMine	BDD	Easy to learn; Easy to use
Capybara	GitHub	RubyDoc	Eclipse, RubyMine	BDD	Slightly more difficult to learn due to BDD topology; Easy to use
Jasmine	Jasmine wiki	Jasmine with rails	RubyMine, Eclipse	BDD	Steeper learning curve; Complexity makes use a bit more challenging

References

<references> <ref name = rubytool> http://ruby-toolbox.com/categories/testing_frameworks.html </ref> <ref name = TDD> http://ruby.about.com/od/testdrivendevelopment/a/teststdd.htm </ref> <ref name = BDD> http://www.oreillynet.com/pub/a/ruby/2007/08/09/behavior-driven-development-using-ruby-part-1.html

</ref>

<ref name = Riot-info> http://thumblemonks.github.com/riot</ref> <ref name = Capybara-README> https://github.com/jnicklas/capybara/blob/master/README.md </ref> <ref name = MiniTest-Info> http://bfts.rubyforge.org/minitest/ </ref> <ref name = MiniSpec-example> http://rubysource.com/test-driven-to-distraction/ </ref> <ref name = testing> E. F. Miller, “Introduction to Software Testing Technology,” Tutorial: Software Testing & Validation Techniques, Second Edition, IEEE Catalog No. EHO 180-0, pp. 4-16 </ref> <ref name = system-testing> http://en.wikipedia.org/wiki/System_testing#cite_note-ieee-0 </ref> <ref name = Rspec> http://en.wikipedia.org/wiki/RSpec </ref> <ref name = Shoulda> http://en.wikipedia.org/wiki/Unit-testing_frameworks_for_Ruby_(programming_language)#Shoulda </ref> <ref name = capybara-example> http://rubydoc.info/github/jnicklas/capybara/ </ref> <ref name = previous> http://expertiza.csc.ncsu.edu/wiki/index.php/CSC/ECE_517_Fall_2011/ch2_2e_kt </ref> <ref name = jasmine-example> http://net.tutsplus.com/tutorials/javascript-ajax/testing-your-javascript-with-jasmine/ </ref>

</references>