CSC/ECE 517 Fall 2012/ch1 1w18 as
Testing frameworks for Ruby
Software Testing
Quality assurance is one of the most important task in software development life cycle that is applied at each step of a software process. Testing is process of executing the program with the intent of finding errors or bugs in it. Software testing ensures that a software functions as expected and confers to the customer's requirements and specifications. An objective, rigorous and comprehensive testing validates and verifies the functionalities of software product but is not restricted to it. Not all software defects are functional. A software is also tested against some non-functional requirements such as scalability, usability, performance, and security etc.
The methodology of the test is governed by the chosen software development methodology. Different software development models will focus the test effort at different points in the development process. In traditional development models, most of the test execution occurs after the requirements have been defined and the coding process has been completed. Newer development models, such as Agile, often employ test-driven development and place an increased portion of the testing in the hands of the developer, before it reaches a formal team of testers.
The general aim of testing is to affirm the quality of software systems by systematically exercising the software in carefully controlled circumstances [1]. Testing cannot establish that a product functions properly under all circumstances but can only establish that it does not function properly under specific circumstances.
Testing Approaches
Testing approaches must have a logico-mathematical form, i.e., have one right answer. There are different approaches to software testing and are classified into different levels, depending on the stage of Software Development Life cycle (SDLC) in which it is done. The different levels are unit testing, integration testing, system testing, system integration testing and performance testing.
Unit testing
Unit testing is a method by which individual units of source code are tested to validate that they are functionally correct. It tests the basic unit of software, which is often called “unit”, “module”, or “component”. In a procedural programming, a unit could be an entire module but is more commonly an individual function or procedure. In an object-oriented programming, a unit is often an entire interface, such as a class, but could be an individual method. Unit tests are created by programmers during the development of the individual units.
Integration testing
Integration Testing is the phase in software testing in which individual unit tested software modules are combined and tested together.
It occurs after unit testing and before system testing. Integration testing takes as its input modules that have been unit tested, groups them in larger aggregates, applies tests defined in an integration test plan to those aggregates, and delivers as its output the integrated system ready for system testing.
In its simplest form, two units that have already been tested are combined into a component and the interface between them is tested. A component, in this sense, refers to an integrated aggregate of more than one unit. In a realistic scenario, many units are combined into components, which are in turn aggregated into even larger parts of the program. The idea is to test combinations of pieces and eventually expand the process to test modules with those of other groups. Eventually all the modules making up a process are tested together.
System Testing
System testing of software or hardware is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing should require no knowledge of the inner design of the code or logic. [1] Once the components of a software are integrated, the system as a whole needs to be rigorously tested to ensure that it meets the Quality Standards.Thus the System testing builds on the previous levels of testing namely unit testing and Integration Testing.
In the SDLC System Testing is the first level where the System is tested as a whole. The System is tested, in an environment that closely resembles the production environment where the application will be finally deployed, to verify if it meets the functional and technical requirements. The System Testing enables us to test, verify and validate both the Business requirements as well as the Application Architecture.
System Integration Testing
System integration testing (SIT) is a testing process that is performed when two systems, each of which that have already passed system testing, are integrated together. The goal of systems integration testing is to validate that the required interactions between the two integrated systems.
Software performance testing
Software performance testing is performed to determine if an application will meet the specified performance requirements. Performance testing can test load by simulating a heavy load, such as many users or large chunks of data going through the system. It can also perform stress tests to determine the upper limits of load that a system can endure. Performance tests monitor the attributes of a system such as throughput, system latency, or response time. These attributes can be monitored post-deployment to help troubleshoot or alert the team of performance delays that might occur. Performance testing can also serve to investigate, measure, validate or verify other quality attributes of the system, such as scalability, reliability and resource usage.
Test Driven Development (TDD)
In Test Driven Development, tests are written even before the code. It defines a rigid set of rules for developing software. It is also called as Test first programming. The code is written only in order to make the test cases successful which are already written. There are different testing frameworks available in Ruby based on TDD e.g. MiniTest, Riot which we will explain in the following sections.<ref name = TDD />
Behavior Driven Development (BDD)
Behaviour Driven Development (BDD) focuses on the languages and interactions used during the process of software development. It stresses on specifying behaviours that are easily understandable by people who are from non technical background. It allows non programmers to write test cases in a natural language. Also, BDD lets you look at the code implementation from a behavioral abstraction perspective. Ruby has BDD based testing frameworks like Capybara, MiniSpec, Cucumber, RSpec etc. for testing rails applications. We will be explaining some of these in the next sections.<ref name = BDD />
Testing in Ruby
Ruby makes it simpler for the developers to write tests. Testing support was present into Ruby from the beginning. It provides a framework in its standard library for setting up organizing, and running tests called Test::Unit. There are many cases where a small bit of code needs to be run before and/or after each test. Test::Unit provides the setup and teardown member functions which are run before and after every test case.
Ruby on Rails also has testing support woven into it from the beginning. While creating the models and controllers, it produces the skeleton test code in the background. Only running these rails tests can also ensure that the code adheres to a desired functionality even after major code refactoring. These tests can even simulate some browser requests and allows us to test an application’s response without testing it through a browser. As Rails applications interact with databases heavily, it provides a test database for testing sample data.
In Ruby, Test::Unit provides a set of different assertions listed below with meaning of each assertion.
| assert( boolean, [message] ) | True if boolean |
| assert_equal( expected, actual, [message] ) assert_not_equal( expected, actual, [message] ) |
True if expected == actual |
| assert_match( pattern, string, [message] ) assert_no_match( pattern, string, [message] ) |
True if string =~ pattern |
| assert_nil( object, [message] ) assert_not_nil( object, [message] ) |
True if object == nil |
| assert_in_delta( expected_float, actual_float, delta, [message] ) | True if (actual_float - expected_float).abs <= delta |
| assert_instance_of( class, object, [message] ) | True if object.class == class |
| assert_kind_of( class, object, [message] ) | True if object.kind_of?(class) |
| assert_same( expected, actual, [message]) assert_not_same( expected, actual, [message] ) |
True if actual.equal?( expected ). |
| assert_raise( Exception,... ) {block} assert_nothing_raised( Exception,...) {block} |
True if the block raises (or doesn't) one of the listed exceptions. |
| assert_throws( expected_symbol, [message] ) {block} assert_nothing_thrown( [message] ) {block} |
True if the block throws (or doesn't) the expected_symbol. |
| assert_respond_to( object, method, [message] ) | True if the object can respond to the given method. |
| assert_send( send_array, [message] ) | True if the method sent to the object with the given arguments return true. |
| assert_operator( object1, operator, object2, [message] ) | Compares the two objects with the given operator, passes if true |
There are different testing frameworks available in Ruby<ref name = rubytool />:
Riot
Overview and Functionality
This is a TDD based framework for testing Ruby applications. As opposed to some other popular testing frameworks in Ruby like Test::Unit, RSepc etc, Riot does not run the setup and teardown function before and after each test. This speeds up test execution quite a bit, but also changes how you write your tests. This speeds up test execution quite a bit, but also changes how you write your tests. In Riot, tests reside in contexts. Within these, a topic object is defined through a setup block. The actual assertions are then made with an asserts or denies block.<ref name = Riot-info />
Running Riot
It is quite simple to run Riot tests. You can put the tests in any folder, but it is recommended to put it under the “test” folder. One can run the individual tests using the normal ruby command or the entire test suite. The normal ruby command would look like following:
ruby test/units/array_empty_test.rb
In order to be able to use Riot framework, you also need to change the test_helper.rb file under the test directory. Add the following line into the test_helper.rb file.
require 'riot'
Code Example
We can write a simple test to see if an array is empty
context "An empty Array" do
setup { Array.new }
asserts("it is empty") { topic.empty? }
end # An Array
As you can see, the setup block doesn’t use any instance variables to save the object under test — rather, the return value of the block is used as the topic. This object can then be accessed in the assertions using the topic attribute. Furthermore, at their very basic level, assertions need only return a boolean. When using asserts, true indicates a pass while false indicates a fail; subsequently, when using denies, true indicates a failure whereas false indicates success. We can also nest contexts within one another. the setup blocks are executed outside-in; as in, the parents’ setups are run before the current context allowing for a setup hierarchy. teardown blocks are run inside out; the current context’s teardowns are run before any of its parents’.
MiniTest
Overview and Functionality
MiniTest is one of the more recent additions to the Ruby Testing frameworks. MiniTest was created to be a small, clean and fast replacement for the Test::Unit framework.
As previously mentioned, MiniTest is included in the standard library as of Ruby 1.9. (If you are running Ruby 1.8, you can run gem install minitest to get Mini::Test. Alternately, you can install the test-unit gem if you have Ruby 1.9 and want to get the Test::Unit functionality.) MiniTest provides both a Test Driven Development framework and a Behavior Driven Development Framework. Test cases in MiniTest::Unit are built similarly to those in Test::Unit - you create test cases with test methods containing assertions along with optional setup and teardown methods .
Because Mini::Test was based on familiar frameworks, using it is realtively intuitive. As creator, Ryan Davis says, "There is no magic". (View Ryan's presentation on Mini::Test at the Cascadia Ruby 2011 conference.) Mini::Test is simple and straightforward.
Most of the assertions in Mini::Test::TestCase are the same as those in its predecessor with some new ones added. The most notable difference is in the negative assertions. In Test::Unit where you have an assert_not_something method, Mini::Test gave them a streamlined name of a refute_something method. (assert_not_raise and assert_not_throws are no longer available.) Mini::Test::Unit provides the following assertions:
| assert | assert_block | assert_empty | refute | refute_empty | |
| assert_equal | assert_in_delta | assert_in_epsilon | refute_equal | refute_in_delta | refute_in_epsilon |
| assert_includes | assert_instance_of | assert_kind_of | refute_includes | refute_instance_of | refute_kind_of |
| assert_match | assert_nil | assert_operator | refute_match | refute_nil | refute_operator |
| assert_respond_to | assert_same | assert_output | refute_respond_to | refute_same | |
| assert_raises | assert_send | assert_silent | assert_throws |
Additional Features
Minitest supports both TDD and BDD. Apart from it, MiniTest also provides benchmarking.
minitest/unit is a small and incredibly fast unit testing framework. It provides a rich set of assertions to make your tests clean and readable. minitest/spec is a functionally complete spec engine. It hooks onto minitest/unit and seamlessly bridges test assertions over to spec expectations.
minitest/benchmark is an awesome way to assert the performance of your algorithms in a repeatable manner.
Aside from the API improvements, MiniTest also provides some additional features such as test randomization. In unit testing, tests should run independent from each other (i.e. the outcome or resulting state(s) of one test should not affect another). By randomizing the order, MiniTest prevents tests from becoming order-dependent. Should you need to repeat a particular order to test for such issues, MiniTest provides the current seed as part of the output and gives you the option to run the test using this same seed.
Mini::Test also gives you the ability to skip tests that are not working correctly (for debug at a later time) as well as additional options for determining the performance of your test suite.<ref name = MiniTest-Info />
Code Example - Mini::Test:Unit
We follow the same steps to create a test case that are done in Test::Unit, except that this class inherits from MiniTest:Unit::TestCase and requires 'minitest/autorun'. (Requiring minitest/unit does not cause the test methods to be automatically invoked when the test case is run, hence we use minitest/autorun which provides this functionality.) We also need to update the assertions to those provided with MiniTest - for this class the assert_not_nil was changed to refute_nil. Let us assume we have an Account class as follows:
class Account
@balance
@name
attr_accessor :balance
attr_accessor :name
def initialize(amount)
@balance = amount
end
def deposit(amount)
@balance += amount
end
def addinterest(rate)
@balance *= (1 + rate)
end
def withdrawal(amount)
@balance -= amount
end
end
The AccountTest test case is below.
require 'minitest/autorun'
require_relative 'account.rb'
class AccountTest < MiniTest::Unit::TestCase
def setup
@a = Account.new(100)
end
def test_deposit
assert_equal(200, @a.deposit(100))
end
def test_withdrawal
assert_equal(50, @a.withdrawal(50))
end
def test_name
@a.name = "Checking"
refute_nil(@a.name())
assert_match("Checking", @a.name)
end
def test_interest
assert_in_delta(130, @a.addinterest(0.333), 5)
end
def test_fail
assert_equal(200, @a.balance())
end
def test_whatru
assert_instance_of(Account, @a)
end
end
MiniSpec
Overview and Functionality
Minispec is one more BDD based framework for testing Ruby on Rails applications. It is the BDD component of MiniTest and borrows concepts from existing BDD frameworks such as RSpec and Shoulda.
MiniSpec is a small library built on top of Test::Unit to provide rspec syntax for systems where rspec may not be available (such as old versions of macruby, or when it's just not feasible to run rspec).
MiniSpec contains the following expectations:
| must_be | must_be_close_to | must_be_empty | wont_be | wont_be_close_to | wont_be_empty |
| must_be_instance_of | must_be_kind_of | must_be_nil | wont_be_instance_of | wont_be_kind_of | wont_be_nil |
| must_be_same_as | must_be_silent | must_be_within_delta | wont_be_same_as | wont_be_within_delta | |
| must_be_within_epsilon | must_equal | must_include | wont_be_within_epsilon | wont_equal | wont_include |
| must_match | must_output | must_raise | wont_match | ||
| must_respond_to | must_send | must_throw | wont_respond_to |
Code Example - MiniSpec
Let us assume we have some accounts files, and we are trying to read a file from this group of files. If the file is not of accounts, the test case would fail. We can write something like following <ref name = MiniSpec-example />:
require 'minitest/spec' require 'minitest/autorun' require './account_file'
describe AccountFile do
describe "parse" do
it "will return an instance of AccountFile" do
AccountFile.parse('test_file.txt').must_be_kind_of(AccountFile)
end
end
end
Capybara
Overview and Functionality
This is a framework used for doing integration testing for rails applications. It is based on Behaviour Driven Development testing. It provides more english like way to write the test cases during development. It helps you test Rails applications by simulating how a real user would interact with your application. It is agnostic about the driver running your tests and comes with Selenium (http://seleniumhq.org/) support built in. Capybara can be used with RSpec and Test::Unit and Cucumber. We will discuss steps of using Capybara with Test::Unit and Cucumber in our further discussion. <ref name = Capybara-README />
In order to be able to use this framework, you will need to add the following lines to the Gemfile of your project in rails.
group :development, :test do gem 'rspec-rails' gem 'capybara' < ---- Add this line. end
Now you have to run “Bundle install” . After that, you will need to add following line to test_helper.rb file.
require 'capybara/rails'
For writing BDD based test cases, you can use Capybara with Cucumber. The cucumber-rails gem comes with Capybara support built-in. If you are not using Rails, manually load the capybara/cucumber module by adding following line:
require 'capybara/cucumber'
Code Example
You can write a simple test case for login by using Capybara DSL as:
When /I sign in/ do
within("#session") do
fill_in 'Login', :with => 'user@example.com'
fill_in 'Password', :with => 'password'
end
click_link 'Sign in'
end
Jasmine
Overview
Jasmine is a Behavior Driven Development (BDD) based testing framework. It can be used to test the JavaScript code and can also be used in Rails framework. Jasmine ships as a Rubygem and leverages familiar rake tasks and generators to automate common workflows. Jasmine has a syntax very similar to RSepc. In order to be able to use Jasmine in Rails, you will need to add the following line in the Gemfile of your project.
gem "jasmine"
Then update your gems, add example specs to your project, and run them:
$ bundle install $ bundle exec rails generate jasmine:install
Better Rails 3/RSpec 2 support is on master, but has not yet been released.
Code Example
Conclusion
Testing has always been an integral part of the Ruby landscape. This culture of testing has given rise to a variety of different testing frameworks. In this article we have reviewed only a few of the most popular offerings<ref name = rubytool /> representing both the Test Driven Development and Behavior Driven Development styles. The matrix below provides a summary of the tools highlighted on this page. Should none of these frameworks meet the Ruby programmer's needs, there are currently a number of others with more being developed every day. See the Ruby Toolbox for a list of additional Ruby testing options.
Framework Matrix
| Framework | Website | Documentation | IDE Integration | Type | Ease of Use |
|---|---|---|---|---|---|
| Riot | RubyDoc | RubyDoc | Eclipse, RubyMine | TDD | Easy to learn; Easy to use |
| MiniTest | GitHub | RubyDoc | Eclipse | TDD/BDD | Easy to learn; Easy to use |
| MiniSpec | http://rspec.info/ | http://rspec.info/ | Eclipse, RubyMine | BDD | Easy to learn; Easy to use |
| Capybara | GitHub | RubyDoc | Eclipse, RubyMine | BDD | Slightly more difficult to learn due to BDD topology; Easy to use |
| Jasmine | Jasmine wiki | Jasmine with rails | RubyMine | BDD | Steeper learning curve; Complexity makes use a bit more challenging |
References
<references> <ref name = rubytool> http://ruby-toolbox.com/categories/testing_frameworks.html </ref> <ref name = TDD> http://ruby.about.com/od/testdrivendevelopment/a/teststdd.htm </ref> <ref name = BDD> http://www.oreillynet.com/pub/a/ruby/2007/08/09/behavior-driven-development-using-ruby-part-1.html
</ref>
<ref name = Riot-info> http://thumblemonks.github.com/riot</ref> <ref name = Capybara-README> https://github.com/jnicklas/capybara/blob/master/README.md </ref> <ref name = MiniTest-Info> http://bfts.rubyforge.org/minitest/ </ref> <ref name = MiniSpec-example> http://rubysource.com/test-driven-to-distraction/ </ref> </references>