CSC/ECE 517 Fall 2012/ch1 1w18 as

Testing frameworks for Ruby

Software Testing

Quality assurance is one of the most important task in software development life cycle that is applied at each step of a software process. Testing is process of executing the program with the intent of finding errors or bugs in it. Software testing ensures that a software functions as expected and confers to the customer's requirements and specifications. An objective, rigorous and comprehensive testing validates and verifies the functionalities of software product but is not restricted to it. Not all software defects are functional. A software is also tested against some non-functional requirements such as scalability, usability, performance, and security etc.

The methodology of the test is governed by the chosen software development methodology. Different software development models will focus the test effort at different points in the development process. In traditional development models, most of the test execution occurs after the requirements have been defined and the coding process has been completed. Newer development models, such as Agile, often employ test-driven development and place an increased portion of the testing in the hands of the developer, before it reaches a formal team of testers.

The general aim of testing is to affirm the quality of software systems by systematically exercising the software in carefully controlled circumstances [1]. Testing cannot establish that a product functions properly under all circumstances but can only establish that it does not function properly under specific circumstances.

Testing Approaches

Testing approaches must have a logico-mathematical form, i.e., have one right answer. There are different approaches to software testing and are classified into different levels, depending on the stage of SDLC in which it is done. The different levels are unit testing, integration testing, system testing, system integration testing and performance testing.

Unit testing

Unit testing is a method by which individual units of source code are tested to determine if they are fit for use. It tests the basic unit of software, which is the smallest testable piece of software, and is often called “unit”, “module”, or “component”. For example, in procedural programming a unit could be an entire module but is more commonly an individual function or procedure or in object-oriented programming a unit is often an entire interface, such as a class, but could be an individual method. Unit tests are created by programmers during the development of the individual units.

Integration testing

Integration Testing is the phase in software testing in which individual software modules are combined and tested as a group.

It occurs after unit testing and before system testing. Integration testing takes as its input modules that have been unit tested, groups them in larger aggregates, applies tests defined in an integration test plan to those aggregates, and delivers as its output the integrated system ready for system testing.

In its simplest form, two units that have already been tested are combined into a component and the interface between them is tested. A component, in this sense, refers to an integrated aggregate of more than one unit. In a realistic scenario, many units are combined into components, which are in turn aggregated into even larger parts of the program. The idea is to test combinations of pieces and eventually expand the process to test modules with those of other groups. Eventually all the modules making up a process are tested together.

System Testing

System testing of software or hardware is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or logic. [1] As a rule, system testing takes, as its input, all of the "integrated" software components that have successfully passed integration testing and also the software system itself integrated with any applicable hardware system(s). The purpose of integration testing is to detect any inconsistencies between the software units that are integrated together (called assemblages) or between any of the assemblages and the hardware. System testing is a more limited type of testing; it seeks to detect defects both within the "inter-assemblages" and also within the system as a whole. System testing is performed on the entire system in the context of a Functional Requirement Specification(s) (FRS) and/or a System Requirement Specification (SRS).

System Integration Testing

In the context of software systems and software engineering, system integration testing (SIT) is a testing process that exercises a software system's coexistence with others. With multiple integrated systems, assuming that each have already passed system testing, SIT proceeds to test their required interactions. Following this, the deliverables are passed on to acceptance testing. SIT is part of the software testing life cycle for collaborative projects. Usually, a round of SIT precedes the user acceptance test (UAT) round. And software providers usually run a pre-SIT round before consumers run their SIT test cases.

Software performance testing

Performance testing is in general executed to determine how a system or sub-system performs in terms of responsiveness and stability under a particular workload. It can also serve to investigate, measure, validate or verify other quality attributes of the system, such as scalability, reliability and resource usage. Load testing is primarily concerned with testing that the system can continue to operate under a specific load, whether that be large quantities of data or a large number of users. This is generally referred to as software scalability. The related load testing activity of when performed as a non-functional activity is often referred to as endurance testing. Volume testing is a way to test software functions even when certain components (for example a file or database) increase radically in size. Stress testing is a way to test reliability under unexpected or rare workloads. Stability testing(often referred to as load or endurance testing) checks to see if the software can continuously function well in or above an acceptable period. There is little agreement on what the specific goals of performance testing are. The terms load testing, performance testing, reliability testing, and volume testing, are often used interchangeably.

Test Driven Development (TDD)

In Test Driven Development, tests are written even before the code. It defines a rigid set of rules for developing software. It is also called as Test first programming. The code is written only in order to make the test cases successful which are already written. There are different testing frameworks available in Ruby based on TDD e.g. MiniTest, Riot which we will explain in the following sections.

Behavior Driven Development (BDD)

Behaviour Driven Development (BDD) focuses on the languages and interactions used during the process of software development. It stresses on specifying behaviours that are easily understandable by people who are from non technical background. It allows non programmers to write test cases in a natural language. Also, BDD lets you look at the code implementation from a behavioral abstraction perspective. Ruby has BDD based testing frameworks like Capybara, MiniSpec, Cucumber, RSpec etc. for testing rails applications. We will be explaining some of these in the next sections.

Testing in Ruby

Ruby makes it simpler for the developers to write tests. Testing support was present into Ruby from the beginning. It provides a framework in its standard library for setting up organizing, and running tests called Test::Unit. There are many cases where a small bit of code needs to be run before and/or after each test. Test::Unit provides the setup and teardown member functions which are run before and after every test case.

Ruby on Rails also has testing support woven into it from the beginning. While creating the models and controllers, it produces the skeleton test code in the background. Only running these rails tests can also ensure that the code adheres to a desired functionality even after major code refactoring. These tests can even simulate some browser requests and allows us to test an application’s response without testing it through a browser. As Rails applications interact with databases heavily, it provides a test database for testing sample data.

In Ruby, Test::Unit provides a set of different assertions listed below with meaning of each assertion.

assert( boolean, [message] )	True if boolean
assert_equal( expected, actual, [message] ) assert_not_equal( expected, actual, [message] )	True if expected == actual
assert_match( pattern, string, [message] ) assert_no_match( pattern, string, [message] )	True if string =~ pattern
assert_nil( object, [message] ) assert_not_nil( object, [message] )	True if object == nil
assert_in_delta( expected_float, actual_float, delta, [message] )	True if (actual_float - expected_float).abs <= delta
assert_instance_of( class, object, [message] )	True if object.class == class
assert_kind_of( class, object, [message] )	True if object.kind_of?(class)
assert_same( expected, actual, [message]) assert_not_same( expected, actual, [message] )	True if actual.equal?( expected ).
assert_raise( Exception,... ) {block} assert_nothing_raised( Exception,...) {block}	True if the block raises (or doesn't) one of the listed exceptions.
assert_throws( expected_symbol, [message] ) {block} assert_nothing_thrown( [message] ) {block}	True if the block throws (or doesn't) the expected_symbol.
assert_respond_to( object, method, [message] )	True if the object can respond to the given method.
assert_send( send_array, [message] )	True if the method sent to the object with the given arguments return true.
assert_operator( object1, operator, object2, [message] )	Compares the two objects with the given operator, passes if true

There are different testing frameworks available in Ruby<ref name = rubytool />:

• Riot
• MiniTest
• MiniSpec
• Capybara
• Jasmine

Back to top

Riot

This is a TDD based framework for testing Ruby applications. As opposed to some other popular testing frameworks in Ruby like Test::Unit, RSepc etc, Riot does not run the setup and teardown function before and after each test. This speeds up test execution quite a bit, but also changes how you write your tests. This speeds up test execution quite a bit, but also changes how you write your tests. In Riot, tests reside in contexts. Within these, a topic object is defined through a setup block. The actual assertions are then made with an asserts or denies block.

Code Example

Back to top

MiniTest

Overview

MiniTest is one of the more recent additions to the Ruby Testing frameworks. MiniTest was created to be a small, clean and fast replacement for the Test::Unit framework.

As previously mentioned, MiniTest is included in the standard library as of Ruby 1.9. (If you are running Ruby 1.8, you can run gem install minitest to get Mini::Test. Alternately, you can install the test-unit gem if you have Ruby 1.9 and want to get the Test::Unit functionality.) MiniTest provides both a Test Driven Development framework and a Behavior Driven Development Framework. Test cases in MiniTest::Unit are built similarly to those in Test::Unit - you create test cases with test methods containing assertions along with optional setup and teardown methods .

Because Mini::Test was based on familiar frameworks, using it is realtively intuitive. As creator, Ryan Davis says, "There is no magic". (View Ryan's presentation on Mini::Test at the Cascadia Ruby 2011 conference.) Mini::Test is simple and straightforward.

Functionality

Most of the assertions in Mini::Test::TestCase are the same as those in its predecessor with some new ones added. The most notable difference is in the negative assertions. In Test::Unit where you have an assert_not_something method, Mini::Test gave them a streamlined name of a refute_something method. (assert_not_raise and assert_not_throws are no longer available.) Mini::Test::Unit provides the following assertions:

assert	assert_block	assert_empty	refute	refute_empty
assert_equal	assert_in_delta	assert_in_epsilon	refute_equal	refute_in_delta	refute_in_epsilon
assert_includes	assert_instance_of	assert_kind_of	refute_includes	refute_instance_of	refute_kind_of
assert_match	assert_nil	assert_operator	refute_match	refute_nil	refute_operator
assert_respond_to	assert_same	assert_output	refute_respond_to	refute_same
assert_raises	assert_send	assert_silent	assert_throws

Additional Features

Aside from the API improvements, Mini::Test also provides some additional features such as test randomization. In unit testing, tests should run independent from each other (i.e. the outcome or resulting state(s) of one test should not affect another). By randomizing the order, Mini::Test prevents tests from becoming order-dependent. Should you need to repeat a particular order to test for such issues, Mini::Test provides the current seed as part of the output and gives you the option to run the test using this same seed.

Mini::Test also gives you the ability to skip tests that are not working correctly (for debug at a later time) as well as additional options for determining the performance of your test suite.

Code Example - Mini::Test:Unit

Back to top

MiniSpec

Overview

MiniSpec (the BDD component of Mini::Test) borrowed concepts from existing BDD frameworks, such as RSpec and Shoulda. Test cases are created using expectations instead of assertions.

MiniSpec contains the following expectations:

must_be	must_be_close_to	must_be_empty	wont_be	wont_be_close_to	wont_be_empty
must_be_instance_of	must_be_kind_of	must_be_nil	wont_be_instance_of	wont_be_kind_of	wont_be_nil
must_be_same_as	must_be_silent	must_be_within_delta	wont_be_same_as	wont_be_within_delta
must_be_within_epsilon	must_equal	must_include	wont_be_within_epsilon	wont_equal	wont_include
must_match	must_output	must_raise	wont_match
must_respond_to	must_send	must_throw	wont_respond_to

Code Example - MiniSpec

Back to top

Jasmine

Code Example

Back to top

Conclusion

Testing has always been an integral part of the Ruby landscape. This culture of testing has given rise to a variety of different testing frameworks. In this article we have reviewed only a few of the most popular offerings<ref name = rubytool /> representing both the Test Driven Development and Behavior Driven Development styles. The matrix below provides a summary of the tools highlighted on this page. Should none of these frameworks meet the Ruby programmer's needs, there are currently a number of others with more being developed every day. See the Ruby Toolbox for a list of additional Ruby testing options.

Framework Matrix

Framework	Website	Documentation	IDE Integration	Type	Ease of Use
Riot	RubyDoc	RubyDoc	Eclipse, RubyMine	TDD	Easy to learn; Easy to use
MiniTest	GitHub	RubyDoc	Eclipse	TDD/BDD	Easy to learn; Easy to use
MiniSpec	http://rspec.info/	http://rspec.info/	Eclipse, RubyMine	BDD	Slightly more difficult to learn due to BDD topology; Easy to use
Capybara	GitHub	RubyDoc	Eclipse, RubyMine	BDD	Easy to learn; Easy to use
Jasmine	Cucumber	Cucumber Wiki	RubyMine	BDD	Steeper learning curve; Complexity makes use a bit more challenging

References

<references> <ref name = dannorth> http://dannorth.net/introducing-bdd </ref> <ref name = rubytool> http://ruby-toolbox.com/categories/testing_frameworks.html </ref> <ref name = progruby> Programming Ruby: The Pragmatic Programmers' Guide, Second Ed. </ref> </references>