CSC/ECE 517 Fall 2012/ch1 1w18 as

Testing frameworks for Ruby

Introduction

Due to the rapid expansion in the use of Ruby through Ruby on Rails, the landscape of the Ruby testing framework has changed dramatically in recent years. Because of this, it provides a great story in the evolution of these testing frameworks.

Software Testing

Across all of the testing frameworks that Ruby has, most of them fall into two categories: Test Driven Development and Behavior Driven Development. These two types of development styles helped to drive the evolution of the Ruby testing frameworks we will be discussing.

Testing in Ruby

There are different testing frameworks available in Ruby<ref name = rubytool />:

• Riot
• MiniTest
• MiniSpec
• Capybara
• Jasmine

Since this will be a comparison across the different frameworks listed, using a common class to test against will provide us with a baseline for our comparisons. As such, we will use the following class implementing a bank account:

 class Account
   @balance
   @name
 
   attr_accessor :balance
   attr_accessor :name
 
   def initialize(amount)
     @balance = amount
   end
 
   def deposit(amount)
     @balance += amount
   end
 
   def addinterest(rate)
     @balance *= (1 + rate)
   end
 
   def withdrawal(amount)
     @balance -= amount
   end
 end

Back to top

Riot

• • • • • HAVE TO MODIFY FROM THIS POINT ********

Functionality

assert	assert_nil	assert_not_nil	assert_equal	assert_not_equal
assert_in_delta	assert_raise	assert_nothing_raised	assert_instance_of	assert_kind_of
assert_respond_to	assert_match	assert_no_match	assert_same	assert_not_same
assert_operator	assert_throws	assert_send	flunk

Code Example

 require "test/unit"
 require_relative("../Account.rb")
 
 class AccountTest < Test::Unit::TestCase
 
   def test_balance
     a = Account.new(100)
     assert_equal(100, a.balance())
   end
 
   def test_deposit 
     a = Account.new(100)
     assert_equal(200, a.deposit(100))
   end
   
   def test_withdrawal
     a = Account.new(100)
     assert_equal(50, a.withdrawal(50))
   end
   
   def test_name
     a = Account.new(100)
     a.name = "Checking"
     assert_not_nil(a.name())
   end
 
   def test_interest
     a = Account.new(100)
     assert_equal(150, a.addinterest(0.5))
   end
 
   def test_fail
     a = Account.new(100)
     assert_equal(200, a.balance())
   end
 end

From this we will get the output:

 Loaded suite AccountTest-ut
 Started
 ..F...
 Finished in 0.000782 seconds.
 
   1) Failure:
 test_fail(AccountTest) [AccountTest-ut.rb:40]:
 <200> expected but was
 <100>.
 
 6 tests, 6 assertions, 1 failures, 0 errors, 0 skips

Here we can see that in total all 6 test methods were run with one of them failing. The failing test method is indicated by name with the code line, expected output and actual output.

Back to top

MiniTest

Overview

MiniTest is one of the more recent additions to the Ruby Testing frameworks. MiniTest was created to be a small, clean and fast replacement for the Test::Unit framework.

As previously mentioned, MiniTest is included in the standard library as of Ruby 1.9. (If you are running Ruby 1.8, you can run gem install minitest to get Mini::Test. Alternately, you can install the test-unit gem if you have Ruby 1.9 and want to get the Test::Unit functionality.) MiniTest provides both a Test Driven Development framework and a Behavior Driven Development Framework. Test cases in MiniTest::Unit are built similarly to those in Test::Unit - you create test cases with test methods containing assertions along with optional setup and teardown methods .

Because Mini::Test was based on familiar frameworks, using it is realtively intuitive. As creator, Ryan Davis says, "There is no magic". (View Ryan's presentation on Mini::Test at the Cascadia Ruby 2011 conference.) Mini::Test is simple and straightforward.

Functionality

Most of the assertions in Mini::Test::TestCase are the same as those in its predecessor with some new ones added. The most notable difference is in the negative assertions. In Test::Unit where you have an assert_not_something method, Mini::Test gave them a streamlined name of a refute_something method. (assert_not_raise and assert_not_throws are no longer available.) Mini::Test::Unit provides the following assertions:

assert	assert_block	assert_empty	refute	refute_empty
assert_equal	assert_in_delta	assert_in_epsilon	refute_equal	refute_in_delta	refute_in_epsilon
assert_includes	assert_instance_of	assert_kind_of	refute_includes	refute_instance_of	refute_kind_of
assert_match	assert_nil	assert_operator	refute_match	refute_nil	refute_operator
assert_respond_to	assert_same	assert_output	refute_respond_to	refute_same
assert_raises	assert_send	assert_silent	assert_throws

Additional Features

Aside from the API improvements, Mini::Test also provides some additional features such as test randomization. In unit testing, tests should run independent from each other (i.e. the outcome or resulting state(s) of one test should not affect another). By randomizing the order, Mini::Test prevents tests from becoming order-dependent. Should you need to repeat a particular order to test for such issues, Mini::Test provides the current seed as part of the output and gives you the option to run the test using this same seed.

Mini::Test also gives you the ability to skip tests that are not working correctly (for debug at a later time) as well as additional options for determining the performance of your test suite.

Code Example - Mini::Test:Unit

Back to top

MiniSpec

Overview

 MiniSpec (the BDD component of Mini::Test) borrowed concepts from existing BDD frameworks, such as RSpec and Shoulda.  Test cases are created using expectations instead of assertions.

MiniSpec contains the following expectations:

must_be	must_be_close_to	must_be_empty	wont_be	wont_be_close_to	wont_be_empty
must_be_instance_of	must_be_kind_of	must_be_nil	wont_be_instance_of	wont_be_kind_of	wont_be_nil
must_be_same_as	must_be_silent	must_be_within_delta	wont_be_same_as	wont_be_within_delta
must_be_within_epsilon	must_equal	must_include	wont_be_within_epsilon	wont_equal	wont_include
must_match	must_output	must_raise	wont_match
must_respond_to	must_send	must_throw	wont_respond_to

Code Example - MiniSpec

Back to top

Jasmine

Code Example

Back to top

Conclusion

Testing has always been an integral part of the Ruby landscape. This culture of testing has given rise to a variety of different testing frameworks. In this article we have reviewed only a few of the most popular offerings<ref name = rubytool /> representing both the Test Driven Development and Behavior Driven Development styles. The matrix below provides a summary of the tools highlighted on this page. Should none of these frameworks meet the Ruby programmer's needs, there are currently a number of others with more being developed every day. See the Ruby Toolbox for a list of additional Ruby testing options.

Framework Matrix

Framework	Website	Documentation	IDE Integration	Type	Ease of Use
Riot	RubyDoc	RubyDoc	Eclipse, RubyMine	TDD	Easy to learn; Easy to use
MiniTest	GitHub	RubyDoc	Eclipse	TDD/BDD	Easy to learn; Easy to use
MiniSpec	http://rspec.info/	http://rspec.info/	Eclipse, RubyMine	BDD	Slightly more difficult to learn due to BDD topology; Easy to use
Capybara	GitHub	RubyDoc	Eclipse, RubyMine	BDD	Easy to learn; Easy to use
Jasmine	Cucumber	Cucumber Wiki	RubyMine	BDD	Steeper learning curve; Complexity makes use a bit more challenging

References

<references> <ref name = dannorth> http://dannorth.net/introducing-bdd </ref> <ref name = rubytool> http://ruby-toolbox.com/categories/testing_frameworks.html </ref> <ref name = progruby> Programming Ruby: The Pragmatic Programmers' Guide, Second Ed. </ref> </references>