|
|
| Line 1: |
Line 1: |
| ='''Testing Frameworks for Ruby'''=
| |
|
| |
|
| This page serves as a knowledge source for understanding the different Testing Frameworks available for [http://en.wikipedia.org/wiki/Ruby_(programming_language) Ruby].<br>
| |
|
| |
| __TOC__
| |
| = Introduction =
| |
| No code is perfect from the word go! Testing plays an important role in System Development Life Cycle. During testing, one follows a taxonomic procedure to uncover defects at various stages. A test framework provides various Test Types, Test Phases, Test Models, Test Metrics and acts as a guideline as to how to perform effective testing <ref> http://www.scribd.com/doc/15909730/Software-Testing-Framework </ref>. Various testing tools are available for [http://en.wikipedia.org/wiki/Ruby Ruby]language, each having some unique features over others. Here is a brief introduction and feature comparisons of popular testing frameworks.
| |
|
| |
| == Testing Approaches ==
| |
| [[File:Capture5.JPG|thumb|right|200px|flow of test driven developmental approach]]
| |
| Before delving into testing for Ruby, in general, these are the followed approaches in industry today
| |
|
| |
| === Test Driven Development ===
| |
|
| |
| This strategy, (abbreviated as TDD) <ref> http://en.wikipedia.org/wiki/Test-driven_development </ref>, though cumbersome due to its methodology develops efficient code. First a unit test is written for a function, even before the code for that function is developed. Based on this test minimal code is developed to ensure the test succeeds; if not the code is modified until test run successfully. In this iterative approach, effort is made to ensure flawless code is developed.
| |
|
| |
| === Behavior Driven Development ===
| |
| BDD <ref> http://en.wikipedia.org/wiki/Behavior_Driven_Development </ref> as its popularly know, upholds the traditional iterative workflow of Test Driven Development, but it focuses on defining behaviors that is easy to understand to naive people (with no programming background) by writing tests in natural language. This approach focuses more on looking at code development from a behavioral abstraction. For example testing the code for a recipe belonging to a category of a cookbook, in BDD would be something like 'A recipe can't be without a category'
| |
|
| |
| = Unit Testing =
| |
| [http://en.wikipedia.org/wiki/Unit_testing Unit Testing] is a method by which one can isolate and test a unit functionality of the program, typically individual methods during and long after the code is written.
| |
| == Test::Unit ==
| |
| [[File:Capture4.JPG|thumb|right|400px|screen capture of TestUnit console in [http://en.wikipedia.org/wiki/RubyMine RubyMine IDE]]]
| |
|
| |
| The in-built, ready to use unit testing mechanism for Ruby is called [http://www.ensta-paristech.fr/~diam/ruby/online/ruby-doc-stdlib/libdoc/test/unit/rdoc/classes/Test/Unit.html Test::Unit].It belongs to the XUnit family unit testing framework. It has a setup method for initialization, a teardown method for cleanup and the actual test methods itself. The tests are bundled separately in a test class in the code it is aimed to test.
| |
| === Features ===
| |
| This salient features of the Test::Unit Framework are:-
| |
| ==== Assertions ====
| |
| One can use Test::Unit to make [http://en.wikipedia.org/wiki/Assertion_(computing) assertions]. The test is successful if the assertion is true and fails if the assertion is false. All the assertion methods provided by test::unit can be found at [http://www.ruby-doc.org/stdlib/libdoc/test/unit/rdoc/classes/Test/Unit/Assertions.html Test::Unit::Assertions].
| |
|
| |
| {|border=1 cellspacing=0 cellpadding=5
| |
| | assert( boolean, [message] )
| |
| | True if ''boolean''
| |
| |-
| |
| | assert_equal( expected, actual, [message] )<br>assert_not_equal( expected, actual, [message] )
| |
| | True if ''expected == actual''
| |
| |-
| |
| | assert_raise( Exception,... ) {block}<br>assert_nothing_raised( Exception,...) {block}
| |
| | True if the block raises (or doesn't) one of the listed exceptions.
| |
| |-
| |
| |}
| |
|
| |
| ==== Test-Fixtures ====
| |
| Using a test fixture, one can initialize (and cleanup) the common set of data for two or more tests hence eliminating unnecessary duplication. Fixtures are written in the setup() and teardown() methods.
| |
| ==== Test-Suite ====
| |
| As unit tests increase in number for a given project, it becomes tough running them one at a time. It is hence useful to combine a bunch of related test cases and run them as batch. Test::Unit provides a class called TestSuite for this purpose<ref>http://www.ensta-paristech.fr/~diam/ruby/online/ruby-doc-stdlib/libdoc/test/unit/rdoc/classes/Test/Unit/TestSuite.html</ref>.
| |
|
| |
| === Example Test ===
| |
| The <i>Test::Unit::TestCase</i> class is the main class and the <i>BinarySearchTest</i> is the subclass, it overrides <i>setup</i> and <i>teardown</i> methods. All test methods start with 'test_' prefix thus differentiating helper methods from main methods. The Test::Unit::TestCase class makes the test methods into tests, wrapping them into a suite and running the individual tests. The results are collected into <i>Test::Unit::TestResult</i> object.
| |
|
| |
| require 'test/unit'
| |
| require 'english_french'
| |
| class EnglishFrenchTest < Test::Unit::TestCase
| |
| include EnglishFrenchTranslator
| |
| def test_simple_word
| |
| s = translate("house")
| |
| assert_equal("mason", s)
| |
| end
| |
| def test_word_beginning_with_vowel
| |
| s = translate("apple")
| |
| assert_equal("pomme", s)
| |
| end
| |
| def test_two_consonant_word
| |
| s = translate("stupid")
| |
| assert_equal("stupides", s)
| |
| end
| |
| end
| |
|
| |
| The tests are run by running the file test_english_french.rb.Sample results for the above test is shown below.
| |
|
| |
| $ ruby test_english_french.rb
| |
| Loaded suite test_english_french
| |
| Started
| |
| FFF
| |
| Finished in 0.01091 seconds.
| |
|
| |
| 3 tests, 3 assertions, 0 failures, 0 errors
| |
|
| |
| == Shoulda ==
| |
| Shoulda is a library rather than a framework which gives the flexibility to write better and easy to understand tests for the Ruby application being tested. Shoulda compliments Test::Unit and RSpec in the sense that it can be used within both of them. Using Shoulda one can provide context to tests so that, tests can be grouped according to a specific feature or scenario.
| |
| ===Features===
| |
|
| |
| [[File:Capture1.JPG|thumb|left|170px|components of Shoulda]]
| |
|
| |
| Shoulda consist of matchers, test-helpers and assertions<ref>http://rubydoc.info/gems/shoulda/2.11.3/frames</ref> -
| |
|
| |
| '''Matchers'''<br>
| |
| Matchers (or more popularly simply [http://en.wikipedia.org/wiki/Macro_(computer_science)#Text_substitution_macros macros]) are Test::Unit- and RSpec-compatible one-liners that test common Rails functionality. The task of developing cumbersome, complex and often erroneous code is made simple with the help of matchers.
| |
|
| |
| '''Helpers'''<br>
| |
| Helpers like ''#context'' and ''#should'' not only gives ''RSpec'' like test blocks in ''Test::Unit'' but also in addition to these, one gets nested contexts and a much more readable syntax.
| |
|
| |
| '''Assertions'''<br>
| |
| Many common Rails testing idioms have been distilled into a set of useful assertions.
| |
|
| |
| <br><br><br>
| |
|
| |
| ===Installation===
| |
| Shoulda for can be installed for [http://weblog.rubyonrails.org/2010/8/29/rails-3-0-it-s-done| Rails 3.0] version by including the following in the [http://gembundler.com/gemfile.html Gemfile]
| |
| <code>
| |
| group :test do
| |
| gem "shoulda"
| |
| gem "rspec-rails", "2.0.0.beta.12"
| |
| end
| |
| </code><br>
| |
| ===Example===
| |
|
| |
| As far as usage of shoulda is concerned, one can write matchers as follows-
| |
|
| |
| Suppose one is testing for the class ''Post'' on a blog, heres how it is achieved using shoulda::Macthers
| |
|
| |
| <code>
| |
| class Post_Test < Test::Unit::TestCase
| |
| should belong_to(:user)
| |
| should have_many(:tags).through(:taggings)<br>
| |
| should validate_uniqueness_of(:title)
| |
| should validate_presence_of(:title)
| |
| should validate_numericality_of(:user_id)
| |
| end
| |
| </code>
| |
|
| |
| Similarly the test for class ''Users'' can be
| |
|
| |
| <code>
| |
| class Users_Test < Test::Unit::TestCase
| |
| should have_many(:posts)<br> #checking validity of e-mail id<br>
| |
| should_not allow_value("blah").for(:email)
| |
| should allow_value("a@b.com").for(:email)
| |
| should_not allow_mass_assignment_of(:password)
| |
| end
| |
| </code>
| |
|
| |
| As can be seen, writing test case for TDD becomes much more elegant and less error prone.
| |
|
| |
| Similarly shoulda::helpers can be written as shown -
| |
|
| |
| Suppose one wants to check the validity of getter method presence for an object, simply an should_have helper can be written like
| |
|
| |
| <code>
| |
| context "the create action" do
| |
| setup do
| |
| @attributes = Posts.attributes_for(:item)
| |
| post :create, :item => @attributes
| |
| end<br>
| |
| should_create_row Item do |item|
| |
| assert_has_attributes @attributes, item #''assert_has_attributes'' - simply a helper method that makes sure the object has getter methods and values corresponding to the key/value pairs in the hash.
| |
| end
| |
| end
| |
| </code>
| |
|
| |
| The most interesting part of shoulda is the assertions which make writing test code efficient. An example of shoulda::Assertion can be seen as below
| |
|
| |
| <code>
| |
| assert_same_elements([:a, :b, :c], [:c, :a, :b]) #checks whether array has same entries
| |
| assert_contains(['a', '1'], /\d/) #checks for presence of a digit
| |
| assert_contains(['a', '1'], 'a') #checks for presence of letter ''a''
| |
| </code>
| |
| == RSpec ==
| |
| [[File:Capture3.JPG|thumb|left|400px|flow of logic in R-Spec]]
| |
|
| |
| RSpec is a Behavioral Driven development tool,created by Dave Astels and Steven Bake <ref>http://blog.emson.co.uk/2008/06/understanding-rspec-stories-a-tutorial</ref> aimed as test driven development.
| |
|
| |
| RSpec is merging of two projects into one -
| |
|
| |
| <i><b>application level</b></i> behaviour described by a <b><i>Story Framework</i></b><br>
| |
| <i><b>object level</b></i> behaviour described by a <b><i>Spec Framework</i></b><br><br>
| |
|
| |
| The primary aim of RSpec is to drift the focus from unit testing to behavioral testing, i.e. to check if the desired functionality is achieved. It is also used to check for redundancy in code and helps refactoring our code without having to re-write every test.
| |
| ===Installation===
| |
| In order to install RSpec, open a command shell, go to /bin folder in Ruby directory and type
| |
| > gem install rspec
| |
|
| |
| ===Features===
| |
| ==== RSpec Story ====
| |
| A Story describes the functionality of a specific software feature, and it describes it in a way that is easy to understand from the point of view of a client. In fact a story can be thought of as a conversation between a client and a programmer over some feature of the software. The key points of a story are:- <i>Title, Narrative, Scenario, Givens, Details</i><sup>[http://blog.emson.co.uk/2008/06/understanding-rspec-stories-a-tutorial/]</sup>
| |
| ==== Message Expectations ====
| |
| A message expectation (a.k.a. mock method)<ref>http://rspec.info</ref> is an expectation that an object should receive a specific message during the execution of an example. This is similar to "Assertions" in test::unit.
| |
| ==== Group ====
| |
| More than one executable example can be grouped into one file.
| |
| ==== Spec File====
| |
| A spec file is a file that contains one or more group examples.
| |
| ==== describe() and it() methods ====
| |
| The describe() method takes an arbitrary number of arguments and returns a sub-class of Spec::Example::ExampleGroup. The it() method takes a single String, an optional Hash and an optional block.
| |
|
| |
| === Example Test ===
| |
| The below code is an example for testing using rspec. The spec file, ruby file and the output are shown. It can be seen that code can be added to the spec file for different test conditions and expectations.
| |
|
| |
| # archery_spec.rb
| |
| require 'archery'
| |
| describe Archery, "#score" do
| |
| it "returns 0 for all mock games" do
| |
| archery = Archery.new
| |
| 20.times { archery.hit(0) }
| |
| archery.score.should == 0
| |
| end
| |
| end
| |
|
| |
| # archery.rb
| |
| class Archery
| |
| def hit(arrows)
| |
| end
| |
| def score
| |
| 0
| |
| end
| |
| end
| |
|
| |
| $ spec archery_spec.rb --format nested
| |
| Archery#score
| |
| returns 0 for all mock games
| |
| Finished in 0.006535 seconds
| |
| 1 example, 0 failures
| |
|
| |
|
| |
| == Cucumber ==
| |
| Cucumber is a testing tool that follows the BDD approach of testing. Cucumber written itself in Ruby programming language allows the execution of feature documentation written in natural language often termed as [http://en.wikipedia.org/wiki/User_story user stories]
| |
| === Features of Cucumber ===
| |
|
| |
| ==== Terminology of Cucumber====
| |
| There are few taxonomy terms related to Cucumber <ref>http://cuke4ninja.com/sec_cucumber_jargon.html</ref> as follows-
| |
| [[File:Capture.JPG|thumb|left|200px|major components of Cucumber]]
| |
| '''Stakeholder''' - A person who gets some value out of the system like an administrator<br>'''Feature''' - A feature is a piece of system functionality that delivers value to one or more stakeholders<br>'''User story''' - It is a rough description of scope for future work used as planning tools. Most common format for the stories are - ''"in order to..."'',''"as a..."'',''"I want"''<br>'''Feature file''' - It describes a feature or a part of feature with illustrative example of expected results<br>'''Key Example''' - Each feature is illustrated with Key Examples which show what are expected in a given test case<br>'''Scenario''' - Scenario captures one key example for a feature file. It represents how stakeholder gets some value from that system. Example of good scenarios for checking login module includes ''user not signed up'', ''password has expired''<br>'''Step''' - Steps are domain language phrases which can be combined to write scenarios. They combine a GWT directive with a regular expression to evaluate something and add a simple ruby code which tells the step what should be done further.
| |
| <br><br><br><br>
| |
| ==== Pattern ====
| |
| Cucumber follows a GWT (Given-When-Then) pattern for developing test cases. In the scenarios written for Cucumber, one states what a ''given'' scenario is, ''when'' one is presented with information and ''then'' what should happen so that the logic of information can be validated.
| |
|
| |
| ===Example===
| |
| Cucumber is a high-level testing tool that defines the feature specs. An typical [[http://www.ultrasaurus.com/sarahblog/2008/12/rails-2-day-3-behavior-driven-development/#spec example]] for a Cucumber Scenario is shown below -
| |
| First one need to install the Cucumber gem on Ruby using this simple command
| |
| <code>
| |
| $gem install cucumber-rails
| |
| Successfully installed cucumber-rails-0.3.0
| |
| 1 gem installed
| |
| Installing ri documentation for cucumber-rails-0.3.0...
| |
| Installing RDoc documentation for cucumber-rails-0.3.0...
| |
| </code>
| |
| When installed, a basic Cucumber test is generated using [http://en.wikipedia.org/wiki/Generator_(computer_programming) generator] script
| |
| <code>
| |
| $script/generate rspec
| |
| $script/generate cucumber
| |
| force config/database.yml
| |
| create config/cucumber.yml
| |
| create config/environments/cucumber.rb
| |
| create script/cucumber
| |
| create features/step_definitions
| |
| create features/step_definitions/web_steps.rb
| |
| create features/support
| |
| create features/support/paths.rb
| |
| create features/support/env.rb
| |
| exists lib/tasks
| |
| create lib/tasks/cucumber.rake
| |
| </code>
| |
| A feature file is next created with name ''feature_name.featrue'' . the Feature is described in the Feature: directive, followed by the story. The story is written in the format shown in the code: <font color="green">''As a <role> , I want <feature> , So that <business Value>.</font>''
| |
| <code>
| |
| $ nano tasklist.feature<br>
| |
| Feature: Tasks<br>
| |
| In order to keep track of tasks
| |
| People should be able to
| |
| Create a list of tasks
| |
| </code>
| |
| Each feature can have multiple scenario. One such scenario for the login feature is as shown below
| |
| <code>
| |
| Scenario: List Tasks
| |
| Given that I have created a task "task 1"
| |
| When I go to the tasks page
| |
| Then I should see "task 1"
| |
| </code>
| |
| When a scenario is ready following command is issued
| |
| <code>
| |
| $script/cucumber features/tasklist.feature
| |
| 0 scenarios
| |
| 0 steps
| |
| </code>
| |
| Steps can be added to scenarios, which are combination of a GWT directive, a [http://en.wikipedia.org/wiki/Regular_expression regular expression] and a block of ruby code which states what this step does
| |
| <code>
| |
| /features/step_definitions$nano tasklist_steps.rb<br>
| |
| Given /^that I have created a task "(.*)"$/ do |desc|
| |
| Task.create!(:description => desc)
| |
| end
| |
| When /^I go to the tasks page$/ do
| |
| visit "/tasks"
| |
| end
| |
| </code>
| |
|
| |
| = Comparison =
| |
| Now that the major frameworks have been explained briefly, here is a comparison among them based on few criteria/metrics
| |
| <br>
| |
| {|class="wikitable sortable" style="font-size: 95%; text-align: center; width: auto;; "cellspacing="0"; border="1"
| |
| !style="width:20%"|Metric
| |
| !style="width:20%"|Test::Unit
| |
| !style="width:20%"|Shoulda
| |
| !style="width:20%"|RSpec
| |
| !style="width:20%"|Cucumber
| |
| |-
| |
| | <p align="justify"> Framework </p>
| |
| | <p align="justify">Test Driven Development framework.</p>
| |
| | <p align="justify">It is an extension to Test::Unit. It has more capabilities and simpler readable syntax compared to Test::Unit.</p>
| |
| | <p align="justify">Behavior Driven Development framework.</p>
| |
| | <p align="justify">Behavior Driven Development framework.</p>
| |
| |-
| |
| | <p align="justify"> Website </p>
| |
| | <p align="justify"> [http://test-unit.rubyforge.org/ Test::Unit] </p>
| |
| | <p align="justify"> [https://github.com/thoughtbot/shoulda Github -Shoulda] </p>
| |
| | <p align="justify"> [http://rspec.info/ RSpec] </p>
| |
| | <p align="justify"> [http://cukes.info/ Cucumber] </p>
| |
| |-
| |
| | <p align="justify"> Base Library </p>
| |
| | <p align="justify"> Yes </p>
| |
| | <p align="justify"> No </p>
| |
| | <p align="justify"> No </p>
| |
| | <p align="justify"> No </p>
| |
| |-
| |
| | <p align="justify"> Allows nested setup</p>
| |
| | <p align="justify"> No </p>
| |
| | <p align="justify"> Yes </p>
| |
| | <p align="justify"> Yes </p>
| |
| | <p align="justify"> Yes </p>
| |
| |-
| |
| | <p align="justify">Opaqueness/Expressiveness </p>
| |
| | <p align="justify"> Opaque to a certain extent, low level coding required </p>
| |
| | <p align="justify"> More descriptive </p>
| |
| | <p align="justify"> More expressive compared to Unit::test and shoulda. </p>
| |
| | <p align="justify"> More expressive compared to Unit::test and shoulda </p>
| |
| |-
| |
| | <p align="justify"> Run Times </p>
| |
| | <p align="justify"> Faster than RSpec because it has fewer features and hence lesser code to run </p>
| |
| | <p align="justify"> Faster than RSpec because it has fewer features and hence lesser code to run </p>
| |
| | <p align="justify"> More features but comes at a cost, run time slower than shoulda and test::unit. </p>
| |
| | <p align="justify"> More features but comes at a cost, run time slower than shoulda and test::unit. </p>
| |
| |-
| |
| | <p align="justify"> General Use </p>
| |
| | <p align="justify"> Unit testing </p>
| |
| | <p align="justify"> Unit testing especially on the Rails framework <sup>[http://www.linuxjournal.com/magazine/forge-testing-rails-applications-shoulda]</sup> </p>
| |
| | <p align="justify"> Unit testing </p>
| |
| | <p align="justify"> Integration testing </p>
| |
| |}
| |
|
| |
| = Conclusion =
| |
| = References =
| |
| <references/>
| |