CSC/ECE 517 Fall 2012/ch1 1w3 pl

Unit-Testing Frameworks for Ruby: Cucumber

This wiki-page serves as a knowledge source for understanding Unit-Testing Frameworks available for Ruby particularly Cucumber .

Introduction

Unit testing is a software development process for testing individual units of source code independently for proper operation <ref>Unit Testing</ref>. An unit testing framework helps the process of unit testing by automating the test cases which are written to ensure the correctness of the system. They are often third-party products that are not offered as part of the suite.Ruby provides a framework in its standard library for setting up, organizing, and running tests called Test::Unit <ref>Ruby Unit Testing Framework</ref>. Other testing frameworks available for ruby are Shoulda, RSpec and Cucumber.

Cucumber

Cucumber is one of the latest unit test frameworks to have come for Ruby as part of the RSpec family of tools. Cucumber itself is written in the Ruby programming language and adheres to behaviour driven development(BDD). Behaviour Driven Development is an Agile Development process that comprises aspects of Acceptance Test Driven Planning , Domain Driven Design and Test Driven Development (TDD). Cucumber is designed specifically to ensure that the acceptance tests can easily be read and written by anyone.

Behaviour Driven Development(BDD)

Behaviour Driven Development<ref>BDD</ref> preserves the basic iterative (fail-pass) workflow of TDD, but stresses on specifying behaviors that are understandable to people (say from non programming background). In this approach we write tests in a natural language that even a non-programmer can understand.

Acceptance Test

Acceptance Test<ref>Acceptance Test</ref><ref>Cucumber Book</ref> Drive Planning is a practice that stresses on the importance of identifying the functions and requirements of the software beforehand so that we exactly get to know when our development phase is over. This practice results in fewer bugs , shorter delivery times and greater customer satisfaction.

Cucumber Acceptance Test

     Feature: Sign up
     Sign up should be quick and friendly.

     Scenario: Successful sign up
     New users should get a confirmation email and be greeted personally by the site once signed in.
     Given I have chosen to sign up
     When I sign up with valid details
     Then I should receive a confirmation email
     And I should see a personalized greeting message

     Scenario: Duplicate email
     Where someone tries to create an account for an email address that already exists.
     Given I have chosen to sign up
     But I enter an email address that has already registered
     Then I should be told that the email is already registered
     And I should be offered the option to recover my password

Acceptance tests written in this style are more than just tests; they are executable specifications. As of January 2012, Cucumber was the second most popular testing framework after RSpec for Ruby

How Cucumber works

Cucumber is a command-line tool. It reads in specifications from plain-language text files called features, examines them for scenarios to test, and runs the scenarios against your system. The feature files should be written according to specific syntax rules called Gherkin. Along with the features, you give Cucumber a set of step definitions, which map the business-readable language of each step into Ruby code to carry out whatever action is being described by the step.

Cucumber takes customer understandable user stories as input and does integration and acceptance tests on the system with those as its input. Cucumber acts as the bridge between the customer and the developer. User stories are customer understandable but at the same time can be used to create test cases from it. This does pose some restrictions on the user stories that are written.

Feature

A Feature is a high level statement of the what the test case actually does and there is only one feature per user story. For every feature that is supported by the system , a separate feature file has to be written and included in the features directory.The feature files usually end with .feature extension.

Scenario

A Scenario consists of one or more steps which describe about the various actions performed by the user with respect to this particular feature. There can be one or more scenarios per feature in a user story. A feature usually contains 3 to 8 steps in it. The Steps of the scenario always begin with one of the below mentioned key words :

1. Given : Steps that begin with given represent the state of the world before an event.(preconditions)
2. When : Steps that represent the event.
3. Then : Steps that represent the expected outcome.
4. And and But : Steps that extend the previous step.

Step and Step Definitions

When testing with Cucumber , The user stories are all kept in one set of files called Steps.The Steps are compared with Step Definitions which are files written in ruby which gives the rules based on which the steps are executed. Step definitions are analogous to method definitions and steps of scenarios are analogous to method calls.Cucumber uses regular expressions to match the English phrases in the steps of scenarios to step definitions.

When the cucumber test framework is run , The steps present in the feature file can be classified as follows based on the result of regex matching with the step definitions.

Successful steps

When Cucumber finds a matching Step Definition it will execute it. If the block in the step definition doesn’t raise an Exception, the step is marked as successful (green). What you return from a Step Definition has no significance what so ever.

Undefined steps

When Cucumber can’t find a matching Step Definition the step gets marked as yellow, and all subsequent steps in the scenario are skipped. If you use --strict this will cause Cucumber to exit with 1.

Pending steps

When a Step Definition’s Proc invokes the #pending method, the step is marked as yellow (as with undefined ones), reminding you that you have work to do. If you use --strict this will cause Cucumber to exit with 1.

Failed steps

When a Step Definition’s Proc is executed and raises an error, the step is marked as red. What you return from a Step Definition has no significance what so ever. Returning nil or false will not cause a step definition to fail.

Cucumber tries to run the steps based on the definitions and thus colors steps based on the execution of the steps. The colors for the steps assigned are as follows :

1. Green : for passing.
2. Yellow : for not yet implemented.
3. Red : for failing.
4. Blue : Once a step fails all the successive steps are marked in blue.

Cucumber Testing Stack

/* Some Content */

Cucumber Testing Stack

Advantages of Cucumber

• Specifications can be written in more than forty different spoken languages.
• Cucumber helps build bridges between the technical and non-technical members of a software team.
• Cucumber tests can be easily read and written by business stakeholders.
• Writing Cucumber tests are quick , easy , efficient and understandable.

Cucumber Installation

To install cucumber we should make sure that the Environment path is set to “..../Ruby/bin”. Then open the command prompt and run

    gem install cucumber 

Once cucumber gem is installed then we would be able to check whether it is installed by issuing the following command:

    cucumber --version

this should give a version number like “1.2.1”

Cucumber uses Rspec for assertions and hence we need Rspec installed to use Cucumber. Use the following command to install rspec

    gem install rspec

Cucumber depends on a particular gherkin version which will be downloaded automatically when we install Cucumber. Suppose we have a later version of gherkin installed in our system then RubyGems get confused and hence the required version of gherkin will not be downloaded and used. In order to resolve this issue we might have to give this command :

    gem install gherkin --version 2.2.9

Testing a Sample Ruby Application using Cucumber

<ref>Sample Cucumber Application</ref>

Step 1 : Create a new directory

    mkdir HelloCucumber 

Step 2 : Install Cucumber and Rspec

    gem install cucumber 
    gem instal rspec

Step 3 : Create a directory features and create a file basic.feature in the features directory mkdir features

     mkdir features

Content of features/basic.feature

    Feature: Hello World Feature

    Scenario: Hello World Scenario
    Given The Action is Hello
    When The Subject is World
    Then The Greeting is Hello, World

Step 4: Create step_definitions directory inside features directory. Create basic_step.rb file inside step_difinitions.

     mkdir features/step_definitions

Content of features/step_definitions/basic_step.rb

     require 'rspec/expectations'

     Given /The Action is ([A-z]*)/ do |action|
        @action = action
     end

     When /The Subject is ([A-z]*)/ do |subject|
       @subject = subject
     end

     Then /The Greeting is (.*)/ do |greeting|
       greeting.should == "#{@action}, #{@subject}"
     end

Cucumber uses regular expression after the keywords Given, When and Then to map lines in a feature file to step definitions.

Step 5: Running cucumber command you will see the following output

    Feature: Hello World Feature

    Scenario: Hello World Scenario    # features\basic.feature:3
    Given The Action is Hello         # features/step_definitions/basic_steps.rb:3
    When The Subject is World         # features/step_definitions/basic_steps.rb:7
    Then The Greeting is Hello, World # features/step_definitions/basic_steps.rb:11

    
    1 scenario (1 passed)
    3 steps (3 passed)
    0m0.016s
    Step 6 : Modify the basic.features file to include 2 scenario

Step 6 : Modify the basic.features file to include 2 scenarios

    Feature: Hello World Feature

    Scenario: Hello World Scenario
    Given The Action is Hello
    When The Subject is World
    Then The Greeting is Hello, World

    Scenario: Bye World Scenario
    Given The Action is Bye
    When The Subject is World
    Then The Greeting is Hello, World

Step 7 : Running the cucumber command you will see the following output

   Scenario: Hello World Scenario      # features\basic.feature:3
   Given The Action is Hello         # features/step_definitions/basic_steps.rb:3
   When The Subject is World         # features/step_definitions/basic_steps.rb:7
   Then The Greeting is Hello, World # features/step_definitions/basic_steps.rb:11

   Scenario: Bye World Scenario       # features\basic.feature:8
   Given The Action is Bye          # features/step_definitions/basic_steps.rb:3
   When The Subject is World        # features/step_definitions/basic_steps.rb:7
   Then The Greeting is Hello, World # features/step_definitions/basic_steps.rb:11
     expected: "Bye, World"
          got: "Hello, World" (using ==) (RSpec::Expectations::ExpectationNotMetError)
     ./features/step_definitions/basic_steps.rb:12:in `/The Greeting is (.*)/'
     features\basic.feature:11:in `Then The Greeting is Hello, World'

   Failing Scenarios:
   cucumber features\basic.feature:8 # Scenario: Bye World Scenario

   2 scenarios (1 failed, 1 passed)
   6 steps (1 failed, 5 passed)
   0m0.033s

The Bye Scenario fails since the expected output is “Hello, World” but the resulting output is “Bye,World”.

Testing a Sample Rails Application using Cucumber

Cucumber easily integrates with Rails and hence can be used with Ruby applications which run on Rails. But there are a few steps which have to be followed before integrating cucumber with Rails. Let us consider we create a new applications using the following command :

   rails new cucumber_app

and the new applications with its folders get created.

we would have to install the gems required for running Cucumber with Rails and there are couple of gems that are required and they can be installed with the following command :

   gem install cucumber
   gem install cucumber cucumber-rails 

We would have to then give “bundle install” command to install these new gems. We would then have to update the Gemfile to include these gems in the test group since they will be needed only in the test environment. There are a few other gems which are required for cucumber namely database_cleaner , rspec-rails and webrat.

   group :test do
      gem 'database_cleaner'
      gem 'cucumber'
      gem "rspec-rails", "~> 2.0", :require => false
      gem "cucumber-rails", "1.0.6"
      gem 'webrat'
   end

Cucumber required files can then be created by giving the following command. The output below the command shows the folders and files that get created for the cucumber test framework.

   rails generate cucumber:install

    identical  script/cucumber
          chmod  script/cucumber
          exist  features/step_definitions
    identical  features/step_definitions/web_steps.rb
          exist  features/support
    identical  features/support/paths.rb
    identical  features/support/selectors.rb
    identical  features/support/env.rb
          exist  lib/tasks
    identical  lib/tasks/cucumber.rake

The Step definitions folder contains ruby files which test the steps which are mentioned in feature file of the cucumber.
The paths.rb file is created in the support folder will contain the rules based on which the step definitions will be compared to the steps and matched. In case we come across any steps whose execution fails while running cucumber we would have to update the paths.rb file to resolve the issue.
env.rb is only loaded when the application is first started up and it sets up the environment required by Cucumber to execute.

   rails generate model article title:string content:text 
   rake db:migrate
   rake db:test:clone
   rails generate controller articles index 

The above four commands generate models with the name article and with two fields title and content and the next two commands migrate the db to production and the last commands creates controller for article and index objects.
We have to go the home directory of our rails application and then run Cucumber with the following command.The result would then be show to the user on how many and which steps succeeded and failed.

   => cucumber

Capybara

We need a tool that acts out the user stories that are part of the feature that is to be tested with Cucumber. Capybara can be used as that tool. Capybara is an integration testing tool for rack based web based applications. It simulates how a user would interact with a website. It can simulate the actions of a web browser and hence can interact with the application to receive web pages. It can also parse the HTML content and submits forms as the user would. One of the restrictions that Capybara has is that it cannot handle JavaScript. There are other tools like webdriver which can handle JavaScript but it runs a lot slower compared to Capybara.

Comparison of Unit Test Frameworks : Rspec, Shoulda and Cucumber

Rspec	Shoulda	Cucumber
RSpec is based on the Behaviour driven development model provided by Ruby. It has its origins in Test Driven Development and Domain Driven Design.	Shoulda is based on the Test:Unit which is another Ruby Test Framework. It is based on Test Driven Development.	Cucumber is a combination of Test Driven Development and Behaviour driven development.
RSpec provides a Domain Specific Language that can express code functionality and behaviour.	Shoulda provides macros, helpers and matchers.	Cucumber allows the test cases to be written in native language of the system
RSpec too allows nested describe structure.	Shoulda has nested contexts which helps create different environment for different set of tests.	Cucumber is similar to RSpec also provides nested structures
RSpec is expressive when it comes to understanding functionalities since they were designed have the human readability in mind.	Shoulda has more descriptive names compared to its predecessor Test::Unit but its less expressive.	Cucumber test are even more descriptive and allows business stakeholder to write and read them.

Reference

<references/>