CSC/ECE 517 Fall 2012/ch1 1w3 pl

Unit-Testing Frameworks for Ruby: Cucumber

This wiki-page serves as a knowledge source for understanding Unit-Testing Frameworks available for Ruby particularly Cucumber .

Introduction

Unit testing <ref>Unit Testing in Ruby</ref> is a software development process for testing individual units of source code independently for proper operation <ref>Unit Testing</ref>. An unit testing framework helps the process of unit testing by automating the test cases written to ensure the correctness of the system. They are often third-party products that are not offered as part of the suite. Ruby provides a framework in its standard library for setting up, organizing, and running tests called Test::Unit <ref>Ruby Unit Testing Framework</ref>. Other testing frameworks available for Ruby are Shoulda, RSpec and Cucumber.

Cucumber

Cucumber is one of the latest unit test frameworks to have come for Ruby as part of the RSpec family of tools. Cucumber itself is written in the Ruby programming language and adheres to Behaviour Driven Development(BDD). Behaviour Driven Development is an Agile Development process that comprises aspects of Acceptance Test Driven Planning , Domain Driven Design and Test Driven Development (TDD). Cucumber is designed specifically to ensure that the acceptance tests can easily be read and written by anyone.

Behaviour Driven Development(BDD)

Behaviour Driven Development<ref>BDD</ref> preserves the basic iterative (fail-pass) workflow of TDD, but stresses on specifying behaviors that are understandable to people (say from non-programming background). In this approach we write tests in a natural language that even a non-programmer can understand.

Acceptance Test

Acceptance Test<ref>Acceptance Test</ref><ref>Cucumber Book</ref> Drive Planning is a practice that stresses on the importance of identifying the functions and requirements of the software beforehand so that we exactly know when the development phase ends. This practice results in fewer bugs , shorter delivery times and greater customer satisfaction.

Cucumber Acceptance Test

     Feature: Sign up
     Sign up should be quick and friendly.

     Scenario: Successful sign up
     New users should get a confirmation email and be greeted personally by the site once signed in.
     Given I have chosen to sign up
     When I sign up with valid details
     Then I should receive a confirmation email
     And I should see a personalized greeting message

     Scenario: Duplicate email
     Where someone tries to create an account for an email address that already exists.
     Given I have chosen to sign up
     But I enter an email address that has already registered
     Then I should be told that the email is already registered
     And I should be offered the option to recover my password

Acceptance tests written in this style are more than just tests; they are executable specifications. As of January 2012, Cucumber was the second most popular testing framework after RSpec for Ruby

How Cucumber works

Cucumber is a command-line tool. It reads in specifications from plain-language text files called features, examines them for scenarios to test, and runs the scenarios against the system. The feature files should be written according to specific syntax rules called Gherkin. Along with the features, you give Cucumber a set of step definitions, which map the business-readable language of each step into Ruby code to carry out whatever action is being described by the step. Cucumber takes customer understandable user stories as input and does integration and acceptance tests on the system with those as its input.

Feature

A Feature is a high level statement of the what the test case does and there is only one feature per user story. For every feature that is supported by the system , a separate feature file has to be written and included in the features directory. The feature files has .feature file extension.

Scenario

A Scenario consists of one or more steps which describe about the various actions performed by the user with respect to this particular feature. There can be one or more scenarios per feature in a user story. A feature usually contains 3 to 8 steps in it. The steps of the scenario always begin with one of the below mentioned key words :

1. Given : Steps that begin with given represent the state of the world before an event.(preconditions)
2. When : Steps that represent the event.
3. Then : Steps that represent the expected outcome.
4. And and But : Steps that extend the previous step.

Step and Step Definitions

When testing with Cucumber , The user stories are all kept in one set of files called Steps.The Steps are compared with Step Definitions which are files written in ruby which gives the rules based on which the steps are executed. Step definitions are analogous to method definitions and steps of scenarios are analogous to method calls. Cucumber uses regular expressions to match the English phrases in the steps of scenarios to step definitions.

When the cucumber test framework is run , The steps present in the feature file can be classified as follows based on the result of regex matching with the step definitions.<ref>Step Definitions in Ruby</ref>

Successful steps

When Cucumber finds a matching Step Definition it will execute it. If the block in the step definition doesn’t raise an Exception, the step is marked as successful (green).

Undefined steps

When Cucumber can’t find a matching Step Definition the step gets marked as yellow, and all subsequent steps in the scenario are skipped. If you use --strict this will cause Cucumber to exit with 1.

Pending steps

When a Step Definition’s Proc invokes the #pending method, the step is marked as yellow (as with undefined ones), reminding you that you have work to do. If you use --strict this will cause Cucumber to exit with 1.

Failed steps

When a Step Definition’s Proc is executed and raises an error, the step is marked as red. Returning nil or false will not cause a step definition to fail.

Cucumber tries to run the steps based on the definitions and thus colors steps based on the execution of the steps. The colors for the steps assigned are as follows :

1. Green : for passing.
2. Yellow : for not yet implemented.
3. Red : for failing.
4. Blue : Once a step fails all the successive steps are marked in blue.

Cucumber Testing Stack

Advantages of Cucumber

• Specifications can be written in more than forty different spoken languages.
• Cucumber helps reduce the gap between technical and non-technical members of a software team.
• Cucumber tests can be easily read and written by business stakeholders.
• Writing Cucumber tests are quick, easy, efficient and understandable.

Cucumber Installation

To install cucumber<ref>Installing Cucumber</ref> we should make sure that the Environment path is set to “..../Ruby/bin”. Then open the command prompt and run

    gem install cucumber 

Once cucumber gem is installed then we would be able to check whether it is installed by issuing the following command:

    cucumber --version

this should give a version number like “1.2.1”

Cucumber uses Rspec for assertions and hence we need Rspec installed to use Cucumber. Use the following command to install rspec

    gem install rspec

Testing a Sample Ruby Application using Cucumber

Lets walk through a example for creating a Hello World Ruby application and testing it using cucumber<ref>Sample Cucumber Application</ref>.

Step 1 : Create a new directory

    mkdir HelloCucumber 

Step 2 : Install Cucumber and Rspec

    gem install cucumber 
    gem instal rspec

Step 3 : Create a directory features and create a file basic.feature in the features directory

     mkdir features

Content of features/basic.feature

    Feature: Hello World Feature

    Scenario: Hello World Scenario
    Given The Action is Hello
    When The Subject is World
    Then The Greeting is Hello, World

Step 4: Create step_definitions directory inside features directory. Create basic_step.rb file inside step_definitions.

     mkdir features/step_definitions

The directory structure looks like this

Directory structure

Content of features/step_definitions/basic_step.rb

     require 'rspec/expectations'

     Given /The Action is ([A-z]*)/ do |action|
        @action = action
     end

     When /The Subject is ([A-z]*)/ do |subject|
       @subject = subject
     end

     Then /The Greeting is (.*)/ do |greeting|
       greeting.should == "#{@action}, #{@subject}"
     end

Cucumber uses regular expression after the keywords Given, When and Then to map lines in a feature file to step definitions.

Step 5: Running cucumber command you will see the following output

    Feature: Hello World Feature

    Scenario: Hello World Scenario    # features\basic.feature:3
    Given The Action is Hello         # features/step_definitions/basic_steps.rb:3
    When The Subject is World         # features/step_definitions/basic_steps.rb:7
    Then The Greeting is Hello, World # features/step_definitions/basic_steps.rb:11

    
    1 scenario (1 passed)
    3 steps (3 passed)
    0m0.016s
    Step 6 : Modify the basic.features file to include 2 scenario

Step 6 : Modify the basic.features file to include 2 scenarios

    Feature: Hello World Feature

    Scenario: Hello World Scenario
    Given The Action is Hello
    When The Subject is World
    Then The Greeting is Hello, World

    Scenario: Bye World Scenario
    Given The Action is Bye
    When The Subject is World
    Then The Greeting is Hello, World

Step 7 : Running the cucumber command you will see the following output

   Scenario: Hello World Scenario      # features\basic.feature:3
   Given The Action is Hello         # features/step_definitions/basic_steps.rb:3
   When The Subject is World         # features/step_definitions/basic_steps.rb:7
   Then The Greeting is Hello, World # features/step_definitions/basic_steps.rb:11

   Scenario: Bye World Scenario       # features\basic.feature:8
   Given The Action is Bye          # features/step_definitions/basic_steps.rb:3
   When The Subject is World        # features/step_definitions/basic_steps.rb:7
   Then The Greeting is Hello, World # features/step_definitions/basic_steps.rb:11
     expected: "Bye, World"
          got: "Hello, World" (using ==) (RSpec::Expectations::ExpectationNotMetError)
     ./features/step_definitions/basic_steps.rb:12:in `/The Greeting is (.*)/'
     features\basic.feature:11:in `Then The Greeting is Hello, World'

   Failing Scenarios:
   cucumber features\basic.feature:8 # Scenario: Bye World Scenario

   2 scenarios (1 failed, 1 passed)
   6 steps (1 failed, 5 passed)
   0m0.033s

The Bye Scenario fails since the expected output is “Hello, World” but the received output is “Bye,World”.

Testing a Sample Rails Application using Cucumber

Cucumber<ref>Sample Cucumber Application on Rails</ref> easily integrates with Rails and hence can be used with Ruby applications which run on Rails. But there are few steps which have to be followed before you will be able to run cucumber with Rails. Lets walk through the steps by creating a new rails application.

   rails new cucumber_app

Install the required cucumber gems

   gem install cucumber
   gem install cucumber cucumber-rails 

Add the following lines to your gemfile. These gems should be added only to the test environment. The gems database_cleaner , rspec-rails and webrat are required for cucumber to function properly.

   group :test do
      gem 'database_cleaner'
      gem 'cucumber'
      gem "rspec-rails", "~> 2.0", :require => false
      gem "cucumber-rails", "1.0.6"
      gem 'webrat'
   end

After modifying the gem file, run bundle update

     bundle update

The required cucumber files are generated using the following command. The output below the command shows the folders and files that gets created for the cucumber test framework.

   rails generate cucumber:install

    identical  script/cucumber
          chmod  script/cucumber
          exist  features/step_definitions
    identical  features/step_definitions/web_steps.rb
          exist  features/support
    identical  features/support/paths.rb
    identical  features/support/selectors.rb
    identical  features/support/env.rb
          exist  lib/tasks
    identical  lib/tasks/cucumber.rake

The step_definitions folder contains ruby files which test the steps which are mentioned in feature file of the cucumber.
The paths.rb file is created in the support folder will contain the rules based on which the step definitions will be compared to the steps and matched. In case we come across any steps whose execution fails while running cucumber we would have to update the paths.rb file to resolve the issue.
env.rb is only loaded when the application is first started up and it sets up the environment required by Cucumber to execute.

   rails generate model article title:string content:text 
   rake db:migrate
   rake db:test:clone
   rails generate controller articles index 

The above four commands generate models with the name article and with two fields title and content and the next two commands migrate the db to production and the last command creates controllers for article and index objects.
We have to go to the home directory of our rails application and then run Cucumber with the following command.The result would then be shown to the user on how many and which steps succeeded and failed.

   => cucumber

Capybara

<ref>Web-app with Cucumber and Capybara</ref>

We need a tool that acts out the user stories that are part of the feature that is to be tested with Cucumber. Capybara can be used as that tool. Capybara is an integration testing tool for rack based web based applications. It simulates how a user would interact with a website. It can simulate the actions of a web browser and hence can interact with the application to receive web pages. It can also parse the HTML content and submits forms as the user would. One of the restrictions that Capybara has is that it cannot handle JavaScript. There are other tools like webdriver which can handle JavaScript but it runs a lot slower compared to Capybara.

Comparison of Unit Test Frameworks : Rspec, Shoulda and Cucumber

<ref>Book on Cucumber and RSpec</ref>

Rspec	Shoulda	Cucumber
RSpec is based on the Behaviour driven development model provided by Ruby. It has its origins in Test Driven Development and Domain Driven Design.	Shoulda is based on the Test:Unit which is another Ruby Test Framework. It is based on Test Driven Development.	Cucumber is a combination of Test Driven Development and Behaviour driven development.
RSpec provides a Domain Specific Language that can express code functionality and behaviour.	Shoulda provides macros, helpers and matchers.	Cucumber allows the test cases to be written in native languages.
RSpec too allows nested describe structure.	Shoulda has nested contexts which helps create different environment for different set of tests.	Cucumber is similar to RSpec and provides nested structures.
RSpec is expressive when it comes to understanding functionalities since it was designed with the human readability in mind.	Shoulda has more descriptive names compared to its predecessor Test::Unit but its less expressive.	Cucumber is more descriptive and allows business stakeholder to read and write test specification.

Summary

Cucumber<ref>Advantage of Cucumber</ref> helps facilitate the discovery and use of a ubiquitous language within the team. Cucumber tests interact directly with the developers’ code, but they’re written in a high level language that business stakeholders can understand. When the team uses this language consistently in their conversations, documentation, and code, the friction of translating between everyone’s different little dialects is gone, and the chances of misunderstandings are greatly reduced. What makes Cucumber stand out from the crowd of other testing tools is that it has been designed specifically to ensure the acceptance tests can easily be read—and written—by anyone on the team.In practice, this means that your documentation, rather than being something that’s written once and then gradually goes out of date, becomes a living thing that reflects the true state of the project

Reference

<references/>