CSC/ECE 517 Fall 2009/wiki1b12 AV: Difference between revisions

From Expertiza_Wiki
Jump to navigation Jump to search
 
(9 intermediate revisions by the same user not shown)
Line 1: Line 1:
XUnit in general, is only one kind of testing framework for object oriented
XUnit in general, is only one kind of testing framework for object oriented languages. This page talks about the major testing frameworks available for the major object oriented languages such as Java, C++, C#,Python, Ruby and C#. Comparisons have been made wherever possible between the major frameworks available in each language.  
languages. This page talks about the major testing frameworks available for
the major object oriented languages such as Java, C++, C#,Python, Ruby and
C#. Comparisons have been made wherever possible between the major
frameworks available in each language.




=Java Testing Frameworks =
=Java Testing Frameworks =


Plenty of testing framework exist for Java. JUnit has been the de facto
Plenty of testing framework exist for Java. JUnit has been the de facto standard for unit testing. However, other frameworks such as TestNG, JTIGER have been built to address various faults and deficiencies with JUnit.  Our primary objective is to compare the various testing frameworks that exist for object oriented languages based on the primary purpose of the framework, features, the platforms on which they operate, etc and thus help in picking up the best testing framework that fits in a particular suitable environment.
standard for unit testing. However, other frameworks such as TestNG, JTIGER
This article DOES NOT introduce JUnit or XUnit. For information on JUnit and XUnit, please refer to the http://www.junit.org or [[http://en.wikipedia.org/wiki/Junit '''APPENDIX 1''']] and [[http://en.wikipedia.org/wiki/XUnit '''APPENDIX 2''']]  
have been built to address various faults and deficiencies with JUnit.  Our
primary objective is to compare the various testing frameworks that exist
for object oriented languages based on the primary purpose of the framework,
features, the platforms on which they operate, etc and thus help in picking
up the best testing framework that fits in a particular suitable
environment.
This article DOES NOT introduce JUnit or XUnit. For information on JUnit,
please refer to http://www.junit.org or [[http://en.wikipedia.org/wiki/Junit '''APPENDIX 1''']]




== TestNG Framework ==
== TestNG Framework ==


TestNG is a testing framework which can help to meet a broad range of
TestNG is a testing framework which can help to meet a broad range of testing such as unit testing, integration testing, functional testing, end-to -end testing. (For definition and more information on the different kinds of testing, refer to [[http://en.wikipedia.org/wiki/Unit_testing ''' APPENDIX 3''']] and [[http://en.wikipedia.org/wiki/Integration_testing ''' APPENDIX 4''']]
testing such as unit testing, integration testing, functional testing,
end-to -end testing. (For definition and more information on the different
kinds of testing, refer to''' APPENDIX''' given below.)


To write a TestNG test, we need to:
To write a TestNG test, we need to:
Write the business logic of the test, and insert the TestNG annotations in
Write the business logic of the test, and insert the TestNG annotations in them.
them.
Add the information about our test (such as the class name, the groups we wish to run, etc) in a testing.xml file or in build.xml.  
Add the information about our test (such as the class name, the groups we
wish to run, etc) in a testing.xml file or in build.xml.
Run the test.
Run the test.
TestNG supports data driven testing.(Data-driven testing is a methodology
TestNG supports data driven testing.(Data-driven testing is a methodology used in Test automation where test scripts are executed and verified based on the data values stored in one or more central data sources or databases. For a detailed description on data driven testing click here: http://en.wikipedia.org/wiki/Data-driven_testing  )
used in Test automation where test scripts are executed and verified based
on the data values stored in one or more central data sources or databases.
For a detailed description on data driven testing click here:
http://en.wikipedia.org/wiki/Data-driven_testing  )


TestNG is invoked in several different ways:
TestNG is invoked in several different ways:  
With a testing.xml file
With a testing.xml file
ANT
ANT
Line 45: Line 24:




It is possible to group the test methods of TestNG.  We can declare methods
It is possible to group the test methods of TestNG.  We can declare methods that belong to these groups and specify groups that contain other groups. It is very flexible as it can help in choosing one set of regular expression and excluding the other sets. This splits the various test groups and there is no need to recompile again if we want to run two different tests back to back.
that belong to these groups and specify groups that contain other groups. It
is very flexible as it can help in choosing one set of regular expression
and excluding the other sets. This splits the various test groups and there
is no need to recompile again if we want to run two different tests back to
back.


Requirements:
Requirements:
JDK 1.4, 1.5
JDK 1.4, 1.5
Invoked in many environments such as command line, ant, ECLIPSE, IDEA,
Invoked in many environments such as command line, ant, ECLIPSE, IDEA, MAVEN, etc
MAVEN, etc


The annotations available in TestNG can be found at:
The annotations available in TestNG can be found at:
http://testng.org/doc/documentation-main.html#annotations
http://testng.org/doc/documentation-main.html#annotations


TestNG is inspired from JUnit and Nunit, with additional functionalities
TestNG is inspired from JUnit and Nunit, with additional functionalities such as Flexible test configuration, support for data driven testing, etc.[[http://testng.org/doc/documentation-main.html 1]]
such as Flexible test configuration, support for data driven testing, etc.[[http://testng.org/doc/documentation-main.html 1]]


== Cactus ==
== Cactus ==




Cactus is a test framework for testing server-side java code such as
Cactus is a test framework for testing server-side java code such as Servlets, EJB, etc. It extends J Unit.
Servlets, EJB, etc. It extends J Unit.
Cactus implements an in-container strategy which means that tests are executed inside the container.  
Cactus implements an in-container strategy which means that tests are
The Cactus testing unit can be viewed as a system that consists of the following components:
executed inside the container.
The Cactus testing unit can be viewed as a system that consists of the
--Cactus Framework:  This provides the API to write Cactus tests.
following components:


--Cactus Framework: This provides the API to write Cactus tests.
--Cactus Integration Modules: They are front ends and frameworks that provide easy ways of using the Cactus Framework. Eg: Eclipse plugin


--Cactus Integration Modules: They are front ends and frameworks that
--Cactus Samples: They demonstrate how to write Cactus tests and how to use a few Integration Modules. [[http://jakarta.apache.org/cactus/index.html 2]]
provide easy ways of using the Cactus Framework. Eg: Eclipse plugin


--Cactus Samples: They demonstrate how to write Cactus tests and how to use
A useful testing framework provided by Cactus is  integration unit testing.  The tests written for this framework will exercise the interactions with the container. Cactus also provides other frameworks such as code logic unit testing, functional unit testing.
a few Integration Modules. [[http://jakarta.apache.org/cactus/index.html 2]]


A useful testing framework provided by Cactus is  integration unit testing.
Cactus was developed to fit only integration unit testing with the idea that it is much easier to have to write tests for a single framework than for several. It is believed that Cactus provides a middle ground that provides a high confidence that your code will run when deployed .[[http://jakarta.apache.org/cactus/index.html 2]]
The tests written for this framework will exercise the interactions with
the container. Cactus also provides other frameworks such as code logic unit
testing, functional unit testing.
 
Cactus was developed to fit only integration unit testing with the idea that
it is much easier to have to write tests for a single framework than for
several. It is believed that Cactus provides a middle ground that provides a
high confidence that your code will run when deployed .[[
http://jakarta.apache.org/cactus/index.html 2]]




Line 97: Line 57:
http://jakarta.apache.org/cactus/how_it_works.html
http://jakarta.apache.org/cactus/how_it_works.html


== '''COMPARISON BETWEEN JUnit,TestNG and Cactus frameworks''' [[
== '''COMPARISON BETWEEN JUnit,TestNG and Cactus frameworks''' [[http://junit.sourceforge.net/doc/cookbook/cookbook.htm 3]] ==
http://junit.sourceforge.net/doc/cookbook/cookbook.htm 3]] ==


{| {{table}}
{| {{table}}
Line 106: Line 65:
| align="center" style="background:#f0f0f0;"|'''Cactus'''
| align="center" style="background:#f0f0f0;"|'''Cactus'''
|-
|-
| Supporting Type of Testing||Suitable more for Unit testing||Supports Unit
| Supporting Type of Testing||Suitable more for Unit testing||Supports Unit testing,Functional testing, end-to-end testing, integration testing ||Primarily unit testing. Also have frameworks for code logic unit testing , functional logic unit testing
testing,Functional testing, end-to-end testing, integration testing
||Primarily unit testing. Also have frameworks for code logic unit testing ,
functional logic unit testing
|-
|-
| Conventions||Rigid  ||Flexible||Not as flexible as TestNG
| Conventions||Rigid  ||Flexible||Not as flexible as TestNG
|-
|-
| Derived from JUnit||Yes, from the previous versions||Inspired from
| Derived from JUnit||Yes, from the previous versions||Inspired from Junit,with  added functionalities ||Extends JUnit
Junit,with  added functionalities ||Extends JUnit
|-
|-
| Flexibility for large suites||Not flexible for large test suites,
| Flexibility for large suites||Not flexible for large test suites, ||Flexible for running large test suites of code  and thus one test\'s failure shouldn\'t mean having to rerun a suite of thousands. || No, it is suitable for unit testing.
||Flexible for running large test suites of code  and thus one test\'s
failure shouldn\'t mean having to rerun a suite of thousands. || No, it is
suitable for unit testing.
|-
|-
| Suitable for ||Best for a single object||Best for a higher level
| Suitable for ||Best for a single object||Best for a higher level testing||Best for server-side java code such as Servlets, EJB, etc.
testing||Best for server-side java code such as Servlets, EJB, etc.
|-
|-
| Works with||Command line, ANT,ECLIPSE||Command line,ANT, ECLIPSE,
| Works with||Command line, ANT,ECLIPSE||Command line,ANT, ECLIPSE, IDEA,MAVEN||ANT
IDEA,MAVEN||ANT
|-
|-
| ||||||
| ||||||
|-
|-
|
|  
|}
|}


Line 137: Line 87:
'''JUnit 4  v/s TestNG'''
'''JUnit 4  v/s TestNG'''


(source: http://www.ibm.com/developerworks/java/library/j-cq08296/)
(source: http://www.ibm.com/developerworks/java/library/j-cq08296/)  




The frameworks differ is in their core design. JUnit has always been an
The frameworks differ is in their core design. JUnit has always been an effective unit-testing framework, meaning that it was built to facilitate testing single objects. TestNG, on the other hand, was built to address testing at higher levels, and consequently, has some features not available in JUnit
effective unit-testing framework, meaning that it was built to facilitate
testing single objects. TestNG, on the other hand, was built to address
JUnit 4 makes clever use of annotations. JUnit no longer requires you to define a test as a method whose name starts with test, and we can now run fixtures just once as opposed to for each test.  However TestNG established itself as an annotations-based framework long before Junit 4. TestNG pioneered testing with annotations in Java programming, which made it a formidable alternative to JUnit. [[http://www.ibm.com/developerworks/java/library/j-cq08296 4]]
testing at higher levels, and consequently, has some features not available
in JUnit
 
JUnit 4 makes clever use of annotations. JUnit no longer requires you to
define a test as a method whose name starts with test, and we can now run
fixtures just once as opposed to for each test.  However TestNG established
itself as an annotations-based framework long before Junit 4. TestNG
pioneered testing with annotations in Java programming, which made it a
formidable alternative to JUnit. [[
http://www.ibm.com/developerworks/java/library/j-cq08296 4]]




Line 169: Line 109:
  }
  }
  @Test
  @Test
  public void verifyDependencies()
  public void verifyDependencies()  
   throws Exception {
   throws Exception {
   String targetClss =
   String targetClss =  
     "test.com.acme.dona.dep.DependencyFind";
     "test.com.acme.dona.dep.DependencyFind";
  Filter[] filtr = new Filter[] {
  Filter[] filtr = new Filter[] {  
       new RegexPackageFilter("java|junit|org")};
       new RegexPackageFilter("java|junit|org")};
  Dependency[] deps =
  Dependency[] deps =  
       finder.findDependencies(targetClss, filtr);
       finder.findDependencies(targetClss, filtr);
  assertNotNull("deps was null", deps);
  assertNotNull("deps was null", deps);
   assertEquals("should be 5 large", 5, deps.length);
   assertEquals("should be 5 large", 5, deps.length);  
   }
   }
  }
  }
Line 199: Line 139:
  }
  }
  @Test
  @Test
  public void verifyDependencies()
  public void verifyDependencies()  
   throws Exception {
   throws Exception {
   String targetClss =
   String targetClss =  
     "test.com.acme.dona.dep.DependencyFind";
     "test.com.acme.dona.dep.DependencyFind";
   Filter[] filtr = new Filter[] {
   Filter[] filtr = new Filter[] {  
       new RegexPackageFilter("java|junit|org")};
       new RegexPackageFilter("java|junit|org")};
   Dependency[] deps =
   Dependency[] deps =  
       finder.findDependencies(targetClss, filtr);
       finder.findDependencies(targetClss, filtr);
   assertNotNull(deps, "deps was null" );
   assertNotNull(deps, "deps was null" );
   assertEquals(5, deps.length, "should be 5 large");
   assertEquals(5, deps.length, "should be 5 large");          
   }
   }
  }
  }
  (source: http://www.ibm.com/developerworks/java/library/j-cq08296/)
  (source: http://www.ibm.com/developerworks/java/library/j-cq08296/)


Both the codes may look a lot similar. However TestNG coding conventions are
Both the codes may look a lot similar. However TestNG coding conventions are more flexible than JUnit 4.
more flexible than JUnit 4.


JUnit forces us to declare the @BeforeClass (See '''APPENDIX''' for more
JUnit forces us to declare the @BeforeClass (See [[http://junit.sourceforge.net/javadoc_40/org/junit/BeforeClass.html '''APPENDIX 5''']] for more information)  method as static, which consequently requires us to also declare the fixture, finder, as static. We also have to declare the init() method as public. Looking at the TestNG code, you can find that such conventions aren't required. Its init() method is neither static nor public.  
information)  method as static, which consequently requires us to also
Flexibility has been one of the strong points of TestNG right from the start. JUnit works well for a unit of code, whereas TestNG is better suited for high-level testing. Its flexibility is especially useful with large test suites.[[http://www.ibm.com/developerworks/java/library/j-cq08296 4]]
declare the fixture, finder, as static. We also have to declare the init()
method as public. Looking at the TestNG code, you can find that such
conventions aren't required. Its init() method is neither static nor
public.
Flexibility has been one of the strong points of TestNG right from the
start. JUnit works well for a unit of code, whereas TestNG is better suited
for high-level testing. Its flexibility is especially useful with large test
suites.[[http://www.ibm.com/developerworks/java/library/j-cq08296 4]]


== Some other JAVA testing units [[
== Some other JAVA testing units [[http://www.opensourcetesting.org/unit_java.php 5]]  ==
http://www.opensourcetesting.org/unit_java.php 5]]  ==


{| {{table}}
{| {{table}}
Line 237: Line 167:
| align="center" style="background:#f0f0f0;"|'''Mockrunner'''
| align="center" style="background:#f0f0f0;"|'''Mockrunner'''
|-
|-
| Features||Generates coverage report Allows virtual mock object||Is a  Java
| Features||Generates coverage report Allows virtual mock object||Is a  Java 2 Micro Edition (J2ME) library containing a unit testing framework for J2ME||Does test data exporting.  JDBC agnostic  Generates DbUnit datasets, hierarchically structured XML, and topologically sorted SQL-DML.||A lightweight framework for unit testing applications in the J2EE environment.
2 Micro Edition (J2ME) library containing a unit testing framework for
J2ME||Does test data exporting.  JDBC agnostic  Generates DbUnit datasets,
hierarchically structured XML, and topologically sorted SQL-DML.||A
lightweight framework for unit testing applications in the J2EE environment.
|-
|-
| Integrates with:||ECLIPSE 3.3||ANT ||Platform Independent||OS independent
| Integrates with:||ECLIPSE 3.3||ANT ||Platform Independent||OS independent
|-
|-
|
|  
|}
|}


Line 251: Line 177:




When it comes to unit testing in Java, a unanimous choice would be JUnit.
When it comes to unit testing in Java, a unanimous choice would be JUnit. However for C++, there is no one clear winner.  CppUnit is the most widely used tool for C++ which is a part of the XUnit family.
However for C++, there is no one clear winner.  CppUnit is the most widely
used tool for C++ which is a part of the XUnit family.


Following is a few basic factors to judge a good testing unit in C++ [[
Following is a few basic factors to judge a good testing unit in C++ [[http://gamesfromwithin.com/exploring-the-c-unit-testing-framework-jungle 6]]  
http://gamesfromwithin.com/exploring-the-c-unit-testing-framework-jungle6]]


{| {{table}}
{| {{table}}
Line 264: Line 187:
| align="center" style="background:#f0f0f0;"|'''CxxUnit'''
| align="center" style="background:#f0f0f0;"|'''CxxUnit'''
|-
|-
| Minimal amount of work needed to add new tests ||No, and thus a major
| Minimal amount of work needed to add new tests ||No, and thus a major downfall.||Yes||Yes, very good.
downfall.||Yes||Yes, very good.
|-
|-
| Easy to modify and port ||Medium||Medium||Simplest
| Easy to modify and port ||Medium||Medium||Simplest
Line 271: Line 193:
| Supports setup/teardown steps ||Yes||||Best
| Supports setup/teardown steps ||Yes||||Best
|-
|-
| Handles exceptions and crashes well.||Yes||Yes, above others||Great
| Handles exceptions and crashes well.||Yes||Yes, above others||Great support.
support.
|-
|-
| Good assert functionality.||Pretty decent.||Yes||Best
| Good assert functionality.||Pretty decent.||Yes||Best
|-
|-
| Supports different outputs. ||Yes. Comes with IDE friendly
| Supports different outputs. ||Yes. Comes with IDE friendly output.||Probably, Not trivial to change.||Yes
output.||Probably, Not trivial to change.||Yes
|-
|-
| Supports  suites||Yes.||Yes, but with a big catch. ||Yes
| Supports  suites||Yes.||Yes, but with a big catch. ||Yes
|-
|-
|
|  
|}
|}




CppUnit passes most of the above tests , except for the first point and thus
CppUnit passes most of the above tests , except for the first point and thus needs a lot of code typing.
needs a lot of code typing.
Boost.Test isn’t exclusively a unit-testing framework, and is not X Unit based.
Boost.Test isn’t exclusively a unit-testing framework, and is not X Unit
based.


Out of the above three , cxxUnit turns out to be a favourite.
Out of the above three , cxxUnit turns out to be a favourite.


It comes close to the requirements of an ideal framework by leveraging the
It comes close to the requirements of an ideal framework by leveraging the power of an external scripting language. It provides some nifty advanced features and great assert functionality. It does require the use of a scripting language as part of the build process, so those uncomfortable with that requirement, might want to look at one of the other two frameworks.
power of an external scripting language. It provides some nifty advanced
features and great assert functionality. It does require the use of a
scripting language as part of the build process, so those uncomfortable with
that requirement, might want to look at one of the other two frameworks.


For work restricted to the PC,  where modifying the framework is not
For work restricted to the PC,  where modifying the framework is not required Boost.Test could  be a very good choice.
required Boost.Test could  be a very good choice.
CppUnit , which has come a long way in years is a solid, complete framework  The major drawbacks are the relative verbosity for adding new tests and fixtures. [[http://gamesfromwithin.com/exploring-the-c-unit-testing-framework-jungle 6]]
CppUnit , which has come a long way in years is a solid, complete framework
  The major drawbacks are the relative verbosity for adding new tests and
fixtures. [[
http://gamesfromwithin.com/exploring-the-c-unit-testing-framework-jungle 6]]


= Python =
= Python =
Line 309: Line 219:




'''Doctest''': It is a simple, ingenuine technology that allows you to write
'''Doctest''': It is a simple, ingenuine technology that allows you to write a narrative in a document which can directly be executed as a test. It is included in Python as a module.
a narrative in a document which can directly be executed as a test. It is
included in Python as a module.


Although doctest makes testing code readable for non-programmers ,the
Although doctest makes testing code readable for non-programmers ,the simplicity of a narrative test like doctest is accompanied by it’s own flaws .Some of the disadvantages of using doctest are
simplicity of a narrative test like doctest is accompanied by it’s own flaws
.Some of the disadvantages of using doctest are


• These tests are difficult to handle in case of failures as they are more
• These tests are difficult to handle in case of failures as they are more generic. Specific tests are narrower and better in communicating intent and ensuring coverage.
generic. Specific tests are narrower and better in communicating intent and
ensuring coverage.


• The test code is executed in a special way thus making it difficult to
• The test code is executed in a special way thus making it difficult to reason about how it will be executed. It is also harder to program around the tests.
reason about how it will be executed. It is also harder to program around
the tests.


• It is actually difficult to get an outline of the tests at glance. There’s
• It is actually difficult to get an outline of the tests at glance. There’s no tool that can give an overview of the unit tests in a doc test file [[http://en.wikipedia.org/wiki/Doctest 7]].
no tool that can give an overview of the unit tests in a doc test file [[
http://en.wikipedia.org/wiki/Doctest 7]].


An detailed information about doctest with thorough examples can be read at
An detailed information about doctest with thorough examples can be read at http://docs.python.org/library/doctest.html
http://docs.python.org/library/doctest.html




'''Unittest''': Unittest are more specific tests which are more functional
'''Unittest''': Unittest are more specific tests which are more functional in nature allowing it to be written for individual functions or the class as a whole.  
in nature allowing it to be written for individual functions or the class as
a whole.


The fact that Unittest is based on jUnit makes people with the xUnit
The fact that Unittest is based on jUnit makes people with the xUnit familiarity pick it up really quick hence it is more Java based than python .It has been introduced in the Python’s library since version 2.1  
familiarity pick it up really quick hence it is more Java based than python
.It has been introduced in the Python’s library since version 2.1


The disadvantages of using the unit test are
The disadvantages of using the unit test are  


• The tests have a different look and feel with respect to the code under
• The tests have a different look and feel with respect to the code under test
test


• The assertions used in the text use custom syntax which makes it difficult
• The assertions used in the text use custom syntax which makes it difficult


• The test code is difficult to understand compared to the Doctest and the
• The test code is difficult to understand compared to the Doctest and the strong basis on jUnit makes it difficult for the pure Python coders [[http://en.wikipedia.org/wiki/Unit_testing 8]]
strong basis on jUnit makes it difficult for the pure Python coders [[
http://en.wikipedia.org/wiki/Unit_testing 8]]


A detailed information about doctest with thorough examples can be read at
A detailed information about doctest with thorough examples can be read at http://docs.python.org/library/unittest.html
http://docs.python.org/library/unittest.html




'''Doctest vs Unittest'''
'''Doctest vs Unittest'''


Here we compare simple features of both the tests thus giving us an idea of
Here we compare simple features of both the tests thus giving us an idea of which one to use based on the requirement
which one to use based on the requirement


{| {{table}}
{| {{table}}
Line 366: Line 256:
| align="center" style="background:#f0f0f0;"|'''unittest'''
| align="center" style="background:#f0f0f0;"|'''unittest'''
|-
|-
| '''Simplicity'''||Best suited for novice programmers||A little complex to
| '''Simplicity'''||Best suited for novice programmers||A little complex to use specially if unaware of Junit
use specially if unaware of Junit
|-
|-
| '''Documentation'''||Better Narratives||Tests have little or no comments
| '''Documentation'''||Better Narratives||Tests have little or no comments
|-
|-
| '''Comprehensive testing'''||Doctests get clutterd for comprehensive tests
| '''Comprehensive testing'''||Doctests get clutterd for comprehensive tests and thus less readable||Best suited for comprehensive tests
and thus less readable||Best suited for comprehensive tests
|-
|-
| '''Flexibility and Control'''||Less flexible||More flexible
| '''Flexibility and Control'''||Less flexible||More flexible
|-
|-
|
|  
|}
|}


Line 382: Line 270:




The basic testing framework for RUBY is xUnit . There are some frameworks
The basic testing framework for RUBY is xUnit . There are some frameworks which have evolved over the years which include RSpec, Trantula, Flog, RunCodeRun, Cucumber etc. We’ll talk about each of them in brief
which have evolved over the years which include RSpec, Trantula, Flog,
RunCodeRun, Cucumber etc. We’ll talk about each of them in brief




Line 391: Line 277:
| style="background:#f0f0f0;"|'''Description'''
| style="background:#f0f0f0;"|'''Description'''
|-
|-
| '''Rspec[[http://www.elctech.com/tutorials/rspec-tutorial 9]]'''||It is
| '''Rspec[[http://www.elctech.com/tutorials/rspec-tutorial 9]]'''||It is Behavior Driven Development whose basic features are
Behavior Driven Development whose basic features are
|-
|-
| ||·        Behavior counts first, testing follows
| ||·        Behavior counts first, testing follows
Line 398: Line 283:
| ||·        It focuses more on business values
| ||·        It focuses more on business values
|-
|-
| ||·        It’s more about getting the words right rather than just
| ||·        It’s more about getting the words right rather than just writing the test first
writing the test first
|-
|-
| ||Advantages:
| ||Advantages:
|-
|-
| ||·        These are extremely easy to read and makes sense to a
| ||·        These are extremely easy to read and makes sense to a non-programmer
non-programmer
|-
|-
| ||·        Rspec lets you unit test while also providing mocking and
| ||·        Rspec lets you unit test while also providing mocking and stubbing support
stubbing support
|-
|-
| ||·        Rspec can help with the coverage tests using Rcov
| ||·        Rspec can help with the coverage tests using Rcov
|-
|-
| ||·        One can also perform integration test using Rspec
| ||·        One can also perform integration test using Rspec  
|-
|-
| ||Disadvantages:
| ||Disadvantages:
|-
|-
| ||·        It involves a lot of behind the scene metaprogramming which
| ||·        It involves a lot of behind the scene metaprogramming which makes the tests very difficult to understand
makes the tests very difficult to understand
|-
|-
| ||A simple Rspec example can be read at http://rspec.info/
| ||A simple Rspec example can be read at http://rspec.info/
|-
|-
| '''Cucumber[[http://www.rubyinside.com/cucumber-the-latest-in-ruby-testing-1342 10]]'''||Cucumber,
| '''Cucumber[[http://www.rubyinside.com/cucumber-the-latest-in-ruby-testing-1342.html 10]]'''||Cucumber, the latest addition to the Rspec family allows one to execute documentation written in plain text(known as ‘stories’)
the latest addition to the Rspec family allows one to
execute documentation written in plain text(known as ‘stories’)
|-
|-
| ||A very good example of writing a test in cucumber can be read at
| ||A very good example of writing a test in cucumber can be read at http://cukes.info/
http://cukes.info/
|-
|-
| ||
| ||
|-
|-
| '''Tarantula[[http://www.ruby-forum.com/topic/144083 11]]'''||Tarantula is
| '''Tarantula[[http://www.ruby-forum.com/topic/144083 11]]'''||Tarantula is a Rail’s plugin that produces a detailed report describing the URL’s that it has crawled across and the responses that it received from each one of them.
a Rail’s plugin that produces a detailed report describing the URL’s that it
has crawled across and the responses that it received from each one of them.
|-
|-
| ||Tarantula\'s functionality somewhat overlaps with the on site testing,
| ||Tarantula\'s functionality somewhat overlaps with the on site testing, particularly if you are practicing test-driven development (TDD). Tarantula runs against your actual Web site thereby helping you to identify the problems associated with your production server, user-generated data, and HTML compliance.
particularly if you are practicing test-driven development (TDD). Tarantula
runs against your actual Web site thereby helping you to identify the
problems associated with your production server, user-generated data, and
HTML compliance.


A well explained example can be seen at
A well explained example can be seen at http://www.railslodge.com/plugins/959-tarantula
http://www.railslodge.com/plugins/959-tarantula


|-
|-
| '''Flog[[http://www.brynary.com/2007/9/13/scourging-your-ruby-code-with-flog 12]]'''||Flog
| '''Flog[[http://www.brynary.com/2007/9/13/scourging-your-ruby-code-with-flog 12]]'''||Flog is a tool which gives you a profile of your code’s complexity. It can given the complexity pertaining to each method in your code. One can also set threshold for each method using flog  
is a tool which gives you a profile of your code’s complexity.
It can given the complexity pertaining to each method in your code. One can
also set threshold for each method using flog
|-
|-
| ||The complexity of a code can be important to
| ||The complexity of a code can be important to  
|-
|-
| ||·        Find the bugs as they are highly probable in complex code
| ||·        Find the bugs as they are highly probable in complex code areas
areas
|-
|-
| ||·        Complex codes can be considered for refactoring
| ||·        Complex codes can be considered for refactoring
|-
|-
| ||A flog report has been explained and shown here
| ||A flog report has been explained and shown here http://www.ruby-forum.com/topic/131931
http://www.ruby-forum.com/topic/131931
|-
|-
| ||
| ||
|-
|-
|
|  
|}
|}


= C# =
= C# =
 
 
The list of frameworks that apply for .NET are applied for C#. A long list
The list of frameworks that apply for .NET are applied for C#. A long list
of testing frameworks are available for C# which are NUnit, csUnit, Pex, dotunit,
of testing frameworks are available for C# which are NUnit, csUnit, Pex, dotunit,
Line 481: Line 346:
| Examples||http://www.nunit.org/blogs/||http://www.csunit.org/tutorials/tutorial7/||http://research.microsoft.com/en-us/um/redmond/projects/pex/wiki/The%20Hello%20World%20of%20Pex.html
| Examples||http://www.nunit.org/blogs/||http://www.csunit.org/tutorials/tutorial7/||http://research.microsoft.com/en-us/um/redmond/projects/pex/wiki/The%20Hello%20World%20of%20Pex.html
|-
|-
| Used as ||Testing framework, GUI and console tool||Testing Framework ,Console tool and VS.NET Add In||It supports NUnit, MbUnit and xUnit.net.Extensions for these test frameworks can be downloaded from the Pex Extensions Project.
| Used as ||Testing framework, GUI and console tool||Testing Framework ,Console tool and VS.NET Add In||It supports NUnit, MbUnit and xUnit.net.Extensions for these test frameworks can be downloaded from the Pex Extensions Project[[http://research.microsoft.com/en-us/um/redmond/projects/pex/wiki/reference%20manual.html 14]].
|-
|-
|  
|  
Line 488: Line 353:
=APPENDIX=
=APPENDIX=


'''JUnit''' - JUnit is a unit testing framework for the Java programming
'''1''' '''JUnit''' - JUnit is a unit testing framework for the Java programming language. JUnit has been important in the development of test-driven development, and comprises a family of unit testing frameworks collectively known as xUnit that originated with SUnit. (For more information refer to http://www.junit.org,http://en.wikipedia.org/wiki/Junit)
language. JUnit has been important in the development of test-driven
development, and comprises a family of unit testing frameworks collectively
known as xUnit that originated with SUnit. (For more information refer to
http://www.junit.org,http://en.wikipedia.org/wiki/Junit)


'''XUnit''' - Various code-driven testing frameworks have come to be known
'''2''' '''XUnit''' - Various code-driven testing frameworks have come to be known collectively as xUnit. These frameworks allow testing of different elements (units) of software, such as functions and classes. (For more information refer to http://en.wikipedia.org/wiki/XUnit)
collectively as xUnit. These frameworks allow testing of different elements
(units) of software, such as functions and classes. (For more information
refer to http://en.wikipedia.org/wiki/XUnit)


'''unit testing''' - unit testing is a software verification and validation
'''3''' '''unit testing''' - unit testing is a software verification and validation method in which a programmer tests if individual units of source code are fit for use, where a unit is the smallest testable part of an application. For more information refer http://en.wikipedia.org/wiki/Unit_testing
method in which a programmer tests if individual units of source code are
fit for use, where a unit is the smallest testable part of an application.
For more information refer http://en.wikipedia.org/wiki/Unit_testing


'''integration testing''' - is the activity of software testing in which
'''4''' '''integration testing''' - is the activity of software testing in which individual software modules are combined and tested as a group. It occurs after unit testing. For more information refer http://en.wikipedia.org/wiki/Integration_testing
individual software modules are combined and tested as a group. It occurs
after unit testing. For more information refer
http://en.wikipedia.org/wiki/Integration_testing


'''BeforeClass''' - Refer to
'''5''' '''BeforeClass''' - Refer to http://junit.sourceforge.net/javadoc_40/org/junit/BeforeClass.html
http://junit.sourceforge.net/javadoc_40/org/junit/BeforeClass.html


'''parameterized testing''' - . A parameterized unit test is simply a method that takes parameters, calls the code under test, and states assertions.
'''6''' '''parameterized testing''' - . A parameterized unit test is simply a method that takes parameters, calls the code under test, and states assertions.


= References =
= References =
Line 527: Line 378:
[5]  http://www.opensourcetesting.org/unit_java.php
[5]  http://www.opensourcetesting.org/unit_java.php


[6]
[6] http://gamesfromwithin.com/exploring-the-c-unit-testing-framework-jungle
http://gamesfromwithin.com/exploring-the-c-unit-testing-framework-jungle


[7] http://en.wikipedia.org/wiki/Doctest
[7] http://en.wikipedia.org/wiki/Doctest

Latest revision as of 03:21, 29 September 2009

XUnit in general, is only one kind of testing framework for object oriented languages. This page talks about the major testing frameworks available for the major object oriented languages such as Java, C++, C#,Python, Ruby and C#. Comparisons have been made wherever possible between the major frameworks available in each language.


Java Testing Frameworks

Plenty of testing framework exist for Java. JUnit has been the de facto standard for unit testing. However, other frameworks such as TestNG, JTIGER have been built to address various faults and deficiencies with JUnit. Our primary objective is to compare the various testing frameworks that exist for object oriented languages based on the primary purpose of the framework, features, the platforms on which they operate, etc and thus help in picking up the best testing framework that fits in a particular suitable environment. This article DOES NOT introduce JUnit or XUnit. For information on JUnit and XUnit, please refer to the http://www.junit.org or [APPENDIX 1] and [APPENDIX 2]


TestNG Framework

TestNG is a testing framework which can help to meet a broad range of testing such as unit testing, integration testing, functional testing, end-to -end testing. (For definition and more information on the different kinds of testing, refer to [ APPENDIX 3] and [ APPENDIX 4]

To write a TestNG test, we need to: Write the business logic of the test, and insert the TestNG annotations in them. Add the information about our test (such as the class name, the groups we wish to run, etc) in a testing.xml file or in build.xml. Run the test. TestNG supports data driven testing.(Data-driven testing is a methodology used in Test automation where test scripts are executed and verified based on the data values stored in one or more central data sources or databases. For a detailed description on data driven testing click here: http://en.wikipedia.org/wiki/Data-driven_testing )

TestNG is invoked in several different ways: With a testing.xml file ANT Command line


It is possible to group the test methods of TestNG. We can declare methods that belong to these groups and specify groups that contain other groups. It is very flexible as it can help in choosing one set of regular expression and excluding the other sets. This splits the various test groups and there is no need to recompile again if we want to run two different tests back to back.

Requirements: JDK 1.4, 1.5 Invoked in many environments such as command line, ant, ECLIPSE, IDEA, MAVEN, etc

The annotations available in TestNG can be found at: http://testng.org/doc/documentation-main.html#annotations

TestNG is inspired from JUnit and Nunit, with additional functionalities such as Flexible test configuration, support for data driven testing, etc.[1]

Cactus

Cactus is a test framework for testing server-side java code such as Servlets, EJB, etc. It extends J Unit. Cactus implements an in-container strategy which means that tests are executed inside the container. The Cactus testing unit can be viewed as a system that consists of the following components:

--Cactus Framework: This provides the API to write Cactus tests.

--Cactus Integration Modules: They are front ends and frameworks that provide easy ways of using the Cactus Framework. Eg: Eclipse plugin

--Cactus Samples: They demonstrate how to write Cactus tests and how to use a few Integration Modules. [2]

A useful testing framework provided by Cactus is integration unit testing. The tests written for this framework will exercise the interactions with the container. Cactus also provides other frameworks such as code logic unit testing, functional unit testing.

Cactus was developed to fit only integration unit testing with the idea that it is much easier to have to write tests for a single framework than for several. It is believed that Cactus provides a middle ground that provides a high confidence that your code will run when deployed .[2]


To know how it works, click here : http://jakarta.apache.org/cactus/how_it_works.html

COMPARISON BETWEEN JUnit,TestNG and Cactus frameworks [3]

' JUnit4 TestNG Cactus
Supporting Type of Testing Suitable more for Unit testing Supports Unit testing,Functional testing, end-to-end testing, integration testing Primarily unit testing. Also have frameworks for code logic unit testing , functional logic unit testing
Conventions Rigid Flexible Not as flexible as TestNG
Derived from JUnit Yes, from the previous versions Inspired from Junit,with added functionalities Extends JUnit
Flexibility for large suites Not flexible for large test suites, Flexible for running large test suites of code and thus one test\'s failure shouldn\'t mean having to rerun a suite of thousands. No, it is suitable for unit testing.
Suitable for Best for a single object Best for a higher level testing Best for server-side java code such as Servlets, EJB, etc.
Works with Command line, ANT,ECLIPSE Command line,ANT, ECLIPSE, IDEA,MAVEN ANT

An example comparing JUNIT 4 and TestNG

JUnit 4 v/s TestNG

(source: http://www.ibm.com/developerworks/java/library/j-cq08296/)


The frameworks differ is in their core design. JUnit has always been an effective unit-testing framework, meaning that it was built to facilitate testing single objects. TestNG, on the other hand, was built to address testing at higher levels, and consequently, has some features not available in JUnit

JUnit 4 makes clever use of annotations. JUnit no longer requires you to define a test as a method whose name starts with test, and we can now run fixtures just once as opposed to for each test. However TestNG established itself as an annotations-based framework long before Junit 4. TestNG pioneered testing with annotations in Java programming, which made it a formidable alternative to JUnit. [4]


Sample J Unit 4 code

package test.com.acme.dona.dep;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import org.junit.BeforeClass;
import org.junit.Test;
public class DependencyFinderTest {
private static DependencyFinder finder;
@BeforeClass
public static void init() throws Exception {
 finder = new DependencyFinder();
}
@Test
public void verifyDependencies() 
 throws Exception {
  String targetClss = 
    "test.com.acme.dona.dep.DependencyFind";
Filter[] filtr = new Filter[] { 
     new RegexPackageFilter("java|junit|org")};
Dependency[] deps = 
     finder.findDependencies(targetClss, filtr);
assertNotNull("deps was null", deps);
  assertEquals("should be 5 large", 5, deps.length);   
 }
}
(source: http://www.ibm.com/developerworks/java/library/j-cq08296/)


Sample TestNG code

package test.com.acme.dona.dep;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertNotNull;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Configuration;
import org.testng.annotations.Test;
public class DependencyFinderTest {
private DependencyFinder finder;
@BeforeClass
private void init(){
 this.finder = new DependencyFinder();
}
@Test
public void verifyDependencies() 
 throws Exception {
  String targetClss = 
    "test.com.acme.dona.dep.DependencyFind";
  Filter[] filtr = new Filter[] { 
     new RegexPackageFilter("java|junit|org")};
  Dependency[] deps = 
     finder.findDependencies(targetClss, filtr);
  assertNotNull(deps, "deps was null" );
  assertEquals(5, deps.length, "should be 5 large");           
 }
}
(source: http://www.ibm.com/developerworks/java/library/j-cq08296/)

Both the codes may look a lot similar. However TestNG coding conventions are more flexible than JUnit 4.

JUnit forces us to declare the @BeforeClass (See [APPENDIX 5] for more information) method as static, which consequently requires us to also declare the fixture, finder, as static. We also have to declare the init() method as public. Looking at the TestNG code, you can find that such conventions aren't required. Its init() method is neither static nor public. Flexibility has been one of the strong points of TestNG right from the start. JUnit works well for a unit of code, whereas TestNG is better suited for high-level testing. Its flexibility is especially useful with large test suites.[4]

Some other JAVA testing units [5]

' JEasyTest J2ME Unit Testing Toolkit Jailer Mockrunner
Features Generates coverage report Allows virtual mock object Is a Java 2 Micro Edition (J2ME) library containing a unit testing framework for J2ME Does test data exporting. JDBC agnostic Generates DbUnit datasets, hierarchically structured XML, and topologically sorted SQL-DML. A lightweight framework for unit testing applications in the J2EE environment.
Integrates with: ECLIPSE 3.3 ANT Platform Independent OS independent

Testing frameworks for C++

When it comes to unit testing in Java, a unanimous choice would be JUnit. However for C++, there is no one clear winner. CppUnit is the most widely used tool for C++ which is a part of the XUnit family.

Following is a few basic factors to judge a good testing unit in C++ [6]

Features CppUnit BOOST.test CxxUnit
Minimal amount of work needed to add new tests No, and thus a major downfall. Yes Yes, very good.
Easy to modify and port Medium Medium Simplest
Supports setup/teardown steps Yes Best
Handles exceptions and crashes well. Yes Yes, above others Great support.
Good assert functionality. Pretty decent. Yes Best
Supports different outputs. Yes. Comes with IDE friendly output. Probably, Not trivial to change. Yes
Supports suites Yes. Yes, but with a big catch. Yes


CppUnit passes most of the above tests , except for the first point and thus needs a lot of code typing. Boost.Test isn’t exclusively a unit-testing framework, and is not X Unit based.

Out of the above three , cxxUnit turns out to be a favourite.

It comes close to the requirements of an ideal framework by leveraging the power of an external scripting language. It provides some nifty advanced features and great assert functionality. It does require the use of a scripting language as part of the build process, so those uncomfortable with that requirement, might want to look at one of the other two frameworks.

For work restricted to the PC, where modifying the framework is not required Boost.Test could be a very good choice. CppUnit , which has come a long way in years is a solid, complete framework The major drawbacks are the relative verbosity for adding new tests and fixtures. [6]

Python

Doctest: It is a simple, ingenuine technology that allows you to write a narrative in a document which can directly be executed as a test. It is included in Python as a module.

Although doctest makes testing code readable for non-programmers ,the simplicity of a narrative test like doctest is accompanied by it’s own flaws .Some of the disadvantages of using doctest are

• These tests are difficult to handle in case of failures as they are more generic. Specific tests are narrower and better in communicating intent and ensuring coverage.

• The test code is executed in a special way thus making it difficult to reason about how it will be executed. It is also harder to program around the tests.

• It is actually difficult to get an outline of the tests at glance. There’s no tool that can give an overview of the unit tests in a doc test file [7].

An detailed information about doctest with thorough examples can be read at http://docs.python.org/library/doctest.html


Unittest: Unittest are more specific tests which are more functional in nature allowing it to be written for individual functions or the class as a whole.

The fact that Unittest is based on jUnit makes people with the xUnit familiarity pick it up really quick hence it is more Java based than python .It has been introduced in the Python’s library since version 2.1

The disadvantages of using the unit test are

• The tests have a different look and feel with respect to the code under test

• The assertions used in the text use custom syntax which makes it difficult

• The test code is difficult to understand compared to the Doctest and the strong basis on jUnit makes it difficult for the pure Python coders [8]

A detailed information about doctest with thorough examples can be read at http://docs.python.org/library/unittest.html


Doctest vs Unittest

Here we compare simple features of both the tests thus giving us an idea of which one to use based on the requirement

' doctest unittest
Simplicity Best suited for novice programmers A little complex to use specially if unaware of Junit
Documentation Better Narratives Tests have little or no comments
Comprehensive testing Doctests get clutterd for comprehensive tests and thus less readable Best suited for comprehensive tests
Flexibility and Control Less flexible More flexible

Ruby

The basic testing framework for RUBY is xUnit . There are some frameworks which have evolved over the years which include RSpec, Trantula, Flog, RunCodeRun, Cucumber etc. We’ll talk about each of them in brief


Testing Framework Description
Rspec[9] It is Behavior Driven Development whose basic features are
· Behavior counts first, testing follows
· It focuses more on business values
· It’s more about getting the words right rather than just writing the test first
Advantages:
· These are extremely easy to read and makes sense to a non-programmer
· Rspec lets you unit test while also providing mocking and stubbing support
· Rspec can help with the coverage tests using Rcov
· One can also perform integration test using Rspec
Disadvantages:
· It involves a lot of behind the scene metaprogramming which makes the tests very difficult to understand
A simple Rspec example can be read at http://rspec.info/
Cucumber[10] Cucumber, the latest addition to the Rspec family allows one to execute documentation written in plain text(known as ‘stories’)
A very good example of writing a test in cucumber can be read at http://cukes.info/
Tarantula[11] Tarantula is a Rail’s plugin that produces a detailed report describing the URL’s that it has crawled across and the responses that it received from each one of them.
Tarantula\'s functionality somewhat overlaps with the on site testing, particularly if you are practicing test-driven development (TDD). Tarantula runs against your actual Web site thereby helping you to identify the problems associated with your production server, user-generated data, and HTML compliance.

A well explained example can be seen at http://www.railslodge.com/plugins/959-tarantula

Flog[12] Flog is a tool which gives you a profile of your code’s complexity. It can given the complexity pertaining to each method in your code. One can also set threshold for each method using flog
The complexity of a code can be important to
· Find the bugs as they are highly probable in complex code areas
· Complex codes can be considered for refactoring
A flog report has been explained and shown here http://www.ruby-forum.com/topic/131931

C#

The list of frameworks that apply for .NET are applied for C#. A long list of testing frameworks are available for C# which are NUnit, csUnit, Pex, dotunit, VSNUnit, Nester[13].

' Nunit csUnit PEX
Description NUnit has been ported from JUnit and has been written completely to take adavntage of the .NET languages. It is a free open source unit testing tool for all the test-driven development .NET languages including C#. The major advancements with NUnit are parameterized tests which provides a concise method to express a set of examples that can used to run individual test cases. It has been inspired from JUnit and can be used as an add-in along with the NUnit It is a tool to generate efficient unit tests with a high code coverage and are converted to C# automatically. It determines the relevant test inputs for the code hunting for boundary conditions, exceptions and assertion failures. Pex enables Parameterized Unit Testing, an extension of Unit Testing that reduces test maintenance costs.
Examples http://www.nunit.org/blogs/ http://www.csunit.org/tutorials/tutorial7/ http://research.microsoft.com/en-us/um/redmond/projects/pex/wiki/The%20Hello%20World%20of%20Pex.html
Used as Testing framework, GUI and console tool Testing Framework ,Console tool and VS.NET Add In It supports NUnit, MbUnit and xUnit.net.Extensions for these test frameworks can be downloaded from the Pex Extensions Project[14].

APPENDIX

1 JUnit - JUnit is a unit testing framework for the Java programming language. JUnit has been important in the development of test-driven development, and comprises a family of unit testing frameworks collectively known as xUnit that originated with SUnit. (For more information refer to http://www.junit.org,http://en.wikipedia.org/wiki/Junit)

2 XUnit - Various code-driven testing frameworks have come to be known collectively as xUnit. These frameworks allow testing of different elements (units) of software, such as functions and classes. (For more information refer to http://en.wikipedia.org/wiki/XUnit)

3 unit testing - unit testing is a software verification and validation method in which a programmer tests if individual units of source code are fit for use, where a unit is the smallest testable part of an application. For more information refer http://en.wikipedia.org/wiki/Unit_testing

4 integration testing - is the activity of software testing in which individual software modules are combined and tested as a group. It occurs after unit testing. For more information refer http://en.wikipedia.org/wiki/Integration_testing

5 BeforeClass - Refer to http://junit.sourceforge.net/javadoc_40/org/junit/BeforeClass.html

6 parameterized testing - . A parameterized unit test is simply a method that takes parameters, calls the code under test, and states assertions.

References

[1] http://testng.org/doc/documentation-main.html

[2] http://jakarta.apache.org/cactus/index.html

[3] http://junit.sourceforge.net/doc/cookbook/cookbook.htm

[4] http://www.ibm.com/developerworks/java/library/j-cq08296

[5] http://www.opensourcetesting.org/unit_java.php

[6] http://gamesfromwithin.com/exploring-the-c-unit-testing-framework-jungle

[7] http://en.wikipedia.org/wiki/Doctest

[8] http://en.wikipedia.org/wiki/Unit_testing

[9] http://www.elctech.com/tutorials/rspec-tutorial

[10] http://www.rubyinside.com/cucumber-the-latest-in-ruby-testing-1342.html

[11] http://www.ruby-forum.com/topic/144083

[12] http://www.brynary.com/2007/9/13/scourging-your-ruby-code-with-flog

[13] http://csharp-source.net/open-source/testing-tools

[14] http://research.microsoft.com/en-us/um/redmond/projects/pex/wiki/reference%20manual.html