CSC/ECE 517 Fall 2009/wiki1b12 AV

XUnit in general, is only one kind of testing framework for object oriented languages. This page talks about the major testing frameworks available for the major object oriented languages such as Java, C++, C#,Python, Ruby and C#. Comparisons have been made wherever possible between the major frameworks available in each language.

Java Testing Frameworks

Plenty of testing framework exist for Java. JUnit has been the de facto standard for unit testing. However, other frameworks such as TestNG, JTIGER have been built to address various faults and deficiencies with JUnit. Our primary objective is to compare the various testing frameworks that exist for object oriented languages based on the primary purpose of the framework, features, the platforms on which they operate, etc and thus help in picking up the best testing framework that fits in a particular suitable environment. This article DOES NOT introduce JUnit or XUnit. For information on JUnit and XUnit, please refer to the http://www.junit.org or [APPENDIX 1] and [APPENDIX 2]

TestNG Framework

TestNG is a testing framework which can help to meet a broad range of testing such as unit testing, integration testing, functional testing, end-to -end testing. (For definition and more information on the different kinds of testing, refer to [ APPENDIX 3] and [ APPENDIX 4]

To write a TestNG test, we need to: Write the business logic of the test, and insert the TestNG annotations in them. Add the information about our test (such as the class name, the groups we wish to run, etc) in a testing.xml file or in build.xml. Run the test. TestNG supports data driven testing.(Data-driven testing is a methodology used in Test automation where test scripts are executed and verified based on the data values stored in one or more central data sources or databases. For a detailed description on data driven testing click here: http://en.wikipedia.org/wiki/Data-driven_testing )

TestNG is invoked in several different ways: With a testing.xml file ANT Command line

It is possible to group the test methods of TestNG. We can declare methods that belong to these groups and specify groups that contain other groups. It is very flexible as it can help in choosing one set of regular expression and excluding the other sets. This splits the various test groups and there is no need to recompile again if we want to run two different tests back to back.

Requirements: JDK 1.4, 1.5 Invoked in many environments such as command line, ant, ECLIPSE, IDEA, MAVEN, etc

The annotations available in TestNG can be found at: http://testng.org/doc/documentation-main.html#annotations

TestNG is inspired from JUnit and Nunit, with additional functionalities such as Flexible test configuration, support for data driven testing, etc.[1]

Cactus

Cactus is a test framework for testing server-side java code such as Servlets, EJB, etc. It extends J Unit. Cactus implements an in-container strategy which means that tests are executed inside the container. The Cactus testing unit can be viewed as a system that consists of the following components:

--Cactus Framework: This provides the API to write Cactus tests.

--Cactus Integration Modules: They are front ends and frameworks that provide easy ways of using the Cactus Framework. Eg: Eclipse plugin

--Cactus Samples: They demonstrate how to write Cactus tests and how to use a few Integration Modules. [2]

A useful testing framework provided by Cactus is integration unit testing. The tests written for this framework will exercise the interactions with the container. Cactus also provides other frameworks such as code logic unit testing, functional unit testing.

Cactus was developed to fit only integration unit testing with the idea that it is much easier to have to write tests for a single framework than for several. It is believed that Cactus provides a middle ground that provides a high confidence that your code will run when deployed .[2]

To know how it works, click here : http://jakarta.apache.org/cactus/how_it_works.html

COMPARISON BETWEEN JUnit,TestNG and Cactus frameworks [3]

'	JUnit4	TestNG	Cactus
Supporting Type of Testing	Suitable more for Unit testing	Supports Unit testing,Functional testing, end-to-end testing, integration testing	Primarily unit testing. Also have frameworks for code logic unit testing , functional logic unit testing
Conventions	Rigid	Flexible	Not as flexible as TestNG
Derived from JUnit	Yes, from the previous versions	Inspired from Junit,with added functionalities	Extends JUnit
Flexibility for large suites	Not flexible for large test suites,	Flexible for running large test suites of code and thus one test\'s failure shouldn\'t mean having to rerun a suite of thousands.	No, it is suitable for unit testing.
Suitable for	Best for a single object	Best for a higher level testing	Best for server-side java code such as Servlets, EJB, etc.
Works with	Command line, ANT,ECLIPSE	Command line,ANT, ECLIPSE, IDEA,MAVEN	ANT

An example comparing JUNIT 4 and TestNG

JUnit 4 v/s TestNG

(source: http://www.ibm.com/developerworks/java/library/j-cq08296/)

The frameworks differ is in their core design. JUnit has always been an effective unit-testing framework, meaning that it was built to facilitate testing single objects. TestNG, on the other hand, was built to address testing at higher levels, and consequently, has some features not available in JUnit

JUnit 4 makes clever use of annotations. JUnit no longer requires you to define a test as a method whose name starts with test, and we can now run fixtures just once as opposed to for each test. However TestNG established itself as an annotations-based framework long before Junit 4. TestNG pioneered testing with annotations in Java programming, which made it a formidable alternative to JUnit. [4]

Sample J Unit 4 code

package test.com.acme.dona.dep;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import org.junit.BeforeClass;
import org.junit.Test;
public class DependencyFinderTest {
private static DependencyFinder finder;
@BeforeClass
public static void init() throws Exception {
 finder = new DependencyFinder();
}
@Test
public void verifyDependencies() 
 throws Exception {
  String targetClss = 
    "test.com.acme.dona.dep.DependencyFind";
Filter[] filtr = new Filter[] { 
     new RegexPackageFilter("java|junit|org")};
Dependency[] deps = 
     finder.findDependencies(targetClss, filtr);
assertNotNull("deps was null", deps);
  assertEquals("should be 5 large", 5, deps.length);   
 }
}
(source: http://www.ibm.com/developerworks/java/library/j-cq08296/)

Sample TestNG code

package test.com.acme.dona.dep;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertNotNull;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Configuration;
import org.testng.annotations.Test;
public class DependencyFinderTest {
private DependencyFinder finder;
@BeforeClass
private void init(){
 this.finder = new DependencyFinder();
}
@Test
public void verifyDependencies() 
 throws Exception {
  String targetClss = 
    "test.com.acme.dona.dep.DependencyFind";
  Filter[] filtr = new Filter[] { 
     new RegexPackageFilter("java|junit|org")};
  Dependency[] deps = 
     finder.findDependencies(targetClss, filtr);
  assertNotNull(deps, "deps was null" );
  assertEquals(5, deps.length, "should be 5 large");           
 }
}
(source: http://www.ibm.com/developerworks/java/library/j-cq08296/)

Both the codes may look a lot similar. However TestNG coding conventions are more flexible than JUnit 4.

JUnit forces us to declare the @BeforeClass (See [APPENDIX 5] for more information) method as static, which consequently requires us to also declare the fixture, finder, as static. We also have to declare the init() method as public. Looking at the TestNG code, you can find that such conventions aren't required. Its init() method is neither static nor public. Flexibility has been one of the strong points of TestNG right from the start. JUnit works well for a unit of code, whereas TestNG is better suited for high-level testing. Its flexibility is especially useful with large test suites.[4]

Some other JAVA testing units [5]

'	JEasyTest	J2ME Unit Testing Toolkit	Jailer	Mockrunner
Features	Generates coverage report Allows virtual mock object	Is a Java 2 Micro Edition (J2ME) library containing a unit testing framework for J2ME	Does test data exporting. JDBC agnostic Generates DbUnit datasets, hierarchically structured XML, and topologically sorted SQL-DML.	A lightweight framework for unit testing applications in the J2EE environment.
Integrates with:	ECLIPSE 3.3	ANT	Platform Independent	OS independent

Testing frameworks for C++

When it comes to unit testing in Java, a unanimous choice would be JUnit. However for C++, there is no one clear winner. CppUnit is the most widely used tool for C++ which is a part of the XUnit family.

Following is a few basic factors to judge a good testing unit in C++ [6]

Features	CppUnit	BOOST.test	CxxUnit
Minimal amount of work needed to add new tests	No, and thus a major downfall.	Yes	Yes, very good.
Easy to modify and port	Medium	Medium	Simplest
Supports setup/teardown steps	Yes		Best
Handles exceptions and crashes well.	Yes	Yes, above others	Great support.
Good assert functionality.	Pretty decent.	Yes	Best
Supports different outputs.	Yes. Comes with IDE friendly output.	Probably, Not trivial to change.	Yes
Supports suites	Yes.	Yes, but with a big catch.	Yes

CppUnit passes most of the above tests , except for the first point and thus needs a lot of code typing. Boost.Test isn’t exclusively a unit-testing framework, and is not X Unit based.

Out of the above three , cxxUnit turns out to be a favourite.

It comes close to the requirements of an ideal framework by leveraging the power of an external scripting language. It provides some nifty advanced features and great assert functionality. It does require the use of a scripting language as part of the build process, so those uncomfortable with that requirement, might want to look at one of the other two frameworks.

For work restricted to the PC, where modifying the framework is not required Boost.Test could be a very good choice. CppUnit , which has come a long way in years is a solid, complete framework The major drawbacks are the relative verbosity for adding new tests and fixtures. [6]

Python

Doctest: It is a simple, ingenuine technology that allows you to write a narrative in a document which can directly be executed as a test. It is included in Python as a module.

Although doctest makes testing code readable for non-programmers ,the simplicity of a narrative test like doctest is accompanied by it’s own flaws .Some of the disadvantages of using doctest are

• These tests are difficult to handle in case of failures as they are more generic. Specific tests are narrower and better in communicating intent and ensuring coverage.

• The test code is executed in a special way thus making it difficult to reason about how it will be executed. It is also harder to program around the tests.

• It is actually difficult to get an outline of the tests at glance. There’s no tool that can give an overview of the unit tests in a doc test file [7].

An detailed information about doctest with thorough examples can be read at http://docs.python.org/library/doctest.html

Unittest: Unittest are more specific tests which are more functional in nature allowing it to be written for individual functions or the class as a whole.

The fact that Unittest is based on jUnit makes people with the xUnit familiarity pick it up really quick hence it is more Java based than python .It has been introduced in the Python’s library since version 2.1

The disadvantages of using the unit test are

• The tests have a different look and feel with respect to the code under test

• The assertions used in the text use custom syntax which makes it difficult

• The test code is difficult to understand compared to the Doctest and the strong basis on jUnit makes it difficult for the pure Python coders [8]

A detailed information about doctest with thorough examples can be read at http://docs.python.org/library/unittest.html

Doctest vs Unittest

Here we compare simple features of both the tests thus giving us an idea of which one to use based on the requirement

'	doctest	unittest
Simplicity	Best suited for novice programmers	A little complex to use specially if unaware of Junit
Documentation	Better Narratives	Tests have little or no comments
Comprehensive testing	Doctests get clutterd for comprehensive tests and thus less readable	Best suited for comprehensive tests
Flexibility and Control	Less flexible	More flexible

Ruby

The basic testing framework for RUBY is xUnit . There are some frameworks which have evolved over the years which include RSpec, Trantula, Flog, RunCodeRun, Cucumber etc. We’ll talk about each of them in brief

Testing Framework	Description
Rspec[9]	It is Behavior Driven Development whose basic features are
	· Behavior counts first, testing follows
	· It focuses more on business values
	· It’s more about getting the words right rather than just writing the test first
	Advantages:
	· These are extremely easy to read and makes sense to a non-programmer
	· Rspec lets you unit test while also providing mocking and stubbing support
	· Rspec can help with the coverage tests using Rcov
	· One can also perform integration test using Rspec
	Disadvantages:
	· It involves a lot of behind the scene metaprogramming which makes the tests very difficult to understand
	A simple Rspec example can be read at http://rspec.info/
Cucumber[10]	Cucumber, the latest addition to the Rspec family allows one to execute documentation written in plain text(known as ‘stories’)
	A very good example of writing a test in cucumber can be read at http://cukes.info/
Tarantula[11]	Tarantula is a Rail’s plugin that produces a detailed report describing the URL’s that it has crawled across and the responses that it received from each one of them.
	Tarantula\'s functionality somewhat overlaps with the on site testing, particularly if you are practicing test-driven development (TDD). Tarantula runs against your actual Web site thereby helping you to identify the problems associated with your production server, user-generated data, and HTML compliance. A well explained example can be seen at http://www.railslodge.com/plugins/959-tarantula
Flog[12]	Flog is a tool which gives you a profile of your code’s complexity. It can given the complexity pertaining to each method in your code. One can also set threshold for each method using flog
	The complexity of a code can be important to
	· Find the bugs as they are highly probable in complex code areas
	· Complex codes can be considered for refactoring
	A flog report has been explained and shown here http://www.ruby-forum.com/topic/131931

C#

The list of frameworks that apply for .NET are applied for C#. A long list of testing frameworks are available for C# which are NUnit, csUnit, Pex, dotunit, VSNUnit, Nester[13].

'	Nunit	csUnit	PEX
Description	NUnit has been ported from JUnit and has been written completely to take adavntage of the .NET languages. It is a free open source unit testing tool for all the test-driven development .NET languages including C#.	The major advancements with NUnit are parameterized tests which provides a concise method to express a set of examples that can used to run individual test cases. It has been inspired from JUnit and can be used as an add-in along with the NUnit	It is a tool to generate efficient unit tests with a high code coverage and are converted to C# automatically. It determines the relevant test inputs for the code hunting for boundary conditions, exceptions and assertion failures. Pex enables Parameterized Unit Testing, an extension of Unit Testing that reduces test maintenance costs.
Examples	http://www.nunit.org/blogs/	http://www.csunit.org/tutorials/tutorial7/	http://research.microsoft.com/en-us/um/redmond/projects/pex/wiki/The%20Hello%20World%20of%20Pex.html
Used as	Testing framework, GUI and console tool	Testing Framework ,Console tool and VS.NET Add In	It supports NUnit, MbUnit and xUnit.net.Extensions for these test frameworks can be downloaded from the Pex Extensions Project[14].

APPENDIX

1 JUnit - JUnit is a unit testing framework for the Java programming language. JUnit has been important in the development of test-driven development, and comprises a family of unit testing frameworks collectively known as xUnit that originated with SUnit. (For more information refer to http://www.junit.org,http://en.wikipedia.org/wiki/Junit)

2 XUnit - Various code-driven testing frameworks have come to be known collectively as xUnit. These frameworks allow testing of different elements (units) of software, such as functions and classes. (For more information refer to http://en.wikipedia.org/wiki/XUnit)

3 unit testing - unit testing is a software verification and validation method in which a programmer tests if individual units of source code are fit for use, where a unit is the smallest testable part of an application. For more information refer http://en.wikipedia.org/wiki/Unit_testing

4 integration testing - is the activity of software testing in which individual software modules are combined and tested as a group. It occurs after unit testing. For more information refer http://en.wikipedia.org/wiki/Integration_testing

5 BeforeClass - Refer to http://junit.sourceforge.net/javadoc_40/org/junit/BeforeClass.html

6 parameterized testing - . A parameterized unit test is simply a method that takes parameters, calls the code under test, and states assertions.

References

[1] http://testng.org/doc/documentation-main.html

[2] http://jakarta.apache.org/cactus/index.html

[3] http://junit.sourceforge.net/doc/cookbook/cookbook.htm

[4] http://www.ibm.com/developerworks/java/library/j-cq08296

[5] http://www.opensourcetesting.org/unit_java.php

[6] http://gamesfromwithin.com/exploring-the-c-unit-testing-framework-jungle

[7] http://en.wikipedia.org/wiki/Doctest

[8] http://en.wikipedia.org/wiki/Unit_testing

[9] http://www.elctech.com/tutorials/rspec-tutorial

[10] http://www.rubyinside.com/cucumber-the-latest-in-ruby-testing-1342.html

[11] http://www.ruby-forum.com/topic/144083

[12] http://www.brynary.com/2007/9/13/scourging-your-ruby-code-with-flog

[13] http://csharp-source.net/open-source/testing-tools

[14] http://research.microsoft.com/en-us/um/redmond/projects/pex/wiki/reference%20manual.html