CSC/ECE 517 Fall 2009/wiki1b 4 iftesting

Overview

Every software product goes though a Software Process or more commonly known as Software Development Cycle. The various stages in the cycle are:

• Planning
• Design
• Development
• Testing
• Release and Maintenance

Each stage can be further broken down into a number of several stages. This article focuses on the Testing phase. Software testing can be defined as inspection or investigation of a product to ascertain its quality and verify if its semantics are implemented the way they are intended to.

Testing can be broadly classified into the following methodologies:

• Unit testing
• Integration testing
• Functional or Validation testing
• System testing

The various testing processes/methods listed can be put in chronological order and can be viewed as a chain process. Each phase is an important part of the overall testing process. Unit test concentrates on the basic building blocks of the product, verifying if each component works correctly. The next phase is the Integration Testing. When many such units are knit together to form a larger, more complex block, it becomes necessary to test if all the smaller units of the bigger block interact properly to implement the semantics of the bigger block, as desired. The process that follows this stage is the Functional or Validation testing. This phase focuses on the behavior and performance in most cases. The question asked here is “If all the complex blocks are properly integrated, are they producing the result which they are required to?” The last phase is the System Testing. This mostly deals with interaction on the hardware-software platform and/or integration of the product on a platform, as viewed from a higher level.

Integration and Functional testing are the topics of discussion in this wiki article.

<diagram?>

Purpose

Testing is a very important phase in the software development cycle. The cycle can either follow the Waterfall model or the Spiral model [ref]. The latter mandates testing continuously as opposed to the former’s discrete phase. No software product is completely free of defects. No matter how much care the design team takes, its impossible to ensure that there are absolutely no bugs. This is where testing comes in. The goal of the testing process is to try and break the software.

Considering a large project, individual modules or units maybe developed by different team or even outsourced to different companies. Each team would have tested their respective modules or units. When, however, these modules are integrated climbing a level up, further testing maybe required to ensure that these units interact with other in a way which does not break the product. An important point in the development plan is the decision on the interface of each module. A common interface eliminates most of the bugs, but a final integration test ensures that everything is clean.

Making sure that the units or modules integrate with each other is not enough. The result produced by integrating the modules and the behavior exhibited after integration is another important aspect of the development plan. A software is designed with a certain behavior and functionality in mind. Functional testing deals with testing the product for functional issues. Its goal is to ensure that the product is doing what it was intended to do in the design phase.

Testing Methodologies

Integration Testing

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:

Top-down testing

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results.

<diagram>

Bottom-up testing

Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another.

<diagram>

Big Bang approach

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly. <add?>

Regression testing

Testing is a continuous process in the Spiral model and in fact, the same is encouraged in the Waterfall model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

---

Functional testing

Functional testing maybe thought as a part of the Quality Control program. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test. [1]

A lot of people are confused between Functional and Unit testing. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the Design specification so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [2]

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

A common testing strategy employed for functional testing is the Black box testing. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Black box. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex. <add?>

<diagram>

A popular way of functional test tracking is by using bug-tracking systems. The most popular open-source BTS is BugZilla [3] developed and used my Mozilla. Another example of a BTS is Launchpad. Popular projects like Ubuntu GNU/Linux, Nautilus make use of Launchpad [4] for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

Testing tools

A number of testing tools and suites are available today. [5] Some of them are:

Eclipse TPTP Enterprise Web Test GNU/Linux Desktop Testing Project XML Test Suite Valgrind WebLOAD Apache JMeter QEngine

References

1. http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
2. https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
3. http://docs.joomla.org/Functional_Testing
4. http://www.opensourcetesting.org/
5. Pressman, Roger S. "Software Engineering: A Practitioner's Approach", Fifth Ed.
6. http://www.bugzilla.org/
7. https://launchpad.net/