CSC/ECE 517 Fall 2009/wiki3 15 assertions
Programming with assertions
Assertions are predicates or statements with expressions which the programmer thinks should be true at the time of execution. Assertions are a powerful way of detecting bugs in programs.
Overview
Every software product goes though a Software Process or more commonly known as Software Development Cycle. The various stages in the cycle are:
- Planning
- Design
- Development
- Testing
- Release and Maintenance
It is impossible for a programmer to make any program a hundred percent bug free. Even worse, there maybe subtle things which the programmer assumes are always true but may not be so in many corner cases. Debugging is an integral part of the Software Process. Bugs can crop up anytime due to various test cases formulated all through these aforementioned stages. To be sure that whatever the programmer assumes is true, assertions can be set up in the program. If ever, the condition evaluates to be false, the program terminates with an assertion. Another way to put it is that assertions are a way to show the programmer that things cannot be taken for granted by the latter. Assertions can be thought of as medium of validating the correctness of modules.
Purpose
The various uses of assertions are:
- Validating correctness of (sub)program
- Supports programming by contract (documentation)
- Helps in debugging
- Exception handling by integrating assertion failures to be detected dynamically
Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught. They are also a very useful tool to detect logical errors. For e.g consider the following code:
switch(scase) {
case 1: ....
case 2: ....
default: assert false;
}
This code fragment clearly says that the programmer assumes that the variable scase will take either of the listed values. If the variables reaches the default segment, then there is a logical error in the program. Numerous other run-time checks can be made. The following example illustrates that:
if (threads % 4 == 0) {....}
else if (threads % 4 == 1) {....}
else if (threads % 4 == 2) {....}
else {
// we know that it has to be 3
assert(threads % 4 == 3);
....
}
This method is called as asserting an invariant.
Types of assertions
Assertions can be categorized primarily as follows:
- Pre-conditions
- Post-conditions
- Class invariants
- Invariants (Control-flow, Loop etc.)
- Loop variants
As Dr.Gehringer mentions, the first three classes are discussed in class and will not be a focus of this wiki. Instead, this will concentrate on the other types of assertions, viz., Control flow, Loop invariants and Loop variants. Invariants are those conditions with a precondition as well as post condition. A loop invariant is an expression for a variable which remains constant through-out the loop. On the other hand, control-flow invariants are those cases which specify that the control should never reach to that part of code. Consider the following code:
for (....)
if (...)
return x;
else if (...)
return y;
/* Control should never reach here */
Now observe how control-flow invariance can be validated by assertions:
for (....)
if (...)
return x;
else if (...)
return y;
assert false
Integration Testing and beyond
Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
Top-down Testing
This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.
Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.
A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].
Bottom-up Testing
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].
Big Bang Approach
This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[2]
Regression Testing
Testing is a continuous process in the Spiral model and in fact, the same is encouraged in the Waterfall model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[3]
Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.
There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.
Functional Testing and beyond
Functional testing maybe thought as a part of the Quality Control program. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [4]. Sometime Functional testing is also called as System Testing.
A lot of people are confused between Functional and Unit testing. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.
During functional testing it is required that the testing team have the Design Specification so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [5]
Functional Decomposition and Blackbox testing
A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.
A common testing strategy employed for functional testing is the Blackbox testing. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.
Bug Tracking Systems
A popular way of functional test tracking is by using Bug-Tracking Systems. A popular open-source BTS is BugZilla [6] developed and used my Mozilla. Another example of a BTS is Launchpad [7]. Popular projects like Ubuntu GNU/Linux, Nautilus etc. make use of Launchpad for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.
Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.
Testing Tools
A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:
- Eclipse TPTP
- Enterprise Web Test
- GNU/Linux Desktop Testing Project
- XML Test Suite
- Valgrind
- WebLOAD
- Apache JMeter
- QEngine
More opensource tools may be found here.
Summary
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.
References
- R. S. Pressman. Software Engineering "A Practitioner Approach" New York McGraw-Hill, 2001, p386-p429.
- http://en.wikipedia.org/wiki/Waterfall_model
- http://en.wikipedia.org/wiki/Spiral_model
- http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
- http://www.cs.umd.edu/~aporter/html/currTesting.html
- https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
- http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
- http://www.bugzilla.org/
- https://launchpad.net/
- http://docs.joomla.org/Functional_Testing
- http://www.opensourcetesting.org/