CSC/ECE 517 Fall 2009/wiki3 15 assertions: Difference between revisions

Revision as of 23:52, 10 November 2009

Programming with assertions

Assertions are predicates or statements with expressions which the programmer thinks should be true at the time of execution. Assertions are a powerful way of detecting bugs in programs.

Overview

Every software product goes though a Software Process or more commonly known as Software Development Cycle. The various stages in the cycle are:

Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 *Fifth Ed.*

Planning
Design
Development
Testing
Release and Maintenance

It is impossible for a programmer to make any program a hundred percent bug free. Even worse, there maybe subtle things which the programmer assumes are always true but may not be so in many corner cases. Debugging is an integral part of the Software Process. Bugs can crop up anytime due to various test cases formulated all through these aforementioned stages. To be sure that whatever the programmer assumes is true, assertions can be set up in the program. If ever, the condition evaluates to be false, the program terminates with an assertion. Another way to put it is that assertions are a way to show the programmer that things cannot be taken for granted by the latter. Assertions can be thought of as medium of validating the correctness of modules.

Purpose

The various uses of assertions are:

Validating correctness of (sub)program
Supports programming by contract (documentation)
Helps in debugging
Exception handling by integrating assertion failures to be detected dynamically

Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught.

Testing Methodologies

Integration Testing and beyond

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:

Top-down approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 *Fifth Ed.*

Top-down Testing

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results^[1].

Bottom-up Testing

Bottom-up approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 *Fifth Ed.*

Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another^[1].

Big Bang Approach

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[2]

Regression Testing

Testing is a continuous process in the Spiral model and in fact, the same is encouraged in the Waterfall model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[3]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

Functional Testing and beyond

Functional testing maybe thought as a part of the Quality Control program. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [4]. Sometime Functional testing is also called as System Testing.

A lot of people are confused between Functional and Unit testing. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the Design Specification so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [5]

Functional Decomposition and Blackbox testing

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

Black-box testing. Image courtesy: Latvian Technological Center[1]

A common testing strategy employed for functional testing is the Blackbox testing. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

Bug Tracking Systems

A popular way of functional test tracking is by using Bug-Tracking Systems. A popular open-source BTS is BugZilla [6] developed and used my Mozilla. Another example of a BTS is Launchpad [7]. Popular projects like Ubuntu GNU/Linux, Nautilus etc. make use of Launchpad for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

Testing Tools

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

More opensource tools may be found here.

Summary

Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

@@ Line 1: / Line 1: @@
-== Integration and Function Testing and beyond ==
+== Programming with assertions ==
-Software testing is the science of breaking software down. During testing we have one and only one aim to develop comprehensive test cases to find bugs in the software and get them fixed.
+Assertions are predicates or statements with expressions which the programmer thinks should be true at the time of execution. Assertions are a powerful way of detecting bugs in programs.
 == Overview ==
@@ Line 11: / Line 11: @@
 * '''Release and Maintenance'''
-Each stage can be further broken down into a number of several stages. This article focuses on the Testing phase. Software testing can be defined as inspection or investigation of a product to ascertain its quality and verify if its semantics are implemented the way they are intended to. Testing can however, be done outside the Software Development cycle too.
+It is impossible for a programmer to make any program a hundred percent bug free. Even worse, there maybe subtle things which the programmer assumes are always true but may not be so in many corner cases. Debugging is an integral part of the Software Process. Bugs can crop up anytime due to various test cases formulated all through these aforementioned stages. To be sure that whatever the programmer assumes is true, assertions can be set up in the program. If ever, the condition evaluates to be false, the program terminates with an assertion. Another way to put it is that assertions are a way to show the programmer that things cannot be taken for granted by the latter. Assertions can be thought of as medium of validating the correctness of modules.
-Testing can be broadly classified into the following methodologies <sup>[1]</sup>:
-* '''Unit testing'''
-* '''Integration testing'''
-* '''Functional or Validation testing'''
-* '''System testing'''
-The various testing processes/methods listed can be put in chronological order and can be viewed as a chain process. Each phase is an important part of the overall testing process. Unit test concentrates on the basic building blocks of the product, verifying if each component works correctly. The next phase is the Integration Testing. When many such units are knit together to form a larger, more complex block, it becomes necessary to test if all the smaller units of the bigger block interact properly to implement the semantics of the bigger block, as desired. The process that follows this stage is the Functional or Validation testing. This phase focuses on the behavior and performance in most cases. The question asked here is “If all the complex blocks are properly integrated, are they producing the result which they are required to?” The last phase is the System Testing. This mostly deals with interaction on the hardware-software platform and/or integration of the product on a platform, as viewed from a higher level. Here we will focus our discussion on Integration and Functional Testing.
 == Purpose ==
-Testing is a very important phase in the software development cycle. The cycle can either follow the ''Waterfall'' [http://en.wikipedia.org/wiki/Waterfall_model] model or the ''Spiral'' model [http://en.wikipedia.org/wiki/Spiral_model]. The latter mandates testing continuously as opposed to the former’s discrete phase. No software product is completely free of defects. No matter how much care the design team takes, its impossible to ensure that there are absolutely no bugs. This is where testing comes in. The goal of the testing process is to try and break the software.
+The various uses of assertions are:
+* '''Validating correctness of (sub)program'''
-Considering a large project, individual modules or units maybe developed by different team or even outsourced to different companies. Each team would have tested their respective modules or units. When, however, these modules are integrated climbing a level up, further testing maybe required to ensure that these units interact with other in a way which does not break the product. An important point in the development plan is the decision on the interface of each module. A common interface eliminates most of the bugs, but a final integration test ensures that everything is clean.
+* '''Supports programming by contract (documentation)'''
+* '''Helps in debugging'''
+* '''Exception handling by integrating assertion failures to be detected dynamically'''
-Making sure that the units or modules integrate with each other is not enough. The result produced by integrating the modules and the behavior exhibited after integration is another important aspect of the development plan. A software is designed with a certain behavior and functionality in mind. Functional testing deals with testing the product for functional issues. Its goal is to ensure that the product is doing what it was intended to do in the design phase.
+Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught.
 == Testing Methodologies ==

CSC/ECE 517 Fall 2009/wiki3 15 assertions: Difference between revisions

Revision as of 23:52, 10 November 2009

Contents

Programming with assertions

Overview

Purpose