CSC/ECE 517 Fall 2009/wiki3 15 assertions: Difference between revisions

From Expertiza_Wiki
Jump to navigation Jump to search
mNo edit summary
 
(12 intermediate revisions by the same user not shown)
Line 16: Line 16:


The various uses of assertions are:
The various uses of assertions are:
* '''Validating correctness of (sub)program'''
* '''Validates correctness of (sub)program'''
* '''Supports programming by contract (documentation)'''
* '''Supports programming by contract (documentation)'''
* '''Helps in debugging'''
* '''Helps in debugging'''
* '''Exception handling by integrating assertion failures to be detected dynamically'''
* '''Exception handling by integrating assertion failures to be detected dynamically'''


Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught. They are also a very useful tool to detect logical errors. For e.g consider the following code:
Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught.  
 
== Types of assertions ==
 
Assertions can be categorized primarily as follows:
 
*''Pre-conditions''
*''Post-conditions''
*''Class invariants''
*''Invariants (Control-flow, Loop etc.)''
*''Loop variants''
 
As Dr.Gehringer mentions, the first three classes are discussed in class and will not be a focus of this wiki. Instead, this will concentrate on the other types of assertions, viz., Control flow, Loop invariants and Loop variants<sup>[1]</sup>. Invariants are those conditions with a precondition as well as post condition. A loop invariant is an expression for a variable which remains constant through-out the loop. On the other hand, control-flow invariants are those cases which specify that the control should never reach to that part of code. Consider the following code<sup>[1]</sup>:
 
<code>
for (....) {
  if (...)
      return x;
  else if (...)
      return y;
 
  /* Control should never reach here */
}
</code>
 
Now observe how control-flow invariance can be validated by assertions:
 
<code>
for (....) {
  if (...)
      return x;
  else if (...)
      return y;
 
  assert false
}
</code>
 
They are also a very useful tool to detect logical errors. For e.g consider the following code<sup>[1]</sup>:


<code>
<code>
Line 33: Line 71:
</code>
</code>


This code fragment clearly says that the programmer assumes that the variable ''scase'' will take either of the listed values. If the variables reaches the ''default'' segment, then there is a logical error in the program. Numerous other run-time checks can be made. The following example illustrates that:
This code fragment clearly says that the programmer assumes that the variable ''scase'' will take either of the listed values. If the variables reaches the ''default'' segment, then there is a logical error in the program.  
 
Numerous other run-time checks can be made. The following example illustrates that<sup>[1]</sup>:


<code>
<code>
if (threads % 4 == 0)
if (threads % 4 == 0) {....}
    ....


else if (threads % 4 == 1)
else if (threads % 4 == 1) {....}
    ....


else if (threads % 4 == 2)
else if (threads % 4 == 2) {....}
    ....


else
else {
   // we know that it has to be 3
   // we know that it has to be 3
   assert(threads % 4 == 3);
   assert(threads % 4 == 3);
   ....
   ....
}
</code>
</code>


This method is called as asserting an invariant.
This method is called as asserting an Internal invariant. The latter means that even though its obvious that the result would be 3, we eliminate any logical errors by asserting that.
 
== Types of assertions ==
 
Assertions can be categorized primarily as follows:


*''Pre-conditions''
== Programming languages supporting native assertions ==
*''Post-conditions''
*''Class invariants'
*''Loop invariants''
*''Loop variants''


=== Integration Testing and beyond ===
The following programming languages support assertions natively<sup>[2]</sup>:


Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
* Cobra
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* D
==== Top-down Testing ====
* Eiffel
* Fortress
* Lisaac
* Nice
* Oxygene (formerly Chrome)
* Sather
* SPARK, via static analysis of Ada programs
* Spec#


This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.
== Advantages and disadvantages of assertions ==


Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.  
Some programming languages natively support assertions (mentioned above) while some support through add-ins (like C/C++, C#, Java, Javascript etc.). However, most of the above listed languages are high level languages with less popularity. Hence the need for supporting assertions in the commonly used languages is important.  


A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results<sup>[1]</sup>.
=== Advantages ===


==== Bottom-up Testing ====
There are many advantages of assertions. Some of them are enumerated here<sup>[3]</sup>:
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another<sup>[1]</sup>.


==== Big Bang Approach ====
* they remove the need for ''passive assertions'' (in the form of comments) and instead actively assert the condition and fail if false during run-time: comments may outlive the code and may not hold for the new code. They act like passive assertions, so its the programmer who has to take care of the condition. In case of ''assert'' statements, they enforce the conditions and never outlive the code. If the code changes, the assert statement has to change too, for the program to execute. They thus act as ''active'' snippets of comments.


This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]
* they make you more confident of your code: The programmer makes a certain assumptions during the coding phase. Assertions validate the programmer's assumptions. If any of them turns out to be false, code without assertions may execute and show unexpected behavior. ''Assertions'' avoid such situations and make the code less buggy.


==== Regression Testing ====
* they help refactor the code: Whenever somebody's refactoring the code, leaving the assertions in place (or modifying them accordingly) ensures that the logic is not flawed and the refactor was actually done (without altering the logic). It is a way to validate the process of refactoring. The same hold true for optimization. Assertions ensure that while in the process, the program logic was not altered.


Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]
*  they act as embedded tests: A very good example is the development of Program 1: MeToo. A lot of assertions were used in the unit tests. Assertions are like small tests embedded within the code. They simplify the testing of the code.


Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.
* they help in debugging: When a program has assertions, it is easy to narrow down the error since the programmer now knows that if there is an assertion error then the problem must be somewhere in the vicinity of the assertion. They thus help in efficient debugging.


There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.  
* they support Programming by Contract: Assertions help establish a relationship between a class and its consumers. It defines what rights does an element have and what is it supposed or capable of doing.


----
=== Disadvantages ===


=== Functional Testing and beyond ===
There are some disadvantages to using assertions too. However, the advantages outnumber the disadvantages. Some of the disadvantages are listed below:


Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.
* the biggest disadvantage is probably the speed: Code with a lot of assertions is slower than one without assertions. However, it can be argued that during development its the accuracy and correctness that is utmost important and thus speed can be sacrificed at the cost of correctness/accuracy<sup>[3]</sup>.


A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.  
* assertions can be tricky to write: Unless the programmer puts down his/her assumptions in the form of assertions accurately, they serve no purpose.  


During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]
* assertions can cause dependency problems: A class may be heavily dependent on every assert class in it. Thus extension maybe a problem


==== Functional Decomposition and Blackbox testing ====
* stimulus must make use of assertions: Extensive and exhaustive testing is still required. The test-suite (or the stimulus) must make use of all the assertions for them to show their benefit. If the stimulus never tests some special corner cases, those assertions will never fire up to trigger an error<sup>[5]</sup>.


A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.
== When ''NOT'' to use assertions ==


[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.
If used incorrectly assertions can be a bane and can significantly alter the program flow<sup>[4]</sup>. Consider the following code:


==== Bug Tracking Systems ====
<code>


A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.
assert((var = a*3*0.45*i) != 0);
var1 = var2/var;


Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.
</code>


== Testing Tools ==
The problem with the above code is that the expression is evaluated inside the assert statement. Most languages have a feature to disable assertions. What if assertions are disabled? ''var'' would never be correctly evaluated and some garbage value maybe passed on to the next statement. If ''var'' is passed on as '0', then the program would generate a 'Divide by zero' exception. A fix to that problem would be to separate the expression evaluation and then assert. The following code shows the fix:


A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:
<code>


* [http://www.eclipse.org/tptp Eclipse TPTP]
var = (a*3*0.45*i);
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
assert(var != 0);
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
var1 = var2/var;
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]


More opensource tools may be found [http://www.opensourcetesting.org here].
</code>


== Summary ==
Thus one must be careful when using the assert statements.
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.


== References ==
== References ==


# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering "A Practitioner Approach" New York McGraw-Hill, 2001, p386-p429.]
# http://java.sun.com/j2se/1.4.2/docs/guide/lang/assert.html
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Design_by_contract
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.stanford.edu/~pgbovine/programming-with-asserts.htm
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://courses.csail.mit.edu/6.170/old-www/2002-Fall/lectures/lecture-10.pdf
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# http://www.esperan.com/pdf/esperan_introduction_to_psl.pdf
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

Latest revision as of 00:11, 13 November 2009

Programming with assertions

Assertions are predicates or statements with expressions which the programmer thinks should be true at the time of execution. Assertions are a powerful way of detecting bugs in programs.

Overview

Every software product goes though a Software Process or more commonly known as Software Development Cycle. The various stages in the cycle are:

Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 Fifth Ed.
  • Planning
  • Design
  • Development
  • Testing
  • Release and Maintenance

It is impossible for a programmer to make any program a hundred percent bug free. Even worse, there maybe subtle things which the programmer assumes are always true but may not be so in many corner cases. Debugging is an integral part of the Software Process. Bugs can crop up anytime due to various test cases formulated all through these aforementioned stages. To be sure that whatever the programmer assumes is true, assertions can be set up in the program. If ever, the condition evaluates to be false, the program terminates with an assertion. Another way to put it is that assertions are a way to show the programmer that things cannot be taken for granted by the latter. Assertions can be thought of as medium of validating the correctness of modules.

Purpose

The various uses of assertions are:

  • Validates correctness of (sub)program
  • Supports programming by contract (documentation)
  • Helps in debugging
  • Exception handling by integrating assertion failures to be detected dynamically

Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught.

Types of assertions

Assertions can be categorized primarily as follows:

  • Pre-conditions
  • Post-conditions
  • Class invariants
  • Invariants (Control-flow, Loop etc.)
  • Loop variants

As Dr.Gehringer mentions, the first three classes are discussed in class and will not be a focus of this wiki. Instead, this will concentrate on the other types of assertions, viz., Control flow, Loop invariants and Loop variants[1]. Invariants are those conditions with a precondition as well as post condition. A loop invariant is an expression for a variable which remains constant through-out the loop. On the other hand, control-flow invariants are those cases which specify that the control should never reach to that part of code. Consider the following code[1]:

for (....) {

  if (...)
      return x;
  else if (...)
      return y;
 
 /* Control should never reach here */

}

Now observe how control-flow invariance can be validated by assertions:

for (....) {

  if (...)
      return x;
  else if (...)
      return y;
 
  assert false

}

They are also a very useful tool to detect logical errors. For e.g consider the following code[1]:

switch(scase) {

  case 1: ....
  case 2: ....
  default: assert false;

}

This code fragment clearly says that the programmer assumes that the variable scase will take either of the listed values. If the variables reaches the default segment, then there is a logical error in the program.

Numerous other run-time checks can be made. The following example illustrates that[1]:

if (threads % 4 == 0) {....}

else if (threads % 4 == 1) {....}

else if (threads % 4 == 2) {....}

else {

  // we know that it has to be 3
  assert(threads % 4 == 3);
  ....

}

This method is called as asserting an Internal invariant. The latter means that even though its obvious that the result would be 3, we eliminate any logical errors by asserting that.

Programming languages supporting native assertions

The following programming languages support assertions natively[2]:

  • Cobra
  • D
  • Eiffel
  • Fortress
  • Lisaac
  • Nice
  • Oxygene (formerly Chrome)
  • Sather
  • SPARK, via static analysis of Ada programs
  • Spec#

Advantages and disadvantages of assertions

Some programming languages natively support assertions (mentioned above) while some support through add-ins (like C/C++, C#, Java, Javascript etc.). However, most of the above listed languages are high level languages with less popularity. Hence the need for supporting assertions in the commonly used languages is important.

Advantages

There are many advantages of assertions. Some of them are enumerated here[3]:

  • they remove the need for passive assertions (in the form of comments) and instead actively assert the condition and fail if false during run-time: comments may outlive the code and may not hold for the new code. They act like passive assertions, so its the programmer who has to take care of the condition. In case of assert statements, they enforce the conditions and never outlive the code. If the code changes, the assert statement has to change too, for the program to execute. They thus act as active snippets of comments.
  • they make you more confident of your code: The programmer makes a certain assumptions during the coding phase. Assertions validate the programmer's assumptions. If any of them turns out to be false, code without assertions may execute and show unexpected behavior. Assertions avoid such situations and make the code less buggy.
  • they help refactor the code: Whenever somebody's refactoring the code, leaving the assertions in place (or modifying them accordingly) ensures that the logic is not flawed and the refactor was actually done (without altering the logic). It is a way to validate the process of refactoring. The same hold true for optimization. Assertions ensure that while in the process, the program logic was not altered.
  • they act as embedded tests: A very good example is the development of Program 1: MeToo. A lot of assertions were used in the unit tests. Assertions are like small tests embedded within the code. They simplify the testing of the code.
  • they help in debugging: When a program has assertions, it is easy to narrow down the error since the programmer now knows that if there is an assertion error then the problem must be somewhere in the vicinity of the assertion. They thus help in efficient debugging.
  • they support Programming by Contract: Assertions help establish a relationship between a class and its consumers. It defines what rights does an element have and what is it supposed or capable of doing.

Disadvantages

There are some disadvantages to using assertions too. However, the advantages outnumber the disadvantages. Some of the disadvantages are listed below:

  • the biggest disadvantage is probably the speed: Code with a lot of assertions is slower than one without assertions. However, it can be argued that during development its the accuracy and correctness that is utmost important and thus speed can be sacrificed at the cost of correctness/accuracy[3].
  • assertions can be tricky to write: Unless the programmer puts down his/her assumptions in the form of assertions accurately, they serve no purpose.
  • assertions can cause dependency problems: A class may be heavily dependent on every assert class in it. Thus extension maybe a problem
  • stimulus must make use of assertions: Extensive and exhaustive testing is still required. The test-suite (or the stimulus) must make use of all the assertions for them to show their benefit. If the stimulus never tests some special corner cases, those assertions will never fire up to trigger an error[5].

When NOT to use assertions

If used incorrectly assertions can be a bane and can significantly alter the program flow[4]. Consider the following code:

assert((var = a*3*0.45*i) != 0); var1 = var2/var;

The problem with the above code is that the expression is evaluated inside the assert statement. Most languages have a feature to disable assertions. What if assertions are disabled? var would never be correctly evaluated and some garbage value maybe passed on to the next statement. If var is passed on as '0', then the program would generate a 'Divide by zero' exception. A fix to that problem would be to separate the expression evaluation and then assert. The following code shows the fix:

var = (a*3*0.45*i); assert(var != 0); var1 = var2/var;

Thus one must be careful when using the assert statements.

References

  1. http://java.sun.com/j2se/1.4.2/docs/guide/lang/assert.html
  2. http://en.wikipedia.org/wiki/Design_by_contract
  3. http://www.stanford.edu/~pgbovine/programming-with-asserts.htm
  4. http://courses.csail.mit.edu/6.170/old-www/2002-Fall/lectures/lecture-10.pdf
  5. http://www.esperan.com/pdf/esperan_introduction_to_psl.pdf