Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-13T00:11:19Z

Ird: /* When and how NOT to use assertions */

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-11T01:32:38Z

Ird: /* When and how NOT to use assertions */

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-11T01:30:38Z

Ird: /* When and how NOT to use assertions */

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-11T01:29:18Z

Ird:

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-11T01:24:09Z

Ird:

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-11T01:04:43Z

Ird:

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-11T01:02:18Z

Ird:

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-11T00:49:41Z

Ird:

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-11T00:47:03Z

Ird:

== Programming with assertions ==
Assertions are predicates or statements with expressions which the programmer thinks should be true at the time of execution. Assertions are a powerful way of detecting bugs in programs.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* ''Planning''
* ''Design''
* ''Development''
* ''Testing''
* ''Release and Maintenance''

It is impossible for a programmer to make any program a hundred percent bug free. Even worse, there maybe subtle things which the programmer assumes are always true but may not be so in many corner cases. Debugging is an integral part of the Software Process. Bugs can crop up anytime due to various test cases formulated all through these aforementioned stages. To be sure that whatever the programmer assumes is true, assertions can be set up in the program. If ever, the condition evaluates to be false, the program terminates with an assertion. Another way to put it is that assertions are a way to show the programmer that things cannot be taken for granted by the latter. Assertions can be thought of as medium of validating the correctness of modules.

== Purpose ==

The various uses of assertions are:
* '''Validates correctness of (sub)program'''
* '''Supports programming by contract (documentation)'''
* '''Helps in debugging'''
* '''Exception handling by integrating assertion failures to be detected dynamically'''

Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught.

== Types of assertions ==

Assertions can be categorized primarily as follows:

*''Pre-conditions''
*''Post-conditions''
*''Class invariants''
*''Invariants (Control-flow, Loop etc.)''
*''Loop variants''

As Dr.Gehringer mentions, the first three classes are discussed in class and will not be a focus of this wiki. Instead, this will concentrate on the other types of assertions, viz., Control flow, Loop invariants and Loop variants. Invariants are those conditions with a precondition as well as post condition. A loop invariant is an expression for a variable which remains constant through-out the loop. On the other hand, control-flow invariants are those cases which specify that the control should never reach to that part of code. Consider the following code:

<code>
for (....) {
if (...)
return x;
else if (...)
return y;

/* Control should never reach here */
}
</code>

Now observe how control-flow invariance can be validated by assertions:

<code>
for (....) {
if (...)
return x;
else if (...)
return y;

assert false
}
</code>

They are also a very useful tool to detect logical errors. For e.g consider the following code:

<code>
switch(scase) {
case 1: ....

case 2: ....

default: assert false;
}
</code>

This code fragment clearly says that the programmer assumes that the variable ''scase'' will take either of the listed values. If the variables reaches the ''default'' segment, then there is a logical error in the program.

Numerous other run-time checks can be made. The following example illustrates that:

<code>
if (threads % 4 == 0) {....}

else if (threads % 4 == 1) {....}

else if (threads % 4 == 2) {....}

else {
// we know that it has to be 3
assert(threads % 4 == 3);
....
}
</code>

This method is called as asserting an Internal invariant. The latter means that even though its obvious that the result would be 3, we eliminate any logical errors by asserting that.

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering "A Practitioner Approach" New York McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-11T00:43:24Z

Ird: /* Types of assertions */

== Programming with assertions ==
Assertions are predicates or statements with expressions which the programmer thinks should be true at the time of execution. Assertions are a powerful way of detecting bugs in programs.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* ''Planning''
* ''Design''
* ''Development''
* ''Testing''
* ''Release and Maintenance''

It is impossible for a programmer to make any program a hundred percent bug free. Even worse, there maybe subtle things which the programmer assumes are always true but may not be so in many corner cases. Debugging is an integral part of the Software Process. Bugs can crop up anytime due to various test cases formulated all through these aforementioned stages. To be sure that whatever the programmer assumes is true, assertions can be set up in the program. If ever, the condition evaluates to be false, the program terminates with an assertion. Another way to put it is that assertions are a way to show the programmer that things cannot be taken for granted by the latter. Assertions can be thought of as medium of validating the correctness of modules.

== Purpose ==

The various uses of assertions are:
* '''Validating correctness of (sub)program'''
* '''Supports programming by contract (documentation)'''
* '''Helps in debugging'''
* '''Exception handling by integrating assertion failures to be detected dynamically'''

Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught. They are also a very useful tool to detect logical errors. For e.g consider the following code:

<code>
switch(scase) {
case 1: ....

case 2: ....

default: assert false;
}
</code>

This code fragment clearly says that the programmer assumes that the variable ''scase'' will take either of the listed values. If the variables reaches the ''default'' segment, then there is a logical error in the program. Numerous other run-time checks can be made. The following example illustrates that:

<code>
if (threads % 4 == 0) {....}

else if (threads % 4 == 1) {....}

else if (threads % 4 == 2) {....}

else {
// we know that it has to be 3
assert(threads % 4 == 3);
....
}
</code>

This method is called as asserting an invariant.

== Types of assertions ==

Assertions can be categorized primarily as follows:

*''Pre-conditions''
*''Post-conditions''
*''Class invariants''
*''Invariants (Control-flow, Loop etc.)''
*''Loop variants''

As Dr.Gehringer mentions, the first three classes are discussed in class and will not be a focus of this wiki. Instead, this will concentrate on the other types of assertions, viz., Control flow, Loop invariants and Loop variants. Invariants are those conditions with a precondition as well as post condition. A loop invariant is an expression for a variable which remains constant through-out the loop. On the other hand, control-flow invariants are those cases which specify that the control should never reach to that part of code. Consider the following code:

<code>
for (....) {
if (...)
return x;
else if (...)
return y;

/* Control should never reach here */
}
</code>

Now observe how control-flow invariance can be validated by assertions:

<code>
for (....) {
if (...)
return x;
else if (...)
return y;

assert false
}
</code>

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering "A Practitioner Approach" New York McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-11T00:42:49Z

Ird:

== Programming with assertions ==
Assertions are predicates or statements with expressions which the programmer thinks should be true at the time of execution. Assertions are a powerful way of detecting bugs in programs.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* ''Planning''
* ''Design''
* ''Development''
* ''Testing''
* ''Release and Maintenance''

It is impossible for a programmer to make any program a hundred percent bug free. Even worse, there maybe subtle things which the programmer assumes are always true but may not be so in many corner cases. Debugging is an integral part of the Software Process. Bugs can crop up anytime due to various test cases formulated all through these aforementioned stages. To be sure that whatever the programmer assumes is true, assertions can be set up in the program. If ever, the condition evaluates to be false, the program terminates with an assertion. Another way to put it is that assertions are a way to show the programmer that things cannot be taken for granted by the latter. Assertions can be thought of as medium of validating the correctness of modules.

== Purpose ==

The various uses of assertions are:
* '''Validating correctness of (sub)program'''
* '''Supports programming by contract (documentation)'''
* '''Helps in debugging'''
* '''Exception handling by integrating assertion failures to be detected dynamically'''

Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught. They are also a very useful tool to detect logical errors. For e.g consider the following code:

<code>
switch(scase) {
case 1: ....

case 2: ....

default: assert false;
}
</code>

This code fragment clearly says that the programmer assumes that the variable ''scase'' will take either of the listed values. If the variables reaches the ''default'' segment, then there is a logical error in the program. Numerous other run-time checks can be made. The following example illustrates that:

<code>
if (threads % 4 == 0) {....}

else if (threads % 4 == 1) {....}

else if (threads % 4 == 2) {....}

else {
// we know that it has to be 3
assert(threads % 4 == 3);
....
}
</code>

This method is called as asserting an invariant.

== Types of assertions ==

Assertions can be categorized primarily as follows:

*''Pre-conditions''
*''Post-conditions''
*''Class invariants''
*''Invariants (Control-flow, Loop etc.)''
*''Loop variants''

As Dr.Gehringer mentions, the first three classes are discussed in class and will not be a focus of this wiki. Instead, this will concentrate on the other types of assertions, viz., Control flow, Loop invariants and Loop variants. Invariants are those conditions with a precondition as well as post condition. A loop invariant is an expression for a variable which remains constant through-out the loop. On the other hand, control-flow invariants are those cases which specify that the control should never reach to that part of code. Consider the following code:

<code>
for (....)
if (...)
return x;
else if (...)
return y;

/* Control should never reach here */
</code>

Now observe how control-flow invariance can be validated by assertions:

<code>
for (....)
if (...)
return x;
else if (...)
return y;

assert false
</code>

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering "A Practitioner Approach" New York McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-11T00:42:23Z

Ird:

== Programming with assertions ==
Assertions are predicates or statements with expressions which the programmer thinks should be true at the time of execution. Assertions are a powerful way of detecting bugs in programs.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* ''Planning''
* ''Design''
* ''Development''
* ''Testing''
* ''Release and Maintenance''

It is impossible for a programmer to make any program a hundred percent bug free. Even worse, there maybe subtle things which the programmer assumes are always true but may not be so in many corner cases. Debugging is an integral part of the Software Process. Bugs can crop up anytime due to various test cases formulated all through these aforementioned stages. To be sure that whatever the programmer assumes is true, assertions can be set up in the program. If ever, the condition evaluates to be false, the program terminates with an assertion. Another way to put it is that assertions are a way to show the programmer that things cannot be taken for granted by the latter. Assertions can be thought of as medium of validating the correctness of modules.

== Purpose ==

The various uses of assertions are:
* '''Validating correctness of (sub)program'''
* '''Supports programming by contract (documentation)'''
* '''Helps in debugging'''
* '''Exception handling by integrating assertion failures to be detected dynamically'''

Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught. They are also a very useful tool to detect logical errors. For e.g consider the following code:

<code>
switch(scase) {
case 1: ....

case 2: ....

default: assert false;
}
</code>

This code fragment clearly says that the programmer assumes that the variable ''scase'' will take either of the listed values. If the variables reaches the ''default'' segment, then there is a logical error in the program. Numerous other run-time checks can be made. The following example illustrates that:

<code>
if (threads % 4 == 0) {....}

else if (threads % 4 == 1) {....}

else if (threads % 4 == 2) {....}

else
// we know that it has to be 3
assert(threads % 4 == 3);
....
</code>

This method is called as asserting an invariant.

== Types of assertions ==

Assertions can be categorized primarily as follows:

*''Pre-conditions''
*''Post-conditions''
*''Class invariants''
*''Invariants (Control-flow, Loop etc.)''
*''Loop variants''

As Dr.Gehringer mentions, the first three classes are discussed in class and will not be a focus of this wiki. Instead, this will concentrate on the other types of assertions, viz., Control flow, Loop invariants and Loop variants. Invariants are those conditions with a precondition as well as post condition. A loop invariant is an expression for a variable which remains constant through-out the loop. On the other hand, control-flow invariants are those cases which specify that the control should never reach to that part of code. Consider the following code:

<code>
for (....)
if (...)
return x;
else if (...)
return y;

/* Control should never reach here */
</code>

Now observe how control-flow invariance can be validated by assertions:

<code>
for (....)
if (...)
return x;
else if (...)
return y;

assert false
</code>

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering "A Practitioner Approach" New York McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-11T00:26:27Z

Ird:

== Programming with assertions ==
Assertions are predicates or statements with expressions which the programmer thinks should be true at the time of execution. Assertions are a powerful way of detecting bugs in programs.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* ''Planning''
* ''Design''
* ''Development''
* ''Testing''
* ''Release and Maintenance''

It is impossible for a programmer to make any program a hundred percent bug free. Even worse, there maybe subtle things which the programmer assumes are always true but may not be so in many corner cases. Debugging is an integral part of the Software Process. Bugs can crop up anytime due to various test cases formulated all through these aforementioned stages. To be sure that whatever the programmer assumes is true, assertions can be set up in the program. If ever, the condition evaluates to be false, the program terminates with an assertion. Another way to put it is that assertions are a way to show the programmer that things cannot be taken for granted by the latter. Assertions can be thought of as medium of validating the correctness of modules.

== Purpose ==

The various uses of assertions are:
* '''Validating correctness of (sub)program'''
* '''Supports programming by contract (documentation)'''
* '''Helps in debugging'''
* '''Exception handling by integrating assertion failures to be detected dynamically'''

Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught. They are also a very useful tool to detect logical errors. For e.g consider the following code:

<code>
switch(scase) {
case 1: ....

case 2: ....

default: assert false;
}
</code>

This code fragment clearly says that the programmer assumes that the variable ''scase'' will take either of the listed values. If the variables reaches the ''default'' segment, then there is a logical error in the program. Numerous other run-time checks can be made. The following example illustrates that:

<code>
if (threads % 4 == 0) {....}

else if (threads % 4 == 1) {....}

else if (threads % 4 == 2) {....}

else
// we know that it has to be 3
assert(threads % 4 == 3);
....
</code>

This method is called as asserting an invariant.

== Types of assertions ==

Assertions can be categorized primarily as follows:

*''Pre-conditions''
*''Post-conditions''
*''Class invariants''
*''Loop invariants''
*''Loop variants''

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering "A Practitioner Approach" New York McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-11T00:25:05Z

Ird:

== Programming with assertions ==
Assertions are predicates or statements with expressions which the programmer thinks should be true at the time of execution. Assertions are a powerful way of detecting bugs in programs.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* ''Planning''
* ''Design''
* ''Development''
* ''Testing''
* ''Release and Maintenance''

It is impossible for a programmer to make any program a hundred percent bug free. Even worse, there maybe subtle things which the programmer assumes are always true but may not be so in many corner cases. Debugging is an integral part of the Software Process. Bugs can crop up anytime due to various test cases formulated all through these aforementioned stages. To be sure that whatever the programmer assumes is true, assertions can be set up in the program. If ever, the condition evaluates to be false, the program terminates with an assertion. Another way to put it is that assertions are a way to show the programmer that things cannot be taken for granted by the latter. Assertions can be thought of as medium of validating the correctness of modules.

== Purpose ==

The various uses of assertions are:
* '''Validating correctness of (sub)program'''
* '''Supports programming by contract (documentation)'''
* '''Helps in debugging'''
* '''Exception handling by integrating assertion failures to be detected dynamically'''

Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught. They are also a very useful tool to detect logical errors. For e.g consider the following code:

<code>
switch(scase) {
case 1: ....

case 2: ....

default: assert false;
}
</code>

This code fragment clearly says that the programmer assumes that the variable ''scase'' will take either of the listed values. If the variables reaches the ''default'' segment, then there is a logical error in the program. Numerous other run-time checks can be made. The following example illustrates that:

<code>
if (threads % 4 == 0)
....

else if (threads % 4 == 1)
....

else if (threads % 4 == 2)
....

else
// we know that it has to be 3
assert(threads % 4 == 3);
....
</code>

This method is called as asserting an invariant.

== Types of assertions ==

Assertions can be categorized primarily as follows:

*''Pre-conditions''
*''Post-conditions''
*''Class invariants'
*''Loop invariants''
*''Loop variants''

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering "A Practitioner Approach" New York McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-11T00:12:54Z

Ird:

== Programming with assertions ==
Assertions are predicates or statements with expressions which the programmer thinks should be true at the time of execution. Assertions are a powerful way of detecting bugs in programs.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* ''Planning''
* ''Design''
* ''Development''
* ''Testing''
* ''Release and Maintenance''

It is impossible for a programmer to make any program a hundred percent bug free. Even worse, there maybe subtle things which the programmer assumes are always true but may not be so in many corner cases. Debugging is an integral part of the Software Process. Bugs can crop up anytime due to various test cases formulated all through these aforementioned stages. To be sure that whatever the programmer assumes is true, assertions can be set up in the program. If ever, the condition evaluates to be false, the program terminates with an assertion. Another way to put it is that assertions are a way to show the programmer that things cannot be taken for granted by the latter. Assertions can be thought of as medium of validating the correctness of modules.

== Purpose ==

The various uses of assertions are:
* '''Validating correctness of (sub)program'''
* '''Supports programming by contract (documentation)'''
* '''Helps in debugging'''
* '''Exception handling by integrating assertion failures to be detected dynamically'''

Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught. They are also a very useful tool to detect errors in control flow. For e.g consider the following code:

<source>

switch(scase) {
case 1: ....

case 2: ....

default: assert false;
}
</source>

This code fragment clearly says that the programmer assumes that the variable ''scase'' will take either of the listed values. If the control flow reaches the ''default'' then there is a logical error in the program.

== Testing Methodologies ==

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering "A Practitioner Approach" New York McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-11T00:12:30Z

Ird:

== Programming with assertions ==
Assertions are predicates or statements with expressions which the programmer thinks should be true at the time of execution. Assertions are a powerful way of detecting bugs in programs.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* ''Planning''
* ''Design''
* ''Development''
* ''Testing''
* ''Release and Maintenance''

It is impossible for a programmer to make any program a hundred percent bug free. Even worse, there maybe subtle things which the programmer assumes are always true but may not be so in many corner cases. Debugging is an integral part of the Software Process. Bugs can crop up anytime due to various test cases formulated all through these aforementioned stages. To be sure that whatever the programmer assumes is true, assertions can be set up in the program. If ever, the condition evaluates to be false, the program terminates with an assertion. Another way to put it is that assertions are a way to show the programmer that things cannot be taken for granted by the latter. Assertions can be thought of as medium of validating the correctness of modules.

== Purpose ==

The various uses of assertions are:
* '''Validating correctness of (sub)program'''
* '''Supports programming by contract (documentation)'''
* '''Helps in debugging'''
* '''Exception handling by integrating assertion failures to be detected dynamically'''

Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught. They are also a very useful tool to detect errors in control flow. For e.g consider the following code:

<source lang="java">
switch(scase) {
case 1: ....

case 2: ....

default: assert false;
}
</source>

This code fragment clearly says that the programmer assumes that the variable ''scase'' will take either of the listed values. If the control flow reaches the ''default'' then there is a logical error in the program.

== Testing Methodologies ==

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering "A Practitioner Approach" New York McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-10T23:57:13Z

Ird:

== Programming with assertions ==
Assertions are predicates or statements with expressions which the programmer thinks should be true at the time of execution. Assertions are a powerful way of detecting bugs in programs.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* ''Planning''
* ''Design''
* ''Development''
* ''Testing''
* ''Release and Maintenance''

It is impossible for a programmer to make any program a hundred percent bug free. Even worse, there maybe subtle things which the programmer assumes are always true but may not be so in many corner cases. Debugging is an integral part of the Software Process. Bugs can crop up anytime due to various test cases formulated all through these aforementioned stages. To be sure that whatever the programmer assumes is true, assertions can be set up in the program. If ever, the condition evaluates to be false, the program terminates with an assertion. Another way to put it is that assertions are a way to show the programmer that things cannot be taken for granted by the latter. Assertions can be thought of as medium of validating the correctness of modules.

== Purpose ==

The various uses of assertions are:
* '''Validating correctness of (sub)program'''
* '''Supports programming by contract (documentation)'''
* '''Helps in debugging'''
* '''Exception handling by integrating assertion failures to be detected dynamically'''

Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught. They are also a very useful tool to detect errors in control flow. For e.g consider the following code:

<code>
switch(scase) {
case 1: ....

case 2:....

default: assert false;
}
</code>

This code fragment clearly says that the programmer assumes that the variable ''scase'' will take either of the listed values. If the control flow reaches the ''default'' then there is a logical error in the program.

== Testing Methodologies ==

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering "A Practitioner Approach" New York McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-10T23:53:25Z

Ird:

== Programming with assertions ==
Assertions are predicates or statements with expressions which the programmer thinks should be true at the time of execution. Assertions are a powerful way of detecting bugs in programs.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* ''Planning''
* ''Design''
* ''Development''
* ''Testing''
* ''Release and Maintenance''

It is impossible for a programmer to make any program a hundred percent bug free. Even worse, there maybe subtle things which the programmer assumes are always true but may not be so in many corner cases. Debugging is an integral part of the Software Process. Bugs can crop up anytime due to various test cases formulated all through these aforementioned stages. To be sure that whatever the programmer assumes is true, assertions can be set up in the program. If ever, the condition evaluates to be false, the program terminates with an assertion. Another way to put it is that assertions are a way to show the programmer that things cannot be taken for granted by the latter. Assertions can be thought of as medium of validating the correctness of modules.

== Purpose ==

The various uses of assertions are:
* '''Validating correctness of (sub)program'''
* '''Supports programming by contract (documentation)'''
* '''Helps in debugging'''
* '''Exception handling by integrating assertion failures to be detected dynamically'''

Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught.

== Testing Methodologies ==

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering "A Practitioner Approach" New York McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-10T23:52:59Z

Ird:

== Programming with assertions ==
Assertions are predicates or statements with expressions which the programmer thinks should be true at the time of execution. Assertions are a powerful way of detecting bugs in programs.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* '''Planning'''
* '''Design'''
* '''Development'''
* '''Testing'''
* '''Release and Maintenance'''

It is impossible for a programmer to make any program a hundred percent bug free. Even worse, there maybe subtle things which the programmer assumes are always true but may not be so in many corner cases. Debugging is an integral part of the Software Process. Bugs can crop up anytime due to various test cases formulated all through these aforementioned stages. To be sure that whatever the programmer assumes is true, assertions can be set up in the program. If ever, the condition evaluates to be false, the program terminates with an assertion. Another way to put it is that assertions are a way to show the programmer that things cannot be taken for granted by the latter. Assertions can be thought of as medium of validating the correctness of modules.

== Purpose ==

The various uses of assertions are:
* '''Validating correctness of (sub)program'''
* '''Supports programming by contract (documentation)'''
* '''Helps in debugging'''
* '''Exception handling by integrating assertion failures to be detected dynamically'''

Assertions can be used by the programmer to make sure that wrong runtime assumptions (by the programmer) can be caught.

== Testing Methodologies ==

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering "A Practitioner Approach" New York McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki3 15 assertions

2009-11-10T23:26:29Z

Ird:

== Integration and Function Testing and beyond ==
Software testing is the science of breaking software down. During testing we have one and only one aim to develop comprehensive test cases to find bugs in the software and get them fixed.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* '''Planning'''
* '''Design'''
* '''Development'''
* '''Testing'''
* '''Release and Maintenance'''

Each stage can be further broken down into a number of several stages. This article focuses on the Testing phase. Software testing can be defined as inspection or investigation of a product to ascertain its quality and verify if its semantics are implemented the way they are intended to. Testing can however, be done outside the Software Development cycle too.

Testing can be broadly classified into the following methodologies [1]:
* '''Unit testing'''
* '''Integration testing'''
* '''Functional or Validation testing'''
* '''System testing'''

The various testing processes/methods listed can be put in chronological order and can be viewed as a chain process. Each phase is an important part of the overall testing process. Unit test concentrates on the basic building blocks of the product, verifying if each component works correctly. The next phase is the Integration Testing. When many such units are knit together to form a larger, more complex block, it becomes necessary to test if all the smaller units of the bigger block interact properly to implement the semantics of the bigger block, as desired. The process that follows this stage is the Functional or Validation testing. This phase focuses on the behavior and performance in most cases. The question asked here is “If all the complex blocks are properly integrated, are they producing the result which they are required to?” The last phase is the System Testing. This mostly deals with interaction on the hardware-software platform and/or integration of the product on a platform, as viewed from a higher level. Here we will focus our discussion on Integration and Functional Testing.

== Purpose ==

Testing is a very important phase in the software development cycle. The cycle can either follow the ''Waterfall'' [http://en.wikipedia.org/wiki/Waterfall_model] model or the ''Spiral'' model [http://en.wikipedia.org/wiki/Spiral_model]. The latter mandates testing continuously as opposed to the former’s discrete phase. No software product is completely free of defects. No matter how much care the design team takes, its impossible to ensure that there are absolutely no bugs. This is where testing comes in. The goal of the testing process is to try and break the software.

Considering a large project, individual modules or units maybe developed by different team or even outsourced to different companies. Each team would have tested their respective modules or units. When, however, these modules are integrated climbing a level up, further testing maybe required to ensure that these units interact with other in a way which does not break the product. An important point in the development plan is the decision on the interface of each module. A common interface eliminates most of the bugs, but a final integration test ensures that everything is clean.

Making sure that the units or modules integrate with each other is not enough. The result produced by integrating the modules and the behavior exhibited after integration is another important aspect of the development plan. A software is designed with a certain behavior and functionality in mind. Functional testing deals with testing the product for functional issues. Its goal is to ensure that the product is doing what it was intended to do in the design phase.

== Testing Methodologies ==

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering "A Practitioner Approach" New York McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-13T18:08:45Z

Ird: /* Example */

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is ''processed'' while instructions are ''executed''. Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior[1].

== Overview ==
Refection-oriented programming is a different paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a ''"watchdog"'' to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program[2].

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as[3]:

* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code[3].

== List of reflective programming languages and platforms ==
Some of the languages that have inherent reflection oriented programming support are[4]:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown [3]:

Non-reflective approach:
<code>
repeat N times {
if STATE is 1
increase A by one
else
decrease A by one

do something with A
}
</code>
The optimized version of the pseudo-code with a reflective approach is:
<code>
repeat N times {
increase A by one
do something with A
}

when STATE has to switch { replace the opcode "increase" above with the opcode to decrease }
</code>

== Definitions ==

; atomic : An atomic operation in computer science refers to a set of operations that can be combined so that they appear to the rest of the system to be a single operation with only two possible outcomes: success or failure.
; metadata : Metadata is "data about data", of any sort in any media. It aids in clarifying and finding the actual data.
; malware : Malware, short for malicious software, is software designed to infiltrate a computer without the owner's informed consent.

== References ==
# [http://cs.indiana.edu/~jsobel/rop.html An Introduction to Reflection-Oriented Programming]
# [http://biophp.net/index.php/Reflective Reflective]
# [http://en.allexperts.com/e/s/se/self-modifying_code.htm Self-modifying code]
# [http://en.allexperts.com/e/l/li/list_of_reflective_programming_languages_and_platforms.htm List of reflective programming languages and platforms]

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-10T04:35:23Z

Ird: /* Example */

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is ''processed'' while instructions are ''executed''. Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior[1].

== Overview ==
Refection-oriented programming is a different paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a ''"watchdog"'' to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program[2].

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as[3]:

* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code[3].

== List of reflective programming languages and platforms ==
Some of the languages that have inherent reflection oriented programming support are[4]:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown:

Non-reflective approach:
<code>
repeat N times {
if STATE is 1
increase A by one
else
decrease A by one

do something with A
}
</code>
The optimized version of the pseudo-code with a reflective approach is:
<code>
repeat N times {
increase A by one
do something with A
}

when STATE has to switch { replace the opcode "increase" above with the opcode to decrease }
</code>

== Definitions ==

; atomic : An atomic operation in computer science refers to a set of operations that can be combined so that they appear to the rest of the system to be a single operation with only two possible outcomes: success or failure.
; metadata : Metadata is "data about data", of any sort in any media. It aids in clarifying and finding the actual data.
; malware : Malware, short for malicious software, is software designed to infiltrate a computer without the owner's informed consent.

== References ==
# [http://cs.indiana.edu/~jsobel/rop.html An Introduction to Reflection-Oriented Programming]
# [http://biophp.net/index.php/Reflective Reflective]
# [http://en.allexperts.com/e/s/se/self-modifying_code.htm Self-modifying code]
# [http://en.allexperts.com/e/l/li/list_of_reflective_programming_languages_and_platforms.htm List of reflective programming languages and platforms]

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:52:37Z

Ird:

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is ''processed'' while instructions are ''executed''. Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior[1].

== Overview ==
Refection-oriented programming is a different paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a ''"watchdog"'' to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program[2].

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as[3]:

* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code[3].

== List of reflective programming languages and platforms ==
Some of the languages that have inherent reflection oriented programming support are[4]:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown:

Non-reflective approach:

repeat N times {
if STATE is 1
increase A by one
else
decrease A by one

do something with A
}

The optimized version of the pseudo-code with a reflective approach is:

repeat N times {
increase A by one
do something with A
}

when STATE has to switch { replace the opcode "increase" above with the opcode to decrease }

== Definitions ==

; atomic : An atomic operation in computer science refers to a set of operations that can be combined so that they appear to the rest of the system to be a single operation with only two possible outcomes: success or failure.
; metadata : Metadata is "data about data", of any sort in any media. It aids in clarifying and finding the actual data.
; malware : Malware, short for malicious software, is software designed to infiltrate a computer without the owner's informed consent.

== References ==
# [http://cs.indiana.edu/~jsobel/rop.html An Introduction to Reflection-Oriented Programming]
# [http://biophp.net/index.php/Reflective Reflective]
# [http://en.allexperts.com/e/s/se/self-modifying_code.htm Self-modifying code]
# [http://en.allexperts.com/e/l/li/list_of_reflective_programming_languages_and_platforms.htm List of reflective programming languages and platforms]

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:49:28Z

Ird:

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is "processed" while instructions are "executed". Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior.[1]

== Overview ==
Refection-oriented programming is a new paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a "watchdog" to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program.

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as:
* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code.

== List of reflective programming languages and platforms ==
Some of the languages that have inherent reflection oriented programming support are:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown:

Non-reflective approach:

repeat N times {
if STATE is 1
increase A by one
else
decrease A by one

do something with A
}

The optimized version of the pseudo-code with a reflective approach is:

repeat N times {
increase A by one
do something with A
}

when STATE has to switch { replace the opcode "increase" above with the opcode to decrease }

== Definitions ==

; atomic : An atomic operation in computer science refers to a set of operations that can be combined so that they appear to the rest of the system to be a single operation with only two possible outcomes: success or failure.
; metadata : Metadata is "data about data", of any sort in any media. It aids in clarifying and finding the actual data.
; malware : Malware, short for malicious software, is software designed to infiltrate a computer without the owner's informed consent.

== References ==
# [http://cs.indiana.edu/~jsobel/rop.html An Introduction to Reflection-Oriented Programming]
# [http://biophp.net/index.php/Reflective Reflective]
# [http://en.allexperts.com/e/s/se/self-modifying_code.htm Self-modifying code]
# [http://en.allexperts.com/e/l/li/list_of_reflective_programming_languages_and_platforms.htm List of reflective programming languages and platforms]

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:47:35Z

Ird: /* References */

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is "processed" while instructions are "executed". Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior.

== Overview ==
Refection-oriented programming is a new paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a "watchdog" to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program.

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as:
* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code.

== List of reflection-oriented programming languages ==
Some of the languages that have inherent reflection oriented programming support are:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown:

Non-reflective approach:

repeat N times {
if STATE is 1
increase A by one
else
decrease A by one

do something with A
}

The optimized version of the pseudo-code with a reflective approach is:

repeat N times {
increase A by one
do something with A
}

when STATE has to switch { replace the opcode "increase" above with the opcode to decrease }

== Definitions ==

* ''atomic'': An atomic operation in computer science refers to a set of operations that can be combined so that they appear to the rest of the system to be a single operation with only two possible outcomes: success or failure.
* ''metadata'': Metadata is "data about data", of any sort in any media. It aids in clarifying and finding the actual data.
* ''malware'': Malware, short for malicious software, is software designed to infiltrate a computer without the owner's informed consent.

== References ==
# [http://cs.indiana.edu/~jsobel/rop.html An Introduction to Reflection-Oriented Programming]
# [http://biophp.net/index.php/Reflective Reflective]
# [http://en.allexperts.com/e/s/se/self-modifying_code.htm Self-modifying code]
# [http://en.allexperts.com/e/l/li/list_of_reflective_programming_languages_and_platforms.htm List of reflective programming languages and platforms]

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:46:40Z

Ird: /* References */

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is "processed" while instructions are "executed". Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior.

== Overview ==
Refection-oriented programming is a new paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a "watchdog" to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program.

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as:
* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code.

== List of reflection-oriented programming languages ==
Some of the languages that have inherent reflection oriented programming support are:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown:

Non-reflective approach:

repeat N times {
if STATE is 1
increase A by one
else
decrease A by one

do something with A
}

The optimized version of the pseudo-code with a reflective approach is:

repeat N times {
increase A by one
do something with A
}

when STATE has to switch { replace the opcode "increase" above with the opcode to decrease }

== Definitions ==

* ''atomic'': An atomic operation in computer science refers to a set of operations that can be combined so that they appear to the rest of the system to be a single operation with only two possible outcomes: success or failure.
* ''metadata'': Metadata is "data about data", of any sort in any media. It aids in clarifying and finding the actual data.
* ''malware'': Malware, short for malicious software, is software designed to infiltrate a computer without the owner's informed consent.

== References ==
# [http://cs.indiana.edu/~jsobel/rop.html An Introduction to Reflection-Oriented Programming]
# http://biophp.net/index.php/Reflective
# http://en.allexperts.com/e/s/se/self-modifying_code.htm
# http://en.allexperts.com/e/l/li/list_of_reflective_programming_languages_and_platforms.htm

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:46:21Z

Ird:

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is "processed" while instructions are "executed". Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior.

== Overview ==
Refection-oriented programming is a new paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a "watchdog" to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program.

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as:
* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code.

== List of reflection-oriented programming languages ==
Some of the languages that have inherent reflection oriented programming support are:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown:

Non-reflective approach:

repeat N times {
if STATE is 1
increase A by one
else
decrease A by one

do something with A
}

The optimized version of the pseudo-code with a reflective approach is:

repeat N times {
increase A by one
do something with A
}

when STATE has to switch { replace the opcode "increase" above with the opcode to decrease }

== Definitions ==

* ''atomic'': An atomic operation in computer science refers to a set of operations that can be combined so that they appear to the rest of the system to be a single operation with only two possible outcomes: success or failure.
* ''metadata'': Metadata is "data about data", of any sort in any media. It aids in clarifying and finding the actual data.
* ''malware'': Malware, short for malicious software, is software designed to infiltrate a computer without the owner's informed consent.

== References ==
# [An Introduction to Reflection-Oriented Programming http://cs.indiana.edu/~jsobel/rop.html]
# http://biophp.net/index.php/Reflective
# http://en.allexperts.com/e/s/se/self-modifying_code.htm
# http://en.allexperts.com/e/l/li/list_of_reflective_programming_languages_and_platforms.htm

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:45:42Z

Ird:

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is "processed" while instructions are "executed". Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior.

== Overview ==
Refection-oriented programming is a new paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a "watchdog" to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program.

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as:
* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code.

== List of reflection-oriented programming languages ==
Some of the languages that have inherent reflection oriented programming support are:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown:

Non-reflective approach:

repeat N times {
if STATE is 1
increase A by one
else
decrease A by one

do something with A
}

The optimized version of the pseudo-code with a reflective approach is:

repeat N times {
increase A by one
do something with A
}

when STATE has to switch { replace the opcode "increase" above with the opcode to decrease }

== Definitions ==

* ''atomic'': An atomic operation in computer science refers to a set of operations that can be combined so that they appear to the rest of the system to be a single operation with only two possible outcomes: success or failure.
* ''metadata'': Metadata is "data about data", of any sort in any media. It aids in clarifying and finding the actual data.
* ''malware'': Malware, short for malicious software, is software designed to infiltrate a computer without the owner's informed consent.

== References ==
# http://cs.indiana.edu/~jsobel/rop.html
# http://biophp.net/index.php/Reflective
# http://en.allexperts.com/e/s/se/self-modifying_code.htm
# http://en.allexperts.com/e/l/li/list_of_reflective_programming_languages_and_platforms.htm

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:40:45Z

Ird: /* Example */

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is "processed" while instructions are "executed". Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior.

== Overview ==
Refection-oriented programming is a new paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a "watchdog" to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program.

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as:
* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code.

== List of reflection-oriented programming languages ==
Some of the languages that have inherent reflection oriented programming support are:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown:

Non-reflective approach:

repeat N times {
if STATE is 1
increase A by one
else
decrease A by one

do something with A
}

The optimized version of the pseudo-code with a reflective approach is:

repeat N times {
increase A by one
do something with A
}

when STATE has to switch { replace the opcode "increase" above with the opcode to decrease }

== Testing Tools ==

== Summary ==

== References ==
# http://cs.indiana.edu/~jsobel/rop.html

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:39:48Z

Ird: /* Example */

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is "processed" while instructions are "executed". Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior.

== Overview ==
Refection-oriented programming is a new paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a "watchdog" to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program.

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as:
* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code.

== List of reflection-oriented programming languages ==
Some of the languages that have inherent reflection oriented programming support are:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown:

Non-reflective approach:

repeat N times {
if STATE is 1
increase A by one
else
decrease A by one

do something with A
}

The optimized version of the pseudo-code with a reflective approach is:

repeat N times {
increase A by one
do something with A
}

when STATE has to switch { replace the opcode "increase" above with the opcode to decrease }

<source lang="csharp">
// Hello World in Microsoft C# ("C-Sharp").

using System;

class HelloWorld
{
public static int Main(String[] args)
{
Console.WriteLine("Hello, World!");
return 0;
}
}
</source>

== Testing Tools ==

== Summary ==

== References ==
# http://cs.indiana.edu/~jsobel/rop.html

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:39:12Z

Ird: /* Example */

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is "processed" while instructions are "executed". Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior.

== Overview ==
Refection-oriented programming is a new paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a "watchdog" to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program.

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as:
* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code.

== List of reflection-oriented programming languages ==
Some of the languages that have inherent reflection oriented programming support are:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown:

Non-reflective approach:

repeat N times {
if STATE is 1
increase A by one
else
decrease A by one

do something with A
}

The optimized version of the pseudo-code with a reflective approach is:

repeat N times {
increase A by one
do something with A
}

when STATE has to switch { replace the opcode "increase" above with the opcode to decrease }

== Testing Tools ==

== Summary ==

== References ==
# http://cs.indiana.edu/~jsobel/rop.html

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:39:03Z

Ird: /* Example */

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is "processed" while instructions are "executed". Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior.

== Overview ==
Refection-oriented programming is a new paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a "watchdog" to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program.

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as:
* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code.

== List of reflection-oriented programming languages ==
Some of the languages that have inherent reflection oriented programming support are:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown:

Non-reflective approach:

=== Functional Testing and beyond ===

== Testing Tools ==

== Summary ==

== References ==
# http://cs.indiana.edu/~jsobel/rop.html

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:38:37Z

Ird:

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is "processed" while instructions are "executed". Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior.

== Overview ==
Refection-oriented programming is a new paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a "watchdog" to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program.

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as:
* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code.

== List of reflection-oriented programming languages ==
Some of the languages that have inherent reflection oriented programming support are:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown:

Non-reflective approach:

repeat N times {
if STATE is 1
increase A by one
else
decrease A by one

do something with A
}

The optimized version of the pseudo-code with a reflective approach is:

repeat N times {
increase A by one
do something with A
}

when STATE has to switch { replace the opcode "increase" above with the opcode to decrease }

=== Functional Testing and beyond ===

== Testing Tools ==

== Summary ==

== References ==
# http://cs.indiana.edu/~jsobel/rop.html

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:38:12Z

Ird:

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is "processed" while instructions are "executed". Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior.

== Overview ==
Refection-oriented programming is a new paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a "watchdog" to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program.

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as:
* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code.

== List of reflection-oriented programming languages ==
Some of the languages that have inherent reflection oriented programming support are:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown:

Non-reflective approach:
<source lang="ruby">
repeat N times {
if STATE is 1
increase A by one
else
decrease A by one

do something with A
}
</source>

The optimized version of the pseudo-code with a reflective approach is:

<source lang="ruby">
repeat N times {
increase A by one
do something with A
}

when STATE has to switch { replace the opcode "increase" above with the opcode to decrease }
</source>
----
=== Functional Testing and beyond ===

== Testing Tools ==

== Summary ==

== References ==
# http://cs.indiana.edu/~jsobel/rop.html

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:37:24Z

Ird:

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is "processed" while instructions are "executed". Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior.

== Overview ==
Refection-oriented programming is a new paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a "watchdog" to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program.

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as:
* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code.

== List of reflection-oriented programming languages ==
Some of the languages that have inherent reflection oriented programming support are:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown:

Non-reflective approach:
<source lang="c">
repeat N times {
if STATE is 1
increase A by one
else
decrease A by one

do something with A
}
</source>

The optimized version of the pseudo-code with a reflective approach is:

<source lang="c">
repeat N times {
increase A by one
do something with A
}

when STATE has to switch { replace the opcode "increase" above with the opcode to decrease }
</source>
----
=== Functional Testing and beyond ===

== Testing Tools ==

== Summary ==

== References ==
# http://cs.indiana.edu/~jsobel/rop.html

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:34:53Z

Ird:

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is "processed" while instructions are "executed". Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior.

== Overview ==
Refection-oriented programming is a new paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a "watchdog" to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program.

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as:
* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code.

== List of reflection-oriented programming languages ==
Some of the languages that have inherent reflection oriented programming support are:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown:

Non-reflective approach:
<code>
repeat N times {
if STATE is 1
increase A by one
else
decrease A by one

do something with A
}
</code>

The optimized version of the pseudo-code with a reflective approach is:

<code>
repeat N times {
increase A by one
do something with A
}

when STATE has to switch { replace the opcode "increase" above with the opcode to decrease }
</code>
----
=== Functional Testing and beyond ===

== Testing Tools ==

== Summary ==

== References ==
# http://cs.indiana.edu/~jsobel/rop.html

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:33:29Z

Ird:

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is "processed" while instructions are "executed". Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior.

== Overview ==
Refection-oriented programming is a new paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a "watchdog" to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program.

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as:
* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too. One interesting observation is the use of self-modifying code to hide copy-protection in DOS applications. It can also be used to hide the presence of a program. This technique may prove harmful if a malware made use of this paradigm. For such reason, some Operating Systems such as OpenBSD have explicit control over self-modifying code.

== List of reflection-oriented programming languages ==
Some of the languages that have inherent reflection oriented programming support are:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

== Example ==
An example of a self-modifying pseudo-code for Optimization of a state dependent loop is shown:

<code>repeat N times {
if STATE is 1
increase A by one
else
decrease A by one

do something with A
}</code>

Self-modifying code in this case would simply be a matter of rewriting the loop like this:

<code>repeat N times {
increase A by one
do something with A
}

when STATE has to switch {
replace the opcode "increase" above with the opcode to decrease } </code>
----
=== Functional Testing and beyond ===

== Testing Tools ==

== Summary ==

== References ==
# http://cs.indiana.edu/~jsobel/rop.html

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:27:55Z

Ird:

== Reflection-oriented programming ==
Traditional programming has always separated data from the instructions. Even though the memory system that stores them doesn't make any distinction between them, the way they are handled is the only differentiator. Data is "processed" while instructions are "executed". Reflection-oriented programming is a technique in which the program is made intelligent enough to modify their behavior.

== Overview ==
Refection-oriented programming is a new paradigm which deals with understanding the computation as a task rather than a program. Conceptually, it can be thought as the program setting up a "watchdog" to observe the changes and self-modify accordingly based on the observations. One thing to be mentioned here is that this paradigm goes well with dynamically typed languages such as Ruby, Lisp etc. The core behind this paradigm is the flexibility of treating the instructions as data so that all the policies of data modification can be applied onto them.

== Reflective Paradigm ==
Reflected-oriented programming is the concept used to write Reflective programs. This allows the program architecture to be decided during the course of execution and is not "hard-wired" to the procedural structure. This paradigm supplements the already-existing object-oriented programming to enhance it with the ''self-modifying-code'' feature. The control flow is something which is decided at runtime; this is important since dynamically altered program can change course of execution of a particular event due to modification. Hence, this paradigm requires that the program contain sequence of decisions be implemented rather than the strictness of decision, based on the run-time data.

On a broad perspective, tasks can be classified into:
* ''atomic'' : this category of operations/tasks cannot be interrupted and usually consist of a single step
* ''complex'' : this category of operations/tasks involve multiple steps (or atomic operations) and can be interrupted in the middle of computation

The reason why these categories are brought up here is that, atomic operations being single step do not have a specific structure associated with them. However, the complex tasks/blocks lose their structure when a program is compiled in a non-reflective language. It is necessary to preserve its architecture for reflective programs. The block can be thought of as a state-machine which has pre-defined states data will be in or transition to in case of specific events. This whole package is the metadata that is saved for preserving the structure or architecture of the program.

== Paradigm implementation ==
In most of the languages as the code is compiled/assembled the structure of the program is lost. However, the key to implementing this programming paradigm is to preserve the structure so that the program can be analyzed and modified accordingly. To enable this, the program structure is preserved as metadata in the code. This ensures that even though the compiled/assembled code doesn't maintain the organization, in favor of optimization mostly, the code can be made self-modified. There are some languages like Lisp which do not separate compile-time and run-time. In such cases, there is no difference in the code.

== Purpose and Uses ==
The basic purpose of reflection oriented programming is to introduce flexibility. Broadly the uses of reflection-oriented programming can be classified as:
* Optimization of a state dependent loop
* Application of a specialized algorithm to a data set depending on the runtime data information
* Obfuscating the code to prevent reverse-engineering
* Protection of code from malware
* Decompression of code at runtime
* Universal system for different situations (adaptation and code simplicity)

Apart from these, there are huge number of uses and many unexplored uses too.

== List of reflection-oriented programming languages ==
Some of the languages that have inherent reflection oriented programming support are:

* APL
* ColdFusion MX
* Curl
* Delphi
* ECMAScript (ActionScript, DMDScript, JavaScript, JScript)
* Java
* Lisp
* Logo
* C#
* Visual Basic .NET
* Objective-C
* Perl
* PHP
* Pico
* Pliant
* POP-11
* Prolog
* Python
* Ruby
* Scheme
* Smalltalk
* Tcl

=== Integration Testing and beyond ===

----
=== Functional Testing and beyond ===

== Testing Tools ==

== Summary ==

== References ==
# http://cs.indiana.edu/~jsobel/rop.html

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:23:19Z

Ird:

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:16:29Z

Ird:

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T20:03:14Z

Ird:

CSC/ECE 517 Fall 2009/wiki2 8 rop

2009-10-08T19:52:24Z

Ird:

CSC/ECE 517 Fall 2009/wiki1b 4 iftesting

2009-09-27T23:03:10Z

Ird: /* References */

CSC/ECE 517 Fall 2009/wiki1b 4 iftesting

2009-09-27T23:02:59Z

Ird: /* References */

== Integration and Function Testing and beyond ==
Software testing is the science of breaking software down. During testing we have one and only one aim to develop comprehensive test cases to find bugs in the software and get them fixed.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* '''Planning'''
* '''Design'''
* '''Development'''
* '''Testing'''
* '''Release and Maintenance'''

Each stage can be further broken down into a number of several stages. This article focuses on the Testing phase. Software testing can be defined as inspection or investigation of a product to ascertain its quality and verify if its semantics are implemented the way they are intended to. Testing can however, be done outside the Software Development cycle too.

Testing can be broadly classified into the following methodologies [1]:
* '''Unit testing'''
* '''Integration testing'''
* '''Functional or Validation testing'''
* '''System testing'''

The various testing processes/methods listed can be put in chronological order and can be viewed as a chain process. Each phase is an important part of the overall testing process. Unit test concentrates on the basic building blocks of the product, verifying if each component works correctly. The next phase is the Integration Testing. When many such units are knit together to form a larger, more complex block, it becomes necessary to test if all the smaller units of the bigger block interact properly to implement the semantics of the bigger block, as desired. The process that follows this stage is the Functional or Validation testing. This phase focuses on the behavior and performance in most cases. The question asked here is “If all the complex blocks are properly integrated, are they producing the result which they are required to?” The last phase is the System Testing. This mostly deals with interaction on the hardware-software platform and/or integration of the product on a platform, as viewed from a higher level. Here we will focus our discussion on Integration and Functional Testing.

== Purpose ==

Testing is a very important phase in the software development cycle. The cycle can either follow the ''Waterfall'' [http://en.wikipedia.org/wiki/Waterfall_model] model or the ''Spiral'' model [http://en.wikipedia.org/wiki/Spiral_model]. The latter mandates testing continuously as opposed to the former’s discrete phase. No software product is completely free of defects. No matter how much care the design team takes, its impossible to ensure that there are absolutely no bugs. This is where testing comes in. The goal of the testing process is to try and break the software.

Considering a large project, individual modules or units maybe developed by different team or even outsourced to different companies. Each team would have tested their respective modules or units. When, however, these modules are integrated climbing a level up, further testing maybe required to ensure that these units interact with other in a way which does not break the product. An important point in the development plan is the decision on the interface of each module. A common interface eliminates most of the bugs, but a final integration test ensures that everything is clean.

Making sure that the units or modules integrate with each other is not enough. The result produced by integrating the modules and the behavior exhibited after integration is another important aspect of the development plan. A software is designed with a certain behavior and functionality in mind. Functional testing deals with testing the product for functional issues. Its goal is to ensure that the product is doing what it was intended to do in the design phase.

== Testing Methodologies ==

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering "A Practitioner Approach". New York: McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki1b 4 iftesting

2009-09-27T23:02:07Z

Ird:

== Integration and Function Testing and beyond ==
Software testing is the science of breaking software down. During testing we have one and only one aim to develop comprehensive test cases to find bugs in the software and get them fixed.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* '''Planning'''
* '''Design'''
* '''Development'''
* '''Testing'''
* '''Release and Maintenance'''

Each stage can be further broken down into a number of several stages. This article focuses on the Testing phase. Software testing can be defined as inspection or investigation of a product to ascertain its quality and verify if its semantics are implemented the way they are intended to. Testing can however, be done outside the Software Development cycle too.

Testing can be broadly classified into the following methodologies [1]:
* '''Unit testing'''
* '''Integration testing'''
* '''Functional or Validation testing'''
* '''System testing'''

The various testing processes/methods listed can be put in chronological order and can be viewed as a chain process. Each phase is an important part of the overall testing process. Unit test concentrates on the basic building blocks of the product, verifying if each component works correctly. The next phase is the Integration Testing. When many such units are knit together to form a larger, more complex block, it becomes necessary to test if all the smaller units of the bigger block interact properly to implement the semantics of the bigger block, as desired. The process that follows this stage is the Functional or Validation testing. This phase focuses on the behavior and performance in most cases. The question asked here is “If all the complex blocks are properly integrated, are they producing the result which they are required to?” The last phase is the System Testing. This mostly deals with interaction on the hardware-software platform and/or integration of the product on a platform, as viewed from a higher level. Here we will focus our discussion on Integration and Functional Testing.

== Purpose ==

Testing is a very important phase in the software development cycle. The cycle can either follow the ''Waterfall'' [http://en.wikipedia.org/wiki/Waterfall_model] model or the ''Spiral'' model [http://en.wikipedia.org/wiki/Spiral_model]. The latter mandates testing continuously as opposed to the former’s discrete phase. No software product is completely free of defects. No matter how much care the design team takes, its impossible to ensure that there are absolutely no bugs. This is where testing comes in. The goal of the testing process is to try and break the software.

Considering a large project, individual modules or units maybe developed by different team or even outsourced to different companies. Each team would have tested their respective modules or units. When, however, these modules are integrated climbing a level up, further testing maybe required to ensure that these units interact with other in a way which does not break the product. An important point in the development plan is the decision on the interface of each module. A common interface eliminates most of the bugs, but a final integration test ensures that everything is clean.

Making sure that the units or modules integrate with each other is not enough. The result produced by integrating the modules and the behavior exhibited after integration is another important aspect of the development plan. A software is designed with a certain behavior and functionality in mind. Functional testing deals with testing the product for functional issues. Its goal is to ensure that the product is doing what it was intended to do in the design phase.

== Testing Methodologies ==

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering. A Practitioner Approach: New York: McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki1b 4 iftesting

2009-09-27T23:01:45Z

Ird: /* Overview */

== Integration and Function Testing and beyond ==
Software testing is the science of breaking software down. During testing we have one and only one aim to develop comprehensive test cases to find bugs in the software and get them fixed.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy: Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* '''Planning'''
* '''Design'''
* '''Development'''
* '''Testing'''
* '''Release and Maintenance'''

Each stage can be further broken down into a number of several stages. This article focuses on the Testing phase. Software testing can be defined as inspection or investigation of a product to ascertain its quality and verify if its semantics are implemented the way they are intended to. Testing can however, be done outside the Software Development cycle too.

Testing can be broadly classified into the following methodologies [1]:
* '''Unit testing'''
* '''Integration testing'''
* '''Functional or Validation testing'''
* '''System testing'''

The various testing processes/methods listed can be put in chronological order and can be viewed as a chain process. Each phase is an important part of the overall testing process. Unit test concentrates on the basic building blocks of the product, verifying if each component works correctly. The next phase is the Integration Testing. When many such units are knit together to form a larger, more complex block, it becomes necessary to test if all the smaller units of the bigger block interact properly to implement the semantics of the bigger block, as desired. The process that follows this stage is the Functional or Validation testing. This phase focuses on the behavior and performance in most cases. The question asked here is “If all the complex blocks are properly integrated, are they producing the result which they are required to?” The last phase is the System Testing. This mostly deals with interaction on the hardware-software platform and/or integration of the product on a platform, as viewed from a higher level. Here we will focus our discussion on Integration and Functional Testing.

== Purpose ==

Testing is a very important phase in the software development cycle. The cycle can either follow the ''Waterfall'' [http://en.wikipedia.org/wiki/Waterfall_model] model or the ''Spiral'' model [http://en.wikipedia.org/wiki/Spiral_model]. The latter mandates testing continuously as opposed to the former’s discrete phase. No software product is completely free of defects. No matter how much care the design team takes, its impossible to ensure that there are absolutely no bugs. This is where testing comes in. The goal of the testing process is to try and break the software.

Considering a large project, individual modules or units maybe developed by different team or even outsourced to different companies. Each team would have tested their respective modules or units. When, however, these modules are integrated climbing a level up, further testing maybe required to ensure that these units interact with other in a way which does not break the product. An important point in the development plan is the decision on the interface of each module. A common interface eliminates most of the bugs, but a final integration test ensures that everything is clean.

Making sure that the units or modules integrate with each other is not enough. The result produced by integrating the modules and the behavior exhibited after integration is another important aspect of the development plan. A software is designed with a certain behavior and functionality in mind. Functional testing deals with testing the product for functional issues. Its goal is to ensure that the product is doing what it was intended to do in the design phase.

== Testing Methodologies ==

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy:Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy:Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering. A Practitioner Approach: New York: McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki1b 4 iftesting

2009-09-27T23:01:18Z

Ird: /* Functional Testing and beyond */

== Integration and Function Testing and beyond ==
Software testing is the science of breaking software down. During testing we have one and only one aim to develop comprehensive test cases to find bugs in the software and get them fixed.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy:Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* '''Planning'''
* '''Design'''
* '''Development'''
* '''Testing'''
* '''Release and Maintenance'''

Each stage can be further broken down into a number of several stages. This article focuses on the Testing phase. Software testing can be defined as inspection or investigation of a product to ascertain its quality and verify if its semantics are implemented the way they are intended to. Testing can however, be done outside the Software Development cycle too.

Testing can be broadly classified into the following methodologies [1]:
* '''Unit testing'''
* '''Integration testing'''
* '''Functional or Validation testing'''
* '''System testing'''

The various testing processes/methods listed can be put in chronological order and can be viewed as a chain process. Each phase is an important part of the overall testing process. Unit test concentrates on the basic building blocks of the product, verifying if each component works correctly. The next phase is the Integration Testing. When many such units are knit together to form a larger, more complex block, it becomes necessary to test if all the smaller units of the bigger block interact properly to implement the semantics of the bigger block, as desired. The process that follows this stage is the Functional or Validation testing. This phase focuses on the behavior and performance in most cases. The question asked here is “If all the complex blocks are properly integrated, are they producing the result which they are required to?” The last phase is the System Testing. This mostly deals with interaction on the hardware-software platform and/or integration of the product on a platform, as viewed from a higher level. Here we will focus our discussion on Integration and Functional Testing.

== Purpose ==

Testing is a very important phase in the software development cycle. The cycle can either follow the ''Waterfall'' [http://en.wikipedia.org/wiki/Waterfall_model] model or the ''Spiral'' model [http://en.wikipedia.org/wiki/Spiral_model]. The latter mandates testing continuously as opposed to the former’s discrete phase. No software product is completely free of defects. No matter how much care the design team takes, its impossible to ensure that there are absolutely no bugs. This is where testing comes in. The goal of the testing process is to try and break the software.

Considering a large project, individual modules or units maybe developed by different team or even outsourced to different companies. Each team would have tested their respective modules or units. When, however, these modules are integrated climbing a level up, further testing maybe required to ensure that these units interact with other in a way which does not break the product. An important point in the development plan is the decision on the interface of each module. A common interface eliminates most of the bugs, but a final integration test ensures that everything is clean.

Making sure that the units or modules integrate with each other is not enough. The result produced by integrating the modules and the behavior exhibited after integration is another important aspect of the development plan. A software is designed with a certain behavior and functionality in mind. Functional testing deals with testing the product for functional issues. Its goal is to ensure that the product is doing what it was intended to do in the design phase.

== Testing Methodologies ==

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy:Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy:Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy: Latvian Technological Center[http://www.innovation.lv]]] A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering. A Practitioner Approach: New York: McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki1b 4 iftesting

2009-09-27T23:00:40Z

Ird: /* Functional Testing and beyond */

== Integration and Function Testing and beyond ==
Software testing is the science of breaking software down. During testing we have one and only one aim to develop comprehensive test cases to find bugs in the software and get them fixed.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy:Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* '''Planning'''
* '''Design'''
* '''Development'''
* '''Testing'''
* '''Release and Maintenance'''

Each stage can be further broken down into a number of several stages. This article focuses on the Testing phase. Software testing can be defined as inspection or investigation of a product to ascertain its quality and verify if its semantics are implemented the way they are intended to. Testing can however, be done outside the Software Development cycle too.

Testing can be broadly classified into the following methodologies [1]:
* '''Unit testing'''
* '''Integration testing'''
* '''Functional or Validation testing'''
* '''System testing'''

The various testing processes/methods listed can be put in chronological order and can be viewed as a chain process. Each phase is an important part of the overall testing process. Unit test concentrates on the basic building blocks of the product, verifying if each component works correctly. The next phase is the Integration Testing. When many such units are knit together to form a larger, more complex block, it becomes necessary to test if all the smaller units of the bigger block interact properly to implement the semantics of the bigger block, as desired. The process that follows this stage is the Functional or Validation testing. This phase focuses on the behavior and performance in most cases. The question asked here is “If all the complex blocks are properly integrated, are they producing the result which they are required to?” The last phase is the System Testing. This mostly deals with interaction on the hardware-software platform and/or integration of the product on a platform, as viewed from a higher level. Here we will focus our discussion on Integration and Functional Testing.

== Purpose ==

Testing is a very important phase in the software development cycle. The cycle can either follow the ''Waterfall'' [http://en.wikipedia.org/wiki/Waterfall_model] model or the ''Spiral'' model [http://en.wikipedia.org/wiki/Spiral_model]. The latter mandates testing continuously as opposed to the former’s discrete phase. No software product is completely free of defects. No matter how much care the design team takes, its impossible to ensure that there are absolutely no bugs. This is where testing comes in. The goal of the testing process is to try and break the software.

Considering a large project, individual modules or units maybe developed by different team or even outsourced to different companies. Each team would have tested their respective modules or units. When, however, these modules are integrated climbing a level up, further testing maybe required to ensure that these units interact with other in a way which does not break the product. An important point in the development plan is the decision on the interface of each module. A common interface eliminates most of the bugs, but a final integration test ensures that everything is clean.

Making sure that the units or modules integrate with each other is not enough. The result produced by integrating the modules and the behavior exhibited after integration is another important aspect of the development plan. A software is designed with a certain behavior and functionality in mind. Functional testing deals with testing the product for functional issues. Its goal is to ensure that the product is doing what it was intended to do in the design phase.

== Testing Methodologies ==

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy:Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy:Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]. Sometime Functional testing is also called as ''System Testing''.

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy:Latvian Technological Center[http://www.innovation.lv]]]A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering. A Practitioner Approach: New York: McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki1b 4 iftesting

2009-09-27T22:59:47Z

Ird:

== Integration and Function Testing and beyond ==
Software testing is the science of breaking software down. During testing we have one and only one aim to develop comprehensive test cases to find bugs in the software and get them fixed.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy:Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
* '''Planning'''
* '''Design'''
* '''Development'''
* '''Testing'''
* '''Release and Maintenance'''

Each stage can be further broken down into a number of several stages. This article focuses on the Testing phase. Software testing can be defined as inspection or investigation of a product to ascertain its quality and verify if its semantics are implemented the way they are intended to. Testing can however, be done outside the Software Development cycle too.

Testing can be broadly classified into the following methodologies [1]:
* '''Unit testing'''
* '''Integration testing'''
* '''Functional or Validation testing'''
* '''System testing'''

The various testing processes/methods listed can be put in chronological order and can be viewed as a chain process. Each phase is an important part of the overall testing process. Unit test concentrates on the basic building blocks of the product, verifying if each component works correctly. The next phase is the Integration Testing. When many such units are knit together to form a larger, more complex block, it becomes necessary to test if all the smaller units of the bigger block interact properly to implement the semantics of the bigger block, as desired. The process that follows this stage is the Functional or Validation testing. This phase focuses on the behavior and performance in most cases. The question asked here is “If all the complex blocks are properly integrated, are they producing the result which they are required to?” The last phase is the System Testing. This mostly deals with interaction on the hardware-software platform and/or integration of the product on a platform, as viewed from a higher level. Here we will focus our discussion on Integration and Functional Testing.

== Purpose ==

Testing is a very important phase in the software development cycle. The cycle can either follow the ''Waterfall'' [http://en.wikipedia.org/wiki/Waterfall_model] model or the ''Spiral'' model [http://en.wikipedia.org/wiki/Spiral_model]. The latter mandates testing continuously as opposed to the former’s discrete phase. No software product is completely free of defects. No matter how much care the design team takes, its impossible to ensure that there are absolutely no bugs. This is where testing comes in. The goal of the testing process is to try and break the software.

Considering a large project, individual modules or units maybe developed by different team or even outsourced to different companies. Each team would have tested their respective modules or units. When, however, these modules are integrated climbing a level up, further testing maybe required to ensure that these units interact with other in a way which does not break the product. An important point in the development plan is the decision on the interface of each module. A common interface eliminates most of the bugs, but a final integration test ensures that everything is clean.

Making sure that the units or modules integrate with each other is not enough. The result produced by integrating the modules and the behavior exhibited after integration is another important aspect of the development plan. A software is designed with a certain behavior and functionality in mind. Functional testing deals with testing the product for functional issues. Its goal is to ensure that the product is doing what it was intended to do in the design phase.

== Testing Methodologies ==

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy:Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy:Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test. [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy:Latvian Technological Center[http://www.innovation.lv]]]A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad'' [https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering. A Practitioner Approach: New York: McGraw-Hill, 2001, p386-p429.]
# http://en.wikipedia.org/wiki/Waterfall_model
# http://en.wikipedia.org/wiki/Spiral_model
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# http://www.bugzilla.org/
# https://launchpad.net/
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/

CSC/ECE 517 Fall 2009/wiki1b 4 iftesting

2009-09-27T22:56:21Z

Ird:

== Integration and Function Testing and beyond ==
Software testing is the science of breaking software down. During testing we have one and only one aim to develop comprehensive test cases to find bugs in the software and get them fixed.

== Overview ==

Every software product goes though a ''Software Process'' or more commonly known as ''Software Development Cycle''. The various stages in the cycle are: [[image:Sw_dev_cycle.jpg|thumb|300px|Waterfall model of Software Development Cycle. Image courtesy:Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]][1]
* '''Planning'''
* '''Design'''
* '''Development'''
* '''Testing'''
* '''Release and Maintenance'''

Each stage can be further broken down into a number of several stages. This article focuses on the Testing phase. Software testing can be defined as inspection or investigation of a product to ascertain its quality and verify if its semantics are implemented the way they are intended to. Testing can however, be done outside the Software Development cycle too.

Testing can be broadly classified into the following methodologies [1]:
* '''Unit testing'''
* '''Integration testing'''
* '''Functional or Validation testing'''
* '''System testing'''

The various testing processes/methods listed can be put in chronological order and can be viewed as a chain process. Each phase is an important part of the overall testing process. Unit test concentrates on the basic building blocks of the product, verifying if each component works correctly. The next phase is the Integration Testing. When many such units are knit together to form a larger, more complex block, it becomes necessary to test if all the smaller units of the bigger block interact properly to implement the semantics of the bigger block, as desired. The process that follows this stage is the Functional or Validation testing. This phase focuses on the behavior and performance in most cases. The question asked here is “If all the complex blocks are properly integrated, are they producing the result which they are required to?” The last phase is the System Testing. This mostly deals with interaction on the hardware-software platform and/or integration of the product on a platform, as viewed from a higher level. Here we will focus our discussion on Integration and Functional Testing.

== Purpose ==

Testing is a very important phase in the software development cycle. The cycle can either follow the ''Waterfall'' [http://en.wikipedia.org/wiki/Waterfall_model] model or the ''Spiral'' model [http://en.wikipedia.org/wiki/Spiral_model]. The latter mandates testing continuously as opposed to the former’s discrete phase. No software product is completely free of defects. No matter how much care the design team takes, its impossible to ensure that there are absolutely no bugs. This is where testing comes in. The goal of the testing process is to try and break the software.

Considering a large project, individual modules or units maybe developed by different team or even outsourced to different companies. Each team would have tested their respective modules or units. When, however, these modules are integrated climbing a level up, further testing maybe required to ensure that these units interact with other in a way which does not break the product. An important point in the development plan is the decision on the interface of each module. A common interface eliminates most of the bugs, but a final integration test ensures that everything is clean.

Making sure that the units or modules integrate with each other is not enough. The result produced by integrating the modules and the behavior exhibited after integration is another important aspect of the development plan. A software is designed with a certain behavior and functionality in mind. Functional testing deals with testing the product for functional issues. Its goal is to ensure that the product is doing what it was intended to do in the design phase.

== Testing Methodologies ==

=== Integration Testing and beyond ===

Integration testing is important considering the fact that the final product would be viewed as a single entity by the users/customers rather than a collection of separate units. There are two approaches to integration testing: Incremental and Non-incremental integration testing. The latter is usually more difficult since testing the program as a whole would unearth a lot of errors and fixing one error may introduce others. Integration testing can be performed though a variety of ways. Following is a list of the popular methods:
[[image:Top-down.jpg|thumb|300px|Top-down approach of Integration testing. Image courtesy:Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]][1]
==== Top-down Testing ====

This is a type of incremental integration testing of the product/program. The name is self-explanatory which implies that the testing assumes a top-down approach where a chain of flow indicates the various paths. After identifying the paths, tests can either be performed horizontally or vertically along the path. Testing horizontally tests modules at the same level, while testing vertically tests a full path.

Depending on whether the test is conducted horizontally or vertically, tests are performed on the modules along the path. Thus depending on the approach, tests are conducted as the modules are integrated. A good coding practice is to separate the data and control modules. The control path should be mostly concentrated in the main/top module. Hence, one advantage of using the vertical approach is that it tests a complete control path at an early stage of integration and thus this gives the development/design team more cushion to work on the modules and fix errors, if any.

A top-down approach may be more time consuming than other methodologies, though. This is because, unless each module is tested chronologically, the bug maybe difficult to trace due to dependencies. While there exists a method to deal with this, viz., replacing the modules in each level with minimal functional test modules, it doesn’t give accurate results[1].

==== Bottom-up Testing ====
[[image:Bottom-up.jpg|thumb|300px|Bottom-up approach of Integration testing. Image courtesy:Pressman, Roger S. Software Engineering: A Practitioner's Approach", 2001 ''Fifth Ed.]][1]
Bottom-up approach is another type of incremental testing where the testing tree is construction from bottom-up. In a way it can be considered as an incremental unit-test as the modules are integrated. Since the approach requires a lot of test modules, the smaller modules are usually grouped to form a meta-module. This is done horizontally so that meta-modules (clusters) are in parallel paths in the final tree. This is helpful, since any information about another meta-module in the same level is available at the same time as the meta-module in consideration is under test. This is important because parallel meta-modules may be dependent on one another[1].

==== Big Bang Approach ====

This is arguably one of the more difficult tests. This approach involves bringing all the methods together and running test on the meta-module. This test can only be run when the product is in the final development stages. One advantage of this method is that it can be quickly performed. Since the test is done on almost complete sets, the test cases are usually more real-scenario like which translates to better testing. The biggest disadvantage of this method is however the manageability. The code-base is so large that it becomes difficult to ensure that everything is tested thoroughly. Also, fixing one part can affect other parts of the program. Integration and final testing become quite cumbersome if not handled properly.[http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing]

==== Regression Testing ====

Testing is a continuous process in the ''Spiral'' model and in fact, the same is encouraged in the ''Waterfall'' model, to test as soon as unit is completes the implementation phase. Regression test refers to the process of continuous test to the bigger meta-module as soon as a new model is added to ensure that the overall cluster is working as intended. While Regression testing may not be strictly classified as an Integration test, it fits in here well.[http://www.cs.umd.edu/~aporter/html/currTesting.html]

Bug-fixing in one part of the module may lead to a bug in another part of the same module. To ensure everything works smoothly, the tests need to be run again on modules that had previously passed the tests. Hence Regression Test in an integral part of the testing phase and an important too. A complex can have a very large set of regression tests. It is always a good idea to categorize the regression test and have one test representing a category.

There are no hard and fast rules about the Integration testing method to be used. Some software packages may benefit from Top-down approach while for some, Bottom-up may be more feasible. Sandwich-method – which is like “best of both worlds” is often employed for different stages of a product.

----

=== Functional Testing and beyond ===

Functional testing maybe thought as a part of the ''Quality Control program''. Functional test is very important since it validates that the product behaves and works, as it was intended to, during the design phase. Functional test is like an accuracy test. [https://wiki.cac.washington.edu/display/SWTest/Functional+Testing]

A lot of people are confused between ''Functional'' and ''Unit testing''. The main thing that differentiates Functional testing from Unit testing is the order of tests conducted. If the order of the module tests is immaterial to the final outcome then the test can be classified as a Unit test, since in a Functional test the final outcome is dependent on the intermediate test results.

During functional testing it is required that the testing team have the ''Design Specification'' so that the results can be validated. The software may also be tested for some performance metric, which also qualifies as functional testing. The Functional testing phase must have definite test plan which includes the process that needs to be carried out as well as deliverables. Also, since this test evaluates the overall behavior of the product, a scope must be defined so that there are a logical number of test cases. [http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing]

==== Functional Decomposition and Blackbox testing ====

A very practical approach to functional testing is to break down the big block into smaller functional blocks. This way, it’s possible to catch a bug at an early stage before it propagates deeper into the tree, where it can be a really difficult process. This breakdown is termed Functional Decomposition. The testing team must also maintain Traceability. This refers to the fact that each test cases or a group of test cases can be associated with certain functional requirement. It would be incorrect to map all the test cases to the same requirement. This further raises the question: Does failure of one test mean, failure to satisfy the requirement? The answer to this question can only be given if a subset of tests is mapped to a certain requirement. This is one aspect that is overlooked by many testing teams.

[[image:Blackbox.gif|thumb|300px|Black-box testing. Image courtesy:Latvian Technological Center[http://www.innovation.lv]]]A common testing strategy employed for functional testing is the ''Blackbox testing''. In this testing strategy, the testing team does not need to be aware of the internal semantics of the product. Instead the team must know what inputs will product what outputs. This eliminates much of the complexity involved in understanding the logic. The higher the level of the meta-module, the bigger the Blackbox. The biggest advantage is that the core development team need not give implementation specific details to the testing team, if outsourced. This is also time saving. One disadvantage however, is that the number of test cases increases exponentially as the meta-module become more and more complex.

==== Bug Tracking Systems ====

A popular way of functional test tracking is by using ''Bug-Tracking Systems''. A popular open-source BTS is ''BugZilla'' [http://www.bugzilla.org/] developed and used my Mozilla. Another example of a BTS is ''Launchpad''[https://launchpad.net/]. Popular projects like ''Ubuntu GNU/Linux'', ''Nautilus'' etc. make use of ''Launchpad'' for bug tracking and functional testing. The database of the bug-tracking system is probably the most important part of the BTS. Releasing the software in alpha or beta version is a common practice in the software industry for testing of the same by a much larger audience.

Documentation is a very important aspect of testing. A periodic report of the tests conducted, results and related comments must be maintained so that the design team is aware of the improvements to be made. This document can include Coverage analysis at the very least.

== Testing Tools ==

A number of testing tools and suites are available today, which help in integration and functional testing. A discussion on them is beyond the scope of this wiki but the reader is free to explore. Here are some links that the reader may find useful:

* [http://www.eclipse.org/tptp Eclipse TPTP]
* [http://sourceforge.net/projects/webunitproj Enterprise Web Test]
* [http://ldtp.freedesktop.org GNU/Linux Desktop Testing Project]
* [http://xmltestsuite.sourceforge.net/ XML Test Suite]
* [http://valgrind.org/ Valgrind]
* [http://www.webload.org/ WebLOAD]
* [http://jakarta.apache.org/jmeter/ Apache JMeter]
* [http://www.manageengine.com/products/qengine/ QEngine]

More opensource tools may be found [http://www.opensourcetesting.org here].

== Summary ==
Software Testing is relatively a new field and we have only just began to develop a systematic approach to it. The scope of research and innovation in this new field is limitless and with the advances in support for testing tools the process is getting more formalized and automated day by day. Nevertheless one thing is guaranteed that a software can never be guaranteed to be bug free and every time a customer uses it, it will undergo yet another test.

== References ==

# [http://www.mhhe.com/engcs/compsci/pressman/index.mhtml R. S. Pressman. Software Engineering. A Practitioner Approach: New York: McGraw-Hill, 2001, p386-p429.]
# http://www.devbistro.com/articles/Testing/Requirements-Based-Functional-Testing
# https://wiki.cac.washington.edu/display/SWTest/Functional+Testing
# http://docs.joomla.org/Functional_Testing
# http://www.opensourcetesting.org/
# http://www.bugzilla.org/
# https://launchpad.net/
# http://www.cs.umd.edu/~aporter/html/currTesting.html
# http://www.testinggeek.com/index.php/testing-types/life-cycle/55-bigbang-integration-testing