CSC/ECE 517 Fall 2009/wiki1b 4 xy

Functional and Integration testing and beyond

Recognition of software test engineering as a specialty area has increased during the last twenty years. Testing is one of the key focus areas in software engineering. It is an essential activity that observes the functionality of a software system validates its intended behavior and identifies the potential malfunctions. Testing is also the primary way to improve software reliability.

The test cases should be carefully designed to cover many aspects in the SDLC such as possible aberrations from user’s requirements, system behavior upon malicious inputs, and robustness to extensive load conditions and so on. As more and more software is used in critical applications, the time and cost involved in testing is also increasing owing to the higher quality required. Although test automation saves a lot of time and can be used for regression load tests, manual testing is still widely used in the industry.

Terms

1. Driver: Supporting code or data used for testing the software system
2. SDLC: Software Development Life Cycle
3. Stub: Dummy procedure that can be used to simulate or substitute an actual portion of a system
4. Sub-system: Component or Module of a software system
5. System Under Test: The software system which has to be tested
6. Test: Sequence of steps to validate the system design and implementation
7. Usecase :Scenario that describes the interaction between an user and a system

Functional testing

Functional testing is designed to ensure whether the system requirements and specifications are achieved. It can start with testing of the requirements and whether the application complies with the organizational policies and procedures. During the design phase, after knowing the functionality of the module/modules and the system, some errors can be predicted in advance. A functional test should incorporate such type of scenarios too. In this case the erroneous transactions/ applications are deliberately introduced and the system should be able to correctly find them. In general, all the use cases in the user’s requirements become the functional test cases.

For example, suppose we are to develop the software for an elevator. For designing the test cases for this software, 
all the scenarios and operations that can take place in the elevator need to be analyzed. To begin from the basic operations, the
elevator should be bidirectional and it should stop at each floor (unless we have two elevators in the requirement for even and 
odd floors separately).The elevator should stop at the desired floor.The behavior of the elevator when the eight inside exceeds 
the maximum allowable limit ( i.e to test the buffer conditions). There can be many other scenarios.

Generally Functional testing is Black Box testing. In this method the SUT (System under Test) is considered as a Black box. In this phase of testing the internals of the SUT like the design or algorithm are not considered by the tester. Inputs are provided to the black box and the outputs are validated to the expected results specified in the design document.

2 approaches can be taken for writing the functional tests

• Requirement based approach: In this approach the specification for the system or software design is used as the base to write functional test.
• Business process based approach: Business process refer to the day-to-day use of the system. This knowledge is used to do functional testing of the system.

Regression testing is another form of functional testing where you test the earlier features and functionality of the module. They had worked on the desired lines in the previous tests but have to be tested currently after some changes have been made in the system. The changes might not be in the particular module but it can be affected due to the inter dependencies.

Integration Testing

Integration testing is a systematic approach to build the complete software structure specified in the design from unit-tested modules.It is the next phase of unit testing. Generally in software development, complete software is divided into various sub-systems mostly developed by different teams. More often than not each sub-system works very well individually but when they are integrated then lot of problems arise. So the intention of integration tests is not to test the functionality of individual modules but the interaction and inter-dependencies between the modules.

For instance in Object oriented model, the classes in a particular module could have been inherited from some other module. 
The interface can be in a different module and the implementation can be elsewhere. In such scenarios, the faults can be 
best found in the integration tests.

Another instance is to test the communication and compatibility between the client and server. Stand alone client and server 
functionality is tested in the unit tests. 

There are many approaches that can be followed for Integration Testing

Big Bang Testing Approach

As the name suggests all the components are integrated at once. After all the integration is completed a set of tests are run which not only validate a single component of sub-system but also test and validate the interaction between different components. This approach is more useful in smaller systems because integration of smaller systems is easier.

• Advantages:
  • Since all the modules are combined and tested at the same time, this approach of integration is less time consuming
• Disadvantages:
  • Debugging and solving a fault is not trivial. The reason being many modules are integrated and not all the bugs are determined in the unit tests.Therefore it is ambiguous for the tester whether the fault is due to the sub-system or due to the integration.
  • Testing process cannot be started until all the sub-systems have finished their unit tests.
  • Designing the test cases is an arduous task.

Incremental Testing Approach

Testing is done after integration of each module. After the successful completion of the unit tests, sub-systems are integrated and tested. Stubs are used for modules in the system which are not integrated till now.

• Advantages:
  • Problems detection is early compared to the Big Bang approach
  • No need to wait till the unit testing of all the modules have finished
  • Debugging is much simpler
• Disadvantages:
  • This approach is a time taking process because the integration tests have to run multiple times
  • Stubs have to be developed to substitute the missing modules

Various Approach can be taken for Incremental Testing

• Top-Down Approach: In this approach the system is tested according to the work flow or code flow. The Top level module is tested first then each lower level module is integrated and tested.Top level modules call other modules and lower level modules are are called by Top level ones. Terminal modules do not call any other module.

In the above figure Module A is the top most module in the system then B,C,D are lower level modules. Generally module A is the user interface and stubs are created for the remaining modules which are used for the tests.As and when the sub-systems are ready after the unit tests, they are replaced by their respective stubs for the integration tests.

• Advantages:
  • The test flow follows the architecture of the system so any bug at the top level design or implementation could be found in the early stages
  • If the top level module calls multiple lower modules then the testing can be done in various combinations

For example in the figure A calls B and C. The testing can be done as follows
A-B-C-D 
A-C-B-D

• Disadvantages :
  • The main problem with this approach is the use of stubs. Mostly stubs are written to print out some trace statement based on the output of module.There can be frequent changes in the stubs as and when the bugs are found out and while testing multiple scenarios.This is an additional overhead.

• Bottom-Up Approach: The bottom-up approach is the opposite of the top-down approach. In this approach the terminal modules are tested first, then the higher modules and finally the top-most one. Drivers are used in this approach.

In the above figure E,D are the terminal modules which are tested first. Then the higher level modules B,C are tested and finally the topmost module is tested

• Advantages:
  • No need to write stubs
  • If the terminal modules does the main work in the system then they will be tested first
• Disadvantages:
  • Need to write drivers
  • The testing is not according to the architecture of the system, in the sense that until all the modules are integrated, the tests do not follow the flow of the design.

Other Type of Testing

System Testing

System testing is testing the working of the system or program with respect to the orginal objectives of the system. This is slightly different from Functional testing. In Functional testing we basically test the use-case of the system while in system testing we are looking for discrepancies between the working of the system and the objectives. There are various categories of system tests some of the main ones are listed below

• Compatibility Tests: They check the compatibility of the system which other system.

For Example: Testing a routing protocol in one router with the one running on a system from another vendor

• Security Tests: They test the security feature of the system.

• Stress Tests: These type of tests stress the system to the maximum to find out how robust they are.

For Example: A good stress test for a web server would be sending HTTP requests at rate for the peak rate for short 
period of time

• Volume Tests: These test subject the system to heavy loads of data.

For Example: Sending HTTP Requests at a high rate ( less than the peak rate) to a web server for a significant amount of 
time is volume testing

• Usability Test: They are used to test if the system is easy to use of the end-user

For Example: Tests that test UI of any product fall under this category

Acceptance Testing

The testing done by user or customer after before accepting the system. The main focus of the testing to establish confidence in the user regarding the system.

For Example: In the case of a program product, such as a computer manufacturer’s operating system or compiler, or a software 
company’s database management system, the sensible customer first performs an acceptance test to determine whether the product  
satisfies its needs