CSC/ECE 517 Fall 2009/wiki1b 4 xy

Functional and Integration testing and beyond

Recognition of software test engineering as a specialty area has increased during the last twenty years. Testing is one of the key focus areas in software engineering. It is an essential activity that observes the functionality of a software system validates its intended behavior and identifies the potential malfunctions. Testing is also the primary way to improve software reliability.

The test cases should be carefully designed to cover many aspects in the SDLC such as possible aberrations from user’s requirements, system behavior upon malicious inputs, and robustness to extensive load conditions and so on. As more and more software is used in critical applications, the time and cost involved in testing is also increasing owing to the higher quality required. Although test automation saves a lot of time and can be used for regression load tests, manual testing is still widely used in the industry.

Terms

1. Driver: Supporting code or data used for testing the software system
2. SDLC: Software Development Life Cycle
3. Stub: Dummy procedure that can be used to simulate or substitute an actual portion of a system
4. Sub-system: Component or Module of a software system
5. System Under Test: The software system which has to be tested
6. Test: Sequence of steps to validate the system design and implementation
7. Usecase :Scenario that describes the interaction between an user and a system

Functional testing

Functional testing is designed to ensure whether the system requirements and specifications are achieved. It can start with testing of the requirements and whether the application complies with the organizational policies and procedures. During the design phase, after knowing the functionality of the module/modules and the system, some errors can be predicted in advance. A functional test should incorporate such type of scenarios too. In this case the erroneous transactions/ applications are deliberately introduced and the system should be able to correctly find them. In general, all the use cases in the user’s requirements become the functional test cases.

For example, suppose we are to develop the software for an elevator. For designing the test cases for this software, 
all the scenarios and operations that can take place in the elevator need to be analyzed. To begin from the basic operations, the
elevator should be bidirectional and it should stop at each floor (unless we have two elevators in the requirement for even and 
odd floors separately).The elevator should stop at the desired floor.The behavior of the elevator when the eight inside exceeds 
the maximum allowable limit ( i.e to test the buffer conditions). There can be many other scenarios.

Generally Functional testing is Black Box testing. In this method the SUT (System under Test) is considered as a Black box. In this phase of testing the internals of the SUT like the design or algorithm are not considered by the tester. Inputs are provided to the black box and the outputs are validated to the expected results specified in the design document.

2 approaches can be taken for writing the functional tests

• Requirement based approach: In this approach the specification for the system or software design is used as the base to write functional test.
• Business process based approach: Business process refer to the day-to-day use of the system. This knowledge is used to do functional testing of the system.

Regression testing is another form of functional testing where you test the earlier features and functionality of the module. They had worked on the desired lines in the previous tests but have to be tested currently after some changes have been made in the system. The changes might not be in the particular module but it can be affected due to the inter dependencies.

Integration Testing

Integration testing is a systematic approach to build the complete software structure specified in the design from unit-tested modules.It is the next phase of unit testing. Generally in software development, complete software is divided into various sub-systems mostly developed by different teams. More often than not each sub-system works very well individually but when they are integrated then lot of problems arise. So the intention of integration tests is not to test the functionality of individual modules but the interaction and inter-dependencies between the modules.

For instance in Object oriented model, the classes in a particular module could have been inherited from some other module. 
The interface can be in a different module and the implementation can be elsewhere. In such scenarios, the faults can be 
best found in the integration tests.

Another instance is to test the communication and compatibility between the client and server. Stand alone client and server 
functionality is tested in the unit tests. 

There are many approaches that can be followed for Integration Testing

Big Bang Testing Approach

As the name suggests all the components are integrated at once. After all the integration is completed a set of tests are run which not only validate a single component of sub-system but also test and validate the interaction between different components. This approach is more useful in smaller systems because integration of smaller systems is easier.

• Advantages:
  • Since all the modules are combined and tested at the same time, this approach of integration is less time consuming
• Disadvantages:
  • Debugging and solving a fault is not trivial. The reason being many modules are integrated and not all the bugs are determined in the unit tests.Therefore it is ambiguous for the tester whether the fault is due to the sub-system or due to the integration.
  • Testing process cannot be started until all the sub-systems have finished their unit tests.
  • Designing the test cases is an arduous task.

Incremental Testing Approach

Testing is done after integration of each module. After the successful completion of the unit tests, sub-systems are integrated and tested. Stubs are used for modules in the system which are not integrated till now.

• Advantages:
  • Problems detection is early compared to the Big Bang approach
  • No need to wait till the unit testing of all the modules have finished
  • Debugging is much simpler
• Disadvantages:
  • This approach is a time taking process because the integration tests have to run multiple times
  • Stubs have to be developed to substitute the missing modules

Various Approach can be taken for Incremental Testing

• Top-Down Approach: In this approach the system is tested according to the work flow or code flow. The Top level module is tested first then each lower level module is integrated and tested.Top level modules call other modules and lower level modules are are called by Top level ones. Terminal modules do not call any other module.

In the above figure Module A is the top most module in the system then B,C,D are lower level modules. Generally module A is the user interface and stubs are created for the remaining modules which are used for the tests.As and when the sub-systems are ready after the unit tests, they are replaced by their respective stubs for the integration tests.

• Advantages:
  • The test flow follows the architecture of the system so any bug at the top level design or implementation could be found in the early stages
  • If the top level module calls multiple lower modules then the testing can be done in various combinations

For example in the figure A calls B and C. The testing can be done as follows
A-B-C-D 
A-C-B-D

• Disadvantages :
  • The main problem with this approach is the use of stubs. Mostly stubs are written to print out some trace statement based on the output of module.There can be frequent changes in the stubs as and when the bugs are found out and while testing multiple scenarios.This is an additional overhead.

• Bottom-Up Approach: The bottom-up approach is the opposite of the top-down approach. In this approach the terminal modules are tested first, then the higher modules and finally the top-most one. Drivers are used in this approach.

In the above figure E,D are the terminal modules which are tested first. Then the higher level modules B,C are tested and finally the topmost module is tested

• Advantages:
  • No need to write stubs
  • If the terminal modules does the main work in the system then they will be tested first
• Disadvantages:
  • Need to write drivers
  • The testing is not according to the architecture of the system, in the sense that until all the modules are integrated, the tests do not follow the flow of the design.

Other Types of Testing

System Testing

Every software system has well defined objectives and a comprehensive functionality.The intention here is to test the working of the system with respect to its original objectives. This is slightly different from functional testing. In functional testing we basically test all possible use-cases of the system while in system testing we are looking for discrepancies between the actual behavior as compared to its objectives. There are various categories of system tests some of the main ones are listed below

• Compatibility Tests: It involves checking the compatibility when interacted or used in cohesion with other systems.

For Example: A routing protocol has a set of standards. The implementation can be vendor specific. When the two different 
implementations of the protocols run on different routers, compatibility tests are conducted to ensure the communication between 
them.

• Security Tests: They test the security feature of the system.

For Example: Security alarms are equipped with a secret code and the people having access have the information about them. They
are tested for all possible incorrect codes to prevent an unauthorized person from entering the protected area.

• Stress Tests: These type of tests stress the system to the maximum to find out the robustness.

For Example: A good stress test for a web server would be sending HTTP requests at the peak rate for short 
period of time

• Volume Tests: These tests subject the system to heavy loads of data.

For Example: Sending HTTP Requests at a high rate ( less than the peak rate) to a web server for a significant amount of 
time

• Usability Test: They are used to test if the system is user friendly

For Example: In a banking software, the employee is unaware of the implementation details and he / she should be able to perform 
the transactions with considerable ease.

Acceptance Testing

The testing done by user or customer before accepting the software. The main focus of this testing is to establish confidence in the user regarding the system.

For Example: In the case of a program product such as a computer manufacturer’s operating system or compiler or a software 
company’s database management system, the customer first performs an acceptance test to determine whether the product  
satisfies his needs

References

1. Art of Software Testing by Glenford J. Myers
2. FOUNDATIONS OF SOFTWARE TESTING ISTQB CERTIFICATION Dorothy Graham Erik van Veenendaal Isabel Evans Rex Black
3. www.softwaretestinghelp.com/types-of-software-testing/
4. [http://www.cs.pitt.edu/~mock/cs1530/lectnotes.html
5. Software Testing Research: Achievements, Challenges, Dreams. Antonia Bertolino. In Future of Software Engineering, 29th International Conference on Software Engineering, May 2007
6. publib.boulder.ibm.com/infocenter
7. Software Testing. Gregory M. Kapfhammer. The Computer Science and Engineering Handbook