CSC/ECE 517 Fall 2010/ch2 2e AR

Writing Meaningful Test Cases

The Prologue

I believe that whenever we start telling any story, it has a prologue associated with it. So before we start discussing the effective testing practices, we will have a quick reference for some concepts which are used in the development and testing of applications. You may skip this section of you to get to the actual discussion on the test cases.

Software Development Life Cycle (SDLC)

This is the “user manual” for the development of any software. It mainly defines how and when all stages of development and testing are to be approached, referred to as the Software Development Model. The article contains many details about the different approaches in SDLC and different models as well. In reality, it is very rare that a certain model will be followed rigidly. In the industry, the models are customized and tailored as per needs of the project

http://en.wikipedia.org/wiki/Software_development_process

Software Testing

This is a topic which is not only responsible for most of the arguments between Developers, Testers and Managers, but also most important when it comes to ensure good quality of the application. So to speak, it is a necessary evil. Testing normally goes in tandem with development and it has been shown that the earlier the testing process is started in the SDLC, the lesser the chance of a defect propagating till the production phase.

http://en.wikipedia.org/wiki/Software_testing

The Pareto Principle

The Pareto Principle is not something that a Manager or Analyst came up with on one boring day. It has been well researched and documented over the years and it has implications in many areas other than software. From the Software testing point of view, a general approximation is that ‘80% of bugs get fixed in 20% of time’ and “By fixing the top 20% of the most reported bugs, 80% of the errors and crashes would be eliminated”

http://en.wikipedia.org/wiki/Pareto_principle http://www.crn.com/news/security/18821726/microsofts-ceo-80-20-rule-applies-to-bugs-not-just-features.htm

Back to the actual topic of discussion.

Introduction

When it comes to testing, the term 100% code coverage can be quite misleading especially when it comes to Object Oriented Programming (OOP). OOP mainly focuses on the object space while structured programming is process driven. So while testing an application at Unit level, Integration testing level or in system testing level, it is very likely that less importance might be given to test the most integral aspects of the code, considering it as a trivial task. Hence, even though there is 100% code coverage, most of the test cases might return incorrect results. We aim to develop test cases that focus on the proper assertions as well as test for seemingly trivial, but important components of the code.

Unit Testing

Unit testing aims at verifying if each line of code adheres to the set guidelines and that each code block accomplishes the function described in the functional and technical requirements. (Technical and functional requirements are based on the business requirements of the clients, so Unit testing can be thought of as the first level of testing.) This testing is mostly done by the developers themselves and is seldom done by others like debuggers. This requires code visibility and is done at the same time that the code is being written. One type of programming which uses this as a base in development activities is the Test Driven Development. Test code is developed based on the requirements and the code initially fails. Failure of the test code is used as a building block and specific code is written keeping the failed Test case in mind. To reduce effort, the test code is written only once and regression cycles are run using the test cases to test the actual code.

Advantages and Issues

Advantages

• The code is visible for testing, so based on personal experience, testing can be done on vulnerable areas quickly. Also, since the size of the code block is small (since unit testing is broken down into many stages), it is easier to catch a bug as compared to searching for it in millions of lines of codes (major drawback of the Waterfall SDLC model)
• Changes made in the code can be tested easily. Also, during coding, redundant code can be eliminated.
• Reduction in the bug propagation to next stages in SDLC. This reduces overall coding effort and time spent for debugging.
• Since bugs are eliminated at the base level, further ripple effects of the bugs in other modules can be avoided.
• Quality assurance for technical and management audits.

Issues

• Separate coding effort is required for writing the unit test cases and regression logic.
• If there is a change in business logic/requirement in later stages of the project, the complete cycle has to be repeated and new test code has to be written for the change.

Best Practices in Unit Testing

Trivial testing

“What code should be tested during Unit testing? Well… all of it”

Although a completely exhaustive testing effort is not possible in the testing life cycle, it is possible to do that during the Unit test level. No code can be considered “trivial” code. (if typos can cause bugs and waste developer time, bugs in meaningful code have far worse ramifications) This can be simplified based on experience. Using personal experience, certain vulnerabilities can be tested easily. Also, common sense dictates that if there is a code change, it is going to have a ripple somewhere. So these have to be tested as well.

e.g.

    private double weight_;
    private double x_, y_;
    public void setWeight(int weight)
    {
      weight = weight_;  // error
    }
    public double getX()
    {
      return x_;
    }
    public double getY()
    {
      return x_;  // error
    }

Having said that, there is always a trade-off that has to be done while defining trivial test cases, since it costs time and money to test them out. Hence a certain approximation is done for the same. A simple example of the trivial test cases is provided here: http://www.petterhesselberg.com/trivial-testing.html

Independence from other test cases and other environments

Dependence on other test cases destroys the robustness of the test case and also makes it difficult for automation. Also, if the test case depends on other external environments, then the condition will have to be followed each time testing has to be done on that code. E.g. Suppose the code is parsing external files from a predefined location. For testing, the files may be included in the test, so that there is no dependence on the other system.

Fast and small test cases

Unit test cases should be small and should concentrate on one scenario alone. The more looping and conditions used, the less useful it is since all conditions have to pass for the test case to pass. Also, the test cases are normally written as part of a test suite. The slow test case acts as a weak link for the test suite.

High execution coverage

Code coverage can be misleading, so it is better to have high execution coverage. This will ensure that the code is actually executed on some input parameters. http://en.wikipedia.org/wiki/Code_coverage

e.g. Consider the following public method:

    void setLength(double length);

By calling setLength(1.0) you might get 100% execution coverage. To acheive 100% actual test coverage the method must be called for every possible double value

Tools for Unit Testing

There are many tools available across various programming platforms. A list of all these unit testing frameworks is provided on this page. Among the most popular are jUnit and TestNG for Java. http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks

Functional Testing

As compared to Unit tests, the functional testing can be done by anyone provided the proper documentation is present. Functional Testing mainly focuses on evaluating the behavior of the code with respect to quality standards and requirements put forward for the application. The functional test approaches the application testing from the User’s perspective, i.e. it checks if the correct input is given to the application, is the application processing the input as per the constraints of the design requirements and most importantly, is it giving the desired input for a certain input.

Advantages and Disadvantages

Advantages

• Anyone can do the testing i.e. programming knowledge or code visibility is not always required (i.e. Black Box testing)
• In all cases (according to the SOX compliance requirements) the role of tester and developer is separate, i.e. a developer cannot sign off on a code he has written. It has to go through the testing phase and has to be signed off by a third person. This avoids any bias and also provides a fresh perspective (Tester) to find bugs
• Any irregularities in the functionality and configuration of environments can be detected since the tester is acting independent of the code.
• The test case documentation can start as soon as the requirement phase is done. By the time the testing actually starts, the documentation can be reviewed through peers and walkthroughs reducing the idle time between test commencement and coding.

Disadvantages

As the tester is not necessarily able to view the code or understand the working, he/she might not be able to understand flow of the code. This calls for dependence on developers for understanding the logic. E.g. the tester might flag a doubt which might be too trivial or non existent since his understanding might be flawed

On the lighter side, arguments between testers and developers are the highest in this phase of testing.

Best Practices

Functional testing mainly acts as a check that the whole application is coded properly i.e. loose ends in the code are tied up. Testers test solely on the basis of the outcome of the requirement analysis phase in SDLC. Hence it is imperative that the documentation (deliverables of the analysis phase) should not be ambiguous and should explain the requirements properly. Test case preparation is of paramount importance since the tester is approaching the testing from the user’s perspective. The general understanding is that given the documentation, anyone should be able to test the application and should not require exhaustive training on the same. The test cases should encompass all the requirements mentioned and tests should cover the functionality of the complete code. For this, testers rely on Functional Requirement Specification document. This gives a mapping of the business logic and functionality of the application. After test cases are done, they are subjected to Static Testing (in the form of reviews, walkthroughs etc.) to iron out any miscommunication. All this can be done before the application actually becomes available for testing, thereby reducing the idle time between coding and testing. A better understanding of this is available here http://en.wikipedia.org/wiki/Software_testing

Writing test cases

Writing Functional test cases requires complete understanding of the requirements. Although the tester doesn’t need coding knowledge, he/she should be able to understand the logical flow of the application. Lack of the same will cause slips and gaps in the test cases. This can be learnt by practice alone. A well experienced tester can check for certain vulnerabilities based on prior experience.It is always advisable to map the functionality to the business requirements and organize them according to the flow, so that nothing is missed.

e.g. Assume a sample scenario where a user is signing up for an email account

<feature 1 Create new user>
 <test for input fields including boundary condition>
 <test for UI issues>
...
…
…
<feature a Login User>
 <test if user exists in database>
 <test for user authentication>
…
…
…
<feature n View Inbox>
 <test if user is logged in>
 <test if user has the necessary permissions to view content>

Feature n depends not only on authentication of feature a, but also on the authenticating the business requirement that the user should be able to view the inbox only if he/she has the permission to do so.

General requirements from the test cases

• They should be easily understandable, i.e. well documented.
• It should outline dependencies for the successful execution of the test case. (such as environment constraints, data requirements etc)
• Proper explanation of procedure to test the scenario identified. This also helps in recreating the defect when the bug is raised.
• It should explain what the expected output was for the test case. This helps in deciding if the test case passed or failed

Bear in mind that the complete exhaustive end to end testing is not a feasible option. Hence the testing is always a trade off based on expense of time and money. The best approach is to randomize and still test as much as possible so that key features and vulnerabilities are avoided.

Regression

Regression is always important since a fixed bug can still impact other sections of the application because of dependencies. Since running the same test cases again and again will take up a lot of time, it is always logical to automate this part.

Tools for Functional Testing

There are many tools available for functional testing. Automation by these saves testing effort.

• Quick Test Professional (Developed and marketed by HP Software & Solutions.)
• Watir(Specifically intended for Web Application testing with Ruby)
• Rational Functional Tester (developed and Marketed by IBM Pvt. Ltd.)
• jFunc (Extension of the jUnit framework)

Unit testing and Functional Testing do not guarantee that there won’t be a bug in later stages. But proper Unit and Functional testing ensures that the number of bugs in later stages is reduced significantly.

References

[1]http://en.wikipedia.org/wiki/Software_development_process

[2]http://en.wikipedia.org/wiki/Software_testing

[3]http://en.wikipedia.org/wiki/Pareto_principle

[4]http://www.crn.com/news/security/18821726/microsofts-ceo-80-20-rule-applies-to-bugs-not-just-features.htm

[5]http://www.petterhesselberg.com/trivial-testing.html

[6]http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks

[7]http://en.wikipedia.org/wiki/Software_testing