CSC/ECE 517 Fall 2012/ch2a 2w29 st

Introduction

This ia a textbook section that covers the online lecture on Testing Overview. Gone are the days when the developers used to write code and toss it to the QA team for testing. Today’s developers are far more responsible for testing their own code. Testing these days is far more automated in the sense that the tester doesn’t need to manually check the output if its correct or not. In this document, we give the history of software testing and introduce some of the newer methods that are being used today such as Behavior Driven Development and Test Driven Development.

History

Brian Kernighan famously quoted that “Debugging is twice as hard as writing the code in the first place.” Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. In other words it meant if you’re as clever as you can be when you write your code, how will you ever debug it.

Another famous computer scientist Dijkstra famously remarked that "testing can be used to show the presence of bugs but never to show their absence”.The fact is that NOTHING, not inspection, not formal proof, not testing, can give 100% certainty of no errors. Yet all these techniques, at some cost, can in fact reduce the errors to whatever level you wish.

In the initial stages i.e when the software industry was in a nascent stage, there was no stress upon software testing. In most of the cases there was just a quick check on the software and then the software was handed over to the client.A thorough testing of application was considered to be huge waste of time and resources.This was basically done so that items in the checklist could just be ticked off and software could be developed and handed over to client as soon as possible.However as time passed the importance of software testing was realized. Now software testing is given as much importance as that of developing a software.The change in trend can be easily explained.

A Study conducted in 2002 by NIST reported that software bugs cost the U.S. economy a loss of atleast 59.5 billion dollars annually.It further reports that more than one third of this cost could have been avoided if better software testing was performed. <ref>http://www.abeacha.com/NIST_press_release_bugs_cost.htm</ref>

It is commonly believed that the earlier a defect is found the cheaper it is to fix it. The following table shows the cost of fixing the defect depending on the stage it was found.For example, if a problem in the requirements is found only post-release, then it would cost 10–100 times more to fix than if it had already been found by the requirements review. With the advent of modern continuous deployment practices and cloud-based services, the cost of re-deployment and maintenance may lessen over time.

Cost to fix a defect		Time detected
		Requirements	Architecture	Construction	System test	Post-release
Time introduced	Requirements	1×	3x	5-10x	10x	10-100x
	Architecture	-	1x	10x	15x	25-100x
	Construction	-	-	1x	10x	10-25x

Another survey by Electric Cloud, the leading provider of software production management (SPM) solutions, conducted in partnership with Osterman Research showed that the majority of software bugs are attributed to poor testing procedures or infrastructure limitations rather than design problems. Additionally, the software test process is generally considered an unpleasant process, with software development professionals rating the use of their companies’ test systems more painful than preparing taxes. Fifty-eight percent of respondents pointed to problems in the testing process or infrastructure as the cause of their last major bug found in delivered or deployed software, not design defects.

Specifically, the survey highlighted the following: Completely automated software testing environments are still rare, with just 12 percent of software development organizations using fully automated test systems. Almost 10 percent reported that all testing was done manually. Forty-six percent of software developers said they do not have time to test as much as they should. More than a third of developers, 36 percent, said they do not believe their companies perform enough pre-release testing. Fifty-three percent said their testing is limited by compute resources.<ref>http://www.electric-cloud.com/news/2010-0602.php</ref>

Importance of Software Testing

Importance of Software Testing:

The importance of software testing can be explained in brief in the following points.

Error Free Software: Software testing needs to be done thoroughly to deliver an error free software to the client. Even a very minute mistake in the the software can have a disastrous effect for the client. For example a small failure in a banking software can result in a wrong balance and millions of dollars worth loss to the client or customers of the client. So it is of prime importance that the software delivered to the client be bug free and accurate.

Variance from the Requirement: One important factor while building a software is to adhere to the client requirements. if a software is not built in accordance to the requirements then the software becomes useless and redundant for the client.This is a lot of trouble and overhead for the software developing firm also as the requirement was not well understood.The budget fixed for the development of th software can easily go overboard.In such scenarios verification and validation process come into picture. The two most important questions that needs to be answered are “Is the product being built right” and “Is the right product being built”.If the answer is negative to any one of these questions it means that the the product developed has a variance from the client requirements and necessary changes needs to be made before going ahead with further development.

Identify Defect and Prevent their Migration: If a defect is detected in the requirement analysis stage ,then rectifying the defect is a lot easier and cheaper. But if the defect is not identified and carried over to the next phases of software development then it become more difficult to fix the defects,So it is highly recommended that software testing process be started right when the software development starts.

Identify Undiscovered Error: If proper importance is not given to software testing i.e it is not done thoroughly and just a superficial testing is done there is high probability that some of the errors will creep through to the next phase. In such case using different software testing methodologies help in identifying the hidden errors.Exploratory testing is one such method. In such case the tester randomly tests the software for bugs and finds out the error. Use the Software in Real Time Environment: Testing a software in development and running it in production are two completely different scenarios.When a developer develops a software he tests it only in a development environment. There is high probability that the software will fail miserably in real time environment.

Do Away with Developer Bias: When a person is designated a tester role his prime responsibility is to test the software. The element of bias is removed. For example when we have a developer testing a software in majority of the cases he is going to be biased towards the software. This is because he has developed the software and its natural human instinct to think one’s product is the best. In such scenario many problems will remain unearthed.

Provide Confidence in the Software: Software testing is also used for asserting the confidence on a developed product. There is a huge difference between being an usable product to operable product. When a software is tested time and over again with a huge degree of success then one can easily approve of the quality of the software developed. <ref>http://www.cbwc-ontario.org/importance-of-software-testing.html</ref>

Testing Today

There are various software development life cycle methodologies available for executing software development projects.Each methodology is unique i.e it is designed for a special purpose and compared to other methodologies has its own advantages and disadvantages.But most methodologies work by dividing the entire life cycle into phases and then sharing tasks across this phases.The common methodologies used for software development and their relationship with respect to testing can be summarized below:

The Waterfall model:

It is one of the most common and earliest structured models for software development.Waterfall models stresses that one should only move to a phase after the completing and perfecting the preceding phase.As a result of this the phases of software development in waterfall model are unique and there is no jumping back and forth between the various stages.

A typical waterfall model consists of the following sequential phases

It consists of the following sequential phases through which the development life cycle progresses: <ref>http://www.guru99.com/testing-methodology.html</ref>

• Requirement analysis: In this phase, software requirements are captured in such a fashion that they can be translated into actual use cases for the system. The requirements can be derived from performance goals, use cases, target deployment, etc.
• System design: In this phase,the interacting components that make up the system are identified, The exposed interfaces and the communication between them,key algorithms and sequence of interaction are defined, In the end of this phase an architecture and design review is conducted to ensure that the design conforms to the previously defined requirements.
• Implementation : This phase is also termed as Coding and unit testing. In this phase,coding is done for the modules that build the system.Code can also be reviewed and functionality of each module individually tested.
• Verification: Also termed as Integration and system testing. In this phase,all the modules in the system are integrated together and tested as a single system for all the use cases.The primary emphasis is upon making sure that all the modules meet the requirements.
• Deployment and maintenance. This is the final phase where the software system is deployed into the production environment. Any errors identified in this phase are corrected and functionality is added/modified to the system based on the updated requirements.

Waterfall model has the following advantages:

• The life cycle can be compartmentalized into various phases which helps in planning the resources and the amount of effort required through the development process.
• Testing is enforced in every stage in the form of unit testing and reviews.During various stages of the lifecycle different form of reviews like design and code reviews and various forms of testing like unit and integration testing are performed.
• After each phase of lifecycle expectations for deliverables can be set.

Waterfall model has the following disadvantages:

• There is no working version of software until its late in the life cycle.So problems can’t be detcted until the system testing phase.Problems are always hard to fix in the latter phases of life cycle.
• Also for a phase to get started the previous phase must be complete.A system design principle cannot start until the requirement analysis phase is complete and the requirements are frozen. So waterfall model cannot accommodate uncertainties that that may persist after a phase is over.This can lead to delays and extended project schedules.

Testing approach: During the requirements phase the project requirements are completely defined.Simultaneously the test team brainstorms the scope of testing,test strategy and drafts a detailed test plan.

Incremental or Iterative Development

The fundamental principle behind incremental or iterative development is to break down the project into small parts. When one iteration is complete a new module is completed or an existing module is improved. The module is then integrated into the structure and finally the structure is then tested as a whole. In the iterative development model a project is usually broken down into 12 iterations of one to four week duration each.Finally the system is tested at the end of each duration and the test feedback is immediately incorporated. Time spent on the successive iterations can be modified on the basis of experience gained from past iterations. So the system grows by incorporating new functions during the development portion of each iteration.Each iteration thus involves adding of new functions during the development phase.testing evolves as the system evolves.



The main advantage of iterative development model is that corrective actions can be taken at end of each iteration.

The main disadvantages of iterative development model are:

• Each iteraton involved giving feedback about the deliverables,timelines,efforts and so on.SO the overhead is considerably higher.
• It is hard to freeze the requirements as requirements may need change based on feedback and increasing customer demands.This can lead to more number of iterations and thus delay in deliverables.
• An efficient control change mechanism is needed to manage the system changes made during each iteration.

Testing approach: As soon as iteration is complete the entire system is subjected to testing.The feedback from testing is immediately available and further incorporated into the next cycle.Testing time required for the successive iterations can be reduced based on the experience gained from past iterations.

Agile Methodology

Previously Majority of the software development life cycle methodologies could be categorised into either iterative or sequential model like waterfall model does.But as software systems evolved and became more complex both of these models couldn’t efficiently adapt to the significant and continuous number of changes.Agile methodology was developed to solve this issue.It was develoepd to respond to changes quicly and smoothly. The drawback with iterative model was that even though it removed the disadvantage of sequential models it was still based on the waterfall model.In Agile methodology , software is developed in incremental, rapid cycles. Interactions amongst customers, developers and client are emphasized rather than processes and tools. Agile methodology focuses on responding to change rather than extensive planning.



The main differences between agile and traditional methodologies are as follows:

• An incremental method of development is followed rather than the traditional sequential method.There are small incremental releases and each release is based on previous functionality.
• Each release is thoroughly tested and that helps to ensure that the defects are addressed in the next iteration.
• There is more emphasis given on people and their interactions rather than processes and tools.The developers,customers and testers continuously interact with each other.This interaction ensures that the tester us aware of the features being developed during a particular iteration and so can easily identify any sort of discrepancy between the system and the requirements.
• More priority is given to working software rather than detailed documentation. Agile methodologies rely on face-to-face communication and collaboration, with people working in pairs. As there is extensive communication between customers and team members, there is no need for comprehensive requirements document. All agile projects have customers as an integral part of the team.When developers have a query regarding program requirements thy can get it immediately clarified from the customers.

The disadvantage is that a constant client interaction leads to and added time pressure on all stakeholders including the client themselves , software development and test teams .

Testing approach: Incremental testing approach is followed and thus every release of the project is tested thoroughly.This ensures that any bugs in the system are fixed before the next release.

Extreme Programming

Extreme programming<ref>http://xprogramming.com/book/whatisxp</ref> is a form of agile methodology that believes in short development cycles.So rather than designing the whole of the system at the start of the project the preliminary design work is shortened down to solve the simple tasks that have already been identified.The developers have to interact frequently with customers and other developers. A simple task is started and as soon as it is developed customer feedback is taken.The system is delivered to the customer as soon as possible and the requirements are then refined on the basis of customer feedback.So the requirements evolve over a period of time and developers are able to respond to changes quickly.Extreme programming emphasizes on pair programming.This means one developer writes the code for a particular feature and the other developer reviews it. In theory, the driver focuses on the code at hand: the syntax, semantics, and algorithm. The navigator focuses less on that, and more on a level of abstraction higher: the test they are trying to get to pass, the technical task to be delivered next, the time elapsed since all the tests were run, the time elapsed since the last repository commit, and the quality of the overall design. The theory is that pairing results in better designs, fewer bugs, and much better spread of knowledge across a development team, and therefore more functionality per unit time, measured over the long term.

Extreme Programming is highly useful in the many situations.

• If a customer doesn't have a clear understanding of the system then the developers can interact continuously with the customer ,deliver small pieces and ask the customer for feedback.Corrective action is then taken.
• If a technology used to develop a system is significantly new and its a completely new platform then frequent test cycles in extreme programing mitigate the risk of incompatibility with other existing systems.
• If you want automated unit and functional tests there may be a need to change system design such that each module can be tested in isolation sing automation. XP(Extreme programming) comes handy in such scenario.

The main advantage of following XP is that customers having a vague software design in mind can go ahead to implement their product. The continuous testing and integration ensures that the software code delivered is of the highest standards.

Testing approach: Extreme programming follows a test driven development.It is explained in brief in the subsequent section.

Test Driven Development (TDD)

Test-driven development is one of the core practices of Extreme Programming. Test cases are written first and then code is written to pass the existing test cases. Then new test cases are added to test the existing functionality, and then the entire test suite is run to ensure that the code fails. Then new functionality is added or existing functionality is modified so that the code can withstand the failed test cases. This cycle continues until the test code passes all of the test cases that the team can create. The code is then refactored to make it DRY and more maintainable.

Test-driven development is totally non conventional in the sense that instead of writing code first and then testing it, you write the tests first and then write code to make the tests pass. This is done iteratively. Only when one test case passes, the developer moves on to the next test case and writes code to make it pass. This process is continued until all tests pass.

With test-driven development we can even start with an unclear set of requirements and then can discuss with the customer later about new requirements or change in existing requirements. Even if the code is not ready for a particular functionality but the tests are written, it will ensure that the functionality is addressing all the requirements given by the customer and unnecessary functionality is not included. It allows you to build your software step-by-step and then as the requirements become more clear it evolves the system.
Other advantages of TDD:

• Loosely coupled and highly cohesive code is promoted by Test-driven development because the functionality is evolved in small steps.
• The tests that we write can act as documentation for the final system’s specifications.
• Time for retesting is reduced because of automated testing. So we don’t have to waste time in retesting existing functionality.
• You know exactly what you have to do in order to make a test pass. Your effort can be measured in terms of number of tests passed.

Unit testing is also ensured through Test-driven development. We will still need to all other kinds of testing such as acceptance testing, system integration testing etc.

Steps in Test-Driven Development

The following figure shows the steps involved in test-driven development process <ref>http://msdn.microsoft.com/en-us/library/ff649520.aspx</ref>

Behavior Driven Development (BDD)

Behavior Driven Development provides a common language between developers, analysts, and customers, thereby, reducing miscommunication between which was common in earlier forms of software development. This language is used by developers to communicate while developing and explaining code. It makes sure that the customers and the developers have a common vocabulary while talking about the system being developed.

While TDD makes sure that the technical quality of software is up to the mark, behavior-driven development makes sure that the needs of the customer are fulfilled. TDD takes care of the verification part i.e. building the thing right, BDD takes care of the validation part i.e. building the right thing.

Building the Right Thing : BDD ensures that the systems is built according to the requirements of the customer and is delivered on time. By having a common language between the stakeholders and the developers we have less chance of going wrong with implementing the requirements. This common language is defining the requirement in terms of user stories.

Reducing Risk : If we are adhering to Behavior Driven Development and our focus is to make a test pass than it is highly unlikely that a developer will go off the way and develop something unnecessary.And with a suite of acceptance tests developed in partnership with the stakeholder, the customer can monitor the system throughout its lifecycle.

With BDD we create a common language which is shared by all stakeholders. It is perfect for projects where the requirements are changing rapidly or are not specified well. Also, it is easier to estimate the effort needed to complete the project because it can be divided into small chunks of user stories.

One of the disadvantages with BDD is that it can be very expensive to maintain constant contact with the customer. Another disadvantage is that we may be able to satisfy all the customer requirements but end up with bad software architecture which would hurt later while maintaining the code. <ref>Aramando, Fox (2012). Engineering Long Lasting Software Pg 153-155. San Francisco: Strawberry Canyon</ref> <ref>http://my.safaribooksonline.com/book/software-engineering-and-development/software-testing/9781449309718</ref> <ref>http://bubusdaybook.blogspot.com/2011/08/extreme-programming-in-nutshell.html</ref>

Cucumber and RSpec

Using cucumber, we can describe, in plain text, how a software should behave. It executes plain-text functional descriptions as automated tests. Cucumber supports Behavior Driven Development. The tests are written before the code is written and is verified by non technical stakeholders. The production code is then written to make the stories pass. <ref> https://github.com/cucumber/cucumber/wiki</ref>

Following is an example of a cucumber scenario in the BackChannel app:

Feature: User can add post

Scenario: Add a post
	Given I am on the posts index page
	When I follow “Add new post”
	Then I should be on the Create New Post page
	When I fill in “Title” with “Fight Club”
	And I fill in “Content” with “The things you own, end up owning you”
	And I press “Save Changes”
	Then I should be on the posts index page
	And I should see “Fight Club”

There are six steps to behavior driven development using Cucumber and Rspec<ref>http://cukes.info/</ref>

1. Describe behavior in plain text using Cucumber
2. Write a step definition in ruby using Rspec
3. Run the test and it will fail because the code is not been written yet
4. Write code to make the step pass
5. Run the test again and see the step pass
6. Repeat 2-5 until all steps pass in the behavior

The test driven development loop is also known as Red-Green-Refactor. <ref>Aramando, Fox (2012). Engineering Long Lasting Software Pg 164-165. San Francisco: Strawberry Canyon</ref>
Red step: Run the test and verify that it fails because you haven’t yet implemented the code.
Green step: Write the simplest possible code that causes this test to pass without breaking any existing tests.
Refactor step: Refactor the code if there is any scope of refactoring.

Conclusion

BDD and TDD may seem strange at first but once you use it you realize you have been using these techniques in conventional development also, while doing unit testing. Rather than coding first and then debugging the code to find the problem, TDD is much better of way of developing a system in that you can isolate the problem really easily because you whole is divided into features and specs.

If we write code first and then debug, we end up using the same techniques as TDD but less efficiently and less productively. Using TDD, bugs can be spotted quickly and regression testing is easy because all the testing is automated.

References

<references/>

The video of the lecture can be found here.