CSC/ECE 517 Fall 2013/ch1 1w35 sa

Introduction to Testing

A project lifecycle has two major components i.e. Development Cycle and Testing Cycle. Any software engineering project life cycle is not complete till the testing phase is complete. It is used for making your work robust and dependable. A lot of sensitive and important information is hosted on a website, it caters to end users, and a part of the deal is that it should work flawlessly and in its entirety.

Approaches to Testing

Testing could be manual or automated. Manual Testing is described as a developer acting as an end user and testing almost all the features of the application and check if they are behaving correctly in order they are supposed to. The manual testing involves writing Test Cases and then running those test cases. Usually the Test cases are generated on the lines of the software’s functional flow.

Manual Testing

Manual Testing includes the testing of software manually i.e. without using any automated tool or any script. In manual testing the tester has to take the role of the end user and test the whole application/ software to discover any abnormality in the functioning of the code or any unexpected behaviour for that matter. This is then classified as a bug, which depending on the protocol being followed is either returned immediately to the developer to fix or is documented into a bug report which is when assigned to developers to fix. Manual Testing has different stages like Unit,Integration, System and User Acceptance Testing. After the reported bug are fixed, the tester will again go through the original set of test cases to test every functionality again, and if it passes this time the defects are classified as fixed and are closed. So basically the main goal for Manual Testing is that verify each part of the code to see if it’s working as it’s supposed to be and hence deliver good quality and robust code to your client. But no all’s hunky dory with Manual Testing because repetitive manual testing can be painstaking and very challenging if you have a very large dataset. Though one way to move ahead of this problem is that you will employ black-box testing technique which will use boundary value analysis or equivalence partitioning so that this large dataset can be tested by managing it into smaller and achievable test suites.

Manual Testing for Small Projects

Manual Testing is more successful and less rigorous when the project is small and it can be manually tested. The tester will create a Test Strategy and test the application on basis of the generic and specific test cases, checking if the functionality of the application is correctly being implemented. He can start with the UI and move ahead to other aspects of the application. He can then file or document bugs found in the application which will be resolved and then re-tested, till the results are satisfactory.

Manual Testing for Large Projects

Manual Testing for Large Test cases often requires a group or community of developers trying to test the application for loop holes and un-expected outputs that harm the website or prevent it from being useful to its end users. A high level Test Plan has to be setup to even undertake Manual Testing on these large projects. It is a rigorous approach where the tester will have to predict and document the detailed outcomes of certain actions that the user will take when they are accessing the website. These results are then documented and the application/website goes into the Bug Fixing mode.

Automated Testing

Manual Testing though is a very good mechanism to make your website/application robust, it might not be the most efficient choice when you have a huge project and has many branches and functionalities to rigorously test specially if make modifications to small components which might affect the final outcome. Every Change Request in your code might trigger a pulse which will produce undesirable and unexpected results and if not caught in time, could result in crashing of your application. Automated tests give you the liberty to run tests on multiple platforms without any need for human interference. It can run anytime for as long period of times and in any order, whereas that’s not the case for Manual Testing as the tester has to be there to execute the test cases. Not to be mistaken, test automation involves a significant human involvement. After creating a test plan, the developer will study the functionalities and write test cases that test those functionalities. But his role ends here; implementing those test cases is the work of the tool being used for Automated Testing.

Pros and Cons of Manual Testing

PROS

• It is used in both big and small projects.
• Manual testing is better in terms of reliability.
• If the test cases have to be run on a smaller scale in a project, it’s always better to use Manual Testing.
• If the tester has to perform Ad-Hoc testing, then Manual Testing is a better fit.
• Manual Testing exposes more real time bugs, as the tester sits on a machine and tries to penetrate the system thus exposing many bugs.

CONS

• In big projects manual testing may not be every efficient and cost effective
• In big projects it’s very difficult to run manual tests on every component within a stipulated deadline and cost budget.
• Manual Testing has to be repeated for every change that was made and hence can be very tiring also exposing to unreliable testing.

Pros and Cons of Automated Testing

PROS

• Extensive testing requires large number of tests to be run every time a change is made, and here the Automated Testing can be really efficient and reliable.
• Automated tests can provide “Compatibility Tests”, when you run your application on various platforms.
• Many tests scripts can be run simultaneously, and since all of them are accurate, they cut short the costs and time consumption on a large scale.
• Automated tests are relatively cheaper investment in the long run.

CONS

• Initial Investments on Automated Testing are bigger than the effort for Manual Testing.
• Sometime GUI Testing will miss out on small components like visibility of elements, placements of fields and text areas.
• Also not everything can be automated; sometimes Manual Testing is the only and the best way to go.

Comparison of Manual and Automated Testing

• Automated Testing can be costly in terms of monetary benefits, but it does eliminate the costs of actual effort on testing. Manual Testing can be very laborious for huge/gigantic projects that have endless room for error.

• Automated Testing involves human labour for writing Test Cases, and sometimes if the tools used for automated testing are very complex or the structure of the Project is so nested that running Test Cases will be more laborious than a community of Testers, it might be a bad idea to choose Automated Testing over Manual.

• Testing Tools are not always “Simple” , i.e. sometimes the results produced by running Automation on a project might be difficult to comprehend, whereas Manual Testing would be very effective as the Developer/Tester knows what’s expected and can better analyze hands on if the code is not working properly, than to find it in hidden in bunch of scrambled digits.

• Though Automation seems like “Extra Work”, it really is designed to make your work easier, imagine a huge application which caters to millions of customers, where the functionality keeps on upgrading and changing. Considering if this application requires 10^3 test cases, how tardy will it be to run these number of tests manually every time a small change is done in the code. Automated Testing is a blessing in disguise, as the effort for writing test cases will be the same, but we can save a fortune of money and effort by making the Tool run the tests fortnight and see the logs early in the morning.

• Automated Testing eliminates Human and Technical Risks, many times a block of code will change drastically overtime and when the tester runs the same test cases manually, he may choose to eliminate some test cases assuming that they might not be impacted. Also sometimes during manual testing it becomes inevitable that some test cases will be left out either by choice or by natural circumstances, resulting in block of code prone to loop holes and crash issues. Which is not the case with automated Testing, as the automation tool is going to run every single test case that it was designed to test, “n” number of times repeatedly if required, and give out exact results to be read and debugged later in the project cycle.

• Certain types of Testing, such as Performance, Regression testing is well suited for Automation tool testing, while Usability Testing is much better if Manual Testing is employed.

Few Tools for Automation of Testing

• Autotest

AutoTest

AutoTest is known for instant feedback on your code i.e. the test runs on its own and you don’t have to worry about it. It is used in scenarios where you need instant result on any change requests that the developer has made or the cases where you change a small functionality and you need to see if that is working fine or has had a bad impact on your working Application. Also AutoTest doesn’t require the user to switch perspectives to each time code and check its results. It’s all automated. AutoTest is for Individual Testing i.e. suppose a developer is working on a module and he decides to change functionality and then wants to test it before he can commit and integrate the piece of code to the final build. In these scenarios AutoTest is very useful.

The catchy feature of AutoTest is that, it only runs the tests that it thinks are required for the changes you made in your application. You need to install a Ruby gem for working with AutoTest.

Installation of AutoTest on OS X and Linux

• Step 1. Install Gem ZenTest

				sudo gem install ZenTest

• Step 2: If Rails development has to be done then

                               sudo gem install autotest-rails

• Step 3: Run AutoTest

                               $ cd <base directory of your rails project>

• Step 4: AutoTest will run all the test cases for the first time
• Step 5: Make some changes in your code
• Step 6: Run Auto Test Again, and you will see that the only section of your tests will run this time.

AutoTest will assume that you are using Test::Unit for writing your test cases. So if you’re using RSpec, you need to manually inform AutoTest about your preference.

On command line type

                                RSPEC= true autotest

Also as an alternative to AutoTest setting true for RSpec you can also use the “autospec” command which is used instead of autotest.

On command line type

                                Autospec

Installation of AutoTest on Windows

• Step 1: Install gem ZenTest from the command line

                                 gem install ZenTest

• Step 2: on the command line read the directory of your rails project

                                 cd <path of rails project>

• Step 3: Set the Environment variable before running AutoTest

                                 set HOME = C:\Documents and Settings\<your name on the machine> 

• Step 4: In the same directory of the rails app type

                                 autotest --rails

CruiseControl.rb

CruiseControl.rb is used for Continuous Integration. Continuous Integration means that the developers integrate their code very frequently i.e. at least on a daily basis. So using CruiseControl.rb here prevents any integration issues that might accumulate and trouble later on at the time of integration.

Installation of CruiseControl.rb

• Step 1: Download CruiseControl.rb

On Command Line type

• Step 2:

                                 cruise add <name> -u <url>

• Step 3:

                                 cruise start

For issues with installation , look at the manual

Conclusion

The decision to automate testing of an application or not has to be taken purely in the context of that application. There cannot be a general set of rules which specifies that manual testing is better or automated testing is better. Following are few questions that can lead us to conclusive answers.

• Time and Effort required for Automation

The effort involved in writing a Test script should not be unreasonably more, as this might lead to more time and effort in developing test scripts than actually implementing the test cases manually. So a large part of this decision lies with how user friendly and customizable the Automation tool is.

• Expected Test Coverage

The amount of code to be tested is another criterion, if the code to be tested is fairly short, then use manual Testing, but if large code coverage is required then Automated tests will save a lot of time.

• Frequency of Testing

A lot depends on how often a piece of code is tested, if it's a critical application and requires constant effort on testing then it's better to use automated as it's less prone to make mistakes, fairly accurate and saves a huge amount of time, which could be utilized elsewhere. But if the functionality is tested only a few times then it's better to test it manually than write scripts for automation.

• Stage of Project

It’s often found that in the initial development stage of a project, it is manual testing that precedes automation to identify bugs. However for mature products, it is automation that primary and manual testing only as and when required.

Thus we need to consider time, resources, and the size of the project as well as the quality of the automated tools while deciding the combination of manual and automated testing to develop the best software possible.

See Also

1. Software Testing Methods and Techniques
2. Context Driven Testing
3. Software Testing Controversies

References

1. http://en.wikipedia.org/wiki/Test_automation
2. http://en.wikipedia.org/wiki/Manual_testing
3. http://www.jobsnewstoday.com/2013/04/discuss-about-adverse-between-manual-testing-vs-automation-testing.html
4. http://www.developsense.com/blog/2013/02/manual-and-automated-testing/
5. http://www.softwaretestingmentor.com/automation/manual-vs-automation/
6. http://blog.typemock.com/2012/04/05/the-difference-between-automated-and-manual-testing
7. http://apprenda.com/blog/manual-v-automated-testing-achieving-consistency/
8. http://www.base36.com/2013/03/automated-vs-manual-testing-the-pros-and-cons-of-each/
9. http://www.tutorialspoint.com/software_testing/testing_types.htm
10. http://www.slideshare.net/Softwarecentral/manual-testing-vs-automated-testing
11. http://selleo.com/blog/software-outsourcing/the-value-of-automated-testing/2/