CSC/ECE 517 Fall 2013/ch1 1w35 sa

Testing

A project lifecycle has two major components i.e. Development Cycle and Testing Cycle. Any software engineering project life cycle is not complete till the testing phase is complete. It is used for making your work robust and dependable. A lot of sensitive and important information is hosted on a website, it caters to end users, and a part of the deal is that it should work flawlessly and in it’s entirety.

Approaches to Testing

Testing could be manual and automated. Manual Testing is described as developer acting as an end user and testing almost all the features of the application and check if they are behaving correctly in order they are supposed to. The manual testing involves writing Test Cases and then running those test cases. Usually the Test cases are generated on the lines of the software’s functional flow.

Manual Testing

Manual Testing for Small Projects

Manual Testing is more successful and less rigorous when the project is small and it can be manually tested. The tester will create a Test Strategy and test the application on basis of the generic and specific test cases, checking if the functionality of the application is correctly being implemented. He can start with the UI and move ahead to other aspects of the application. He then can file bugs in the application which will be resolved and then re-tested, till the results are satisfactory.

Manual Testing for Large Projects

Manual Testing for Large Test cases often requires a group or community of developers trying to test the application for loop holes and un-expected outputs that harm the website or prevent it from being useful to its end users. A high level Test Plan has to be setup to even undertake Manual Testing. It is a rigorous approach where the tester will have to predict and document the detailed outcomes of certain actions that the user will take when they are accessing the website. These results are then documented and the application/website goes into the Bug Fixing mode.

Automated Testing

Manual Testing though is a very good mechanism to make your website/application robust, it might not be the most efficient choice when you have a huge project and has many branches and functionalities to rigorously test specially alter modifications to small components which might alter the outcomes. Every Change Request in you code might trigger a pulse which will produce undesirable and unexpected results and if not caught in time , could result in temporary crash of your application. Automated tests give you the liberty to run tests on multiple platforms without any need for human interference. It can run anytime for as long period of times and in any order, whereas that’s not the case for Manual Testing as the tester has to be there to execute the test cases. Not to be mistaken , test automation involves a significant human role. After creating a test plan , the developer will study the functionalities and write test cases that test those functionalities. But his role ends here, implementing those test cases is the work of the tool being implemented for Automated Testing.

Pros and Cons of Manual Testing

PROS

• It is used in both big and small projects.
• Manual testing is better in terms of reliability.
• If the test cases have to be run on a smaller scale in a project, it’s always better to use Manual Testing.
• If the tester has to perform Ad-Hoc testing, then Manual Testing is a better fit.
• Manual Testing exposes more real time bugs, as the tester sits n a machine and tried to penetrate the system thus exposing many bugs.

CONS

• In big projects manual testing maynot be every efficient and cost effective
• In big projects it’s very difficult to run manual tests on every component within a stipulated deadline and cost budget.
• Manual Testing has to be repeated for every change, can be very tiring and thus exposing to unreliable testing.

Pros and Cons of Automated Testing

PROS

• Extensive testing requires, large number of tests to be run every time a change is made, and hee the Automated Testing can be really efficient and reliable.
• Automated tests can provide “Compatibility Tests”, when you run your application on various platforms
• Many tests scripts can be run simultaneously , and thus all of them being accurate they cut short the costs and time consumption on a large scale.
• Automated tests are relatively cheaper investment in the long run.

CONS

• Initial Investments on Automated Testing are bigger than the effort for Manual Testing
• Sometime GUI Testing will miss out on small components like visibility of elements, placements of fields andt ext areas
• Also no everything can be automated, sometimes Manual Testing is the only and the best way to go.

Comparison of Manual and Automated Testing

• Automated Testing can be costly in terms of monetary benefits, but it does eliminate the costs of actual effort on testing. Manual Testing can be very laborious for huge/gigantic projects that have endless room for error.

• Automated Testing involves human labor for writing Test Cases , and sometimes if the tools used for Automated testing are very complex or the structure of the Project is so nested that running Test Cases will be more laborious than a community of Testers, it might be a bad idea to choose Automated Testing over Manual.

• Testing Tools are not always “Simple” , i.e. sometimes the results produced by running Automation on a project might be difficult to comprehend, whereas Manual Testing would be very effective as the Developer/Tester knows what’s expected and can better analyze hands on if the code is working properly, than to find it in hidden in bunch of codes.

• Though Automation seems like a “Extra Work” , it really is designed to make your work easier, imagine a huge application which caters to millions of customers, where the functionality keeps on upgrading and changing . Considering if this application requires 10^3 test cases, how tardy will it be to run these number of tests manually every time a small change is done in the code. Automated Testing is a blessing in disguise, as the effort for writing test cases will be the same, but we can save a fortune of money and effort by making the Tool run the tests fortnight and see the logs early in the morning.

• Automated Testing eliminates Human and Technical Risks, many times a block of code will change drastically overtime and when the tester runs the same test cases manually, he may choose to eliminate some test cases assuming that they might not be impacted. Also sometimes during manual testing it becomes inevitable that some test cases will be left out either by choice or by natural circumstances, resulting in block of code eligible for loop holes and crash issues.Which is not the case with automated Testing, as the automation tool is going to run every single test cases that it was designed to test, n number of times repeatedly if required, and give out exact results to be read and debugged later in the project cycle.

• Certain types of Testing, such as Performance, Regression testing are well suitable for Automation tool testing, while Usability Testing is much better if Manual Testing is employed.

Few Tools for Automation of Testing

• Autotest
• CruiseControl.rb

AutoTest

AutoTest is known for instant feedback on your code i.e. the test runs on it’s own and you don’t have to worry about it. It is used in scenarios where you need instant result on any change requests that the developer has made or the cases where you change a small functionality and you need to see if that is working fine or has had a bad impact on your working Application. Also Autotest , doesn’t require the user to switch perspectives to each time code and check it’s results. It’s all automated . Autotest is for Individual Testing i.e. supoose a developer is working on a module and he deides to change a fucntioanlty and thene wants to test it before he can commit and integrate the piece of code to the final buid. In these scenarios Autotest is very useful.

The catchy feature of Autotest is that, it only runs the tests that it thinks are required for the changes you made in your application. You need to install a Ruby gem for working with AutoTest.

Installation of AutoTest on OS X and Linux

• Step 1. Install Gem ZenTest

				sudo gem install ZenTest

• Step 2: If Rails development has to be done then

                               sudo gem install autotest-rails

• Step 3: Run AutoTest

                               $ cd <base directory of your rails project>

• Step 4: AutoTest will run all the test cases for the first time
• Step 5: Make some changes in your code
• Step 6: Run Auto Test Again , and you will see that the only section of your tests will run this time.

AutoTest will assume that you are using Test::Unit for writing your test cases. So if your using RSpec , you need to manually inform AutoTest about your preference.

On command line type

                                RSPEC= true autotest

Also as an alternative to AutoTest setting true for RSpec you can also use the “autospec” command which is used instead of autotest.

On command line type

                                Autospec

Installation of AutoTest on Windows

• Step 1: Install gem ZenTest from the command line

                                 gem install ZenTest

• Step 2: on the command line read the directory of your rails project

                                 cd <path of rails project>

• Step 3: Set the Environment variable before running AutoTest

                                 set HOME = C:\Documents and Settings\<your name on the machine> 

• Step 4: In the same directory of the rails app type

                                 autotest --rails

CruiseControl.rb

CruiseControl.rb is used for Continuous Integration. Continuous Integration means that the developers integrate their code very frequently i.e. at least on a daily basis. So using CruiseControl.rb here prevents any integration issues that might accumulate and trouble later on at the time of integration.

Installation of CruiseControl.rb

• Step 1: Download CruiseControl.rb

On Command Line type

• Step 2:

                                 cruise add <name> -u <url>

• Step 3:

                                 cruise start

For issues with installation , look at the manual < http://cruisecontrolrb.thoughtworks.com/documentation>

Conclusion

The decision to automate testing of an application or not has to be taken purely in the context of that application. There cannot be a general set of rules which specifies that manual testing is better or automated testing is better. It is this aspect which really makes any discussion on the topic "Manual vs Automated Testing" very interesting.Following are few questions that can lead us to conclusive answers.

1. How long does it to automate a test scenario? What is the effort involved? The effort to prepare automation scripts should be reasonably less. Otherwise the effort spent in creating automation scripts might turn out to be greater than the effort required for manual testing. From a pure effort perspective, automation could become a pain. This depends to a large extent on the automation tool and how easy it is to use the tool.

2. How frequently is the testing done? Some applications are tested daily and some are tested less frequently. It makes more sense to automate testing of an application wherever the frequence of testing is very high. It reduces the manual effort a lot. Also, in a majority of cases, if an application is being tested very frequently, then it means that it is a very critical application and it is better to avoid human error in testing as much as possible by resorting to automation.

3. What is the test coverage expected? It is obvious that automation can test more lines of code than manual testing in the same amount of time. But one has to check whether it is really required to have a tremendously high code coverage. There could be certain functionalities which are very rarely used or used only once (maybe during the setting up of the system). There is no point in automating such functionalities. A compromise has to be made between code coverage by an automation tool and the importance of functionality to be tested.

4. How adaptable is it? This question basically arises from the fact that there might be multiple test systems in the landscape (for example, multiple test Portals) and the master data in these Portals could be different. In this scenario, it should be possible to use the same script to test all the test Portals and the option to change the master data easily and with minimal effort should be available (like using placeholders in the script instead of actual master data)

References

1. http://en.wikipedia.org/wiki/Test_automation
2. http://en.wikipedia.org/wiki/Manual_testing
3. http://www.jobsnewstoday.com/2013/04/discuss-about-adverse-between-manual-testing-vs-automation-testing.html
4. http://www.developsense.com/blog/2013/02/manual-and-automated-testing/
5. http://www.softwaretestingmentor.com/automation/manual-vs-automation/
6. http://blog.typemock.com/2012/04/05/the-difference-between-automated-and-manual-testing
7. http://apprenda.com/blog/manual-v-automated-testing-achieving-consistency/
8. http://www.base36.com/2013/03/automated-vs-manual-testing-the-pros-and-cons-of-each/
9. http://www.tutorialspoint.com/software_testing/testing_types.htm
10. http://www.slideshare.net/Softwarecentral/manual-testing-vs-automated-testing
11. http://selleo.com/blog/software-outsourcing/the-value-of-automated-testing/2/
12. http://apprenda.com/blog/manual-v-automated-testing-achieving-consistency/