CSC/ECE 517 Fall 2013/ch1 1w35 sa: Difference between revisions

From Expertiza_Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(14 intermediate revisions by one other user not shown)
Line 1: Line 1:
== Testing ==
[[File:207.jpg|right]]
A project lifecycle has two major components i.e. Development Cycle and Testing Cycle. Any software engineering project life cycle is not complete till the testing phase is complete. It is used for making your work robust and dependable. A lot of sensitive and important information is hosted on a website, it caters to end users, and a part of the deal is that it should work flawlessly and in it’s entirety.  
 
== Introduction to Testing ==
A project lifecycle has two major components i.e. Development Cycle and Testing Cycle. Any software engineering project life cycle is not complete till the testing phase is complete. It is used for making your work robust and dependable. A lot of sensitive and important information is hosted on a website, it caters to end users, and a part of the deal is that it should work flawlessly and in its entirety.  




== Approaches to Testing ==
== Approaches to Testing ==
Testing could be manual and automated. Manual Testing is described as developer acting as an end user and testing almost all the features of the application and check if they are behaving correctly in order they are supposed to. The manual testing involves writing Test Cases and then running those test cases. Usually the Test cases are generated on the lines of the software’s functional flow.
Testing could be manual or automated. Manual Testing is described as a developer acting as an end user and testing almost all the features of the application and check if they are behaving correctly in order they are supposed to. The manual testing involves writing Test Cases and then running those test cases. Usually the Test cases are generated on the lines of the software’s functional flow.


=== Manual Testing ===
=== Manual Testing ===
Manual Testing includes the testing of software manually i.e. without using any automated tool or any script. In manual testing the tester has to take the role of the end user and test the whole application/ software to discover any abnormality in the functioning of the code or any unexpected behaviour for that matter. This is then classified as a bug, which depending on the protocol being followed is either returned immediately to the developer to fix or is documented into a bug report which is when assigned to developers to fix. Manual Testing has different stages like [http://en.wikipedia.org/wiki/Unit_testing Unit],[http://en.wikipedia.org/wiki/Integration_testing Integration], [http://en.wikipedia.org/wiki/System_testing System] and [http://en.wikipedia.org/wiki/Acceptance_testing User Acceptance Testing].
After the reported bug are fixed, the tester will again go through the original set of test cases to test every functionality again, and if it passes this time the defects are classified as fixed and are closed. So basically the main goal for Manual Testing is that verify each part of the code to see if it’s working as it’s supposed to be and hence deliver good quality and robust code to your client.
But no all’s hunky dory with Manual Testing because repetitive manual testing can be painstaking and very challenging if you have a very large dataset. Though one way to move ahead of this problem is that you will employ black-box testing technique which will use boundary value analysis or equivalence partitioning so that this large dataset can be tested by managing it into smaller and achievable [http://en.wikipedia.org/wiki/Test_suite test suites].


==== Manual Testing for Small Projects ====
==== Manual Testing for Small Projects ====
Manual Testing is more successful and less rigorous when the project is small and it can be manually tested. The tester will create a Test Strategy and test the application on basis of the generic and specific test cases, checking if the functionality of the application is correctly being implemented. He can start with the UI and move ahead to other aspects of the application. He then can file bugs in the application which will be resolved and then re-tested, till the results are satisfactory.
Manual Testing is more successful and less rigorous when the project is small and it can be manually tested. The tester will create a Test Strategy and test the application on basis of the generic and specific test cases, checking if the functionality of the application is correctly being implemented. He can start with the UI and move ahead to other aspects of the application. He can then file or document bugs found in the application which will be resolved and then re-tested, till the results are satisfactory.


==== Manual Testing for Large Projects ====
==== Manual Testing for Large Projects ====
Manual Testing for Large Test cases often requires a group or community of developers trying to test the application for loop holes and un-expected outputs that harm the website or prevent it from being useful to its end users. A high level Test Plan has to be setup to even undertake Manual Testing. It is a rigorous approach where the tester will have to predict and document the detailed outcomes of certain actions that the user will take when they are accessing the website. These results are then documented and the application/website goes into the Bug Fixing mode.
Manual Testing for Large Test cases often requires a group or community of developers trying to test the application for loop holes and un-expected outputs that harm the website or prevent it from being useful to its end users. A high level Test Plan has to be setup to even undertake Manual Testing on these large projects. It is a rigorous approach where the tester will have to predict and document the detailed outcomes of certain actions that the user will take when they are accessing the website. These results are then documented and the application/website goes into the Bug Fixing mode.


=== Automated Testing ===
=== Automated Testing ===
Manual Testing though is a very good mechanism to make your website/application robust, it might not be the most efficient choice when you have a huge project and has many branches and functionalities to rigorously test specially alter modifications to small components which might alter the outcomes. Every Change Request in you code might trigger a pulse which will produce undesirable and unexpected results and if not caught in time , could result in temporary crash of your application. Automated tests give you the liberty to run tests on multiple platforms without any need for human interference. It can run anytime for as long period of times and in any order, whereas that’s not the case for Manual Testing as the tester has to be there to execute the test cases.
Manual Testing though is a very good mechanism to make your website/application robust, it might not be the most efficient choice when you have a huge project and has many branches and functionalities to rigorously test specially if make modifications to small components which might affect the final outcome. Every Change Request in your code might trigger a pulse which will produce undesirable and unexpected results and if not caught in time, could result in crashing of your application. Automated tests give you the liberty to run tests on multiple platforms without any need for human interference. It can run anytime for as long period of times and in any order, whereas that’s not the case for Manual Testing as the tester has to be there to execute the test cases.
Not to be mistaken , test automation involves a significant human role. After creating a test plan , the developer will study the functionalities and write test cases that test those functionalities. But his role ends here, implementing those test cases is the work of the tool being implemented for Automated Testing.  
Not to be mistaken, test automation involves a significant human involvement. After creating a test plan, the developer will study the functionalities and write test cases that test those functionalities. But his role ends here; implementing those test cases is the work of the tool being used for Automated Testing.  


== Pros and Cons of Manual Testing ==
== Pros and Cons of Manual Testing ==
Line 26: Line 31:
* If the test cases have to be run on a smaller scale in a project, it’s always better to use Manual Testing.
* If the test cases have to be run on a smaller scale in a project, it’s always better to use Manual Testing.
* If the tester has to perform Ad-Hoc testing, then Manual Testing is a better fit.
* If the tester has to perform Ad-Hoc testing, then Manual Testing is a better fit.
* Manual Testing exposes more real time bugs, as the tester sits n a machine and tried to penetrate the system thus exposing many bugs.
* Manual Testing exposes more real time bugs, as the tester sits on a machine and tries to penetrate the system thus exposing many bugs.


'''CONS'''
'''CONS'''


* In big projects manual testing maynot be every efficient and cost effective
* In big projects manual testing may not be every efficient and cost effective
* In big projects it’s very difficult to run manual tests on every component within a stipulated deadline and cost budget.
* In big projects it’s very difficult to run manual tests on every component within a stipulated deadline and cost budget.
* Manual Testing has to be repeated for every change, can be very tiring and thus exposing to unreliable testing.
* Manual Testing has to be repeated for every change that was made and hence can be very tiring also exposing to unreliable testing.


== Pros and Cons of Automated Testing ==
== Pros and Cons of Automated Testing ==
Line 38: Line 43:
'''PROS'''
'''PROS'''


* Extensive testing requires, large number of tests to be run every time a change is made, and hee the Automated Testing can be really efficient and reliable.
* Extensive testing requires large number of tests to be run every time a change is made, and here the Automated Testing can be really efficient and reliable.
* Automated tests can provide “Compatibility Tests”, when you run your application on various platforms
* Automated tests can provide “Compatibility Tests”, when you run your application on various platforms.
* Many tests scripts can be run simultaneously , and thus all of them being accurate they cut short the costs and time consumption on a large scale.
* Many tests scripts can be run simultaneously, and since all of them are accurate, they cut short the costs and time consumption on a large scale.
* Automated tests are relatively cheaper investment in the long run.
* Automated tests are relatively cheaper investment in the long run.


'''CONS'''
'''CONS'''


* Initial Investments on Automated Testing are bigger than the effort for Manual Testing
* Initial Investments on Automated Testing are bigger than the effort for Manual Testing.
* Sometime GUI Testing will miss out on small components like visibility of elements, placements of fields andt ext areas
* Sometime GUI Testing will miss out on small components like visibility of elements, placements of fields and text areas.
* Also no everything can be automated, sometimes Manual Testing is the only and the best way to go.
* Also not everything can be automated; sometimes Manual Testing is the only and the best way to go.




Line 55: Line 60:
* Automated Testing can be costly in terms of monetary benefits, but it does eliminate the costs of actual effort on testing. Manual Testing can be very laborious for huge/gigantic projects that have endless room for error.  
* Automated Testing can be costly in terms of monetary benefits, but it does eliminate the costs of actual effort on testing. Manual Testing can be very laborious for huge/gigantic projects that have endless room for error.  


* Automated Testing involves human labor for writing Test Cases , and sometimes if the tools used for Automated testing are very complex or the structure of the Project is so nested that running Test Cases will be more laborious than a community of Testers, it might be a bad idea to choose Automated Testing over Manual.
* Automated Testing involves human labour for writing Test Cases, and sometimes if the tools used for automated testing are very complex or the structure of the Project is so nested that running Test Cases will be more laborious than a community of Testers, it might be a bad idea to choose Automated Testing over Manual.
 
* Testing Tools are not always “Simple” , i.e. sometimes the results produced by running Automation on a project might be difficult to comprehend, whereas Manual Testing would be very effective as the Developer/Tester knows what’s expected and can better analyze hands on if the code is not working properly, than to find it in hidden in bunch of scrambled digits.


* Testing Tools are not always “Simple” , i.e. sometimes the results produced by running Automation on a project might be difficult to comprehend, whereas Manual Testing would be very effective as the Developer/Tester knows what’s expected and can better analyze hands on if the code is working properly, than to find it in hidden in bunch of codes.
* Though Automation seems like “Extra Work”, it really is designed to make your work easier, imagine a huge application which caters to millions of customers, where the functionality keeps on upgrading and changing. Considering if this application requires 10^3 test cases, how tardy will it be to run these number of tests manually every time a small change is done in the code.  Automated Testing is a blessing in disguise, as the effort for writing test cases will be the same, but we can save a fortune of money and effort by making the Tool run the tests fortnight and see the logs early in the morning.


* Though Automation seems like a “Extra Work” , it really is designed to make your work easier, imagine a huge application which caters to millions of customers, where the functionality keeps on upgrading and changing . Considering if this application requires 10^3 test cases, how tardy will it be to run these number of tests manually every time a small change is done in the code.  Automated Testing is a blessing in disguise, as the effort for writing test cases will be the same, but we can save a fortune of money and effort by making the Tool run the tests fortnight and see the logs early in the morning.
* Automated Testing eliminates Human and Technical Risks, many times a block of code will change drastically overtime and when the tester runs the same test cases manually, he may choose to eliminate some test cases assuming that they might not be impacted. Also sometimes during manual testing it becomes inevitable that some test cases will be left out either by choice or by natural circumstances, resulting in block of code prone to loop holes and crash issues. Which is not the case with automated Testing, as the automation tool is going to run every single test case that it was designed to test, “n” number of times repeatedly if required, and give out exact results to be read and debugged later in the project cycle.  


* Automated Testing eliminates Human and Technical Risks, many times a block of code will change drastically overtime and when the tester runs the same test cases manually, he may choose to eliminate some test cases assuming that they might not be impacted. Also sometimes during manual testing it becomes inevitable that some test cases will be left out either by choice or by natural circumstances, resulting in block of code eligible for loop holes and crash issues.Which is not the case with automated Testing, as the automation tool is going to run every single test cases that it was designed to test, n number of times repeatedly if required, and give out exact results to be read and debugged later in the project cycle.  
* Certain types of Testing, such as Performance, Regression testing is well suited for Automation tool testing, while Usability Testing is much better if Manual Testing is employed.


* Certain types of Testing, such as Performance, Regression testing are well suitable for Automation tool testing, while Usability Testing is much better if Manual Testing is employed.


== Few Tools for Automation of Testing ==
== Few Tools for Automation of Testing ==


* Autotest
* Autotest
* CruiseControl.rb
=== AutoTest ===
=== AutoTest ===
AutoTest is known for instant feedback on your code i.e. the test runs on it’s own and you don’t have to worry about it. It is used in scenarios where you need instant result on any change requests that the developer has made or the cases where you change a small functionality and you need to see if that is working fine or has had a bad impact on your working Application. Also Autotest , doesn’t require the user to switch perspectives to each time code and check it’s results. It’s all automated . Autotest is for Individual Testing i.e. supoose a developer is working on a module and he deides to change a fucntioanlty and thene wants to test it before he can commit and integrate the piece of code to the final buid. In these scenarios Autotest is very useful.
AutoTest is known for instant feedback on your code i.e. the test runs on its own and you don’t have to worry about it. It is used in scenarios where you need instant result on any change requests that the developer has made or the cases where you change a small functionality and you need to see if that is working fine or has had a bad impact on your working Application. Also AutoTest doesn’t require the user to switch perspectives to each time code and check its results. It’s all automated. AutoTest is for Individual Testing i.e. suppose a developer is working on a module and he decides to change functionality and then wants to test it before he can commit and integrate the piece of code to the final build. In these scenarios AutoTest is very useful.


The catchy feature of Autotest is that, it only runs the tests that it thinks are required for the changes you made in your application.
The catchy feature of AutoTest is that, it only runs the tests that it thinks are required for the changes you made in your application.
You need to install a Ruby gem for working with AutoTest.
You need to install a Ruby gem for working with AutoTest.


==== Installation of AutoTest on OS X and Linux ====
==== Installation of AutoTest on OS X and Linux ====
* Step 1. Install Gem ZenTest
* Step 1. Install Gem [http://zentest.rubyforge.org/ZenTest/ ZenTest ]
  sudo gem install ZenTest
  sudo gem install ZenTest
* Step 2: If Rails development has to be done then
* Step 2: If Rails development has to be done then
Line 85: Line 89:
* Step 4: AutoTest will run all the test cases for the first time
* Step 4: AutoTest will run all the test cases for the first time
* Step 5: Make some changes in your code
* Step 5: Make some changes in your code
* Step 6: Run Auto Test Again , and you will see that the only section of your tests will run this time.
* Step 6: Run Auto Test Again, and you will see that the only section of your tests will run this time.


AutoTest will assume that you are using Test::Unit for writing your test cases. So if your using RSpec , you need to manually inform AutoTest about your preference.
AutoTest will assume that you are using [http://test-unit.rubyforge.org/ Test::Unit] for writing your test cases. So if you’re using [https://relishapp.com/rspec/rspec-rails/docs RSpec], you need to manually inform AutoTest about your preference.


On command line type
On command line type
Line 96: Line 100:
On command line type  
On command line type  
                                 Autospec
                                 Autospec


==== Installation of AutoTest on Windows ====
==== Installation of AutoTest on Windows ====
Line 111: Line 114:
=== CruiseControl.rb ===
=== CruiseControl.rb ===


CruiseControl.rb is used for Continuous Integration. Continuous Integration means that the developers integrate their code very frequently i.e. at least on a daily basis. So using CruiseControl.rb here prevents any integration issues that might accumulate and trouble later on at the time of integration.  
[http://cruisecontrolrb.thoughtworks.com/ CruiseControl.rb] is used for Continuous Integration. Continuous Integration means that the developers integrate their code very frequently i.e. at least on a daily basis. So using CruiseControl.rb here prevents any integration issues that might accumulate and trouble later on at the time of integration.  


==== Installation of CruiseControl.rb ====
==== Installation of CruiseControl.rb ====
Line 123: Line 126:
                                   cruise start
                                   cruise start


For issues with installation , look at the manual  < http://cruisecontrolrb.thoughtworks.com/documentation>
For issues with installation , look at the [http://cruisecontrolrb.thoughtworks.com/documentation manual]


== Conclusion ==
== Conclusion ==


The decision to automate testing of an application or not has to be taken purely in the context of that application. There cannot be a general set of rules which specifies that manual testing is better or automated testing is better. It is this aspect which really makes any discussion on the topic "Manual vs Automated Testing" very interesting.Following are few questions that can lead us to conclusive answers.
The decision to automate testing of an application or not has to be taken purely in the context of that application. There cannot be a general set of rules which specifies that manual testing is better or automated testing is better. Following are few questions that can lead us to conclusive answers.
 
* Time and Effort required for Automation
The effort involved in writing a Test script should not be unreasonably more, as this might lead to more time and effort in developing test scripts than actually implementing the test cases manually. So a large part of this decision lies with how user friendly and customizable the Automation tool is.
 
* Expected Test Coverage
The amount of code to be tested is another criterion, if the code to be tested is fairly short, then use manual Testing, but if large code coverage is required then Automated tests will save a lot of time.
 
* Frequency of Testing
A lot depends on how often a piece of code is tested, if it's a critical application and requires constant effort on testing then it's better to use automated as it's less prone to make mistakes, fairly accurate and saves a huge amount of time, which could be utilized elsewhere. But if the functionality is tested only a few times then it's better to test it manually than write scripts for automation.


1. How long does it to automate a test scenario? What is the effort involved?
* Stage of Project
The effort to prepare automation scripts should be reasonably less. Otherwise the effort spent in creating automation scripts might turn out to be greater than the effort required for manual testing. From a pure effort perspective, automation could become a pain. This depends to a large extent on the automation tool and how easy it is to use the tool.
It’s often found that in the initial development stage of a project, it is manual testing that precedes automation to identify bugs. However for mature products, it is automation that primary and manual testing only as and when required.


2. How frequently is the testing done?
Thus we need to consider time, resources, and the size of the project as well as the quality of the automated tools while deciding the combination of manual and automated testing to develop the best software possible.
Some applications are tested daily and some are tested less frequently. It makes more sense to automate testing of an application wherever the frequence of testing is very high. It reduces the manual effort a lot. Also, in a majority of cases, if an application is being tested very frequently, then it means that it is a very critical application and it is better to avoid human error in testing as much as possible by resorting to automation.


3. What is the test coverage expected?
== See Also ==
It is obvious that automation can test more lines of code than manual testing in the same amount of time. But one has to check whether it is really required to have a tremendously high code coverage. There could be certain functionalities which are very rarely used or used only once (maybe during the setting up of the system). There is no point in automating such functionalities. A compromise has to be made between code coverage by an automation tool and the importance of functionality to be tested.


4. How adaptable is it?
# [http://www.internetjournals.net/journals/tir/2009/January/Paper%2006.pdf Software Testing Methods and Techniques]
This question basically arises from the fact that there might be multiple test systems in the landscape (for example, multiple test Portals) and the master data in these Portals could be different. In this scenario, it should be possible to use the same script to test all the test Portals and the option to change the master data easily and with minimal effort should be available (like using placeholders in the script instead of actual master data)
# [http://context-driven-testing.com/ Context Driven Testing]
# [http://en.wikipedia.org/wiki/Software_testing_controversies Software Testing Controversies]


== References ==
== References ==
Line 154: Line 165:
# http://www.slideshare.net/Softwarecentral/manual-testing-vs-automated-testing
# http://www.slideshare.net/Softwarecentral/manual-testing-vs-automated-testing
# http://selleo.com/blog/software-outsourcing/the-value-of-automated-testing/2/
# http://selleo.com/blog/software-outsourcing/the-value-of-automated-testing/2/
# http://apprenda.com/blog/manual-v-automated-testing-achieving-consistency/

Latest revision as of 03:06, 15 October 2013

Introduction to Testing

A project lifecycle has two major components i.e. Development Cycle and Testing Cycle. Any software engineering project life cycle is not complete till the testing phase is complete. It is used for making your work robust and dependable. A lot of sensitive and important information is hosted on a website, it caters to end users, and a part of the deal is that it should work flawlessly and in its entirety.


Approaches to Testing

Testing could be manual or automated. Manual Testing is described as a developer acting as an end user and testing almost all the features of the application and check if they are behaving correctly in order they are supposed to. The manual testing involves writing Test Cases and then running those test cases. Usually the Test cases are generated on the lines of the software’s functional flow.

Manual Testing

Manual Testing includes the testing of software manually i.e. without using any automated tool or any script. In manual testing the tester has to take the role of the end user and test the whole application/ software to discover any abnormality in the functioning of the code or any unexpected behaviour for that matter. This is then classified as a bug, which depending on the protocol being followed is either returned immediately to the developer to fix or is documented into a bug report which is when assigned to developers to fix. Manual Testing has different stages like Unit,Integration, System and User Acceptance Testing. After the reported bug are fixed, the tester will again go through the original set of test cases to test every functionality again, and if it passes this time the defects are classified as fixed and are closed. So basically the main goal for Manual Testing is that verify each part of the code to see if it’s working as it’s supposed to be and hence deliver good quality and robust code to your client. But no all’s hunky dory with Manual Testing because repetitive manual testing can be painstaking and very challenging if you have a very large dataset. Though one way to move ahead of this problem is that you will employ black-box testing technique which will use boundary value analysis or equivalence partitioning so that this large dataset can be tested by managing it into smaller and achievable test suites.

Manual Testing for Small Projects

Manual Testing is more successful and less rigorous when the project is small and it can be manually tested. The tester will create a Test Strategy and test the application on basis of the generic and specific test cases, checking if the functionality of the application is correctly being implemented. He can start with the UI and move ahead to other aspects of the application. He can then file or document bugs found in the application which will be resolved and then re-tested, till the results are satisfactory.

Manual Testing for Large Projects

Manual Testing for Large Test cases often requires a group or community of developers trying to test the application for loop holes and un-expected outputs that harm the website or prevent it from being useful to its end users. A high level Test Plan has to be setup to even undertake Manual Testing on these large projects. It is a rigorous approach where the tester will have to predict and document the detailed outcomes of certain actions that the user will take when they are accessing the website. These results are then documented and the application/website goes into the Bug Fixing mode.

Automated Testing

Manual Testing though is a very good mechanism to make your website/application robust, it might not be the most efficient choice when you have a huge project and has many branches and functionalities to rigorously test specially if make modifications to small components which might affect the final outcome. Every Change Request in your code might trigger a pulse which will produce undesirable and unexpected results and if not caught in time, could result in crashing of your application. Automated tests give you the liberty to run tests on multiple platforms without any need for human interference. It can run anytime for as long period of times and in any order, whereas that’s not the case for Manual Testing as the tester has to be there to execute the test cases. Not to be mistaken, test automation involves a significant human involvement. After creating a test plan, the developer will study the functionalities and write test cases that test those functionalities. But his role ends here; implementing those test cases is the work of the tool being used for Automated Testing.

Pros and Cons of Manual Testing

PROS

  • It is used in both big and small projects.
  • Manual testing is better in terms of reliability.
  • If the test cases have to be run on a smaller scale in a project, it’s always better to use Manual Testing.
  • If the tester has to perform Ad-Hoc testing, then Manual Testing is a better fit.
  • Manual Testing exposes more real time bugs, as the tester sits on a machine and tries to penetrate the system thus exposing many bugs.

CONS

  • In big projects manual testing may not be every efficient and cost effective
  • In big projects it’s very difficult to run manual tests on every component within a stipulated deadline and cost budget.
  • Manual Testing has to be repeated for every change that was made and hence can be very tiring also exposing to unreliable testing.

Pros and Cons of Automated Testing

PROS

  • Extensive testing requires large number of tests to be run every time a change is made, and here the Automated Testing can be really efficient and reliable.
  • Automated tests can provide “Compatibility Tests”, when you run your application on various platforms.
  • Many tests scripts can be run simultaneously, and since all of them are accurate, they cut short the costs and time consumption on a large scale.
  • Automated tests are relatively cheaper investment in the long run.

CONS

  • Initial Investments on Automated Testing are bigger than the effort for Manual Testing.
  • Sometime GUI Testing will miss out on small components like visibility of elements, placements of fields and text areas.
  • Also not everything can be automated; sometimes Manual Testing is the only and the best way to go.


Comparison of Manual and Automated Testing

  • Automated Testing can be costly in terms of monetary benefits, but it does eliminate the costs of actual effort on testing. Manual Testing can be very laborious for huge/gigantic projects that have endless room for error.
  • Automated Testing involves human labour for writing Test Cases, and sometimes if the tools used for automated testing are very complex or the structure of the Project is so nested that running Test Cases will be more laborious than a community of Testers, it might be a bad idea to choose Automated Testing over Manual.
  • Testing Tools are not always “Simple” , i.e. sometimes the results produced by running Automation on a project might be difficult to comprehend, whereas Manual Testing would be very effective as the Developer/Tester knows what’s expected and can better analyze hands on if the code is not working properly, than to find it in hidden in bunch of scrambled digits.
  • Though Automation seems like “Extra Work”, it really is designed to make your work easier, imagine a huge application which caters to millions of customers, where the functionality keeps on upgrading and changing. Considering if this application requires 10^3 test cases, how tardy will it be to run these number of tests manually every time a small change is done in the code. Automated Testing is a blessing in disguise, as the effort for writing test cases will be the same, but we can save a fortune of money and effort by making the Tool run the tests fortnight and see the logs early in the morning.
  • Automated Testing eliminates Human and Technical Risks, many times a block of code will change drastically overtime and when the tester runs the same test cases manually, he may choose to eliminate some test cases assuming that they might not be impacted. Also sometimes during manual testing it becomes inevitable that some test cases will be left out either by choice or by natural circumstances, resulting in block of code prone to loop holes and crash issues. Which is not the case with automated Testing, as the automation tool is going to run every single test case that it was designed to test, “n” number of times repeatedly if required, and give out exact results to be read and debugged later in the project cycle.
  • Certain types of Testing, such as Performance, Regression testing is well suited for Automation tool testing, while Usability Testing is much better if Manual Testing is employed.


Few Tools for Automation of Testing

  • Autotest

AutoTest

AutoTest is known for instant feedback on your code i.e. the test runs on its own and you don’t have to worry about it. It is used in scenarios where you need instant result on any change requests that the developer has made or the cases where you change a small functionality and you need to see if that is working fine or has had a bad impact on your working Application. Also AutoTest doesn’t require the user to switch perspectives to each time code and check its results. It’s all automated. AutoTest is for Individual Testing i.e. suppose a developer is working on a module and he decides to change functionality and then wants to test it before he can commit and integrate the piece of code to the final build. In these scenarios AutoTest is very useful.

The catchy feature of AutoTest is that, it only runs the tests that it thinks are required for the changes you made in your application. You need to install a Ruby gem for working with AutoTest.

Installation of AutoTest on OS X and Linux

				sudo gem install ZenTest
  • Step 2: If Rails development has to be done then
                               sudo gem install autotest-rails
  • Step 3: Run AutoTest
                               $ cd <base directory of your rails project>
  • Step 4: AutoTest will run all the test cases for the first time
  • Step 5: Make some changes in your code
  • Step 6: Run Auto Test Again, and you will see that the only section of your tests will run this time.

AutoTest will assume that you are using Test::Unit for writing your test cases. So if you’re using RSpec, you need to manually inform AutoTest about your preference.

On command line type

                                RSPEC= true autotest

Also as an alternative to AutoTest setting true for RSpec you can also use the “autospec” command which is used instead of autotest.

On command line type

                                Autospec

Installation of AutoTest on Windows

  • Step 1: Install gem ZenTest from the command line
                                 gem install ZenTest
  • Step 2: on the command line read the directory of your rails project
                                 cd <path of rails project>
  • Step 3: Set the Environment variable before running AutoTest
                                 set HOME = C:\Documents and Settings\<your name on the machine> 
  • Step 4: In the same directory of the rails app type
                                 autotest --rails

CruiseControl.rb

CruiseControl.rb is used for Continuous Integration. Continuous Integration means that the developers integrate their code very frequently i.e. at least on a daily basis. So using CruiseControl.rb here prevents any integration issues that might accumulate and trouble later on at the time of integration.

Installation of CruiseControl.rb

  • Step 1: Download CruiseControl.rb

On Command Line type

  • Step 2:
                                 cruise add <name> -u <url>
  • Step 3:
                                 cruise start

For issues with installation , look at the manual

Conclusion

The decision to automate testing of an application or not has to be taken purely in the context of that application. There cannot be a general set of rules which specifies that manual testing is better or automated testing is better. Following are few questions that can lead us to conclusive answers.

  • Time and Effort required for Automation

The effort involved in writing a Test script should not be unreasonably more, as this might lead to more time and effort in developing test scripts than actually implementing the test cases manually. So a large part of this decision lies with how user friendly and customizable the Automation tool is.

  • Expected Test Coverage

The amount of code to be tested is another criterion, if the code to be tested is fairly short, then use manual Testing, but if large code coverage is required then Automated tests will save a lot of time.

  • Frequency of Testing

A lot depends on how often a piece of code is tested, if it's a critical application and requires constant effort on testing then it's better to use automated as it's less prone to make mistakes, fairly accurate and saves a huge amount of time, which could be utilized elsewhere. But if the functionality is tested only a few times then it's better to test it manually than write scripts for automation.

  • Stage of Project

It’s often found that in the initial development stage of a project, it is manual testing that precedes automation to identify bugs. However for mature products, it is automation that primary and manual testing only as and when required.

Thus we need to consider time, resources, and the size of the project as well as the quality of the automated tools while deciding the combination of manual and automated testing to develop the best software possible.

See Also

  1. Software Testing Methods and Techniques
  2. Context Driven Testing
  3. Software Testing Controversies

References

  1. http://en.wikipedia.org/wiki/Test_automation
  2. http://en.wikipedia.org/wiki/Manual_testing
  3. http://www.jobsnewstoday.com/2013/04/discuss-about-adverse-between-manual-testing-vs-automation-testing.html
  4. http://www.developsense.com/blog/2013/02/manual-and-automated-testing/
  5. http://www.softwaretestingmentor.com/automation/manual-vs-automation/
  6. http://blog.typemock.com/2012/04/05/the-difference-between-automated-and-manual-testing
  7. http://apprenda.com/blog/manual-v-automated-testing-achieving-consistency/
  8. http://www.base36.com/2013/03/automated-vs-manual-testing-the-pros-and-cons-of-each/
  9. http://www.tutorialspoint.com/software_testing/testing_types.htm
  10. http://www.slideshare.net/Softwarecentral/manual-testing-vs-automated-testing
  11. http://selleo.com/blog/software-outsourcing/the-value-of-automated-testing/2/