CSC/ECE 517 Fall 2012/ch2a 2w10 rc: Difference between revisions
| (5 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
<p> | <p> | ||
<font size="5"> | <font size="5">Automated Testing</font> | ||
</p> | </p> | ||
<p>Every software development group tests its products but defects are bound to be present in the end product. Test engineers<ref>http://jobsearchtech.about.com/od/careersintechnology/p/SWTest.htm</ref> try to detect these errors but they always creep in, even with the best manual testing processes. Automated software testing is the best way to increase the effectiveness, efficiency and coverage of software testing. It is the use of strategies, tools and artifacts that augment or reduce the need of manual or human involvement or interaction in unskilled, repetitive or redundant tasks. It involves the use of software to control execution of tests, comparison of actual outcomes to predicted outcomes, setting up of test preconditions, test design<ref>http://istqbexamcertification.com/what-is-test-design-technique/</ref>, and other test control and test reporting functions. There are many automation tools available in the market today. | <p>Every software development group tests its products but defects are bound to be present in the end product. Test engineers<ref>http://jobsearchtech.about.com/od/careersintechnology/p/SWTest.htm</ref> try to detect these errors but they always creep in, even with the best manual testing processes. Automated software testing is the best way to increase the effectiveness, efficiency and coverage of software testing. It is the use of strategies, tools and artifacts that augment or reduce the need of manual or human involvement or interaction in unskilled, repetitive or redundant tasks. It involves the use of software to control execution of tests, comparison of actual outcomes to predicted outcomes, setting up of test preconditions, test design<ref>http://istqbexamcertification.com/what-is-test-design-technique/</ref>, and other test control and test reporting functions. There are many automation tools available in the market today. | ||
</p> | </p> | ||
== | == Manual Testing == | ||
<p> | <p> | ||
Manual testing is the method in which testing of a software is done manually. Here, a tester acts like an end-user. All the features of a software are tested to know if the behavior of the software is in accordance with the expectations of the customer. The tester uses a test plan<ref>http://testingsoftware.blogspot.com/2005/11/what-is-test-plan_30.html</ref> and writes test cases<ref>http://softwaretestingfundamentals.com/test-case/</ref>, which are used for implementing the test plan. Although manual tests may find many defects in a software application, it is a laborious and time consuming process. Moreover, it may not be effective in finding certain classes of defects. | Manual testing is the method in which testing of a software is done manually. Here, a tester acts like an end-user. All the features of a software are tested to know if the behavior of the software is in accordance with the expectations of the customer. The tester uses a test plan<ref>http://testingsoftware.blogspot.com/2005/11/what-is-test-plan_30.html</ref> and writes test cases<ref>http://softwaretestingfundamentals.com/test-case/</ref>, which are used for implementing the test plan. Although manual tests may find many defects in a software application, it is a laborious and time consuming process. Moreover, it may not be effective in finding certain classes of defects. | ||
</p> | </p> | ||
== | == Why Automation Testing == | ||
<p>Automated software testing has long been considered critical for big software development organizations ;especially for applications requiring recurring testing by helping them deliver software more quickly, more consistently and at lower cost. Many companies are recognizing the importance of automating the work of testers and including the auto-test as a part of the regular build process. The results of automated tests are seen as a measure of the current quality of the software. Combined with a code coverage tool it answers all-elusive question: "How much of my code is currently running ok?."</p> | <p>Automated software testing has long been considered critical for big software development organizations ;especially for applications requiring recurring testing by helping them deliver software more quickly, more consistently and at lower cost. Many companies are recognizing the importance of automating the work of testers and including the auto-test as a part of the regular build process. The results of automated tests are seen as a measure of the current quality of the software. Combined with a code coverage tool it answers all-elusive question: "How much of my code is currently running ok?."</p> | ||
<p><b> <h4>The below factors make automation testing important:</h4> | <p><b> <h4>The below factors make automation testing important:</h4> | ||
| Line 24: | Line 24: | ||
A controlled web application test with thousands of users cannot be performed by even the largest software departments. Automated testing can simulate tens, hundreds or thousands of virtual users interacting with network or web software<ref>http://en.wikipedia.org/wiki/Category:Web_software</ref> and applications.</li> | A controlled web application test with thousands of users cannot be performed by even the largest software departments. Automated testing can simulate tens, hundreds or thousands of virtual users interacting with network or web software<ref>http://en.wikipedia.org/wiki/Category:Web_software</ref> and applications.</li> | ||
<li><b> Helps Developers and Testers </b><br> | <li><b> Helps Developers and Testers </b><br> | ||
Shared automated tests can be used by developers to find problems quickly before sending the code for Quality assurance. Tests would run automatically whenever source code changes are checked in and in case of failure error reports would be automatically sent to the team or the developer. Features like these save developers time and increase their confidence.</li> | Shared automated tests can be used by developers to find problems quickly before sending the code for Quality assurance<ref>http://en.wikipedia.org/wiki/Quality_assurance</ref>. Tests would run automatically whenever source code changes are checked in and in case of failure error reports would be automatically sent to the team or the developer. Features like these save developers time and increase their confidence.</li> | ||
<li><b> Improves Team Morale </b><br> | <li><b> Improves Team Morale </b><br> | ||
Automating repetitive tasks with automated software testing gives the project team , time to spend on more challenging and rewarding projects. Team members improve their skill sets and confidence and, in turn, pass those gains on to their organization.</li> | Automating repetitive tasks with automated software testing gives the project team , time to spend on more challenging and rewarding projects. Team members improve their skill sets and confidence and, in turn, pass those gains on to their organization.</li> | ||
</ol> | </ol> | ||
== | ==Approaches to Test Automation== | ||
===<small>Code-driven testing</small>=== | ===<small>Code-driven testing</small>=== | ||
| Line 38: | Line 38: | ||
<p> In this kind of testing ,user interface events such as keystrokes and mouse clicks are generated by the testing framework which observes the changes that result in the user interface and validates whether the observable behavior of the program is correct or not. A variation of this type of tool is for testing web sites wherein the web page is the “interface”. Such a framework utilizes an entirely different set of techniques because it involves reading HTML instead of observing window events. Another variation is scriptless test automation that does not use record and playback, instead builds a model of the Application Under Test (AUT) and then enables the tester to create test cases by simply editing the test parameters and conditions. Test-case maintenance is easy, as there is no code to maintain .As the AUT changes, the software objects can simply be re-learned or added. It can be applied to any GUI-based software application. The problem with this model is that AUT is actually implemented using test scripts, which have to be changed whenever there's change to the AUT.</p> | <p> In this kind of testing ,user interface events such as keystrokes and mouse clicks are generated by the testing framework which observes the changes that result in the user interface and validates whether the observable behavior of the program is correct or not. A variation of this type of tool is for testing web sites wherein the web page is the “interface”. Such a framework utilizes an entirely different set of techniques because it involves reading HTML instead of observing window events. Another variation is scriptless test automation that does not use record and playback, instead builds a model of the Application Under Test (AUT) and then enables the tester to create test cases by simply editing the test parameters and conditions. Test-case maintenance is easy, as there is no code to maintain .As the AUT changes, the software objects can simply be re-learned or added. It can be applied to any GUI-based software application. The problem with this model is that AUT is actually implemented using test scripts, which have to be changed whenever there's change to the AUT.</p> | ||
== | ==What to automate== | ||
<p>Test automation tools can be expensive, and are usually employed in combination with manual testing. Hence It is not possible to automate everything in the Software. Software systems which are used by a large amount of users simultaneously should be automated. For example, automating test that tests the combinatorial interactions of interdependent parameters at the API level or through the UI makes a lot of sense because the number of combinatorial tests for any complex feature would take much longer to test manually as compared to a 'data-driven' automated test.</p> | <p>Test automation tools can be expensive, and are usually employed in combination with manual testing. Hence It is not possible to automate everything in the Software. Software systems which are used by a large amount of users simultaneously should be automated. For example, automating test that tests the combinatorial interactions of interdependent parameters at the API level or through the UI makes a lot of sense because the number of combinatorial tests for any complex feature would take much longer to test manually as compared to a 'data-driven' automated test.</p> | ||
== | ==When to automate== | ||
<p>Automated tests, that are written before the code, capture the intention of the code, inform design decisions, provide rapid feedback and let the developers know when the code is done. Test automation could also be written after the system code has been written so the automation has to cope with less change. But Automated tests that are written after the code do not directly inform the design nor do they provide rapid feedback. If these tests help us build a better product then they should be written before the code. If we are using automation to do exploratory testing and we intend to throw the automation code away afterwards then we can write the tests later.</p> | <p>Automated tests, that are written before the code, capture the intention of the code, inform design decisions, provide rapid feedback and let the developers know when the code is done. Test automation could also be written after the system code has been written so the automation has to cope with less change. But Automated tests that are written after the code do not directly inform the design nor do they provide rapid feedback. If these tests help us build a better product then they should be written before the code. If we are using automation to do exploratory testing and we intend to throw the automation code away afterwards then we can write the tests later.</p> | ||
== | == Automation tools == | ||
<p> | <p> | ||
Tools are a driving agent for an automation process. Tools are specifically designed to target some particular test environment, such as Windows and web automation tools, etc. However, an automation framework is an infrastructure that provides the solution where different tools can do their job in a unified manner. This provides a common platform for the automation engineer.<br><br> Below is a table that lists out some of the automation tools that are being used in the software industry today:</p> | Tools are a driving agent for an automation process. Tools are specifically designed to target some particular test environment, such as Windows and web automation tools, etc. However, an automation framework is an infrastructure that provides the solution where different tools can do their job in a unified manner. This provides a common platform for the automation engineer.<br><br> Below is a table that lists out some of the automation tools that are being used in the software industry today:</p> | ||
| Line 122: | Line 122: | ||
|} | |} | ||
== | == Disadvantages of automation testing == | ||
<ul> | <ul> | ||
<li>Creating an automated test is more time-consuming and expensive than running it once manually. </li> | <li>Creating an automated test is more time-consuming and expensive than running it once manually. </li> | ||
<li>Proficiency is required to write | <li>Proficiency is required to write automation test scripts.</li> | ||
<li>Debugging the test script is major issue. If any error | <li>Debugging the test script is a major issue. If there is any error in the test script, sometimes it may lead to deadly consequences.</li> | ||
<li>Test maintenance is costly in case of playback methods. Even | <li>Test maintenance is costly in case of playback methods. Even if a minor change occurs in the GUI, the test script has to be re-recorded or replaced by a new test script.</li> | ||
<li>Maintenance of test data files is difficult, if the test script tests more screens.</li> | <li>Maintenance of test data files is difficult, if the test script tests more screens.</li> | ||
</ul> | </ul> | ||
== | == Conclusion == | ||
<p> | <p> | ||
In a nut shell, Test automation is the process of writing a computer program to do testing that would otherwise have to be done manually. Once tests have been automated, they can be run quickly and repeatedly. This is often the most cost-effective method for software products that have a long maintenance life, because even minor patches over the lifetime of the application can cause features that were working, to break. It is the best way to accomplish most of the testing goals and to effectively use resources and time. There are several automation tools available in the market | In a nut shell, Test automation is the process of writing a computer program to do testing that would otherwise have to be done manually. Once tests have been automated, they can be run quickly and repeatedly. This is often the most cost-effective method for software products that have a long maintenance life, because even minor patches over the lifetime of the application can cause features that were working, to break. It is the best way to accomplish most of the testing goals and to effectively use resources and time. There are several automation tools available in the market.However before buying a tool one should make sure it is the best fit to all the requirements that are at hand. | ||
</p> | </p> | ||
== | == References == | ||
<references/> | <references/> | ||
== | == Additional reading == | ||
<ol> | <ol> | ||
<li>http://www.tutorialspoint.com/software_testing/testing_types.htm</li> | <li>http://www.tutorialspoint.com/software_testing/testing_types.htm</li> | ||
Latest revision as of 16:57, 30 October 2012
Automated Testing
Every software development group tests its products but defects are bound to be present in the end product. Test engineers<ref>http://jobsearchtech.about.com/od/careersintechnology/p/SWTest.htm</ref> try to detect these errors but they always creep in, even with the best manual testing processes. Automated software testing is the best way to increase the effectiveness, efficiency and coverage of software testing. It is the use of strategies, tools and artifacts that augment or reduce the need of manual or human involvement or interaction in unskilled, repetitive or redundant tasks. It involves the use of software to control execution of tests, comparison of actual outcomes to predicted outcomes, setting up of test preconditions, test design<ref>http://istqbexamcertification.com/what-is-test-design-technique/</ref>, and other test control and test reporting functions. There are many automation tools available in the market today.
Manual Testing
Manual testing is the method in which testing of a software is done manually. Here, a tester acts like an end-user. All the features of a software are tested to know if the behavior of the software is in accordance with the expectations of the customer. The tester uses a test plan<ref>http://testingsoftware.blogspot.com/2005/11/what-is-test-plan_30.html</ref> and writes test cases<ref>http://softwaretestingfundamentals.com/test-case/</ref>, which are used for implementing the test plan. Although manual tests may find many defects in a software application, it is a laborious and time consuming process. Moreover, it may not be effective in finding certain classes of defects.
Why Automation Testing
Automated software testing has long been considered critical for big software development organizations ;especially for applications requiring recurring testing by helping them deliver software more quickly, more consistently and at lower cost. Many companies are recognizing the importance of automating the work of testers and including the auto-test as a part of the regular build process. The results of automated tests are seen as a measure of the current quality of the software. Combined with a code coverage tool it answers all-elusive question: "How much of my code is currently running ok?."
The below factors make automation testing important:
- Automated Testing is both Time and Money Saving
During software development cycle, software tests have to be repeated very often to ensure quality. Software tests need to be repeated, every time a change in the source code happens. For each release of the software it may be tested on all supported operating systems<ref>http://www.computerhope.com/os.htm</ref> and hardware configurations. Repeating these tests manually is costly and time consuming. Once created, automated tests can be run over and over again at no additional cost and they are much faster than manual tests. The time to run these repetitive tests is reduced from days to hours by automated software testing.Saving the time directly translates into cost savings. - Improves Accuracy
Manual testers are bound to make mistakes during monotonous manual testing. With Automated tests, the probability of errors occuring is relatively low. Automated tests perform the same steps precisely every time they are executed and never forget to record detailed results. - Increases Test Coverage
Automated software testing can increase the depth and scope of tests to help improve software quality. Lengthy tests that are difficult to be conducted manually can be run unattended in Automation testing. They can even be run on multiple computers with different configurations. Automated software tests can easily execute thousands of different complex test cases during every test run providing test coverage that is impossible with manual tests. - Automated Software Testing Does What Manual Testing Cannot
A controlled web application test with thousands of users cannot be performed by even the largest software departments. Automated testing can simulate tens, hundreds or thousands of virtual users interacting with network or web software<ref>http://en.wikipedia.org/wiki/Category:Web_software</ref> and applications. - Helps Developers and Testers
Shared automated tests can be used by developers to find problems quickly before sending the code for Quality assurance<ref>http://en.wikipedia.org/wiki/Quality_assurance</ref>. Tests would run automatically whenever source code changes are checked in and in case of failure error reports would be automatically sent to the team or the developer. Features like these save developers time and increase their confidence. - Improves Team Morale
Automating repetitive tasks with automated software testing gives the project team , time to spend on more challenging and rewarding projects. Team members improve their skill sets and confidence and, in turn, pass those gains on to their organization.
Approaches to Test Automation
Code-driven testing
Code driven test automation is a key feature of agile software development<ref>http://en.wikipedia.org/wiki/Agile_software_development</ref>, where it is known as test-driven development (TDD)<ref>http://en.wikipedia.org/wiki/Test-driven_development</ref>. With the help of a variety of input arguments, the public (usually) interfaces to classes, modules or libraries are tested to validate the results that are returned. Unit tests are written to define the functionality before the code is written. Only when all tests pass the code considered to be complete. Proponents argue that this kind of testing produces software that is more reliable and more cost-effective than the code that is tested by manual exploration. It is considered to be more reliable because the code coverage is better and it is run repeatedly during development rather than just once at the end of a waterfall development cycle. Defects are detected as and when a change is made and hence it costs less to fix the errors. Finally, code refactoring<ref>http://refactoring.com/</ref> is safer; transforming the code into a simpler form with less code duplication, but equivalent behavior, is much less likely to introduce new defects.
Graphical User Interface (GUI) testing
In this kind of testing ,user interface events such as keystrokes and mouse clicks are generated by the testing framework which observes the changes that result in the user interface and validates whether the observable behavior of the program is correct or not. A variation of this type of tool is for testing web sites wherein the web page is the “interface”. Such a framework utilizes an entirely different set of techniques because it involves reading HTML instead of observing window events. Another variation is scriptless test automation that does not use record and playback, instead builds a model of the Application Under Test (AUT) and then enables the tester to create test cases by simply editing the test parameters and conditions. Test-case maintenance is easy, as there is no code to maintain .As the AUT changes, the software objects can simply be re-learned or added. It can be applied to any GUI-based software application. The problem with this model is that AUT is actually implemented using test scripts, which have to be changed whenever there's change to the AUT.
What to automate
Test automation tools can be expensive, and are usually employed in combination with manual testing. Hence It is not possible to automate everything in the Software. Software systems which are used by a large amount of users simultaneously should be automated. For example, automating test that tests the combinatorial interactions of interdependent parameters at the API level or through the UI makes a lot of sense because the number of combinatorial tests for any complex feature would take much longer to test manually as compared to a 'data-driven' automated test.
When to automate
Automated tests, that are written before the code, capture the intention of the code, inform design decisions, provide rapid feedback and let the developers know when the code is done. Test automation could also be written after the system code has been written so the automation has to cope with less change. But Automated tests that are written after the code do not directly inform the design nor do they provide rapid feedback. If these tests help us build a better product then they should be written before the code. If we are using automation to do exploratory testing and we intend to throw the automation code away afterwards then we can write the tests later.
Automation tools
Tools are a driving agent for an automation process. Tools are specifically designed to target some particular test environment, such as Windows and web automation tools, etc. However, an automation framework is an infrastructure that provides the solution where different tools can do their job in a unified manner. This provides a common platform for the automation engineer.
Below is a table that lists out some of the automation tools that are being used in the software industry today:
| Tool | Developed by | Latest version | Important supported technologies |
|---|---|---|---|
| Ranorex | Ranorex GmbH | 3.3.3 | .NET, HTML, HTML5, Javascript, AJAX, Silverlight, SAP, IE, Safari, Firefox, Chrome |
| HP QuickTest Professional | HP Software Division | 11.0 | Web, Java (Core and Advanced), .Net, WPF, SAP, Oracle, Siebel, PeopleSoft, Delphi, Power Builder, Stingray 1, Terminal Emulator, Flex, Web Services, Windows Mobile, VisualAge Smalltalk, Silverlight |
| Selenium | OpenSource | 2.25 | Firefox,Chrome,IE,Opera |
| IBM Rational Functional Tester | IBM | 8.2.2 | List of supported platforms |
| LabVIEW | National Instruments | 2011 | Macintosh,Linux,Windows |
| Maveryx | Maveryx | 1.3.1 | Microsoft Windows, Linux, Mac OS,Java,NetBeans |
| Oracle Application Testing Suite | Oracle Corporation | 12.1 | Web/HTML, Adobe Flex, Siebel, Oracle E-business Suite, Oracle Fusion Applications, JD Edwards EnterpriseOne, Application Development Framework (ADF), Oracle Forms, Web Services, Oracle Databases |
| QF-Test | Quality First Software GmbH | 3.4.11 | Java/Swing, SWT, Eclipse plug-ins and RCP applications, ULC, Java applets, Java Web Start and cross-browser test automation of web-based applications (HTML, AJAX, ExtJS, GWT, RAP, Qooxdoo, RichFaces |
| Visual Studio Test Professional | Microsoft | 2012 | Windows API/SDK,.NET,Silverlight,XNA,ASP.NET,AJAX,LINQ,ADO.NET,Xbox development kit,managed DirectX,
Visual Basic,Visual Basic .NET,Visual C++,Visual C#,Visual J#,Visual FoxPro,IronPython,Expression Blend,Expression Web,Robotics Developer Studio,SharePoint Designer,FrontPage,Small Basic,WebMatrix |
| Eggplant (GUI testing tool) | TestPlant | 2012 | SenseTalk |
| TestComplete | SmartBear Software | 9.1 | Delphi, C++Builder, .NET, WPF, Java and Visual Basic,Microsoft Visual Studio 2005, 2008, 2010 Team System, BugZilla and AutomatedQA AQdevTeam |
| Testing Anywhere | Automation Anywhere | 7.5 | Java, Silverlight, .Net, mainframe, C++, etc. |
| Watir | Open source | 3.0 | Ruby, Internet Explorer, Firefox, Chrome, Opera and Safari, HTML |
Disadvantages of automation testing
- Creating an automated test is more time-consuming and expensive than running it once manually.
- Proficiency is required to write automation test scripts.
- Debugging the test script is a major issue. If there is any error in the test script, sometimes it may lead to deadly consequences.
- Test maintenance is costly in case of playback methods. Even if a minor change occurs in the GUI, the test script has to be re-recorded or replaced by a new test script.
- Maintenance of test data files is difficult, if the test script tests more screens.
Conclusion
In a nut shell, Test automation is the process of writing a computer program to do testing that would otherwise have to be done manually. Once tests have been automated, they can be run quickly and repeatedly. This is often the most cost-effective method for software products that have a long maintenance life, because even minor patches over the lifetime of the application can cause features that were working, to break. It is the best way to accomplish most of the testing goals and to effectively use resources and time. There are several automation tools available in the market.However before buying a tool one should make sure it is the best fit to all the requirements that are at hand.
References
<references/>
Additional reading
- http://www.tutorialspoint.com/software_testing/testing_types.htm
- http://support.smartbear.com/articles/testcomplete/manager-overview/
- http://www.buzzle.com/articles/manual-testing-tutorial.html
- http://www.outsourcebazaar.com/index_Article_AutomatedVsManualTesing.html
- http://www.aptest.com/tools.html