CSC/ECE 517 Fall 2009/wiki2 10 mk

GUI Testing Frameworks

Introduction

Most software developed in recent years has a graphical user interface (GUI). The only way for the end-user to interact with the software application is through the GUI. Hence, acceptance and system testing of the software requires GUI testing. In this wiki we aim at covering the different approaches, including patterns and tools for GUI testing.

Some problems of GUI testing

• GUIs are tested manually, often by the developers themselves. This is very unreliable and expensive. For new GUIs or those being significantly changed, quality is low, and failures at integration time or during user acceptance tests are common.
• ScreenScrapper based GUI test does a nice job but to a certain extent. Even though they are cheap, the problem with these tests is that if you change the screen layout all existing tests become useless, which means you have no regression tests. Another problem here is that test creators can't start writing tests till the GUIs are finished. Example: test harnesses, capture/replay tools(Example provided in the appendix), and model-based methods
• The user has an extremely wide choice of actions. The user could click on any pixel on the screen. Using manual tools to mimic the usage of the GUI only provides limited testing.
• There are tools which try to capture GUI widgets rather than mouse coordinates. These tools, however, require a significant amount of manual effort to be effective, including developing test scripts and manually detecting failures.Modifications to the GUI require changes to the scripts as well. Example: Winrunner, Abbot, and Rational Robot

Approaches for GUI testing

1. An Ontology-Based Approach for GUI Testing [1]

In the approach an GUI testing ontology is established by analyzing the source code with reverse engineering techniques. Then from the user experience the generation rules are extracted to create test cases. GUI testing is proposed for the purpose of making use of the knowledge provided by GUI systems and testers’ experience. GUI ontology is used to store potential information in a GUI system, while test case generation rules extract useful information from testers’ experience. In a word, ontology based GUI testing is a new branch of software testing, which not only takes the knowledge intensive features of GUI testing into account, but also sufficiently make use of them.

2. Automation of GUI testing using a model-driven approach [2]

In this approach the generated test cases are based on Unified Modelling Language. This introduces data into the UML model via the Category-Partition method. The functions that have to be tested are specified using the use cases and activity diagrams. This also specifies how they have to be tested. A combination like this has the potential to generate large number of test cases.The test can be managed in two ways.Firstly the Category-partitioned data which allows the designer full control over the possible and impossible paths for the system to run.Secondly automation allows different configuration for both data and graph coverage.Using all this we can generate test scripts which can be used for GUI testing.

Figure 1:Example Use Case Diagram

3. Plan Generation GUI testing [3]

This is based on the AI techniques, for partially automating GUI testing.In this method of testing the tester specifies the initial and the final goal states for the users of the system.The automated system produces a set of sequences or plans which will start with the initial state and end with the final state specified by the user. Each of the plans generated will represent a test case of a user of the system.

4. A practical approach to testing GUI systems [4]

In this approach, GUI is divided into two tires. One the component and other the system. Flow graphs will be created for each GUI component. The flow graph represents a set of preconditions, event sequences and post conditions of the corresponding component. On the system tire we build a viewpoint by integrating the components of the system. This will ensure that the components are working fine and are interacting as required. This is a simple, effective and practical method of performing GUI testing.

5. A Dynamic Partitioning Approach for GUI Testing [5]

The above approaches specify how to generate the test cases without actually specifying how to run them. This approach specifies how the test cases have to run in order to make GUI testing effective and useful. Th GUI primitive actions are partitioned into two classes. They are prioritized primitive actions and non-prioritized primitive actions. This further divides the testing into two stages which contains two feed back loops.The first stage prioritizes primitive actions and the second stage selects and performs prioritized primitive actions. The first feedback loop is local and occurs in the second stage, which adjusts the memberships of primitive actions after they are performed. The second feedback loop is global and occurs between the first and second stages. It switches GUI testing from the second stage to the first stage when no prioritized primitive actions are available. The two testing experiments with real GUI applications show that the proposed dynamic partitioning approach can really work in practice and may significantly outperform the random testing approach.

Tools for GUI testing

1. GUITAR

The GUITAR (GUI Testing frAmewoRk) project helps in simplifying GUI testing by automatically creating test cases that intelligently challenge a GUI's functionality. It currently contains a rich collection of plug-ins that may be used to test an application through its graphical user interface. For example, the “test case generator” plug-in, a tester can automatically generate various types of test cases for the Application Under Test (AUT); the “replayer” plug-in may be used to execute these test cases on the AUT automatically; during the various development phases of the AUT, the “regression tester” plug-in can be used to efficiently perform regression testing on the AUT.

figure 1:Guitar Framework

In order to test the GUI of the AUT, the tester has to perform a certain set of steps. These steps are detailed below

• Initialize configurations in GUITAR for the AUT. This can be done using the below window

figure 2:Preferences window for initializing the application type

• Rip the GUI structure of the AUT : To begin testing an application, the tester first needs to determine the GUI structure of the AUT. Running the Ripper applications on the AUT automatically does this. Types of rippers are available for analyzing the AUT. For analyzing an AUT developed using
  • Java use the JavaGUIRipper
  • Native Win32 use Windows GUI Ripper

• Generate the Event Flow Graph : The Event Flow Graph is generated from the GUI structure, ripped in the above step. The EFGGenerator generates the event-flow graph for an AUT’s GUI. To see how to analyzes a window-based application and understand its GUI integration tree. This is explained in detail here

• Generate Test cases : To generate testcases from the event flow graphs use the TCGenerator

• Replay the Testcases :

figure 3:Testcase execution

• Coverage Evaluation

Execute the coverage evaluator to analyze the coverage generated when the testcases were executed on the instrumented AUT. A coverage report is generated by the instrumented code, when the testcases are replayed on it. The coverage evaluator analyzes this report and a summary report is generated.

figure 4:Running the coverage evaluator

2. Planning Assisted Tester for grapHical user interface Systems (PATHS)

This is based on the event interaction sequences. This tests the GUI software using interactions which are mostly likely to be used in actual scenarios. This accepts an operator, initial state and a final state, with which the planning sequence produces a series of sequences which transforms the system form the initial state to the final state. The GUI tester can use this to generate interactions sequences by specifying the final state.

3. GUIdancer

GUIdancer is an Eclipse-based automated GUI test-tool which runs as a standalone application or as an Eclipse Plugin.

GUIdancer is different from other GUI test-tools because automated tests can be written before the Application Under Test (AUT) is ready. This means that GUIdancer is not a tool which tests an application by recording user actions and replaying them. Tests can be created from the requirements without access to the AUT, and involve no programming, script or code. GUIdancer tests can be created, run and maintained without support from automation experts.

Each Test Step (the smallest unit in GUIdancer) consists of three pieces of information chosen from interactive dialogs: the GUI-component to be tested, the action to execute on this component, and the parameters (or data) the action requires. A Test Step to enter “hello” into a text field would look like this:

* GUI-component: Text field
* Action: Enter Text
* Parameter: Hello

4. SeliniumHQ

Selenium is a robust set of tools that supports rapid development of test automation for web-based applications. Selenium provides a rich set of testing functions specifically geared to the needs of testing of a web application. These operations are highly flexible, allowing many options for locating UI elements and comparing expected test results against actual application behavior. How Selenium Works

Figure 1:How Selenium Works

6. Cucumber:

Cucumber is a tool that can execute plain-text functional descriptions as automated tests. The language that Cucumber understands is called Gherkin. Here is an example source:

Feature: Search courses
 In order to ensure better utilization of courses
 Potential students should be able to search for courses

 Scenario: Search by topic
   Given there are 240 courses which do not have the topic "biology"
   And there are 2 courses A001, B205 that each have "biology" as one of the topics
   When I search for "biology"
   Then I should see the following courses:
     | Course code |
     | A001        |
     | B205        |

Cucumber itself is written in Ruby, but it can be used to “test” code written in Ruby or other languages including but not limited to Java, C# and Python. Cucumber only requires minimal use of Ruby programming and Ruby is easy, so don’t be afraid even if the code you’re developing in is not Ruby.

7. jRapture:

jRapture is a tool for capturing and replying Java program execution in the field.This works with the Java binaries and the Java virtual machine. It employs a lightweight , transparent capture process that permits unobtrusive of a Java programs executions. jRapture captures the interactions between a Java program and the system, which includes GUI.It has a profiling interface that permits a Java program to be instrumented for profiling after its executions have been captured.Using an XML-based profiling specification language a tester can specify various forms of profiling to be carried out during replay.

Figure 1:Future API Modifications

Appendix

• ScreenScrapper - Software that automatically extracts data from interactive screens without user intervention.
• Regression testing - Regression testing is any type of software testing that seeks to uncover software regressions. Such regressions occur whenever previously working software functionality stops working as intended
• Test harness - In software testing, a test harness or automated test framework is a collection of software and test data configured to test a program unit by running it under varying conditions and monitoring its behavior and outputs.
• Model-based testing - Model-based testing is software testing in which test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system under test (SUT).
• WinRunner - WinRunner is an automated functional GUI testing tool that allows a user to record and play back UI interactions as test scripts.
• Abbot framework - The Abbot framework is a Java library that provides methods to reproduce user actions and examine the state of GUI components.
• Rational Robot - Rational Robot is a test automation tool for functional testing of client/server applications.
• Reverse engineering - Reverse engineering (RE) is the process of discovering the technological principles of a device, object or system through analysis of its structure, function and operation.
• Ontology - a rigorous and exhaustive organization of some knowledge domain that is usually hierarchical and contains all the relevant entities and their relations.
• Unified Modeling Language - Unified Modeling Language is the industry-standard language for the specification, visualization, construction, and documentation of the components of software systems. UML helps to simplify the process of software design, making a model for construction with a number of different views.
• Use case - A use case in software engineering and systems engineering is a description of a system’s behavior as it responds to a request that originates from outside of that system.
• Activity diagram - Activity diagrams are diagram technique showing workflows of stepwise activities and actions, with support for choice, iteration and concurrency.
• Flow graph - A control flow graph (CFG) in computer science is a representation, using graph notation, of all paths that might be traversed through a program during its execution.
• GUITAR - GUITAR is a suite of models, components, and tools for automated testing of software applications that have a Graphical User Interface (GUI) front-end.
• Application under test - refers to a system that is being tested for correct operation. The term is used mostly in software testing.

Conclusion

In summary, GUI testing is a complicated task. Systematic test design helps us to focus on the important tests and gives us an objective way of addressing risks. Tools are appropriate for many but not all tests and a staged approach to testing enables us to identify which tests to automate much more easily. Tools can therefore be used to detect errors pro-actively as well as to execute regression tests.

References

[1] http://c2.com/cgi/wiki?GuiTesting
[2] http://chandlerproject.org/Journal/AutomatedGuiTestingProject
[3] http://en.wikipedia.org/wiki/GUI_software_testing
[4] http://www.open-xchange.com/wiki/index.php?title=Automated_GUI_Tests
[5] http://agilistas.org/presentations/codecamp06/
[6] http://www.ranorex.com/?gclid=CNvA_YnesJ0CFchW2godpT5YrQ
[7] http://www.gerrardconsulting.com/GUI/TestGui.html
[8] http://www.slideshare.net/rpires/GUI-Test-Patterns
[9] http://en.wikipedia.org/wiki/List_of_GUI_testing_tools
[10] http://www.junit.org/taxonomy/term/6
[11] http://www.cs.umd.edu/~atif/papers/MemonSQW2000.pdf
[12] http://www.testingfaqs.org/t-gui.html
[13] http://www.springerlink.com/content/d08681k5081553r7/
[14] http://www.cs.umd.edu/~atif/GUITAR-distribution/manuals/Perform%20Tests%20on%20the%20AUT.htm#Rip
[15] http://seleniumhq.org/
[16] http://www2.computer.org/portal/web/csdl/doi/10.1109/COMPSAC.2006.94