CSC/ECE 517 Fall 2009/wiki2 10 28

GUI testing frameworks

GUI (graphical user interface) allows users to interact with the system [1]. The user has more natural approach to the applications and feels more in control of the system. Today, every application provides GUI support and the application's ease-of-use makes it popular among users. The process of testing the user interface to verify if it meets the specifications is called GUI testing [2]. The most popular approach is to maintain checklists to perform manual testing. The other approach is to perform usability testing which includes testing navigation, user friendliness and other non-functional requirements [10].

Challenges in GUI testing

Testing Graphical User Interfaces (GUI) is a difficult task. The event driven nature, large number of states, different permutations of inputs and events, window management are few features of GUI that provides hurdle in automating GUI testing. Following is the detailed explanation :-

Event driven nature : Graphical user interface provides user with full control of the system. There are many possible user inputs as user may click on any pixel on the screen. User may click on any field or on a button, and this may bring up a new window. It’s possible that this window belongs to another application. Since users are provided with extremely wide options, large number of actions and inputs can be expected. This makes testing hard and increases the error probability as it’s difficult for tester to write test cases for every possible event [3]. Also, Users can navigate from one feature to another unrelated feature using the options like a button clicks. Software has to handle these potential unknown context, and it’s possible that the previous feature is still alive, left in partially completed state. The navigation feature of GUI raises the “infinite path” problem as there are many possible ways to navigate from one feature to another. Thus, navigation feature makes it difficult for tester to test all possible paths in the GUI framework [3].

Infinite input domain : Testing graphical user interface requires different permutations of input. This not only increases the complexity but also increases the input size domain. These inputs are user interactions or events while output could be an event or some graphical object [8]. There are “many ways in” possible for the user to reach any point in the software [3]. GUI tester has to decide how many of these input paths existing and whether all these paths has to be tested. Similarly there may be ‘many ways out’ like keyboard shortcut, a button click, a menu option, click on another window etc. This further increases the difficulty in GUI testing [3].

Window Management : Testing standard features of windows management like window movement, resizing, maximization, minimization and closure etc. are also part of GUI testing. Using available standard buttons and key board commands, programmer implements the above features and must also hand the impact on the application. The behavior of these windows is controlled by operating system [3]. In some circumstances, programmer disables these window features to avoid user closing the application before transaction is complete. But this prevents user to undo or reverse actions. The difficult in testing arises when tester has to decide which controls, navigation path forward or backward or both needs to be tested [3].

Regression testing : GUI forms large part of software and are usually developed using rapid prototyping. The correctness of systems GUI ensures the overall proper functioning of whole system. To ensure correctness, the systems GUI need to be test comprehensively. As GUIs are created, modified using rapid prototyping on continuous basis, the test cases developed must be reusable across versions. But continuous modifications make it difficult for tester to run regression suites [11]. For example, a test case written to test an event by clicking on a button may not be run again as this button must have been deleted or moved to some other form. Thus, continuous modification makes it difficult for regression testing as it makes large number of test cases unusable and requires expensive regeneration [11].

GUI Test Patterns

A pattern is a written document that describes a general recurring solution to a repeatedly occurring problem in software engineering projects. GUI testing patterns also follow the same testing procedure used for testing other conventional software's.The general testing patterns is to determine what is to be tested, generating test input, generating expected output and to execute test cases and to verify its output 12.

GUI testing frameworks

Capture/Replay framework

Capture/Replay is the most popular technique in GUI testing domain. In this technique the capture tool generates a script that records every interaction of the tester with the application GUI. This file is later replayed by the replay component of the software [6]. However, this process of GUI testing is resource intensive and requires significant human intervention [6].

Pitfalls of Capture/Replay technique

Major drawback of Capture/Replay technique is that when GUI changes, previously recorded input sequence may not be valid [7]. It relies on test engineer to provide all possible GUI interactions. Automation of this technique requires the programmer to consider all possible decision points in the GUI [7]. Bitmap comparison using this technique is not reliable as even a desirable 1 bit shift in pixel will be reported as Test failure. Thus, it’s a very time consuming, requires extensive human intervention and their are high chances of missing important decision points while supporting automation [7].

Test Monkeys

Manual testing may not detect all the bugs in an application. Even if automation of manual testing is done by generating test scripts, it will still lack human common sense; making it difficult to cover high risk part of an application[9]. Test monkeys are automated tools that randomly assign test cases, resulting in large range input sizes. Thus, working along with the test scripts , with no human intervention and large input size combination, makes it possible to test an application effectively. The advantage of using Test Monkeys is that its user interface independent, thus any change in GUI will not affect its performance[9].In addition, a test monkey will keep running the tests until the application crashes, thus increasing the confidence in the system [13].

Automated GUI Testing Tools

IBM Rational Test Tools

IBM’s Rational software branch produces large number of testing tools for every stage of development.

Rational Robot

The user needs to activate the capture/replay recorder in Microsoft Visual Studio 6.It captures the user interactions and generate script written in visual basic type language called SQABasic. This lacks effective capturing and verification of objects. For more details, please visit http://www-01.ibm.com/software/awdtools/tester/robot/index.html

Rational Visual Test

This tool allows testers either to write test cases or record test sessions. Applications built on Microsoft visual studio are tested using this tool. Testers can also provide test data while performing recording sessions.For more details, please visit http://www-01.ibm.com/software/awdtools/visualtest/support/index.html

Mercury Interactive tools

Mercury Interactive toolsprovides tool for testing internet applications and customized packages.

WinRunner

WinRunner is a testing tool for windows and web applications based on Capture/Replay technique. The recorded scripts are written in language called Test Script Language (TSL). After generating test scripts, the testers can add or remove GUI components, insert test data etc.

LoadRunner

LoadRunner acts as a virtual user to simulate the real user actions on the scripts generated by other tools. Its main feature is to perform stress testing, measure responsiveness, identify bottlenecks and other performance related issues.

Segue’s Silk Test

Segue’s Silk testing tool is used for testing GUI components built using Microsoft foundation Class library.It comes with capture/replay component and a interactive wizard that allow testers to select the GUI components to be tested.The testers actions are recorded into test scripts an language used is 4Test.

Open Source Testing tools

Most of the open source GUI testing tool are for java platform. Few of them are listed down.

Abbot

Abbot is a testing tool for Java GUI test framework and performs unit and functional testing. User actions are reproduced and the state of GUI components are examined. Java code or scripts can be used to invoke this tool. Following example shows how GUI unit test code looks like using Abbot (Source)

 // Suppose MyComponent has a text field and a button...
    MyComponent comp = new MyComponent();
    // Display a frame containing the given component
    showFrame(comp);

    JTextField textField = (JTextField)getFinder().
        find(new ClassMatcher(JTextField.class));
    JButton button = (JButton)getFinder().find(new Matcher() {
        public boolean matches(Component c) {
            // Add as much information as needed to distinguish the component
            return c instanceof JButton && ((JButton)c).getText().equals("OK");
        }
    });
    JTextComponentTester tester = new JTextComponentTester();
    tester.actionEnterText(textField, "This text is typed in the text field");
    tester.actionClick(button);
    // Perform some tests on the state of your UI or model
    assertEquals("Wrong button tooltip", "Click here to accept", button.getToolTipText());

GUIdancer

GUIdancer an eclipse based automatic testing tool for generation of java and swing test programs. This automated tool is more helpful in Test driven development [14]. It does not involve any programming or scripts to be written. The GUIdancer are made up of Test Steps that defines the GUI component under test, action or event associated with component and the input parameters (or data) (Source).

 A Test Step to enter “hello” into a text field would look like this:

    * GUI-component: Text field
    * Action: Enter Text
    * Parameter: Hello

Jacareto

Jacareto is a testing tool for java application and applets. This open source capture/replay tool has 3 parts: Jacareto: event record and playback, Picorder: command line interface and CleverPHL: graphical user interface. It allows customization i.e. testers can design their own capture and replay modules. It can also be used to perform quantitative analysis.

Pounder

Poundera utility that does dynamic loading of Java GUI and recorded scripts are used in test harnessing.

Cucumber

Cucumber is a testing tool that allows tester to write user stories i.e. to execute feature documentation written in plain text. Gherkin is the language supported by cucumber [15]. The test cases are written as user stories as follows [15] (source) :

Scenario: See all vendors
    Given I am logged in as a user in the administrator role
    And There are 3 vendors
    When I go to the manage vendors page
    Then I should see the first 3 vendor names

Cucumber supports Behavior Driven Development (BDD) i.e. before developing the application, running these tests can be verified by domain experts,customers etc. and then later on write production code to pass these test cases. This tool can be used for languages like Ruby, C#, Java and Python [16].

GUITAR

GUITAR is an automated GUI testing tool that provides a suite of model, component and tools for testing []. Using GUITAR framework, testers can create new “process plug-ins” (i.e., tools/techniques/algorithms) to perform different testing activities. It also provides a collection of plug-ins like "test case generator”,“replayer” and “regression tester” plug-ins. "test case generator” plug-in is used for generating test case for the application under test, “replayer” plug-in automatically replays these tests on the application and “regression tester” plug-ins is used for driving regression test in application development phase [18].

Conclusion

Today, GUI is developed using rapid prototyping and this has driven testing market to develop automated testing tools. Most of these testing tools are based on Capture/Replay technique. Test monkeys is an effective tool as it can randomly assign test cases and keep running these test cases till the application crashes. A brief description of commercially available testing tools was discussed. The new open source testing tools like Cucumber, GUITAR etc. was introduced.

References

[1] http://en.wikipedia.org/wiki/Graphical_user_interface
[2] http://en.wikipedia.org/wiki/GUI_software_testing
[3] http://www.gerrardconsulting.com/GUI/TestGui.html
[4] http://en.wikipedia.org/wiki/List_of_GUI_testing_tools
[5] http://www.geocities.com/xtremetesting/TestingDictionary.html#S
[6] GUI Ripping: Reverse Engineering of Graphical User Interfaces for Testing, Ishan Banerjee, Adithya Nagarajan, and Atif Memon
[7] http://www.cs.umd.edu/~atif/papers/MemonIEEEComputer2002.pdf
[8] Lee J. White, "Regression Testing of GUI Event Interactions," icsm, pp.350, 12th International Conference on Software Maintenance (ICSM'96), 1996
[9] Effective GUI test automation: developing an automated GUI testing tool, by Kanglin Li, Mengqi Wu.
[10] http://www.comparesuite.com/solutions/tests-automation/hbb_windows_application_gui_test.htm
[11] MEMON, A. M. AND SOFFA,M. L. 2003. Regression testing of GUIs. In ESEC/FSE-11: Proceedings of the 9th European Software Engineering Conference Held Jointly with 11th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM Press, New York, NY,118–127.
[12] http://msdn.microsoft.com/en-us/library/ms979209.aspx
[13] http://media.wiley.com/product_ancillary/12/07821435/DOWNLOAD/4351ch02.pdf
[14] http://en.wikipedia.org/wiki/GUIdancer
[15] http://www.rubyinside.com/cucumber-the-latest-in-ruby-testing-1342.html
[16] http://wiki.github.com/aslakhellesoy/cucumber
[17] http://guitar.sourceforge.net/
[18] http://www.cs.umd.edu/~atif/GUITAR-distribution/manuals/index.html

External Links

[1] http://www.rubyinside.com/cucumber-the-latest-in-ruby-testing-1342.html
[2] http://www.members.tripod.com/~bazman/checklist.html
[3] http://www.geekinterview.com/question_details/15476
[4] http://guitar.sourceforge.net/