CSC/ECE 517 Fall 2009/wiki2 10 Gui Testing Frameworks aa
GUI TESTING FRAMEWORKS
Introduction
A Graphical User Interface is a program interface that takes advantage of the computer's graphics capabilities to make the program easier to use. They are the assumed user interface for virtually all systems development using modern technologies.They deliver the system functionality for the clients in the client/server system.Ideally, user interfaces should only be thin shells on top of the next layer in an application displaying information in a useful fashion, and passing user input as quickly as possible to some code that knows what to do with it.However testing systmes with GUIs becomes very difficult because of the event-driven nature of GUI, unsolicited events and a never ending input domain.
The article here describes the problems associated with GUI testing and several possible patterns and tools used for GUI Testing.
GUI Testing Difficulties
GUI testing is comparatively difficult because of the following reasons:
- There is an infinite number of events that occur when a user is using the application. This makes it really difficult for a coder to anticipate all contexts and include event handlers for all of them.Also the number of potential paths from feature to feature within the application is so high that the scope for programmers to make errors is dramatically increased. The ‘infinite paths’ problem also makes it extremely unlikely that they will all be tested.
- Unsolicited events like a LAN printer going offline and an operating system popping up a dialog box asking the user to feed more paper can occur any time.Here not only is the number of test cases high but also special test drivers may be necessary to generate such events within the operating systems.
- A major problem involved with GUI testing is a large number of input spaces wherein user can click on any point on the screen at any time and there may be an undefined number of applications running simultaneously. This means that the application code needs to check the next event irrespective of whether it’s meant for it. This event driven approach makes testing an extremely complicated issue.
- GUIs are better suited to object oriented paradigm. A large number of methods and attributes are associated with each object that deals with GUI events thus making it even more complex. For example even for a simple text box there may be around thirty attributes which means a larger set of test cases to deal with.
- GUI environment deals with windows and user take the standard features of window management (re-size, move, maximize and minimize) for granted. These events do affect the application and thus need to be taken care of. For instance testing backward, forward, refresh etc makes testing a challenging task.
- GUI tests can’t be written until you finish the implementation phase. This is an issue considering the fact that there may be contractors developing the GUI who may not be available for testing thus forcing the tester to reverse engineer making life miserable for him.
- GUIs involve complex dependencies thereby resulting in a synchronization problem. For example if an application involves two windows one showing a portfolio of your stock market investments and the other making transactions then as soon as you buy a stock the user would like to see it in the portfolio, now for that to happen you need the two windows to be synchronized. With a large number of windows involved with the application the dependencies are extremely complex. Now a user may also want his transaction history window, the order book window etc be synchronized too with the transaction window. This shows the problem of a large number of dependencies which makes synchronization complicated.
- There can be many ways a user can reach a point in the application which may be a keyboard shortcut, function keys or mouse movements creating a never ending list of test cases.
- An infinite input domain problem is an issue wherein a user has an option of clicking anywhere on the window in view and entering data in any order he wishes too. This may result in a n factorial ways of inputting data where n is the number of fields in the GUI application. Greater the n larger the input domain and more the complexity for a tester.
GUI Test Principles
To develop a test process which is applicable for GUI testing in general we need to follow a set of principles. These principles deal with errors involved with GUI and using the GUI to exercise tests so is very-oriented toward black-box testing.
• The errors can be broken down into types, and we can design tests to diagnose each type of error. So, we can channelize the efforts in the direction of testing.
• A divide and conquer approach helps to separate out concerns. A complex problem is broken down into a large number of simple ones thereby concentrating on designing test cases for specific types of error.
• Automating GUI tests is complicated as seen before. Thus we need to ensure that we include this feature only when needed in stead of using it in general for everything.
• Layering as for everything helps separate complexity. Similarly layering of tests can be an effective solution as we implement the tests of individual components(low level tests) first and then test the integrated application at the end.
• We can use traditional techniques that are used for form based applications called black box test techniques wherever appropriate.
Different Approaches used for GUI Testing
The different approaches used for GUI testing are as follows:
Capture and Replay Tool
As the name suggests,Capture & Replay tool captures the user interactions with the system under test.Inputs, outputs, and other information needed to reproduce a session with the system under test need to be recorded during the capture process.It is used mostly for regression testing.Some Java testing frameworks employing this approach are Jacareto and Abbot.
During the capture process, the tool will register as an event listener.Event notification method for the tool will record the details of all events that occurred.During the replay process, the tool will register as an event source (possibly also as a listener)and compare results.For mouse and keyboard events, the tool has to substitute for the actual devices as the event source.At test checkpoints, the tool can ask various controls for their state and record it, for eg GUI radio button properties.Replay events should be initiated at the same relative time as during the capture.
Drawbacks
- As mentioned in the problems before, this can be tested only after the development has finished.
- A layout may change over the time and hence the test cases may become invalid owing to the changes in the physical interface.
- It lacks the necessary support to design the test cases and coverage criteria evaluation.
MVC Design Pattern
It is typically more difficult to perform automated tests on code residing in the user interface, and the more logic contained there, the greater the need for automated testing. The solution is the Model View Controller pattern. The basic idea behind this approach is to add a new level of indirection behind the user interface and the domain model with the intention of isolating UI changes and prevent them from requiring changes to the domain logic of the application. MVC divides an application into three concerns:
- Model - Encapsulates core application data and functionality domain logic.
- View - obtains data from the model and presents it to the user.
- Controller - receives and translates input to requests on the model or the view.
It pulls the complex part of the user interface code into a library that can be unit-tested, thus making GUI testing simpler.
Test Stubs and Drivers
A test stub is a dummy software component or object used (during development and testing) to simulate the behavior of a real component. More specifically while intergrating all the modules in top to bottom approach whenever any mandatory module is missing that module is replaced by a temperoary program which is called as stub.
If the function A you are testing, calls another function B,then use a simplified version of function B, called a stub.
void function_under_test(int& x, int& y) {
...
p = price(x);
...
}
double price(int x) {return 10.00;}
The value returned by function price is good enough for testing.The real price() function may not yet have been tested, or even
written.
Similarly, while integrating all the modules in bottom to top approach whenever any mandatory module is missing that module is replaced by a temperoary program which is called as test driver.
//Driver program for the function get_input.
#include <iostream.h>
void get_input(x& cost, int& y);
//Precondition: User is ready to enters values correctly.Postconditions: The value of cost has been set to the.
//wholesale cost of one item. The value of turnover has been set to the expected number of days until the item is sold.
int main( )
{ double a;
int b;
char ans;
do {
get_input(a, b);
cout.setf(ios::fixed);
cout.setf(ios::showpoint);
cout.precision(2);
cout << "a is " << a << endl;
cout << "b is " << b << endl;
cout << "Test again?"
<< " (Type y for yes or n for no): ";
cin >> ans;
cout << endl;
} while (ans == 'y' || ans == 'Y');
return 0;
}
Capture and Replay Tool
As the name suggests,Capture & Replay tool captures the user interactions with the system under test.Inputs, outputs, and other information needed to reproduce a session with the system under test need to be recorded during the capture process.It is used mostly for regression testing.Some Java testing frameworks employing this approach are Jacareto and Abbot.
During the capture process, the tool will register as an event listener.Event notification method for the tool will record the details of all events that occurred.During the replay process, the tool will register as an event source (possibly also as a listener)and compare results.For mouse and keyboard events, the tool has to substitute for the actual devices as the event source.At test checkpoints, the tool can ask various controls for their state and record it, for eg GUI radio button properties.Replay events should be initiated at the same relative time as during the capture.
Drawbacks
- As mentioned in the problems before, this can be tested only after the development has finished.
- A layout may change over the time and hence the test cases may become invalid owing to the changes in the physical interface.
- It lacks the necessary support to design the test cases and coverage criteria evaluation.
Guitar GUI Testing Framework
GUITAR is an extensible framework for testing software applications. It presents a unified solution to the software testing problem. GUITAR’s architecture allows testers to create new “process plug-ins” (i.e., tools/techniques/algorithms) to perform different testing activities. It currently contains a rich collection of plug-ins that may be used to test an application through its graphical user interface. For example, the “test case generator” plug-in, a tester can automatically generate various types of test cases for the Application Under Test (AUT); the “replayer” plug-in may be used to execute these test cases on the AUT automatically; during the various development phases of the AUT, the “regression tester” plug-in can be used to efficiently perform regression testing on the AUT.
Cucumber
Cucumber is a tool that can execute plain-text functional descriptions as automated tests. The language that Cucumber understands is called Gherkin. Here is an example:
Feature: Search courses
In order to ensure better utilization of courses
Potential students should be able to search for courses
Scenario: Search by topic
Given there are 240 courses which do not have the topic "biology"
And there are 2 courses A001, B205 that each have "biology" as one of the topics
When I search for "biology"
Then I should see the following courses:
| Course code |
| A001 |
| B205 |
While Cucumber can be thought of as a “testing” tool, the intent of the tool is to support BDD. This means that the “tests” (plain text feature descriptions with scenarios) are typically written before anything else and verified by business analysts, domain experts, etc. non technical stakeholders. The production code is then written outside-in, to make the stories pass.