CSC/ECE 517 Fall 2013/ch1 1w32 av

GUI Testing

Open Source GUI Testing Tools

Graphical user interface testing is the process of testing a product's graphical user interface to ensure it meets its designed specifications.

GUI testing is difficult mainly due to following reasons :

• Large space of possibilities. Each sequence of GUI commands can result in various states. Evaluation of all these states is essential for validation of the GUI of the given software.
• Event-driven architecture. The test suite has to simulate the various user created actions.

Following are the challenges encountered while testing the graphical user interface of any software or system:

• Automation is difficult.
• The scope for testing is enormous
• Inability to determine the state of the program from GUI.
• Regression testing is difficult. Testcases may not be reusable

The current trend in user interfaces is geared towards graphical user interfaces (GUIs). This presents a problem for designing tests for software, since GUIs are very complex and hence GUI testing is very time consuming. Automation is a requirement for testing any larger graphical user interfaces, but automating GUI tests is not a straightforward task. Techniques which are familiar from the command line interface (CLI) age, don’t translate to the GUI world without problems.

Test Automation

The advantages of test automation are as follows.

• Frequent regression testing
• Rapid feedback to developers - The earlier the problems are found, the cheaper it is to fix them. An automated unit test suite finds problems effectively as early as possible, long before the software reaches a customer, and even before it reaches the QA team.
• Virtually unlimited iterations of test case execution
• Support for Agile and extreme development methodologies
• Disciplined documentation of test cases
• Customized defect reporting
• Finding defects missed by manual testing

The challenge is in extending these advantages to GUI testing.

GUI testing in Ruby and Rails

Ruby prefers the term scripted testing over the more traditional automated testing. For testing a GUI application there can be several kind of tests in the background to verify the functionality and requirements of an application.

Various frameworks that support testing in Ruby are:

• Test:Unit - It is a unit testing framework for Ruby.
• Capybara[1] - It is an integration testing tool for web applications.
• Rspec[2] - Behaviour Driven Development (BDD) framework/ test tool
• Cucumber[3] - It is a tool for running automated acceptance tests written in a behavior-driven development (BDD) style

Ruby on Rails testing frameworks are much more advanced than other web frameworks.

GUI Testing - Pitfalls

• Coverage criteria cannot be determined accurately with GUI testing.
• Verifying whether the GUI test executes correctly poses a problem as the logging is difficult.
• Developing a stable regression suite is difficult since simple changes in GUI can render the test suite obsolete.

GUI Testing Tools

GUI testing tools serve the purpose of automating the testing process of software with graphical user interfaces. Following are few of the available open source GUI testing tools:

• AutoHotkey
• CubicTest
• Dogtail
• Dojo
• Selenium
• Watir
• GNU Xnee
• Sikuli
• Webrat

Selenium

Introduction

Selenium is a portable software testing framework for web applications. Selenium provides a record/playback tool for authoring tests without learning a test scripting language (Selenium IDE). It also provides a test domain-specific language (Selenese) to write tests in a number of popular programming languages, including Java, C#, Groovy, Perl, PHP, Python and Ruby. The tests can then be run against most modern web browsers. Selenium deploys on Windows, Linux, and Macintosh platforms.

Selenium Tool Suite

Selenium is composed of multiple software tools.

Selenium IDE

• Prototyping tool for building testscripts.
• Firefox plugin provides easy to use interface for developing automated tests.
• Supports 'record and play' feature to create reusable scripts for testing.

Selenium1

• Also known as Selenium RC or Remote Control
• Provides flexibility in porting tests across browsers and platforms.
• Currently in maintainence mode. No active development.

Selenium 2

• Also known as Selenium Webdriver
• Supports WebDriver API thus providing maximum flexibility in porting the tests.
• Runs Selenium 1’s Selenium RC interface for backwards compatibility.

Selenium Grid

• Selenium-Grid allows the Selenium RC solution to scale for large test suites and for test suites that must be run in multiple environments.
• Selenium Grid runs the tests in parallel, that is, different tests can be run at the same time on different remote machines.
• Selenium Grid greatly improves the time it takes to run the suite by making use of parallel processing.

Supported Browsers and Platforms

Selenium 2

• Google Chrome 12.0.712.0+
• Internet Explorer 6, 7, 8, 9 - 32 and 64-bit where applicable
• Firefox 3.0, 3.5, 3.6, 4.0, 5.0, 6, 7
• Opera 11.5+
• HtmlUnit 2.9
• Android – 2.3+ for phones and tablets (devices & emulators)
• iOS 3+ for phones (devices & emulators) and 3.2+ for tablets (devices & emulators)

Selenium 1 and Selenium IDE

Browser	Selenium IDE	Selenium 1 (RC)	Operating Systems
Firefox 2, 3	Record and playback tests	Start browser, run tests	Windows, Linux, Mac
IE 6, 7, 8	Unsupported	Start browser, run tests	Windows
Safari 2, 3, 4	Unsupported	Start browser, run tests	Windows, Mac
Opera 8, 9, 10	Unsupported	Start browser, run tests	Windows, Linux, Mac
Google Chrome	Unsupported	Start browser, run tests	Windows, Linux, Mac

Advantages

The chief advantages of Selenium are :

• Flexibility - Test automation can be customized by adding functionality to Selenium test scripts and Selenium framework.
• Open Source - The source code can be downloaded and modified as needed to suit new browsers.
• Deploys on multiple paltforms - Windows, Linux, and Macintosh.

Disadvantages

• Identifying the locators and common attributes like id, names etc as well as XPATH, javascript DOM and others is difficult.
• Results are not available in good detail. Debugging failures migt prove to be difficult sometimes.
• Loop and data driven testing is not supported by Selenium IDE.
• Options to verify the images are not available.

Auto Hot Key

AutoHotkey automates tests by sending keystrokes and mouse clicks to the target application with the help of a macro recorder. Virtually any key, button, or combination can become a hotkey. Following are the notable features of hotkey:

• Expand abbreviations as you type them. For example, typing "gn" can automatically produce "good night".
• Create custom data-entry forms, user interfaces, and menu bars.
• Remap keys and buttons on your keyboard, joystick, and mouse.
• Convert any script into an EXE file that can be run on computers that don't have AutoHotkey installed.

Quick start tutorial : http://www.autohotkey.com/docs/Tutorial.htm

Watir

Watir, pronounced water, stands for Web Application Testing in Ruby. Watir is an open-source (BSD) family of Ruby libraries for automating web browsers. It is distributed as a Ruby GEM named 'Watir'. Watir drives browsers the same way people do. It clicks links, fills in forms, presses buttons. Watir also checks results, such as whether expected text appears on the page or not.

There are 2 version available :

• Watir-Classic : supports only Internet Explorer on Windows
• Watir-WebDriver : supports Chrome, Firefox, Internet Explorer, Opera and also running in headless mode (HTMLUnit).

GNU Xnee

GNU Xnee is a suite of programs that can record, replay and distribute user actions under the X11 environment. Xnee has the capability to :

• Automate tests like nightly regression
• Demonstrate programs by recording and replaying a user interaction with the system
• Distribute actions from system admin to end users.

The Record and replay ‘macro’ is the highlight of Xnee.

DOJO

Dojo is a JavaScript framework targeting the many needs of large-scale client-side web development.

• Dojo abstracts the differences among diverse browsers to provide APIs that will work on all of them (it can even run on the server under Node.js)
• It establishes a framework for defining modules of code and managing their inter-dependencies
• Supports internationalization, localization, and accessibility
• Provides a rich suite of commonly needed utility classes and user-interface widgets.

CubicTest

CubicTest is a graphical Eclipse plug-in for writing Selenium and Watir tests. It makes tests faster and easier to write, and provides abstractions to make tests more robust and reusable.

• CubicTest's test editor is centered around pages/states and transitions between these pages/states.
• The model is intuitive for both Ajax and traditional web applications and supports most user interaction types.
• CubicTest features an innovative test recorder and test runner based on Selenium RC which are fully integrated with the graphical test editor.

Dogtail

Dogtail is a GUI test tool and automation framework written in ​Python. It uses Accessibility (a11y) technologies to communicate with desktop applications. Dogtail scripts are written in Python and executed like any other Python program. Follwoing are the notable features of Dogtail:

• Object-Oriented API: With the object oriented API, you are under control of the UI tree and its elements, allowing you to write sophisticated scripts utilizing benefits of OO right deep from dogtail.
• Procedural API: Ace programmer credentials are not necessary to write useful automated scripts with procedural API.
• Easily Extensible: Writing custom classes and helper libraries are easy.
• Results and debug reporting: Separate logs for test case comparisons and debug information are generated.

Sikuli

Sikuli automates GUI testing by using image recognition to identify and control GUI components. It is useful when there is no easy access to a GUI's internal or source code.

• Sikuli uses an innovative approach of automating GUI tests by using screenshots and powerpoint slides.
• Sikuli Script is a visual scripting API using Jython to create automation scripts.
• Sikuli includes an integrated development environment (IDE) for writing visual Sikuli scripts with screenshots.

Webrat

Webrat lets you quickly write expressive and robust acceptance tests for a Ruby web application.

• Webrat differs from Selenium or Webrat in that it uses a library called Nokogiri to parse the (X)HTML response from a Rails application rather than an actual web browser.
• Testing in Webrat has an advantage of making the tests browser-independent and quicker to run.

See Also

• Web testing tools [4]
• Selenium Framework [5]
• Selenium [6]
• Using Selenium [7]
• Selenium by example [8]
• Scalable UI Testing with Selenium [9]

References

• http://docs.seleniumhq.org/docs/
• http://www.eecs.yorku.ca/course_archive/2004-05/W/4313/Slides/Module10-GUI.pdf - 2004-05
• http://www.cs.helsinki.fi/u/paakki/mikkolainen.pdf
• http://watir.com/
• http://www.gnu.org/software/xnee/
• https://fedorahosted.org/dogtail/
• http://www.cubictest.org/
• http://www.autohotkey.com/
• http://www.sikuli.org/
• http://sivaeluri.blogspot.com/2012/08/advantages-and-disadvantages-of.html
• http://dojotoolkit.org/reference-guide/1.9/util/dohrobot.html
• http://railscasts.com/episodes/156-webrat
• GUI Testing: Pitfalls and Process, Atif M. Memon, University of Maryland, IEEE Computer 2002, p. 87 - 88 [10]

Note to reviewers

The topic is very vast. Hence the wiki is written based on the guidelines provided by the TA's and the Prof.

• List all GUI tools used in OSS that you can quickly identify.
• Try to describe the tools to the same level of detail, except that you may choose to go into more detail on tools that are used with Rails.
• Listing of commands is not a good idea. You are not writing a reference manual; you are writing a descriptive article that someone might read to learn about what's available before choosing a tool or tools. You should aim for readability.

We have referred the respective tool websites to provide the overview for each one of them. We have provided appropriate citations as well. Additionally the tool and the documentation itself are under GNU/ open source licence and hence will not fall under copyright violation. Please correct us if we are wrong.