CSC/ECE 517 Fall 2017/M1754 Mutation Testing on Servo

Servo uses the WPT test suite for integration testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

Introduction

Servo

Servo is a modern, high-performance browser engine designed for both application and embedded use. [ https://github.com/servo/servo ] is a web browser layout engine written in Rustand is currently being developed by Mozilla Research. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [1]

Rust

Rust is an open source, systems programming language sponsored by Mozilla Research.

Rust performs the majority of its safety checks and memory management decisions at compile time, so that your program’s runtime performance isn’t impacted. This makes it useful in a number of use cases that other languages aren’t good at: programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems. It’s also great for web applications: it powers the Rust package registry site.

Mutation Testing

Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as Fault based testing strategy as it involves creating fault in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases." [2]

Scope

The scope of the project was to complete the initial steps mentioned here.

The steps are as follows:

• implement a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
• build the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: and verify if test failures are generated.
• automate this process by writing scripts in a new python/servo/mutation directory, and calling them from scripts in /etc/ci.

Implementation

The following subsequent steps will be followed to meet the project requirements as per this [3].

• implement mutations,like replacing if statements by if true/if false,duplicating statements, reordering statements, changing arithmetic & atomic string constant.
• improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
• find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
• find search heuristics for identifying mutations that cause no test failures.

Testing

Following are the steps to run Mutation Testing for WPT on XMLHttpRequest.rs:

1. Install the pre-requisites required for servo as mentioned here
2. Run the following commands

git clone https://github.com/dsandeephegde/servo.git

cd servo

git checkout -b test origin/MutationTesting

python init.py

Note: It may take around 30-45 mins to build

Initial Mutation Testing Code Snippet

Mutation Testing Log

You will see that some tests may fail as expected.This indicates that WPT has covered in breadth the mutant that is currently been tested.

Note: We have not added any new tests to the test suite for servo nor we have we modified any component of servo source code. We have written scripts that generate mutants on the source code and run their corresponding WPT tests to determine the extent of test coverage.

Testing From UI

Our scripts can be called from command line interface, the corresponding results are displayed on the terminal

Pull Request

Here is our pull request. In the link you can see all code snippets changed due to implementing the above steps that resulted in the test logs. Also information regarding the mapping of test scripts and source code is available that get invoked when Mutation Testing script is called.

References

<references/>

1. http://wiki.expertiza.ncsu.edu/index.php?title=CSC/ECE_517_Fall_2015/Mozilla_Refactor_GLES2&action=edit&section=7
2. https://www.guru99.com/mutation-testing.html
3. https://github.com/servo/servo/wiki/Mutation-testing-project
4. https://servo.org/
5. https://github.com/servo/servo/wiki/Mutation-testing-project