CSC/ECE 517 Fall 2017/M1754 Mutation Testing on Servo

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

Introduction

Servo

Servo is a modern, high-performance browser engine designed for both application and embedded use. Servo is a web browser layout engine written in Rustand is currently being developed by Mozilla. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [1]

Rust

Rust is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

Mutation Testing

Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as Fault based testing strategy as it involves creating fault in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[2]

Project description

Build process

• The steps to setup the environment were followed as mentioned in readme file link. The servo code was built in release mode on a Mac environment using the below command:

./mach build --release

The whole set of WPT was run on the servo code base using the below command:

./mach test -wpt --release

Implemented steps

The approach adopted was based on the requirements mentioned here. The below steps implement the initial steps mentioned in the project description.

• Step 1: A python script was written to mutate one source file and the corresponding WPT was run on it to check if the mutant was killed.

• Step 2: A test framework was defined to identify the source files that require mutation testing along with their corresponding WPTs. This framework is implemented in /servo/components/script/dom.

• Step 3: The script was expanded to include the test framework and automate the process of generating mutations for multiple source files and running their corresponding WPTs based on the test_mapping.json. The script also traverses through sub folders of the parsed path to check for the .json file. The script also logs on the terminal any mutants that the WPT failed to kill.

• Step 4: Integrated the script so that it can be invoked from the CI tool.

Execution

The script can be called by running in terminal:

python python/servo/mutation/init.py components/scirpt/dom

Output

The log for success to kill a mutant is as shown below:

The log for failure to kill a mutant is as shown below:

Subsequent steps

The following subsequent steps will be followed to meet the project requirements as per this [3].

• implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
• improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
• find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
• find search heuristics for identifying mutations that cause no test failures.

Challenges

1. Setting up environment in local machine:

The amount of time taken to build and test the WPT in local machine was pretty long. Additionally faced problems regarding in which not all the WPT cases passed.

2. Defining the test mapping framework:

Servo consists of multiple components for which there many tests written for each component. The test cases are organised according to the functionality they test, where as the servo source code is organised according to the specific components.

Pull Request

Here is our pull request link.

References

<references/> 1. https://en.wikipedia.org/wiki/Servo_(layout_engine)
2. https://www.guru99.com/mutation-testing.html
3. https://github.com/servo/servo
4. https://github.com/servo/servo/wiki/Mutation-testing-project