CSC/ECE 517 Fall 2017/M1754 Mutation Testing on Servo

Servo uses the WPT test suite for integration testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

Introduction

Servo

Servo is a modern, high-performance browser engine designed for both application and embedded use. [ https://github.com/servo/servo ] is a web browser layout engine written in Rustand is currently being developed by Mozilla Research. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [1]

Rust

Rust is an open source, systems programming language sponsored by Mozilla Research.

Rust performs the majority of its safety checks and memory management decisions at compile time, so that your program’s runtime performance isn’t impacted. This makes it useful in a number of use cases that other languages aren’t good at: programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems. It’s also great for web applications: it powers the Rust package registry site.

Mutation Testing

Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as Fault based testing strategy as it involves creating fault in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases." [2]

Project description

Build process

• The steps to setup the environment were followed as mentioned here. The servo code was built in release mode on a Mac environment using the below command:

./mach build --release

The whole set of WPT was run on the servo code base using the below command:

./mach test -wpt --release

Implemented steps

• Step 1: A python script was written to mutate one source file and the corresponding WPT was run on it to check if the mutant was killed.

• Step 2: A test framework was defined to identify the source files that require mutation testing along with their corresponding WPTs. This framework is implemented in

        /servo/components/script/dom.

• Step 3: The script was expanded to include the test framework and automate the process of generating mutations for multiple source files and running their corresponding WPTs based on the test_mapping.json. The script also traverses through sub folders of the parsed path to check for the .json file. The script also logs on the terminal any mutants that the WPT failed to kill.

• Step 4: Integrated the script so that it can be invoked from the CI tool.

• Step 5: The log for failure to kill a mutant is as shown:

On Debian-based Linuxes:

   sudo apt-get install curl freeglut3-dev \
   libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
   msttcorefonts gperf g++ cmake python-virtualenv \
   libssl-dev libglfw-dev

• We then forked the original servo repository and cloned the forked repository on our local machines and then compiled and ran the servo browser.

   cd servo
   ./mach build --dev
   ./mach run tests/html/about-mozilla.html

Initial steps

• Add new enum that represents each possible validity check in ValidityState.

• Define a Validatable trait that contains a method which accepts this enum as an argument.

• Implement this trait for different form element types (HTMLInputElement, HTMLButtonElement, etc.), and define an as_maybe_validatable method on Element which returns an &Validatable value if the element implements the trait.

• Use JS<Element> member to call these new methods in ValidityState.

• We pushed the code to the servo and the pull request is #13969. The servo team requires only one commit so that all modifications we made are in that single commit.

Implemented steps

• Change using enum to bitflags that represents each possible validity check in ValidityState.

• Defined a trait Validatable in validation.rs file that contains methods to be used for validating an html element.

• The traits for different form elements like Input text, Input submit, Input Button, Input Image are defined in htmlInputElement.rs

• The code below will check the type of elements from the form and if they are input elements then it will call the is_instance_validatable function to check the validation.

Challenges faced

• While building the servo as shown above we were getting GLSL 1.50 is not supported error. We investigated this issue and found that there was an open issue #13515. We found that the error was because the rendering was done with Nvidia graphics card and so we changed it to the intel graphics card by ./mach run tests/html/about-mozilla.html -c and were able to run the browser. We even updated the open thread so that other people can take advantage of that.

Subsequent steps

• We will be adding methods so that we can statically and dynamically validate the constraints of the form element.
• We will implement the checkValidity and ReportValidity API for HTML form element.
• We will implement the validity state defined in the spec and implement the form submission algorithm to include the constraint validation.

Design patterns & o-o practices

This project is to add some features to servo, so we were not supposed to change the design of original servo project. However, after further discuss with servo project member, we found a disadvantage of the original design of using enum. Therefore we have made an agreement on changing this into using bitflags. Throughout the development process, we tried to follow good o-o practices by following DRY, giving the elements and functions the correct names and we tried and maintained the servo and rust coding standards.

Test

Test with tidy

Servo has its own code standard. We managed to meet its code standard by using the following command to run code style check.

   cd servo
   ./mach test-tidy

When we ran tidy we got 20 code style errors and we fixed all the errors.

Compile test

After adding code, we compiled the servo by the following command

   cd servo
   ./mach build -d

No error showed up, servo worked as expected with the new code.

Test from UI

At this stage, the project can't be tested from UI. There are still some subsequent steps to be implemented after this.

For reviewers

In addition to the reasons mentioned above, the servo is a web browser and not a web application. Therefore it can't be deployed on the cloud server. All these would make it hard for reviewers to test the correctness of code at this time.

References

<references />