CSC/ECE 517 Fall 2017/M1754 Mutation Testing: Difference between revisions

From Expertiza_Wiki
Jump to navigation Jump to search
Line 44: Line 44:


===Initial steps===
===Initial steps===
The approach adopted was based on the requirements mentioned [https://github.com/servo/servo/wiki/Mutation-testing-project here]. The below steps implement the initial steps mentioned in the project description.


* Step 1: A python script was written to mutate one source file and the corresponding WPT was run on it to check if the mutant was killed.
*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
 
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt:  This produced test failures.
* Step 2: A test framework was defined to identify the source files that require mutation testing along with their corresponding WPTs. This framework is implemented in /servo/components/script/dom.
*Introduced test mapping framework to map source files to WPT test.
 
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Step 3: The script was expanded to include the test framework and automate the process of generating mutations for multiple source files and running their corresponding WPTs based on the test_mapping.json. The script also traverses through subfolders of the parsed path to check for the .json file. The script also logs on the terminal any mutants that the WPT failed to kill.
 
* Step 4: Integrated the script so that it can be invoked from the CI tool.


===Subsequent steps===
===Subsequent steps===

Revision as of 01:27, 7 November 2017

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

Introduction

Servo

Servo is a modern, high-performance browser engine designed for both application and embedded use. Servo is a web browser layout engine written in Rustand is currently being developed by Mozilla. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [1]

Rust

Rust is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

Web-platform-tests

The web-platform-tests Project is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

Mutation Testing

Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[2]

Project description

Environment Setup

  • Clone the repository from link
  • The steps to setup the environment for different OS are mentioned in readme file link.

Install packages

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

Running test after setup

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

Initial steps

  • Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
  • Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
  • Introduced test mapping framework to map source files to WPT test.
  • Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.

Subsequent steps

  • implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
  • improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
  • find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
  • find search heuristics for identifying mutations that cause no test failures.

Execution

  • Mutation test can be ran using either of the following command from the servo directory:
python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

Excecution Flow

Given a path

mutation test(path)
Look for test mapping file in that path
for each file the in the test mapping
mutate a random line using some strategy in the file
build project and run WPT tests corresponding to it
if wpt tests pass?
report mautation test failure
else
report mutation test success

for each subfolder folder in the path

mutation test(subfolder)

Testing

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

References

<references/> 1. https://en.wikipedia.org/wiki/Servo_(layout_engine)
2. https://www.guru99.com/mutation-testing.html
3. https://github.com/servo/servo
4. https://github.com/servo/servo/wiki/Mutation-testing-project