Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-12-07T02:07:46Z

Sdhegde: /* Subsequent steps */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to set up the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:
Before Mutation:
if a && b {
c = 1;
} else {
c = 0;
}
After Mutation:
if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Change String literals
* Change Comparision operators
* Duplicate statements
* Change arithmetic operations.

As the build and WPT test run take a lot of time to run. Mutation Test Strategies are implemented with these key features
* The mutants produces only run-time bugs.
* The mutants do not cause any compilation failure.
* The mutants do not produce False Positives.

===Mutation Strategy Examples===
* The example for this strategy has already been mentioned in the [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2017/M1754_Mutation_Testing#Existing_Mutation_Strategy Existing Mutation Strategy] section.
* Make 'if condition' to 'if true'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if true
{
c = 1;
}
* Make 'if condition' to 'if false'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if false
{
c = 1;
}
* Change Comparision operators
Example:
Before Mutation:

if a >= 1
{
c = a * 10;
}

After Mutation:

if a > 1
{
c = a * 10;
}

*Change String literals
Example:
Before Mutation:

if a > 0
{
c = "a is possitive";
}

After Mutation:

if a > 0
{
c = " ";
}

* Duplicate statements
Before Mutation:
for (i=0;i<5;i++)
{
counter++;
}
if (counter%2==0)
{
x=x+y;
}
After Mutation:

for (i=0;i<5;i++)
{
counter++;
counter++;
}
if (counter%2==0)
{
x=x+y;
}
* Change arithmetic operations.
Before Mutation:

if (counter%2==0)
{
x=x+y;
}
After Mutation:

if (counter%2==0)
{
x=x-y;
}

===Other Enhancements===
* Randomized the test order to increase the performance of testing.
* Introduced logging instead of print statements.
* Added Retry Mechanism when mutation cannot be performed on a file by a strategy.

===Excecution Flow===

[[File:Mutation_testing.jpg]]

==Design Plan==

===Strategy Design Pattern===
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing_(1).jpg]]

===Code snippet===

class Mutator:
def __init__(self, strategy):
self._strategy = strategy
def mutate(self, file_name):
return self._strategy.mutate_random_line(file_name)

class Strategy:
def __init__(self):
self._replace_strategy = {}

def mutate_random_line(self, file_name):
line_numbers = []
for line in fileinput.input(file_name):
if re.search(self._replace_strategy['regex'], line):
line_numbers.append(fileinput.lineno())
if len(line_numbers) == 0:
return -1
else:
mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
for line in fileinput.input(file_name, inplace=True):
if fileinput.lineno() == mutation_line_number:
line = re.sub(self._replace_strategy['regex'], self._replace_strategy['replaceString'], line)
print line.rstrip()
return mutation_line_number

class AndOr(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'\s&&\s',
'replaceString': ' || '
}

class IfTrue(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'true'
}

class IfFalse(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'false'
}

==Test Plan==

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-12-07T02:06:26Z

Sdhegde: /* Mutation Strategy Examples */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to set up the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:
Before Mutation:
if a && b {
c = 1;
} else {
c = 0;
}
After Mutation:
if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Change String literals
* Change Comparision operators
* Duplicate statements
* Change arithmetic operations.

As the build and WPT test run take a lot of time to run. Mutation Test Strategies are implemented with these key features
* The mutants produces only run-time bugs.
* The mutants do not cause any compilation failure.
* The mutants do not produce False Positives.

===Mutation Strategy Examples===
* The example for this strategy has already been mentioned in the [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2017/M1754_Mutation_Testing#Existing_Mutation_Strategy Existing Mutation Strategy] section.
* Make 'if condition' to 'if true'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if true
{
c = 1;
}
* Make 'if condition' to 'if false'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if false
{
c = 1;
}
* Change Comparision operators
Example:
Before Mutation:

if a >= 1
{
c = a * 10;
}

After Mutation:

if a > 1
{
c = a * 10;
}

*Change String literals
Example:
Before Mutation:

if a > 0
{
c = "a is possitive";
}

After Mutation:

if a > 0
{
c = " ";
}

* Duplicate statements
Before Mutation:
for (i=0;i<5;i++)
{
counter++;
}
if (counter%2==0)
{
x=x+y;
}
After Mutation:

for (i=0;i<5;i++)
{
counter++;
counter++;
}
if (counter%2==0)
{
x=x+y;
}
* Change arithmetic operations.
Before Mutation:

if (counter%2==0)
{
x=x+y;
}
After Mutation:

if (counter%2==0)
{
x=x-y;
}

===Other Enhancements===
* Randomized the test order to increase the performance of testing.
* Introduced logging instead of print statements.
* Added Retry Mechanism when mutation cannot be performed on a file by a strategy.

===Excecution Flow===

[[File:Mutation_testing.jpg]]

==Design Plan==

===Strategy Design Pattern===
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing_(1).jpg]]

===Code snippet===

class Mutator:
def __init__(self, strategy):
self._strategy = strategy
def mutate(self, file_name):
return self._strategy.mutate_random_line(file_name)

class Strategy:
def __init__(self):
self._replace_strategy = {}

def mutate_random_line(self, file_name):
line_numbers = []
for line in fileinput.input(file_name):
if re.search(self._replace_strategy['regex'], line):
line_numbers.append(fileinput.lineno())
if len(line_numbers) == 0:
return -1
else:
mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
for line in fileinput.input(file_name, inplace=True):
if fileinput.lineno() == mutation_line_number:
line = re.sub(self._replace_strategy['regex'], self._replace_strategy['replaceString'], line)
print line.rstrip()
return mutation_line_number

class AndOr(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'\s&&\s',
'replaceString': ' || '
}

class IfTrue(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'true'
}

class IfFalse(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'false'
}

==Test Plan==

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-12-07T02:04:34Z

Sdhegde: /* Mutation Strategy Examples */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to set up the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:
Before Mutation:
if a && b {
c = 1;
} else {
c = 0;
}
After Mutation:
if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Change String literals
* Change Comparision operators
* Duplicate statements
* Change arithmetic operations.

As the build and WPT test run take a lot of time to run. Mutation Test Strategies are implemented with these key features
* The mutants produces only run-time bugs.
* The mutants do not cause any compilation failure.
* The mutants do not produce False Positives.

===Mutation Strategy Examples===
* The example for this strategy has already been mentioned in the [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2017/M1754_Mutation_Testing#Existing_Mutation_Strategy Existing Mutation Strategy] section.
* Make 'if condition' to 'if true'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if true
{
c = 1;
}
* Make 'if condition' to 'if false'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if false
{
c = 1;
}
* Change Comparision operators
Example:
Before Mutation:

if a >= 1
{
c = a * 10;
}

After Mutation:

if a > 1
{
c = a * 10;
}

* Duplicate statements
Before Mutation:
for (i=0;i<5;i++)
{
counter++;
}
if (counter%2==0)
{
x=x+y;
}
After Mutation:

for (i=0;i<5;i++)
{
counter++;
counter++;
}
if (counter%2==0)
{
x=x+y;
}
* Change arithmetic operations.
Before Mutation:

if (counter%2==0)
{
x=x+y;
}
After Mutation:

if (counter%2==0)
{
x=x-y;
}

===Other Enhancements===
* Randomized the test order to increase the performance of testing.
* Introduced logging instead of print statements.
* Added Retry Mechanism when mutation cannot be performed on a file by a strategy.

===Excecution Flow===

[[File:Mutation_testing.jpg]]

==Design Plan==

===Strategy Design Pattern===
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing_(1).jpg]]

===Code snippet===

class Mutator:
def __init__(self, strategy):
self._strategy = strategy
def mutate(self, file_name):
return self._strategy.mutate_random_line(file_name)

class Strategy:
def __init__(self):
self._replace_strategy = {}

def mutate_random_line(self, file_name):
line_numbers = []
for line in fileinput.input(file_name):
if re.search(self._replace_strategy['regex'], line):
line_numbers.append(fileinput.lineno())
if len(line_numbers) == 0:
return -1
else:
mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
for line in fileinput.input(file_name, inplace=True):
if fileinput.lineno() == mutation_line_number:
line = re.sub(self._replace_strategy['regex'], self._replace_strategy['replaceString'], line)
print line.rstrip()
return mutation_line_number

class AndOr(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'\s&&\s',
'replaceString': ' || '
}

class IfTrue(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'true'
}

class IfFalse(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'false'
}

==Test Plan==

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-12-07T02:02:43Z

Sdhegde: /* Mutation Strategies */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to set up the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:
Before Mutation:
if a && b {
c = 1;
} else {
c = 0;
}
After Mutation:
if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Change String literals
* Change Comparision operators
* Duplicate statements
* Change arithmetic operations.

As the build and WPT test run take a lot of time to run. Mutation Test Strategies are implemented with these key features
* The mutants produces only run-time bugs.
* The mutants do not cause any compilation failure.
* The mutants do not produce False Positives.

===Mutation Strategy Examples===
* The example for this strategy has already been mentioned in the [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2017/M1754_Mutation_Testing#Existing_Mutation_Strategy Existing Mutation Strategy] section.
* Make 'if condition' to 'if true'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if true
{
c = 1;
}
* Make 'if condition' to 'if false'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if false
{
c = 1;
}

* Duplicate statements
Before Mutation:
for (i=0;i<5;i++)
{
counter++;
}
if (counter%2==0)
{
x=x+y;
}
After Mutation:

for (i=0;i<5;i++)
{
counter++;
counter++;
}
if (counter%2==0)
{
x=x+y;
}
* Change arithmetic operations.
Before Mutation:

if (counter%2==0)
{
x=x+y;
}
After Mutation:

if (counter%2==0)
{
x=x-y;
}

===Other Enhancements===
* Randomized the test order to increase the performance of testing.
* Introduced logging instead of print statements.
* Added Retry Mechanism when mutation cannot be performed on a file by a strategy.

===Excecution Flow===

[[File:Mutation_testing.jpg]]

==Design Plan==

===Strategy Design Pattern===
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing_(1).jpg]]

===Code snippet===

class Mutator:
def __init__(self, strategy):
self._strategy = strategy
def mutate(self, file_name):
return self._strategy.mutate_random_line(file_name)

class Strategy:
def __init__(self):
self._replace_strategy = {}

def mutate_random_line(self, file_name):
line_numbers = []
for line in fileinput.input(file_name):
if re.search(self._replace_strategy['regex'], line):
line_numbers.append(fileinput.lineno())
if len(line_numbers) == 0:
return -1
else:
mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
for line in fileinput.input(file_name, inplace=True):
if fileinput.lineno() == mutation_line_number:
line = re.sub(self._replace_strategy['regex'], self._replace_strategy['replaceString'], line)
print line.rstrip()
return mutation_line_number

class AndOr(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'\s&&\s',
'replaceString': ' || '
}

class IfTrue(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'true'
}

class IfFalse(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'false'
}

==Test Plan==

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-12-07T02:02:28Z

Sdhegde: /* Mutation Strategies */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to set up the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:
Before Mutation:
if a && b {
c = 1;
} else {
c = 0;
}
After Mutation:
if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Change String literals
* Change Comparision operators
* Duplicate statements
* Change arithmetic operations.

As the build and WPT test run take a lot of time to run. Mutation Test Strategies are implemented with these key features
* The mutants produces only run-time bugs.
* The mutants do not cause any compilation failure.
* The mutants do not produce False Positives.

===Mutation Strategy Examples===
* The example for this strategy has already been mentioned in the [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2017/M1754_Mutation_Testing#Existing_Mutation_Strategy Existing Mutation Strategy] section.
* Make 'if condition' to 'if true'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if true
{
c = 1;
}
* Make 'if condition' to 'if false'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if false
{
c = 1;
}

* Duplicate statements
Before Mutation:
for (i=0;i<5;i++)
{
counter++;
}
if (counter%2==0)
{
x=x+y;
}
After Mutation:

for (i=0;i<5;i++)
{
counter++;
counter++;
}
if (counter%2==0)
{
x=x+y;
}
* Change arithmetic operations.
Before Mutation:

if (counter%2==0)
{
x=x+y;
}
After Mutation:

if (counter%2==0)
{
x=x-y;
}

===Other Enhancements===
* Randomized the test order to increase the performance of testing.
* Introduced logging instead of print statements.
* Added Retry Mechanism when mutation cannot be performed on a file by a strategy.

===Excecution Flow===

[[File:Mutation_testing.jpg]]

==Design Plan==

===Strategy Design Pattern===
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing_(1).jpg]]

===Code snippet===

class Mutator:
def __init__(self, strategy):
self._strategy = strategy
def mutate(self, file_name):
return self._strategy.mutate_random_line(file_name)

class Strategy:
def __init__(self):
self._replace_strategy = {}

def mutate_random_line(self, file_name):
line_numbers = []
for line in fileinput.input(file_name):
if re.search(self._replace_strategy['regex'], line):
line_numbers.append(fileinput.lineno())
if len(line_numbers) == 0:
return -1
else:
mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
for line in fileinput.input(file_name, inplace=True):
if fileinput.lineno() == mutation_line_number:
line = re.sub(self._replace_strategy['regex'], self._replace_strategy['replaceString'], line)
print line.rstrip()
return mutation_line_number

class AndOr(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'\s&&\s',
'replaceString': ' || '
}

class IfTrue(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'true'
}

class IfFalse(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'false'
}

==Test Plan==

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-12-07T02:01:54Z

Sdhegde: /* Mutation Strategies */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to set up the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:
Before Mutation:
if a && b {
c = 1;
} else {
c = 0;
}
After Mutation:
if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Change String literals
* Change Comparision operators
* Duplicate statements
* Change arithmetic operations.

As the build and WPT test run take a lot of time to run. Mutation Test Strategies are implemented with these key features
* The mutants produces only run-time bugs.
* The mutants do not cause any compilation failure.
* The mutants do not produce False Positives.

===Mutation Strategy Examples===
* The example for this strategy has already been mentioned in the [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2017/M1754_Mutation_Testing#Existing_Mutation_Strategy Existing Mutation Strategy] section.
* Make 'if condition' to 'if true'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if true
{
c = 1;
}
* Make 'if condition' to 'if false'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if false
{
c = 1;
}

* Duplicate statements
Before Mutation:
for (i=0;i<5;i++)
{
counter++;
}
if (counter%2==0)
{
x=x+y;
}
After Mutation:

for (i=0;i<5;i++)
{
counter++;
counter++;
}
if (counter%2==0)
{
x=x+y;
}
* Change arithmetic operations.
Before Mutation:

if (counter%2==0)
{
x=x+y;
}
After Mutation:

if (counter%2==0)
{
x=x-y;
}

===Other Enhancements===
* Randomized the test order to increase the performance of testing.
* Introduced logging instead of print statements.
* Added Retry Mechanism when mutation cannot be performed on a file by a strategy.

===Excecution Flow===

[[File:Mutation_testing.jpg]]

==Design Plan==

===Strategy Design Pattern===
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing_(1).jpg]]

===Code snippet===

class Mutator:
def __init__(self, strategy):
self._strategy = strategy
def mutate(self, file_name):
return self._strategy.mutate_random_line(file_name)

class Strategy:
def __init__(self):
self._replace_strategy = {}

def mutate_random_line(self, file_name):
line_numbers = []
for line in fileinput.input(file_name):
if re.search(self._replace_strategy['regex'], line):
line_numbers.append(fileinput.lineno())
if len(line_numbers) == 0:
return -1
else:
mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
for line in fileinput.input(file_name, inplace=True):
if fileinput.lineno() == mutation_line_number:
line = re.sub(self._replace_strategy['regex'], self._replace_strategy['replaceString'], line)
print line.rstrip()
return mutation_line_number

class AndOr(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'\s&&\s',
'replaceString': ' || '
}

class IfTrue(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'true'
}

class IfFalse(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'false'
}

==Test Plan==

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-12-07T01:55:22Z

Sdhegde: /* Project description */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to set up the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:
Before Mutation:
if a && b {
c = 1;
} else {
c = 0;
}
After Mutation:
if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Change String literals
* Change Comparision operators
* Duplicate statements
* Change arithmetic operations.

===Mutation Strategy Examples===
* The example for this strategy has already been mentioned in the [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2017/M1754_Mutation_Testing#Existing_Mutation_Strategy Existing Mutation Strategy] section.
* Make 'if condition' to 'if true'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if true
{
c = 1;
}
* Make 'if condition' to 'if false'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if false
{
c = 1;
}

* Duplicate statements
Before Mutation:
for (i=0;i<5;i++)
{
counter++;
}
if (counter%2==0)
{
x=x+y;
}
After Mutation:

for (i=0;i<5;i++)
{
counter++;
counter++;
}
if (counter%2==0)
{
x=x+y;
}
* Change arithmetic operations.
Before Mutation:

if (counter%2==0)
{
x=x+y;
}
After Mutation:

if (counter%2==0)
{
x=x-y;
}

===Other Enhancements===
* Randomized the test order to increase the performance of testing.
* Introduced logging instead of print statements.
* Added Retry Mechanism when mutation cannot be performed on a file by a strategy.

===Excecution Flow===

[[File:Mutation_testing.jpg]]

==Design Plan==

===Strategy Design Pattern===
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing_(1).jpg]]

===Code snippet===

class Mutator:
def __init__(self, strategy):
self._strategy = strategy
def mutate(self, file_name):
return self._strategy.mutate_random_line(file_name)

class Strategy:
def __init__(self):
self._replace_strategy = {}

def mutate_random_line(self, file_name):
line_numbers = []
for line in fileinput.input(file_name):
if re.search(self._replace_strategy['regex'], line):
line_numbers.append(fileinput.lineno())
if len(line_numbers) == 0:
return -1
else:
mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
for line in fileinput.input(file_name, inplace=True):
if fileinput.lineno() == mutation_line_number:
line = re.sub(self._replace_strategy['regex'], self._replace_strategy['replaceString'], line)
print line.rstrip()
return mutation_line_number

class AndOr(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'\s&&\s',
'replaceString': ' || '
}

class IfTrue(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'true'
}

class IfFalse(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'false'
}

==Test Plan==

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-12-07T01:40:46Z

Sdhegde: /* Introduction */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to set up the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:
Before Mutation:
if a && b {
c = 1;
} else {
c = 0;
}
After Mutation:
if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Change String literals
* Change Comparision operators
* Duplicate statements
* Change arithmetic operations.

===Mutation Strategy Examples===
* The example for this strategy has already been mentioned in the [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2017/M1754_Mutation_Testing#Existing_Mutation_Strategy Existing Mutation Strategy] section.
* Make 'if condition' to 'if true'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if true
{
c = 1;
}
* Make 'if condition' to 'if false'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if false
{
c = 1;
}

* Duplicate statements
Before Mutation:
for (i=0;i<5;i++)
{
counter++;
}
if (counter%2==0)
{
x=x+y;
}
After Mutation:

for (i=0;i<5;i++)
{
counter++;
counter++;
}
if (counter%2==0)
{
x=x+y;
}
* Change arithmetic operations.
Before Mutation:

if (counter%2==0)
{
x=x+y;
}
After Mutation:

if (counter%2==0)
{
x=x-y;
}

===Excecution Flow===

[[File:Mutation_testing.jpg]]

==Design Plan==

===Strategy Design Pattern===
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing_(1).jpg]]

===Code snippet===

class Mutator:
def __init__(self, strategy):
self._strategy = strategy
def mutate(self, file_name):
return self._strategy.mutate_random_line(file_name)

class Strategy:
def __init__(self):
self._replace_strategy = {}

def mutate_random_line(self, file_name):
line_numbers = []
for line in fileinput.input(file_name):
if re.search(self._replace_strategy['regex'], line):
line_numbers.append(fileinput.lineno())
if len(line_numbers) == 0:
return -1
else:
mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
for line in fileinput.input(file_name, inplace=True):
if fileinput.lineno() == mutation_line_number:
line = re.sub(self._replace_strategy['regex'], self._replace_strategy['replaceString'], line)
print line.rstrip()
return mutation_line_number

class AndOr(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'\s&&\s',
'replaceString': ' || '
}

class IfTrue(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'true'
}

class IfFalse(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'false'
}

==Test Plan==

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-12-07T01:38:53Z

Sdhegde: /* Mutation Strategy Examples */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to set up the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:
Before Mutation:
if a && b {
c = 1;
} else {
c = 0;
}
After Mutation:
if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Change String literals
* Change Comparision operators
* Duplicate statements
* Change arithmetic operations.

===Mutation Strategy Examples===
* The example for this strategy has already been mentioned in the [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2017/M1754_Mutation_Testing#Existing_Mutation_Strategy Existing Mutation Strategy] section.
* Make 'if condition' to 'if true'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if true
{
c = 1;
}
* Make 'if condition' to 'if false'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if false
{
c = 1;
}

* Duplicate statements
Before Mutation:
for (i=0;i<5;i++)
{
counter++;
}
if (counter%2==0)
{
x=x+y;
}
After Mutation:

for (i=0;i<5;i++)
{
counter++;
counter++;
}
if (counter%2==0)
{
x=x+y;
}
* Change arithmetic operations.
Before Mutation:

if (counter%2==0)
{
x=x+y;
}
After Mutation:

if (counter%2==0)
{
x=x-y;
}

===Excecution Flow===

[[File:Mutation_testing.jpg]]

==Design Plan==

===Strategy Design Pattern===
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing_(1).jpg]]

===Code snippet===

class Mutator:
def __init__(self, strategy):
self._strategy = strategy
def mutate(self, file_name):
return self._strategy.mutate_random_line(file_name)

class Strategy:
def __init__(self):
self._replace_strategy = {}

def mutate_random_line(self, file_name):
line_numbers = []
for line in fileinput.input(file_name):
if re.search(self._replace_strategy['regex'], line):
line_numbers.append(fileinput.lineno())
if len(line_numbers) == 0:
return -1
else:
mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
for line in fileinput.input(file_name, inplace=True):
if fileinput.lineno() == mutation_line_number:
line = re.sub(self._replace_strategy['regex'], self._replace_strategy['replaceString'], line)
print line.rstrip()
return mutation_line_number

class AndOr(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'\s&&\s',
'replaceString': ' || '
}

class IfTrue(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'true'
}

class IfFalse(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'false'
}

==Test Plan==

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-12-07T01:38:28Z

Sdhegde: /* Mutation Strategies */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to set up the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:
Before Mutation:
if a && b {
c = 1;
} else {
c = 0;
}
After Mutation:
if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Change String literals
* Change Comparision operators
* Duplicate statements
* Change arithmetic operations.

===Mutation Strategy Examples===
* The example for this strategy has already been mentioned in the [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2017/M1754_Mutation_Testing#Existing_Mutation_Strategy Existing Mutation Strategy] section.
* Make 'if condition' to 'if true'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if true
{
c = 1;
}
* Make 'if condition' to 'if false'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if false
{
c = 1;
}
* Re-order statements
Example:
Before Mutation:

int a=6, b=11;
float c,d;
c= a/b;
d= c/a;

After Mutation:
int a=6, b=11;
float c,d;
d= c/a;
c= a/b;

* Duplicate statements
Before Mutation:
for (i=0;i<5;i++)
{
counter++;
}
if (counter%2==0)
{
x=x+y;
}
After Mutation:

for (i=0;i<5;i++)
{
counter++;
counter++;
}
if (counter%2==0)
{
x=x+y;
}
* Change arithmetic operations.
Before Mutation:

if (counter%2==0)
{
x=x+y;
}
After Mutation:

if (counter%2==0)
{
x=x-y;
}

===Excecution Flow===

[[File:Mutation_testing.jpg]]

==Design Plan==

===Strategy Design Pattern===
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing_(1).jpg]]

===Code snippet===

class Mutator:
def __init__(self, strategy):
self._strategy = strategy
def mutate(self, file_name):
return self._strategy.mutate_random_line(file_name)

class Strategy:
def __init__(self):
self._replace_strategy = {}

def mutate_random_line(self, file_name):
line_numbers = []
for line in fileinput.input(file_name):
if re.search(self._replace_strategy['regex'], line):
line_numbers.append(fileinput.lineno())
if len(line_numbers) == 0:
return -1
else:
mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
for line in fileinput.input(file_name, inplace=True):
if fileinput.lineno() == mutation_line_number:
line = re.sub(self._replace_strategy['regex'], self._replace_strategy['replaceString'], line)
print line.rstrip()
return mutation_line_number

class AndOr(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'\s&&\s',
'replaceString': ' || '
}

class IfTrue(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'true'
}

class IfFalse(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'false'
}

==Test Plan==

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-12-07T01:29:10Z

Sdhegde: /* Excecution Flow */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to set up the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:
Before Mutation:
if a && b {
c = 1;
} else {
c = 0;
}
After Mutation:
if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Re-order statements
* Duplicate statements
* Change arithmetic operations.

===Mutation Strategy Examples===
* The example for this strategy has already been mentioned in the [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2017/M1754_Mutation_Testing#Existing_Mutation_Strategy Existing Mutation Strategy] section.
* Make 'if condition' to 'if true'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if true
{
c = 1;
}
* Make 'if condition' to 'if false'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if false
{
c = 1;
}
* Re-order statements
Example:
Before Mutation:

int a=6, b=11;
float c,d;
c= a/b;
d= c/a;

After Mutation:
int a=6, b=11;
float c,d;
d= c/a;
c= a/b;

* Duplicate statements
Before Mutation:
for (i=0;i<5;i++)
{
counter++;
}
if (counter%2==0)
{
x=x+y;
}
After Mutation:

for (i=0;i<5;i++)
{
counter++;
counter++;
}
if (counter%2==0)
{
x=x+y;
}
* Change arithmetic operations.
Before Mutation:

if (counter%2==0)
{
x=x+y;
}
After Mutation:

if (counter%2==0)
{
x=x-y;
}

===Excecution Flow===

[[File:Mutation_testing.jpg]]

==Design Plan==

===Strategy Design Pattern===
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing_(1).jpg]]

===Code snippet===

class Mutator:
def __init__(self, strategy):
self._strategy = strategy
def mutate(self, file_name):
return self._strategy.mutate_random_line(file_name)

class Strategy:
def __init__(self):
self._replace_strategy = {}

def mutate_random_line(self, file_name):
line_numbers = []
for line in fileinput.input(file_name):
if re.search(self._replace_strategy['regex'], line):
line_numbers.append(fileinput.lineno())
if len(line_numbers) == 0:
return -1
else:
mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
for line in fileinput.input(file_name, inplace=True):
if fileinput.lineno() == mutation_line_number:
line = re.sub(self._replace_strategy['regex'], self._replace_strategy['replaceString'], line)
print line.rstrip()
return mutation_line_number

class AndOr(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'\s&&\s',
'replaceString': ' || '
}

class IfTrue(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'true'
}

class IfFalse(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'false'
}

==Test Plan==

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

File:Mutation testing.jpg

2017-12-07T01:28:58Z

Sdhegde: Control flow diagram - mutation testing

Control flow diagram - mutation testing

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-12-07T01:27:07Z

Sdhegde: /* Strategy Design Pattern */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to set up the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:
Before Mutation:
if a && b {
c = 1;
} else {
c = 0;
}
After Mutation:
if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Re-order statements
* Duplicate statements
* Change arithmetic operations.

===Mutation Strategy Examples===
* The example for this strategy has already been mentioned in the [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2017/M1754_Mutation_Testing#Existing_Mutation_Strategy Existing Mutation Strategy] section.
* Make 'if condition' to 'if true'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if true
{
c = 1;
}
* Make 'if condition' to 'if false'
Example:
Before Mutation:

if a == 0
{
c = 1;
}

After Mutation:

if false
{
c = 1;
}
* Re-order statements
Example:
Before Mutation:

int a=6, b=11;
float c,d;
c= a/b;
d= c/a;

After Mutation:
int a=6, b=11;
float c,d;
d= c/a;
c= a/b;

* Duplicate statements
Before Mutation:
for (i=0;i<5;i++)
{
counter++;
}
if (counter%2==0)
{
x=x+y;
}
After Mutation:

for (i=0;i<5;i++)
{
counter++;
counter++;
}
if (counter%2==0)
{
x=x+y;
}
* Change arithmetic operations.
Before Mutation:

if (counter%2==0)
{
x=x+y;
}
After Mutation:

if (counter%2==0)
{
x=x-y;
}

===Excecution Flow===

[[File:Oodd servo.png]]

==Design Plan==

===Strategy Design Pattern===
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing_(1).jpg]]

===Code snippet===

class Mutator:
def __init__(self, strategy):
self._strategy = strategy
def mutate(self, file_name):
return self._strategy.mutate_random_line(file_name)

class Strategy:
def __init__(self):
self._replace_strategy = {}

def mutate_random_line(self, file_name):
line_numbers = []
for line in fileinput.input(file_name):
if re.search(self._replace_strategy['regex'], line):
line_numbers.append(fileinput.lineno())
if len(line_numbers) == 0:
return -1
else:
mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
for line in fileinput.input(file_name, inplace=True):
if fileinput.lineno() == mutation_line_number:
line = re.sub(self._replace_strategy['regex'], self._replace_strategy['replaceString'], line)
print line.rstrip()
return mutation_line_number

class AndOr(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'\s&&\s',
'replaceString': ' || '
}

class IfTrue(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'true'
}

class IfFalse(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'false'
}

==Test Plan==

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

File:UML Mutation Testing (1).jpg

2017-12-07T01:26:12Z

Sdhegde: UML diagram of strategy pattern

UML diagram of strategy pattern

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-14T03:03:13Z

Sdhegde: /* Design Plan */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:

if a && b {
c = 1;
} else {
c = 0;
}

if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Re-order statements
* Duplicate statements
* Change arithmetic operations.

===Excecution Flow===

[[File:Oodd servo.png]]

==Design Plan==

===Strategy Design Pattern===
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing.jpg]]

===Code snippet===

class Mutator:
def __init__(self, strategy):
self._strategy = strategy
def mutate(self, file_name):
return self._strategy.mutate_random_line(file_name)

class Strategy:
def __init__(self):
self._replace_strategy = {}

def mutate_random_line(self, file_name):
line_numbers = []
for line in fileinput.input(file_name):
if re.search(self._replace_strategy['regex'], line):
line_numbers.append(fileinput.lineno())
if len(line_numbers) == 0:
return -1
else:
mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
for line in fileinput.input(file_name, inplace=True):
if fileinput.lineno() == mutation_line_number:
line = re.sub(self._replace_strategy['regex'], self._replace_strategy['replaceString'], line)
print line.rstrip()
return mutation_line_number

class AndOr(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'\s&&\s',
'replaceString': ' || '
}

class IfTrue(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'true'
}

class IfFalse(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'false'
}

==Test Plan==

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-14T03:02:51Z

Sdhegde: /* Design Plan */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:

if a && b {
c = 1;
} else {
c = 0;
}

if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Re-order statements
* Duplicate statements
* Change arithmetic operations.

===Excecution Flow===

[[File:Oodd servo.png]]

==Design Plan==

===Strategy Design Pattern===
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing.jpg]]

Code snippet:

class Mutator:
def __init__(self, strategy):
self._strategy = strategy
def mutate(self, file_name):
return self._strategy.mutate_random_line(file_name)

class Strategy:
def __init__(self):
self._replace_strategy = {}

def mutate_random_line(self, file_name):
line_numbers = []
for line in fileinput.input(file_name):
if re.search(self._replace_strategy['regex'], line):
line_numbers.append(fileinput.lineno())
if len(line_numbers) == 0:
return -1
else:
mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
for line in fileinput.input(file_name, inplace=True):
if fileinput.lineno() == mutation_line_number:
line = re.sub(self._replace_strategy['regex'], self._replace_strategy['replaceString'], line)
print line.rstrip()
return mutation_line_number

class AndOr(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'\s&&\s',
'replaceString': ' || '
}

class IfTrue(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'true'
}

class IfFalse(Strategy):
def __init__(self):
Strategy.__init__(self)
self._replace_strategy = {
'regex': r'(?<=if\s)(.*)(?=\s\{)',
'replaceString': 'false'
}

==Test Plan==

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-10T20:07:19Z

Sdhegde:

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:

if a && b {
c = 1;
} else {
c = 0;
}

if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Re-order statements
* Duplicate statements
* Change arithmetic operations.

===Excecution Flow===

[[File:Oodd servo.png]]

==Design Plan==

===Strategy Design Pattern===
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing.jpg]]

==Test Plan==

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-10T20:07:04Z

Sdhegde:

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:

if a && b {
c = 1;
} else {
c = 0;
}

if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Re-order statements
* Duplicate statements
* Change arithmetic operations.

===Excecution Flow===

[[File:Oodd servo.png]]

==Design Plan==

===Strategy Design Pattern===
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing.jpg]]

==Testing==

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-09T17:48:11Z

Sdhegde: /* Mutation Strategies */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:

if a && b {
c = 1;
} else {
c = 0;
}

if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like the existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Re-order statements
* Duplicate statements
* Change arithmetic operations.

===Excecution Flow===

[[File:Oodd servo.png]]

===Design Pattern===

====Strategy Design Pattern====
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing.jpg]]

===Testing===

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-09T17:47:50Z

Sdhegde:

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:

if a && b {
c = 1;
} else {
c = 0;
}

if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Mutation Strategies===
There will be many strategies to mutate a file, like existing one. And the strategies will be picked randomly during the execution of the program.
Some of the strategies are:
* Replace && to ||
* Make 'if condition' to 'if true'
* Make 'if condition' to 'if false'
* Re-order statements
* Duplicate statements
* Change arithmetic operations.

===Excecution Flow===

[[File:Oodd servo.png]]

===Design Pattern===

====Strategy Design Pattern====
There are multiple strategies for mutating a file. Each strategy has the implementation of its strategy in the mutate_line method. The Mutator picks up random strategy during the execution of the program and mutates the file.

[[File:UML_Mutation_Testing.jpg]]

===Testing===

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-09T17:41:52Z

Sdhegde: /* Existing Excecution Flow */

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-09T17:40:21Z

Sdhegde: /* Design Pattern */

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-09T17:40:13Z

Sdhegde: /* Design Pattern */

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-09T17:39:52Z

Sdhegde: /* Design Pattern */

File:UML Mutation Testing.jpg

2017-11-09T17:35:38Z

Sdhegde:

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-09T17:33:09Z

Sdhegde: /* Design Pattern */

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-08T05:19:02Z

Sdhegde: Undo revision 112685 by Sdhegde (talk)

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-08T05:18:42Z

Sdhegde: Undo revision 112686 by Sdhegde (talk)

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-08T05:17:57Z

Sdhegde:

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-08T05:16:50Z

Sdhegde: /* References */

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-07T02:43:07Z

Sdhegde: /* Existing Mutation Strategy */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example:

if a && b {
c = 1;
} else {
c = 0;
}

if a || b {
c = 1;
} else {
c = 0;
}

This will produce a bug. When a is true and b is false also when a is false and b is true the value of c will be wrong. If our WPT test catches this bug then we can say mutant is killed. Otherwise mutant has lived.

===Existing Excecution Flow===

Given a path
:mutation test(path)
::Look for test mapping file in that path
::for each file the in the test mapping
:::skip this file if there are local uncommitted changes
:::mutate a random line using some strategy in the file
:::build project and run WPT tests corresponding to it
:::if wpt tests pass?
::::report mautation test failure
:::else
::::report mutation test success
for each subfolder folder in the path
:mutation test(subfolder)

===Design Pattern===

Design patterns are not applicable as our task is involved in implementing the script to modify files and invoking tests.

===Testing===

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-07T02:38:49Z

Sdhegde: /* Existing Mutation Strategy */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

Example

if a && b {
c = 1;
} else {
c = 0;
}

if a || b {
c = 1;
} else {
c = 0;
}

===Existing Excecution Flow===

Given a path
:mutation test(path)
::Look for test mapping file in that path
::for each file the in the test mapping
:::skip this file if there are local uncommitted changes
:::mutate a random line using some strategy in the file
:::build project and run WPT tests corresponding to it
:::if wpt tests pass?
::::report mautation test failure
:::else
::::report mutation test success
for each subfolder folder in the path
:mutation test(subfolder)

===Design Pattern===

Design patterns are not applicable as our task is involved in implementing the script to modify files and invoking tests.

===Testing===

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-07T02:38:15Z

Sdhegde: /* Existing Mutation Strategy */

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-07T02:33:16Z

Sdhegde: /* Existing Mutation Strategy */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===

*Implemented a simple mutator as a Python script that finds random uses of && in Servo's code base and replaces them by ||.
*Built the mutated implementation of Servo with ./mach build -r and run it on the WPT test suite with ./mach test-wpt: This produced test failures.
*Introduced test mapping framework to map source files to WPT test.
*Optimized the number of test runs by using test mapping to run only relevant tests for a mutant.
*Automated this process by writing scripts in a new python/servo/mutation directory, and called them from scripts in /etc/ci.
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Existing Mutation Strategy===

* Replace random occurrence of && with ||

`if n < 0 {
print!("{} is negative", n);
} else if n > 0 {
print!("{} is positive", n);
} else {
print!("{} is zero", n);
}`

===Existing Excecution Flow===

Given a path
:mutation test(path)
::Look for test mapping file in that path
::for each file the in the test mapping
:::skip this file if there are local uncommitted changes
:::mutate a random line using some strategy in the file
:::build project and run WPT tests corresponding to it
:::if wpt tests pass?
::::report mautation test failure
:::else
::::report mutation test success
for each subfolder folder in the path
:mutation test(subfolder)

===Design Pattern===

Design patterns are not applicable as our task is involved in implementing the script to modify files and invoking tests.

===Testing===

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-07T02:31:30Z

Sdhegde:

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-07T01:47:20Z

Sdhegde:

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-07T01:36:40Z

Sdhegde: /* Excecution Flow */

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-07T01:35:20Z

Sdhegde: /* Initial steps */

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-07T01:34:58Z

Sdhegde:

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-07T01:30:43Z

Sdhegde: /* Initial steps */

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-07T01:27:57Z

Sdhegde: /* Initial steps */

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-07T01:21:26Z

Sdhegde: /* Subsequent steps */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===
The approach adopted was based on the requirements mentioned [https://github.com/servo/servo/wiki/Mutation-testing-project here]. The below steps implement the initial steps mentioned in the project description.

* Step 1: A python script was written to mutate one source file and the corresponding WPT was run on it to check if the mutant was killed.

* Step 2: A test framework was defined to identify the source files that require mutation testing along with their corresponding WPTs. This framework is implemented in /servo/components/script/dom.

* Step 3: The script was expanded to include the test framework and automate the process of generating mutations for multiple source files and running their corresponding WPTs based on the test_mapping.json. The script also traverses through subfolders of the parsed path to check for the .json file. The script also logs on the terminal any mutants that the WPT failed to kill.

* Step 4: Integrated the script so that it can be invoked from the CI tool.

===Subsequent steps===

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Execution===
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Excecution Flow===

Given a path
:mutation test(path)
::Look for test mapping file in that path
::for each file the in the test mapping
:::mutate a random line using some strategy in the file
:::build project and run WPT tests corresponding to it
:::if wpt tests pass?
::::report mautation test failure
:::else
::::report mutation test success
for each subfolder folder in the path
:mutation test(subfolder)

===Testing===

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-07T01:20:52Z

Sdhegde: /* Implemented steps */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Initial steps===
The approach adopted was based on the requirements mentioned [https://github.com/servo/servo/wiki/Mutation-testing-project here]. The below steps implement the initial steps mentioned in the project description.

* Step 1: A python script was written to mutate one source file and the corresponding WPT was run on it to check if the mutant was killed.

* Step 2: A test framework was defined to identify the source files that require mutation testing along with their corresponding WPTs. This framework is implemented in /servo/components/script/dom.

* Step 3: The script was expanded to include the test framework and automate the process of generating mutations for multiple source files and running their corresponding WPTs based on the test_mapping.json. The script also traverses through subfolders of the parsed path to check for the .json file. The script also logs on the terminal any mutants that the WPT failed to kill.

* Step 4: Integrated the script so that it can be invoked from the CI tool.

===Subsequent steps===

The following subsequent steps will be followed to meet the project requirements as per this [https://github.com/servo/servo/wiki/Mutation-testing-project].

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Execution===
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Excecution Flow===

Given a path
:mutation test(path)
::Look for test mapping file in that path
::for each file the in the test mapping
:::mutate a random line using some strategy in the file
:::build project and run WPT tests corresponding to it
:::if wpt tests pass?
::::report mautation test failure
:::else
::::report mutation test success
for each subfolder folder in the path
:mutation test(subfolder)

===Testing===

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-07T01:18:29Z

Sdhegde:

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Implemented steps===
The approach adopted was based on the requirements mentioned [https://github.com/servo/servo/wiki/Mutation-testing-project here]. The below steps implement the initial steps mentioned in the project description.

* Step 1: A python script was written to mutate one source file and the corresponding WPT was run on it to check if the mutant was killed.

* Step 2: A test framework was defined to identify the source files that require mutation testing along with their corresponding WPTs. This framework is implemented in /servo/components/script/dom.

* Step 3: The script was expanded to include the test framework and automate the process of generating mutations for multiple source files and running their corresponding WPTs based on the test_mapping.json. The script also traverses through subfolders of the parsed path to check for the .json file. The script also logs on the terminal any mutants that the WPT failed to kill.

* Step 4: Integrated the script so that it can be invoked from the CI tool.

===Subsequent steps===

The following subsequent steps will be followed to meet the project requirements as per this [https://github.com/servo/servo/wiki/Mutation-testing-project].

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Execution===
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Excecution Flow===

Given a path
:mutation test(path)
::Look for test mapping file in that path
::for each file the in the test mapping
:::mutate a random line using some strategy in the file
:::build project and run WPT tests corresponding to it
:::if wpt tests pass?
::::report mautation test failure
:::else
::::report mutation test success
for each subfolder folder in the path
:mutation test(subfolder)

===Testing===

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-07T01:13:48Z

Sdhegde:

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Implemented steps===
The approach adopted was based on the requirements mentioned [https://github.com/servo/servo/wiki/Mutation-testing-project here]. The below steps implement the initial steps mentioned in the project description.

* Step 1: A python script was written to mutate one source file and the corresponding WPT was run on it to check if the mutant was killed.

* Step 2: A test framework was defined to identify the source files that require mutation testing along with their corresponding WPTs. This framework is implemented in /servo/components/script/dom.

* Step 3: The script was expanded to include the test framework and automate the process of generating mutations for multiple source files and running their corresponding WPTs based on the test_mapping.json. The script also traverses through subfolders of the parsed path to check for the .json file. The script also logs on the terminal any mutants that the WPT failed to kill.

* Step 4: Integrated the script so that it can be invoked from the CI tool.

===Execution===
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Excecution Flow===

Given a path
:mutation test(path)
::Look for test mapping file in that path
::for each file the in the test mapping
:::mutate a random line using some strategy in the file
:::build project and run WPT tests corresponding to it
:::if wpt tests pass?
::::report mautation test failure
:::else
::::report mutation test success
for each subfolder folder in the path
:mutation test(subfolder)

===Testing===

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

===Subsequent steps===

The following subsequent steps will be followed to meet the project requirements as per this [https://github.com/servo/servo/wiki/Mutation-testing-project].

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Challenges===

1. Setting up the environment on the local machine:

*The amount of time taken to build and test the WPT on the local machine was pretty long due to machine memory. Additionally faced problems on intermittent WPT test failures. It is expected that any reviewer will face a similar challenge.

We overcame this challenge by discussing possibilities with Mozilla team who suggested us to use Janitor Technology. The servo image would be already set up with the latest code, we can edit code in cloud9 and execute it. Using this platform for our testing made the project agnostic of the environment on the local machine and any hassle associated with same.

2. Defining the test mapping framework:

*Servo consists of multiple components of which there are many WPT tests written for each component. The test cases are organized according to the functionality they test, whereas the servo source code is organized according to the specific components. Due to a large number of source files and WPT tests for the entire project, it was difficult for one individual to have information on the mapping of components to their corresponding WPTs.

Hence, we had to have a mapping to run the corresponding tests for a source file. The mapping framework enabled us to run tests on a few components and provide a base for other members from Mozilla team to add and replicate the same framework as per their requirement. We still need to find if there can be any automated way to populate the mapping.

3. The extent of mutation testing to be performed.

*There are standard techniques that define what a mutant is. For example, replacing '&&' with '||'. However, there is no specification for the number of replacements that must be made within a single source code, nor is there any specification on the permutations of possible mutations for each technique. This is an ambiguity in the project and our initial implementation replaces all instances of '&&' to '||' to generate a single type of mutant for each file of the source code. But now our implementation mutates random lines using a strategy. The code needs to evolve based on how runs/performs. And there is no predefined answer for it.

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017

2017-11-07T01:07:50Z

Sdhegde:

'''Design Project Documentation'''
*[[CSC/ECE 517 Fall 2017/E17A2 Lightweight Badging System]]
*[[CSC/ECE 517 Fall 2017/E17A8. Use a profiler to identify the problems / pages that take some time to load & fix them]]
*[[CSC/ECE 517 Fall 2017/E1793. Help students find teams to join]]
*[[CSC/ECE 517 Fall 2017/E17A4 Allow calibration to be part of an assignment]]
*[[CSC/ECE 517 Fall 2017/E17A0 Team-based reviewing]]
*[[CSC/ECE 517 Fall 2017/E1793. Help students find teams to join_Team1964]]
*[[CSC/ECE 517 Fall 2017/E1791. Track the time that students look at the other submissions - logging improvement]]
*[[CSC/ECE 517 Fall 2017/E1799 Improve self-review Link peer review & self-review to derive grades]]
*[[CSC/ECE_517_Fall_2017/E1797_Timestamps_for_students_submissions]]
*[[CSC/ECE_517_Fall_2017/E17A1|CSC/ECE_517_Fall_2017/E17A1 - Let experts as well as students do reviews]]
*[[CSC/ECE 517 Fall 2017/M1754_Mutation_Testing]]
'''Writing Assignment 2'''
*[[CSC/ECE 517 Fall 2017/E1779. Fix teammate advertisements and requests to join a team]]
*[[CSC/ECE_517_Fall_2017/E1773 Investigate and Fix Expertiza Production Version Runtime Exceptions.rb]]
*[[CSC/ECE_517_Fall_2017/E1774 Metareview fixes and improvements.rb]]
*[[CSC/ECE_517_Fall_2017/E1788 OSS project Maroon Heatmap fixes]]
*[[CSC/ECE 517 Fall 2017/E1781 Topic Management]]
*[[CSC/ECE_517_Fall_2017/E1745_Refactor_response_controller.rb]]
*[[CSC/ECE_517_Fall_2017/E1752 Refactor assignments controller]]
*[[CSC/ECE_517_Fall_2017/E1776_Enhance_Imports]]
*[[CSC/ECE_517_Fall_2017/E1756 TLD Refactor response.rb]]
*[[CSC/ECE_517_Fall_2017/E1772_Refactor reputation_web_service_controller.rb]]
*[[CSC/ECE_517_Fall_2017/M1754_Mutation Testing on Servo]]
*[[CSC/ECE_517_Fall_2017/E1753 OSS project bidding tests]]
*[[CSC/ECE_517_Fall_2017/E1766_Test team functionality]]
*[[CSC/ECE 517 Fall 2017/E1767 Improve imports]]
*[[CSC/ECE_517_Fall_2017/E1786_OSS project Juniper Bookmark Enhancements]]
*[[CSC/ECE_517_Fall_2017/E1787_OSS project Bronze Score calculations]]
*[[CSC/ECE_517_Fall_2017/E1763 Fix Staggered-Deadline Assignments]]
*[[CSC/ECE 517 Fall 2017/E1757 Introduce a Student View for instructors]]
*[[CSC/ECE 517 Fall 2017/E1769_Refactor assignment_form.rb]]
*[[CSC/ECE 517 Fall 2017/E1754_Feature_test_of_rubric_advice]]
*[[CSC/ECE 517 Fall 2017/E1782 OSS Project Red Assignment Directories]]
*[[CSC/ECE 517 Fall 2017/E1748 Add past-due assignments to task list]]
*[[CSC/ECE 517 Fall 2017/E1761 Test First Development Refactor assignment.rb]]
*[[CSC/ECE_517_Fall_2017/E1780_OSS_Project_Teal_Email_Notification_Enhancements]]
*[[CSC/ECE_517_Fall_2017/E1762 Test various kinds of response-map hierarchies]]
*[[CSC/ECE_517_Fall_2017/E1759 ]]
*[[CSC/ECE_517_Fall_2017/E1749_Test First Development Refactor questionnaire_controller.rb]]
*[[CSC/ECE 517 Fall 2017/E1777 Coherent specification of review requirements.rb]]
*[[CSC/ECE 517 Fall 2017/E1771_Refactor_team.rb]]

CSC/ECE 517 Fall 2017/M1754 Mutation Testing

2017-11-07T01:07:06Z

Sdhegde: Created page with "Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutat..."

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Implemented steps===
The approach adopted was based on the requirements mentioned [https://github.com/servo/servo/wiki/Mutation-testing-project here]. The below steps implement the initial steps mentioned in the project description.

* Step 1: A python script was written to mutate one source file and the corresponding WPT was run on it to check if the mutant was killed.

* Step 2: A test framework was defined to identify the source files that require mutation testing along with their corresponding WPTs. This framework is implemented in /servo/components/script/dom.

* Step 3: The script was expanded to include the test framework and automate the process of generating mutations for multiple source files and running their corresponding WPTs based on the test_mapping.json. The script also traverses through subfolders of the parsed path to check for the .json file. The script also logs on the terminal any mutants that the WPT failed to kill.

* Step 4: Integrated the script so that it can be invoked from the CI tool.

===Execution===
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Excecution Flow===

Given a path
:mutation test(path)
::Look for test mapping file in that path
::for each file the in the test mapping
:::mutate a random line using some strategy in the file
:::build project and run WPT tests corresponding to it
:::if wpt tests pass?
::::report mautation test failure
:::else
::::report mutation test success
for each subfolder folder in the path
:mutation test(subfolder)

===Output===

The log for success to kill a mutant is as shown below:

[[File:Oss4.png]]

The log for failure to kill a mutant is as shown below:

[[File:Oss1.png]]

===Testing===

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

===Subsequent steps===

The following subsequent steps will be followed to meet the project requirements as per this [https://github.com/servo/servo/wiki/Mutation-testing-project].

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Challenges===

1. Setting up the environment on the local machine:

*The amount of time taken to build and test the WPT on the local machine was pretty long due to machine memory. Additionally faced problems on intermittent WPT test failures. It is expected that any reviewer will face a similar challenge.

We overcame this challenge by discussing possibilities with Mozilla team who suggested us to use Janitor Technology. The servo image would be already set up with the latest code, we can edit code in cloud9 and execute it. Using this platform for our testing made the project agnostic of the environment on the local machine and any hassle associated with same.

2. Defining the test mapping framework:

*Servo consists of multiple components of which there are many WPT tests written for each component. The test cases are organized according to the functionality they test, whereas the servo source code is organized according to the specific components. Due to a large number of source files and WPT tests for the entire project, it was difficult for one individual to have information on the mapping of components to their corresponding WPTs.

Hence, we had to have a mapping to run the corresponding tests for a source file. The mapping framework enabled us to run tests on a few components and provide a base for other members from Mozilla team to add and replicate the same framework as per their requirement. We still need to find if there can be any automated way to populate the mapping.

3. The extent of mutation testing to be performed.

*There are standard techniques that define what a mutant is. For example, replacing '&&' with '||'. However, there is no specification for the number of replacements that must be made within a single source code, nor is there any specification on the permutations of possible mutations for each technique. This is an ambiguity in the project and our initial implementation replaces all instances of '&&' to '||' to generate a single type of mutant for each file of the source code. But now our implementation mutates random lines using a strategy. The code needs to evolve based on how runs/performs. And there is no predefined answer for it.

==Pull Request==

Here is our pull request [https://github.com/servo/servo/pull/18984 link].

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing on Servo

2017-11-02T23:47:11Z

Sdhegde:

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This, in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Implemented steps===
The approach adopted was based on the requirements mentioned [https://github.com/servo/servo/wiki/Mutation-testing-project here]. The below steps implement the initial steps mentioned in the project description.

* Step 1: A python script was written to mutate one source file and the corresponding WPT was run on it to check if the mutant was killed.

* Step 2: A test framework was defined to identify the source files that require mutation testing along with their corresponding WPTs. This framework is implemented in /servo/components/script/dom.

* Step 3: The script was expanded to include the test framework and automate the process of generating mutations for multiple source files and running their corresponding WPTs based on the test_mapping.json. The script also traverses through subfolders of the parsed path to check for the .json file. The script also logs on the terminal any mutants that the WPT failed to kill.

* Step 4: Integrated the script so that it can be invoked from the CI tool.

===Execution===
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Excecution Flow===

Given a path
:mutation test(path)
::Look for test mapping file in that path
::for each file the in the test mapping
:::mutate a random line using some strategy in the file
:::build project and run WPT tests corresponding to it
:::if wpt tests pass?
::::report mautation test failure
:::else
::::report mutation test success
for each subfolder folder in the path
:mutation test(subfolder)

===Output===

The log for success to kill a mutant is as shown below:

[[File:Oss4.png]]

The log for failure to kill a mutant is as shown below:

[[File:Oss1.png]]

===Testing===

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

===Subsequent steps===

The following subsequent steps will be followed to meet the project requirements as per this [https://github.com/servo/servo/wiki/Mutation-testing-project].

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Challenges===

1. Setting up the environment on the local machine:

*The amount of time taken to build and test the WPT on the local machine was pretty long due to machine memory. Additionally faced problems on intermittent WPT test failures. It is expected that any reviewer will face a similar challenge.

We overcame this challenge by discussing possibilities with Mozilla team who suggested us to use Janitor Technology. The servo image would be already set up with the latest code, we can edit code in cloud9 and execute it. Using this platform for our testing made the project agnostic of the environment on the local machine and any hassle associated with same.

2. Defining the test mapping framework:

*Servo consists of multiple components of which there are many WPT tests written for each component. The test cases are organized according to the functionality they test, whereas the servo source code is organized according to the specific components. Due to a large number of source files and WPT tests for the entire project, it was difficult for one individual to have information on the mapping of components to their corresponding WPTs.

Hence, we had to have a mapping to run the corresponding tests for a source file. The mapping framework enabled us to run tests on a few components and provide a base for other members from Mozilla team to add and replicate the same framework as per their requirement. We still need to find if there can be any automated way to populate the mapping.

3. The extent of mutation testing to be performed.

*There are standard techniques that define what a mutant is. For example, replacing '&&' with '||'. However, there is no specification for the number of replacements that must be made within a single source code, nor is there any specification on the permutations of possible mutations for each technique. This is an ambiguity in the project and our initial implementation replaces all instances of '&&' to '||' to generate a single type of mutant for each file of the source code. But now our implementation mutates random lines using a strategy. The code needs to evolve based on how runs/performs. And there is no predefined answer for it.

==Pull Request==

Here is our pull request [https://github.com/servo/servo/pull/18984 link].

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing on Servo

2017-11-02T19:27:30Z

Sdhegde: /* Challenges */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Implemented steps===
The approach adopted was based on the requirements mentioned [https://github.com/servo/servo/wiki/Mutation-testing-project here]. The below steps implement the initial steps mentioned in the project description.

* Step 1: A python script was written to mutate one source file and the corresponding WPT was run on it to check if the mutant was killed.

* Step 2: A test framework was defined to identify the source files that require mutation testing along with their corresponding WPTs. This framework is implemented in /servo/components/script/dom.

* Step 3: The script was expanded to include the test framework and automate the process of generating mutations for multiple source files and running their corresponding WPTs based on the test_mapping.json. The script also traverses through sub folders of the parsed path to check for the .json file. The script also logs on the terminal any mutants that the WPT failed to kill.

* Step 4: Integrated the script so that it can be invoked from the CI tool.

===Execution===
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Excecution Flow===

Given a path
:mutation test(path)
::Look for test mapping file in that path
::for each file the in the test mapping
:::mutate a random line using some strategy in the file
:::build project and run wpt tests corresponding to it
:::if wpt tests pass?
::::report mautation test failure
:::else
::::report mautation test success
for each subfolder folder in the path
:mutation test(subfolder)

===Output===

The log for success to kill a mutant is as shown below:

[[File:Oss4.png]]

The log for failure to kill a mutant is as shown below:

[[File:Oss1.png]]

===Testing===

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

===Subsequent steps===

The following subsequent steps will be followed to meet the project requirements as per this [https://github.com/servo/servo/wiki/Mutation-testing-project].

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Challenges===

1. Setting up the environment on the local machine:

*The amount of time taken to build and test the WPT in the local machine was pretty long due to machine memory. Additionally faced problems on intermittent WPT test failures. It is expected that any reviewer will face a similar challenge.

We overcame this challenge by discussing possibilities with Mozilla team who suggested us to use Janitor Technology. The servo image would be already set up with the latest code, we can edit code in cloud9 and execute it. Using this platform for our testing made the project agnostic of the environment on the local machine and any hassle associated with same.

2. Defining the test mapping framework:

*Servo consists of multiple components of which there are many WPT tests written for each component. The test cases are organized according to the functionality they test, whereas the servo source code is organized according to the specific components. Due to a large number of source files and WPT tests for the entire project, it was difficult for one individual to have information on the mapping of components to their corresponding WPTs.

Hence, we had to have a mapping to run the corresponding tests for a source file. The mapping framework enabled us to run tests on a few components and provide a base for other members from Mozilla team to add and replicate the same framework as per their requirement. We still need to find if there can be any automated way to populate the mapping.

3. The extent of mutation testing to be performed.

*There are standard techniques that define what a mutant is. For example, replacing '&&' with '||'. However, there is no specification for the number of replacements that must be made within a single source code, nor is there any specification on the permutations of possible mutations for each technique. This is an ambiguity in the project and our initial implementation replaces all instances of '&&' to '||' to generate a single type of mutant for each file of the source code. But now our implementation mutates random lines using a strategy. The code needs to evolve based on how runs/performs. And there is no predefined answer for it.

==Pull Request==

Here is our pull request [https://github.com/servo/servo/pull/18984 link].

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project

CSC/ECE 517 Fall 2017/M1754 Mutation Testing on Servo

2017-11-02T18:26:58Z

Sdhegde: /* Running test after setup */

Servo uses the Web Platform Test (WPT) suite for testing, but does not perform an evaluation of the breadth of the tests. The goal of this project is to use techniques from mutation testing to evaluate the performance of the WPT suite when bugs are deliberately introduced into the code base.

The implementation of this project was done by writing python scripts that would modify the source code to generate mutants and run tests on them expecting failures. The scripts would temporarily modify the source codes, call the corresponding tests and revert back to the original code by reversing the changes that were made earlier. This process was repeated for multiple iterations by modifying various parts of the source code in random order.

==Introduction==

===Servo===

Servo is a modern, high-performance browser engine designed for both application and embedded use. [https://servo.org Servo] is a web browser layout engine written in [https://github.com/rust-lang/rust Rust]and is currently being developed by [https://en.wikipedia.org/wiki/Mozilla Mozilla]. The aim of the project is not to create a full browser but is rather to create a highly parallel environment that allows for many components be handled by fine-grained, isolated tasks. [https://en.wikipedia.org/wiki/Servo_(layout_engine)]

===Rust===

[https://doc.rust-lang.org/book/second-edition/ Rust] is an open source, systems programming language sponsored by Mozilla Research. Rust performs the majority of its safety checks and memory management decisions at compile time, so that program’s runtime performance is not impacted. Making it useful in programs with predictable space and time requirements, embedding in other languages, and writing low-level code, like device drivers and operating systems.

===Web-platform-tests===
The [https://github.com/w3c/web-platform-tests web-platform-tests Project] is a W3C-coordinated attempt to build a cross-browser test suite for the Web-platform stack. Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software that is compatible with other implementations, and that later implementations will be compatible with their implementations. This in turn, gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementors.

===Mutation Testing===
Mutation Testing is a type of software testing where we mutate (change) certain statements in the source code and check if the test cases are able to find the errors.The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as a Fault-based testing strategy as it involves creating faults in the program.Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.[https://www.guru99.com/mutation-testing.html ]

=='''Project description'''==
===Environment Setup===
* Clone the repository from [https://github.com/dsandeephegde/servo link]
* The steps to setup the environment for different OS are mentioned in readme file [https://github.com/servo/servo link].

====Install packages====

Example, In OSX(homebrew):

brew install automake pkg-config python cmake yasm
pip install virtualenv

====Running test after setup====

After the setup, make sure that everything works in your machine by building and running tests.

build in release mode:

./mach build --release

run WPT tests:

./mach test -wpt --release

===Implemented steps===
The approach adopted was based on the requirements mentioned [https://github.com/servo/servo/wiki/Mutation-testing-project here]. The below steps implement the initial steps mentioned in the project description.

* Step 1: A python script was written to mutate one source file and the corresponding WPT was run on it to check if the mutant was killed.

* Step 2: A test framework was defined to identify the source files that require mutation testing along with their corresponding WPTs. This framework is implemented in /servo/components/script/dom.

* Step 3: The script was expanded to include the test framework and automate the process of generating mutations for multiple source files and running their corresponding WPTs based on the test_mapping.json. The script also traverses through sub folders of the parsed path to check for the .json file. The script also logs on the terminal any mutants that the WPT failed to kill.

* Step 4: Integrated the script so that it can be invoked from the CI tool.

===Execution===
* Mutation test can be ran using either of the following command from the servo directory:

python python/servo/mutation/init.py components/script/dom

or

python etc/ci/mutation_test.py

===Excecution Flow===

Given a path
:mutation test(path)
::Look for test mapping file in that path
::for each file the in the test mapping
:::mutate a random line using some strategy in the file
:::build project and run wpt tests corresponding to it
:::if wpt tests pass?
::::report mautation test failure
:::else
::::report mautation test success
for each subfolder folder in the path
:mutation test(subfolder)

===Output===

The log for success to kill a mutant is as shown below:

[[File:Oss4.png]]

The log for failure to kill a mutant is as shown below:

[[File:Oss1.png]]

===Testing===

The project is about writing a python script to change the source code and run tests. It does not add any functionality to Servo. So there is no scope for testing in this project.

===Subsequent steps===

The following subsequent steps will be followed to meet the project requirements as per this [https://github.com/servo/servo/wiki/Mutation-testing-project].

* implement mutations like replacing if statements by if true/if false, duplicating statements, reordering statements, changing arithmetic & atomic string constant.
* improving the performance of the testing, for example randomizing the test order, fast-failing, running tests with faster builds (e.g. ./mach build -d).
* find heuristics for identifying false positives, that is mutations which are expected to have no effect, for example removing logging.
* find search heuristics for identifying mutations that cause no test failures.

===Challenges===

1. Setting up the environment on the local machine:

*The amount of time taken to build and test the WPT in the local machine was pretty long due to machine memory. Additionally faced problems on intermittent WPT test failures. It is expected that any reviewer will face a similar challenge.

We overcame this challenge by discussing possibilities with Mozilla team who suggested us to use Janitor Technology. The servo image would be already set up with the latest code, we can edit code in cloud9 and execute it. Using this platform for our testing made the project agnostic of the environment on the local machine and any hassle associated with same.

2. Defining the test mapping framework:

*Servo consists of multiple components of which there are many WPT tests written for each component. The test cases are organized according to the functionality they test, whereas the servo source code is organized according to the specific components. Due to a large number of source files and WPT tests for the entire project, it was difficult for one individual to have information on the mapping of components to their corresponding WPTs.

Hence, we had to have a mapping to run the corresponding tests for a source file. The mapping framework enabled us to run tests on a few components and provide a base for other members from Mozilla team to add and replicate the same framework as per their requirement.

3. Understanding the extent of mutation testing to be performed.

*There are standard techniques that define what a mutant is. For example, replacing '&&' with '||'. However, there is no specification for the number of replacements that must be made within a single source code, nor is there any specification on the permutations of possible mutations for each technique. This is an ambiguity in the project and our implementation replaces all instances of '&&' to '||' to generate a single type of mutant for the each file of the source code.

==Pull Request==

Here is our pull request [https://github.com/servo/servo/pull/18984 link].

==References==
<references/>
1. https://en.wikipedia.org/wiki/Servo_(layout_engine) 
2. https://www.guru99.com/mutation-testing.html 
3. https://github.com/servo/servo 
4. https://github.com/servo/servo/wiki/Mutation-testing-project