CSC/ECE 517 Spring 2017/OSS M1706 Tracking intermittent test failures over time: Difference between revisions
No edit summary |
No edit summary |
||
Line 1: | Line 1: | ||
This | == Introduction == | ||
This wiki provides details on new functionality programmed for the Servo OSS project. | |||
== | ===Background=== | ||
The intermittent test failure tracker | "[https://github.com/servo/servo/wiki/Design Servo] is a project to develop a new Web browser engine. Our goal is to create an architecture that takes advantage of parallelism at many levels while eliminating common sources of bugs and security vulnerabilities associated with incorrect memory management and data races." Servo can be used through Browser.html, embedded in a website, or natively in Mozilla Firefox. It is designed to load web pages more efficiently and more securely. | ||
* Build a | |||
===Motivation=== | |||
This project is a request from the Servo OSS project to reduce the impact intermittent test failures have on the software. The [https://github.com/servo/servo/wiki/Tracking-intermittent-failures-over-time-project request] made is for a [http://flask.pocoo.org/docs/0.12/ Flask] service using [https://en.wikipedia.org/wiki/Python_(programming_language) Python 2.7]. The intermittent test failure tracker stores information regarding a test that fails intermittently and also provides means to quickly query for tests that have failed. | |||
===Tasks=== | |||
The intermittent test failure tracker initial steps (for the OSS project) include: | |||
* Build a Flask service | |||
* Use a JSON file to store information | * Use a JSON file to store information | ||
* Record required parameters: Test file, platform, test machine (builder), and related GitHub pull request number | * Record required parameters: Test file, platform, test machine (builder), and related GitHub pull request number | ||
Line 9: | Line 16: | ||
* Use the known intermittent issue tracker as an example of a Simple flask server | * Use the known intermittent issue tracker as an example of a Simple flask server | ||
Subsequent steps include: | Subsequent steps (for the final project) include: | ||
* Add ability to query the service by a date range, to find out which were occurred the most often | * Add ability to query the service by a date range, to find out which were occurred the most often | ||
* Build an HTML front-end to the service that queries using JS and reports the results | * Build an HTML front-end to the service that queries using JS and reports the results | ||
Line 19: | Line 26: | ||
== Design == | == Design == | ||
=== | ===Application Flow=== | ||
==== Saving a Test ==== | |||
The | The Servo build agent calls a webhook (a way for an app to provide other applications with real-time information) inside the test tracker. The webhook then calls a handler that contains any business logic necessary to transform the request. Finally the handler persists the request into the db, in this case a json file. This flow can be seen in the graph below. | ||
<pre> | <pre> | ||
Line 41: | Line 48: | ||
== Implementation == | == Implementation == | ||
The implementation is entirely influenced by the request, the | The implementation is entirely influenced by the request, the Servo team clearly defines what the service should do and how it would be made. | ||
=== Data model === | === Data model === | ||
Line 88: | Line 95: | ||
The first method returns 'index page' at the root URL. The second method accepts a URL param after user and returns the user from a database. | The first method returns 'index page' at the root URL. The second method accepts a URL param after user and returns the user from a database. | ||
== | == Test Plan == | ||
The sample [http://csc517oss.zachncst.com/ index page] is for illustrating the project's functionality only. The links on the page lead to demonstrations of the query and record handlers, as well as a display of the JSON file containing all the Intermittent Test Failure records. | The sample [http://csc517oss.zachncst.com/ index page] is for illustrating the project's functionality only. The links on the page lead to demonstrations of the query and record handlers, as well as a display of the JSON file containing all the Intermittent Test Failure records. | ||
Line 94: | Line 101: | ||
== Submission == | == Submission == | ||
The work has been started in a new GitHub repo located [https://github.com/adamw17/csc517ossproject/tree/832969c1cf01d94be340731c744854c25fdbb441 here]. When | The work has been started in a new GitHub repo located [https://github.com/adamw17/csc517ossproject/tree/832969c1cf01d94be340731c744854c25fdbb441 here]. When Servo developers are ready, the project will be pulled in to the Servo project on GitHub. |
Revision as of 17:57, 28 March 2017
Introduction
This wiki provides details on new functionality programmed for the Servo OSS project.
Background
"Servo is a project to develop a new Web browser engine. Our goal is to create an architecture that takes advantage of parallelism at many levels while eliminating common sources of bugs and security vulnerabilities associated with incorrect memory management and data races." Servo can be used through Browser.html, embedded in a website, or natively in Mozilla Firefox. It is designed to load web pages more efficiently and more securely.
Motivation
This project is a request from the Servo OSS project to reduce the impact intermittent test failures have on the software. The request made is for a Flask service using Python 2.7. The intermittent test failure tracker stores information regarding a test that fails intermittently and also provides means to quickly query for tests that have failed.
Tasks
The intermittent test failure tracker initial steps (for the OSS project) include:
- Build a Flask service
- Use a JSON file to store information
- Record required parameters: Test file, platform, test machine (builder), and related GitHub pull request number
- Query the store results given a particular test file name
- Use the known intermittent issue tracker as an example of a Simple flask server
Subsequent steps (for the final project) include:
- Add ability to query the service by a date range, to find out which were occurred the most often
- Build an HTML front-end to the service that queries using JS and reports the results
- Links to GitHub
- Sorting
- Make filter-intermittents command record a separate failure for each intermittent failure encountered
- Propogate the required information for recording failures in saltfs
Design
Application Flow
Saving a Test
The Servo build agent calls a webhook (a way for an app to provide other applications with real-time information) inside the test tracker. The webhook then calls a handler that contains any business logic necessary to transform the request. Finally the handler persists the request into the db, in this case a json file. This flow can be seen in the graph below.
+---------------------------------------------+ | Intermittent Test Failure Tracker | | | +--------------+ | +-----------+ +---------+ +------+ | | | | | | | | | | | +--------+ | Servo | | | | | | | | | | | | Build +------> webhook +------> handler +----> db +---------> json | | Server | | | | | | | | | | file | | | | | | | | | | | | | +--------------+ | +-----------+ +---------+ +------+ | +--------+ | | +---------------------------------------------+
Implementation
The implementation is entirely influenced by the request, the Servo team clearly defines what the service should do and how it would be made.
Data model
The model for an intermittent test is defined mostly by the request with a few additions to help with querying in later steps of the OSS request.
Name | Type | Description |
---|---|---|
test_file | String | Name of the intermittent test file |
platform | String | Platform the test failed on |
builder | String | The test machine (builder) the test failed on |
number | Integer | The GitHub pull request number |
fail_date | ISO date (String) | Date of the failure |
Datastore
To store the intermittent test failures, a library called TinyDB is used. This library is a native python library that provides convenient SQL command like helpers around a JSON file to more easily use it like a database. The format of the JSON file is simply an array of JSON objects, making the file easily human readable.
Flask Service
Flask is a microservice framework written in Python. A flask service is a REST (representational state transfer) API that maps URL and HTTP verbs to python functions. Some basic examples of flask routes:
@app.route('/') def index(): return 'Index page' @app.route('/user/<username>') def show_user(username): return db.lookup(username)
The first method returns 'index page' at the root URL. The second method accepts a URL param after user and returns the user from a database.
Test Plan
The sample index page is for illustrating the project's functionality only. The links on the page lead to demonstrations of the query and record handlers, as well as a display of the JSON file containing all the Intermittent Test Failure records.
Submission
The work has been started in a new GitHub repo located here. When Servo developers are ready, the project will be pulled in to the Servo project on GitHub.