CSC/ECE 517 Spring 2017/OSS M1706 Tracking intermittent test failures over time

Introduction

This wiki provides details on new functionality programmed for the Servo OSS project.

Background

"Servo is a project to develop a new Web browser engine. Our goal is to create an architecture that takes advantage of parallelism at many levels while eliminating common sources of bugs and security vulnerabilities associated with incorrect memory management and data races." Servo can be used through Browser.html, embedded in a website, or natively in Mozilla Firefox. It is designed to load web pages more efficiently and more securely.

Motivation

This project is a request from the Servo OSS project to reduce the impact intermittent test failures have on the software. Specifically, the requestors have specified that a Flask service be developed using Python 2.7 to aid in tracking failures. Intermittent failures frequently occur during development but are normally ignored during continuous integration. The frequency of each intermittent failure signature, though not currently logged, would be useful in allowing developers to identify and resolve the most prevalent issues. The intermittent test failure tracker would store information regarding frequencies of specific failing tests and also provide a means to quickly query for tests that have failed.

Tasks

Round 1

The intermittent test failure tracker initial steps (for the OSS project) include:

• Build a Flask service
• Use a JSON file to store information
• Record required parameters: Test file, platform, test machine (builder), and related GitHub pull request number
• Query the store results given a particular test file name
• Use the known intermittent issue tracker as an example of a Simple flask server

Round 2

Subsequent steps (for the final project) include:

• Add ability to query the service by a date range, to find out which were occurred the most often
• Build an HTML front-end to the service that queries using JS and reports the results
  • Links to GitHub
  • Sorting
• Make filter-intermittents command record a separate failure for each intermittent failure encountered
• Propogate the required information for recording failures in saltfs

Design

Design Pattern

The Servo and this project's code follow a Service Layer design pattern. This design pattern breaks up functionality into smaller "services" and applies the services to the topmost "layer" of the project for which they are needed.

Application Flow

Saving a Test

The Servo build agent calls a webhook (a way for an app to provide other applications with real-time information) inside the test tracker. The webhook then calls a handler that contains any business logic necessary to transform the request. Finally the handler persists the request into the db, in this case a json file. This flow can be seen in the graph below.

                    +---------------------------------------------+
                    |       Intermittent Test Failure Tracker     |
                    |                                             |
+--------------+    | +-----------+      +---------+    +------+  |
|              |    | |           |      |         |    |      |  |      +--------+
|    Servo     |    | |           |      |         |    |      |  |      |        |
|    Build     +------>  webhook  +------> handler +---->  db  +--------->  json  |
|    Server    |    | |           |      |         |    |      |  |      |  file  |
|              |    | |           |      |         |    |      |  |      |        |
+--------------+    | +-----------+      +---------+    +------+  |      +--------+
                    |                                             |
                    +---------------------------------------------+

Round 2 Design Plan

The first request is to add the ability to query the service by a date range, to find out which failures were most frequent. Given that the fail_date is already being populated in the database as an ISO date string when records are added, it should be possible to build a function to query this date using standard date functions and a range given from the user. This will require a new query function that takes the range as parameters.

Use Case	Filter intermittent failures by date
Primary Actor	Servo Developer
Preconditions	Previously saved intermittent failure records exist in the database during specific time frame
Triggers	A user submits a query for intermittent failures occurring within a specific date range
Postconditions	A collection of intermittent failure records is returned

The second request is to build an HTML front-end to the service to be queried using JavaScript. The user interface should report the results in a useful manner (linking to the pertinent Github location, sorting by different criteria, etc.). A sample HTML page demonstrating the ability to query by filename was written as part of phase I. For phase II, a more complex HTML page will be required which queries for filename and date range. Records will also be populated into an HTML table instead of returning raw JSON. Javascript functions will also be added to sort columns using the DataTables jQuery plug-in.

Use Case	Interface with Flask service via HTML front-end
Primary Actor	Servo Developer
Preconditions	Previously saved intermittent failure records exist in the database, Flask server is operational
Triggers	Users select a date range via an HTML form
Postconditions	An HTML view is presented showing a pareto failing tests, each linked to the pertinent Github library and sortable by various criteria.

The last two steps involve the full integration of this product into the Servo pipeline and will require the forking of the Servo project on Github.

It was requested that we use an existing command (filter-intermittents) In the existing testing_commands.py code as the initiator for saving records. Separate failure records should be recorded for each intermittent failure encountered. This modification will provide the actual integration into the Servo framework to allow communication with this tracking system.

Use Case	Modify "filter-intermittents" command to add entries to database
Primary Actor	Servo Developer
Preconditions	Failure database exists with functionality to add records, Flask server is operational.
Triggers	The "filter-intermittents command is issued as part of the automated failure reporting that occurs during continuous integration. Instead of being immediately filtered, the details of the intermittent failures are firstly recorded in the database before being filtered.
Postconditions	Entries exist in the database for each occurrence of an intermittent failure.

The second integration into the Servo project for this tracker will be to propagate the required information for recording failures in Salt Stack File System (saltfs). This will also require a testing setup for the saltfs or at least a mock setup for mimicking the integration into saltfs.

Use Case	Propagate failure data into Salt Stack Filesystem
Primary Actor	Servo Developer
Preconditions	Failure database exists with functionality to add records, Flask server is operational.
Triggers	The "filter-intermittents command is issued as part of the automated failure reporting that occurs during continuous integration. Failure information is propagated to the master Salt Stack Filesystem for archival.
Postconditions	Entries exist in the Salt Stack Filesystem for each occurrence of an intermittent failure.

Implementation

The implementation is entirely dictated by the Servo team - they have defined what the service should do and how it should be designed. A standalone working version of the design has been implemented in a Github repository. The complete list of files created for this project can be examined in this repository. Eventually, this project will be incorporated into the Servo development repository and interfaced with other Servo development software. The details of the implementation are given below.

Data model

The model for an intermittent test is defined mostly by the request with a few additions to help with querying in later steps of the OSS request.

Name	Type	Description
test_file	String	Name of the intermittent test file
platform	String	Platform the test failed on
builder	String	The test machine (builder) the test failed on
number	Integer	The GitHub pull request number
fail_date	ISO date (String)	Date of the failure in the form YYYYMMDD
fail_time	ISO date (String)	Time of the failure in the form HH:MM:SS.MMMMMM

Datastore

To store the intermittent test failures, a library called TinyDB is used. This library is a native python library that provides convenient SQL command like helpers around a JSON file to more easily use it like a database. The format of the JSON file is simply an array of JSON objects, making the file easily human-readable.

Flask Service

Flask is a microservice framework written in Python. A flask service is a REST (representational state transfer) API that maps URL and HTTP verbs to python functions. Some basic examples of flask routes are given below:

 @app.route('/')
 def index():
   return 'Index page'
 
 @app.route('/user/<username>')
 def show_user(username):
   return db.lookup(username)

The first method returns 'index page' at the root URL. The second method accepts a URL param after user and returns the user from a database.

Test Plan

Functional Testing

As a convenience to the testers included in this code base is a set of testing web applications and is only for illustrating the project's functionality. This simple set of forms allow a tester to exercise the functionality of the REST endpoints without having to write any REST code. The links on the page lead to demonstrations of the query and record handlers, as well as a display of the JSON file containing all the Intermittent Test Failure records. All usable for thorough integration testing.

Unit Testing

The Unit Tests included in the code exercise the major functions of this system. The tests exercise the addition of a record into the database, the removal of a record given a filename, the retrieval of a record, and the assertion that a record will not be added if any of the record parameters (test_file, platform, builder, number) is missing. All unit tests are in tests.py.

Unit Test Summary
Test Purpose	Function Tested	Parameters
Add a record to a database	db.add	params[:self, :test_file, :platform, :builder, :number, :fail_date, :fail_time]
Delete a record from database	db.remove	params[:test_file]
Record a new Intermittent failure	handlers.record	params[:db, :test_file, :platform, :builder, :number]
Query the Intermittent failure records	handlers.query	params[:db, :test_file]
Record a new Intermittent failure, test invalid values - 4 tests for blanks for each input item	handlers.record	params[:db, :test_file, :platform, :builder, :number]

Round 2 Testing Plan

Feature	Testing Infrastructure Required
Date Range Query Functionality	New unit tests to ensure query returns proper records
Servo Integration via filter-intermittents	Unit and functional testing via Servo
Servo integration with Salt Stack File System	Unit and functional testing via Servo

Running Unit Tests and the App

Before attempting either of the following, clone the repo.

To Run Unit Tests

• In the cloned repo folder, use the command python tests.py

To Run The App Locally

• Set the Flask environment variable by typing export FLASK_APP=flask_sever (use set instead of export for Windows)
• In the cloned repo folder, use the command python -m flask_server or alternatively type flask run
• To launch the app, go to http://localhost:5000

Submission/Pull Requests

There is no Pull Request because Servo manager Josh Matthews requested that we start a new (non-branched) repository for this project. The work has been started in a new GitHub repo located here. When Servo developers are ready, the project will be pulled in to the Servo project on GitHub. In the interim, we shared our repo with Josh, whose reply was "this looks really great! Thanks for tackling it!"