CSC/ECE 517 Spring 2017/OSS M1706 Tracking intermittent test failures over time

This is a request from the Servo OSS project to reduce the impact intermittent test failures have on the software. The request made was for a Flask service using Python 2.7. The intermittent test failure tracker will store information regarding a test that fails intermittently and also provide means to quickly query for tests that have failed.

Scope

The intermittent test failure tracker initial steps included

• Build a flask service
• Use a JSON file to store information
• Test file, platform, test machine (builder), and related Github pull request number are required parameters
• Allow querying the store results given a particular test file name.
• Use the known intermittent issue tracker as an example of a Simple flask server

Subsequent steps included

• Add ability to query the service by a date range, to find out which were occurred the most often
• Build an HTML front-end to the service that queries using JS and reports the results
  • Links to github
  • Sorting
• Make filter-intermittents command record a separate failure for each intermittent failure encountered
• Propgate the rqeuired information for recording failures in saltfs

Design

Application Flow

                    +---------------------------------------------+
                    |       Intermittent Test Failure Tracker     |
                    |                                             |
+--------------+    | +-----------+      +---------+    +------+  |
|              |    | |           |      |         |    |      |  |      +--------+
|    Servo     |    | |           |      |         |    |      |  |      |        |
|    Build     +------>  webhook  +------> handler +---->  db  +--------->  json  |
|    Server    |    | |           |      |         |    |      |  |      |  file  |
|              |    | |           |      |         |    |      |  |      |        |
+--------------+    | +-----------+      +---------+    +------+  |      +--------+
                    |                                             |
                    +---------------------------------------------+

Implementation

The implementation was entirely influenced by the request, the servo team had clearly defined what the service should do and how it would be made.

Data model

The model for an intermittent test is defined mostly by the request with a few additions to help with querying in later steps of the OSS request.

Name	Type	Description
test_file	String	Name of the intermittent test file
platform	String	Platform the test failed on
builder	String	The test machine (builder) the test failed on
number	Integer	The github pull request number
fail_date	ISO date (String)	Date of the failure

Datastore

To store the intermittent test failures, a library called TinyDB was used. This library is a native python library that provides convenient SQL command like helpers around a JSON file to more easily use it like a database. The format of the JSON file is simply an array of JSON objects, making the file easily human readable.

Flask Service

Flask is a microservice framework written in Python. A flask service is a REST (representational state transfer) API that maps URL and HTTP verbs to python functions. Some basic examples of flask routes:

 @app.route('/')
 def index():
   return 'Index page'
 
 @app.route('/user/<username>')
 def show_user(username):
   return db.lookup(username)

The first method returns 'index page' at the root URL. The second method accepts a URL param after user and returns the user from a database.

Submission

The work has been started in a new github repo located here. When servo developers are ready, the project will be pulled in to the servo project on github.