CSC/ECE 517 Spring 2017/OSS M1706 Tracking intermittent test failures over time
This is a request from the Servo OSS project to reduce the impact intermittent test failures have on the software. The request made was for a Flask service using Python 2.7. The intermittent test failure tracker will store information regarding a test that fails intermittently and also provide means to quickly query for tests that have failed.
Introduction
Servo
Python
Scope
Implementation
The implementation was entirely influenced by the request, the servo team had clearly defined what the service should do and how it would be made.
Data model
The model for an intermittent test is defined mostly by the request with a few additions to help with querying in later steps of the OSS request.
| Name | Type | Description |
|---|---|---|
| test_file | String | Name of the intermittent test file |
| platform | String | Platform the test failed on |
| builder | String | The test machine (builder) the test failed on |
| number | Integer | The github pull request number |
| fail_date | ISO date (String) | Date of the failure |
Datastore
To store the intermittent test failures, a library called TinyDB was used. This library is a native python library that provides convenient SQL command like helpers around a JSON file to more easily use it like a database. The format of the JSON file is simply an array of JSON objects, making the file easily human readable.