CSC/ECE 517 Spring 2016/M1704 Servo Dependency Tool: Difference between revisions
(~~~~) |
|||
| (32 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
''' | The '''[https://github.com/chbrown13/servo-dependency-tool Servo Dependency Tool]''' is a set of python scripts that automatically upgrades Cargo dependencies for the [https://servo.org/ Servo] web browser engine. | ||
== | == Background == | ||
[https://servo.org/ Servo] is a prototype web browser engine created by [https://www.mozilla.org/en-US/ Mozilla Research] and written in the [https://www.rust-lang.org/en-US/index.html Rust] programming language. It is currently developed on 64-bit OS X, 64-bit Linux, and Android platforms. Servo depends on numerous other Rust libraries that are published on the package manager [https://crates.io/ crates.io]. There are no notifications for when packages are updated so it's up to developers to keep track of when they need to upgrade the dependencies within their projects. Though this tool is designed for Servo, it is independent from the Servo code base and does not require a pull request from the [https://github.com/servo/servo servo] Github repository. | |||
'''The goal of the Servo Dependency Tool is to automatically upgrade Servo's dependencies whenever new versions are released.''' | |||
=== | == Requirements[https://github.com/servo/servo/wiki/Cargo-upgrade-service-project] == | ||
* Use [https://www.python.org/ Python], [https://nodejs.org/en/ Node.js], or [https://www.ruby-lang.org/en/ Ruby] to develop the tool | |||
* Tool must be easily ran on hosted services ([https://www.heroku.com/ Heroku], [https://aws.amazon.com/ Amazon AWS], etc) | |||
* Parse a Cargo.lock file to determine the list of dependencies and their current versions | |||
* Compare the list of dependencies and their current versions to a local clone of the crates.io index to find out the latest versions available | |||
* Edit all Cargo.toml files within Servo with the latest versions | |||
* Update the packages by running the cargo-update command using Servo's Mach tool | |||
* Interact with GitHub to create a branch for the updates and then open a pull request on the main Servo repo | |||
== Tool Workflow Overview == | |||
# Perform a pull to ensure the servo fork is up-to-date | |||
# Create a new branch on the servo fork to contain the changes performed by the tool | |||
# Parse the Cargo.lock file to determine all the packages within Servo and their current versions | |||
# Make a local clone of the crates.io index and compare the current versions of each package to the most recent version in the crates.io index | |||
# Iterate through all the Cargo.toml files in the Servo project and update all listed versions to the most recent version found in the previous step | |||
# For each package with an update, run the cargo-update command using the Mach tool | |||
# Push all updates to the newly created upstream branch | |||
# Submit a pull request containing these updates to the main Servo repository | |||
=== Project Links | == Project Structure == | ||
=== servo_dependency_tool.py === | |||
servo_dependency_tool.py is the main driver file of our tool. This file imports all of the other files and controls the overall flow of the tool from start to finish. | |||
=== cargo_lock_parser.py === | |||
This module handles parsing Cargo.lock files to get metadata on project dependencies and store in Python objects. The purpose of Cargo.lock files are to store data on dependencies for Rust packages. cargo_lock_parse completes this by creating several class objects. LockFile represents the project's Cargo.lock file. LockRoot represents the root from the [[root]] tag which is present in all Cargo.lock files. LockPackage stores package information for version, an upgrade is available, and dependencies. A package can have 0 or more dependencies that come from a specific source and are stored in a LockDependency object. | |||
=== crates_io_checker.py === | |||
The purpose of this module is to check each package in the crates.io repository to see if an upgrade is available. It clones the latest version of the crates.io Github repository, checks if the package is in the crates index, and determines if the package version is the latest version. | |||
=== cargo_toml_updater === | |||
This module updates all Cargo.toml files within the project to display the latest version. In order for the upgrades to take place, the .toml files for dependencies must be updated in the repository. [https://en.wikipedia.org/wiki/TOML TOML] is a markup language that is minimal, easy to read, maps key and value pairs similar to a hash table, and has been implemented in various programming languages. | |||
=== run_cargo_update.py === | |||
run_cargo_update is responsible for upgrading the project dependencies based on the results from the parser and crates.io checker. This module runs the following command to upgrade a crate in Servo after the .toml files have been updated: | |||
<code> | |||
./mach cargo-update -p [crate name] | |||
</code> | |||
For non-Servo projects, the following command can be used to upgrade dependencies from crates.io packages: | |||
<code> | |||
cargo update -p [name] | |||
</code> | |||
=== repo_management.py === | |||
The repo_management module is focused on handling the Github operations for project repositories. The code depends on external Python Github APIs [http://gitpython.readthedocs.io/en/stable/ gitpython] and [https://github3py.readthedocs.io/en/master/ github3]. It can pull the master branch of a project repo, create new a new branch for the repository, commit and push to the new branch, and create pull requests on the repository with the upgraded versions. | |||
== Running the Tool == | |||
The Servo Dependency Tool has been designed to be easy to setup and use. Detailed information can be found in the README on [https://github.com/chbrown13/servo-dependency-tool GitHub]. | |||
=== Prerequisites === | |||
The tool runs using Python3 and requires two python libraries in order to run: GitPython and github3.py. In addition, it is expected that the machine running the tool has a local copy of the forked Servo repo. | |||
=== Installation and Execution === | |||
Installing the tool requires cloning the Servo Dependency Tool into the root folder of the local copy of Servo: | |||
<code> | |||
git clone https://github.com/chbrown13/servo-dependency-tool | |||
</code> | |||
Call servo_dependency_tool.py to run the tool: | |||
<code> | |||
python3 servo_dependency_tool.py | |||
</code> | |||
== Testing == | |||
Tests for the Servo Dependency Tool were implemented using the Python 3 [https://docs.python.org/3/library/unittest.html unit testing framework]. These tests were created to ensure certain functionality of the tool works appropriately and compares actual values to expected output. The unit tests can be found in the test folder of the project's repository. To run the unit tests, use the following command: | |||
<code> | |||
python3 -m unittest <module-name> | |||
</code> | |||
Below is more information on the unit tests created to test the functionality of our located in the test directory: | |||
=== test_cargo_lock_parser.py === | |||
This tests parsing the Cargo lock files. A sample Cargo.lock file is included in the test folder specifically for testing purposes. This module tests that the example file is parsed correctly by asserting that the LockFile, LockRoot, LockDependency, and LockPackage objects created have the correct values after parsing. | |||
=== test_cargo_toml_updater.py === | |||
This module tests the tools ability to update a Cargo.toml file if a dependency has an upgrade available. A sample Cargo.toml file is placed in the test directory specifically for testing the updates. This test makes sure the file was changed by asserting the text of the Cargo.toml file differs after ''toml_file_update'' is run. The test .toml file is modified during the test and then returned to it's original state at ''tearDown'' when the test completes to facilitate multiple runs. | |||
=== test_crates_io_checker.py === | |||
This module tests various functions from crates_io_checker.py. In the test folder, a sample file is stored in the directory cargo_test/te/st/testing to simulate the file structure of the [https://github.com/rust-lang/crates.io-index crates.io] repository. ''test_check_upgrade'' asserts that the correct boolean value is returned based on whether or not an upgrade is available for a dependency, ''test_read_file'' tests that the /cargo_test/te/st/testing file is read properly and the dependencies are updated in the depends dictionary, ''test_check_folder'' tests that an invalid file returns None while an existing file returns the correct path, ''test_check_package'' checks if the package exists and is loaded in the fake crates.io directory, and ''test_check'' ensures that the entire crates.io checking process works correctly combining all the functionality with mock data. | |||
* Note that we did not include tests for repo_management.py, because it depends on external APIs that connect to Github repositories, run_cargo_update.py, because it contains code runs scripts to actually apply updates to outdated dependencies, and servo_dependency_tool.py, because it is the module that controls the entire workflow of the tool. | |||
== Project Links == | |||
* Servo Dependency Tool (Our GitHub): https://github.com/chbrown13/servo-dependency-tool | * Servo Dependency Tool (Our GitHub): https://github.com/chbrown13/servo-dependency-tool | ||
* Project Request Link: https://github.com/servo/servo/wiki/Cargo-upgrade-service-project | * Project Request Link: https://github.com/servo/servo/wiki/Cargo-upgrade-service-project | ||
* GitHub Issue Tracker: https://github.com/servo/servo/issues/15600 | * GitHub Issue Tracker: https://github.com/servo/servo/issues/15600 | ||
Latest revision as of 21:01, 29 March 2017
The Servo Dependency Tool is a set of python scripts that automatically upgrades Cargo dependencies for the Servo web browser engine.
Background
Servo is a prototype web browser engine created by Mozilla Research and written in the Rust programming language. It is currently developed on 64-bit OS X, 64-bit Linux, and Android platforms. Servo depends on numerous other Rust libraries that are published on the package manager crates.io. There are no notifications for when packages are updated so it's up to developers to keep track of when they need to upgrade the dependencies within their projects. Though this tool is designed for Servo, it is independent from the Servo code base and does not require a pull request from the servo Github repository.
The goal of the Servo Dependency Tool is to automatically upgrade Servo's dependencies whenever new versions are released.
Requirements[1]
- Use Python, Node.js, or Ruby to develop the tool
- Tool must be easily ran on hosted services (Heroku, Amazon AWS, etc)
- Parse a Cargo.lock file to determine the list of dependencies and their current versions
- Compare the list of dependencies and their current versions to a local clone of the crates.io index to find out the latest versions available
- Edit all Cargo.toml files within Servo with the latest versions
- Update the packages by running the cargo-update command using Servo's Mach tool
- Interact with GitHub to create a branch for the updates and then open a pull request on the main Servo repo
Tool Workflow Overview
- Perform a pull to ensure the servo fork is up-to-date
- Create a new branch on the servo fork to contain the changes performed by the tool
- Parse the Cargo.lock file to determine all the packages within Servo and their current versions
- Make a local clone of the crates.io index and compare the current versions of each package to the most recent version in the crates.io index
- Iterate through all the Cargo.toml files in the Servo project and update all listed versions to the most recent version found in the previous step
- For each package with an update, run the cargo-update command using the Mach tool
- Push all updates to the newly created upstream branch
- Submit a pull request containing these updates to the main Servo repository
Project Structure
servo_dependency_tool.py
servo_dependency_tool.py is the main driver file of our tool. This file imports all of the other files and controls the overall flow of the tool from start to finish.
cargo_lock_parser.py
This module handles parsing Cargo.lock files to get metadata on project dependencies and store in Python objects. The purpose of Cargo.lock files are to store data on dependencies for Rust packages. cargo_lock_parse completes this by creating several class objects. LockFile represents the project's Cargo.lock file. LockRoot represents the root from the root tag which is present in all Cargo.lock files. LockPackage stores package information for version, an upgrade is available, and dependencies. A package can have 0 or more dependencies that come from a specific source and are stored in a LockDependency object.
crates_io_checker.py
The purpose of this module is to check each package in the crates.io repository to see if an upgrade is available. It clones the latest version of the crates.io Github repository, checks if the package is in the crates index, and determines if the package version is the latest version.
cargo_toml_updater
This module updates all Cargo.toml files within the project to display the latest version. In order for the upgrades to take place, the .toml files for dependencies must be updated in the repository. TOML is a markup language that is minimal, easy to read, maps key and value pairs similar to a hash table, and has been implemented in various programming languages.
run_cargo_update.py
run_cargo_update is responsible for upgrading the project dependencies based on the results from the parser and crates.io checker. This module runs the following command to upgrade a crate in Servo after the .toml files have been updated:
./mach cargo-update -p [crate name]
For non-Servo projects, the following command can be used to upgrade dependencies from crates.io packages:
cargo update -p [name]
repo_management.py
The repo_management module is focused on handling the Github operations for project repositories. The code depends on external Python Github APIs gitpython and github3. It can pull the master branch of a project repo, create new a new branch for the repository, commit and push to the new branch, and create pull requests on the repository with the upgraded versions.
Running the Tool
The Servo Dependency Tool has been designed to be easy to setup and use. Detailed information can be found in the README on GitHub.
Prerequisites
The tool runs using Python3 and requires two python libraries in order to run: GitPython and github3.py. In addition, it is expected that the machine running the tool has a local copy of the forked Servo repo.
Installation and Execution
Installing the tool requires cloning the Servo Dependency Tool into the root folder of the local copy of Servo:
git clone https://github.com/chbrown13/servo-dependency-tool
Call servo_dependency_tool.py to run the tool:
python3 servo_dependency_tool.py
Testing
Tests for the Servo Dependency Tool were implemented using the Python 3 unit testing framework. These tests were created to ensure certain functionality of the tool works appropriately and compares actual values to expected output. The unit tests can be found in the test folder of the project's repository. To run the unit tests, use the following command:
python3 -m unittest <module-name>
Below is more information on the unit tests created to test the functionality of our located in the test directory:
test_cargo_lock_parser.py
This tests parsing the Cargo lock files. A sample Cargo.lock file is included in the test folder specifically for testing purposes. This module tests that the example file is parsed correctly by asserting that the LockFile, LockRoot, LockDependency, and LockPackage objects created have the correct values after parsing.
test_cargo_toml_updater.py
This module tests the tools ability to update a Cargo.toml file if a dependency has an upgrade available. A sample Cargo.toml file is placed in the test directory specifically for testing the updates. This test makes sure the file was changed by asserting the text of the Cargo.toml file differs after toml_file_update is run. The test .toml file is modified during the test and then returned to it's original state at tearDown when the test completes to facilitate multiple runs.
test_crates_io_checker.py
This module tests various functions from crates_io_checker.py. In the test folder, a sample file is stored in the directory cargo_test/te/st/testing to simulate the file structure of the crates.io repository. test_check_upgrade asserts that the correct boolean value is returned based on whether or not an upgrade is available for a dependency, test_read_file tests that the /cargo_test/te/st/testing file is read properly and the dependencies are updated in the depends dictionary, test_check_folder tests that an invalid file returns None while an existing file returns the correct path, test_check_package checks if the package exists and is loaded in the fake crates.io directory, and test_check ensures that the entire crates.io checking process works correctly combining all the functionality with mock data.
- Note that we did not include tests for repo_management.py, because it depends on external APIs that connect to Github repositories, run_cargo_update.py, because it contains code runs scripts to actually apply updates to outdated dependencies, and servo_dependency_tool.py, because it is the module that controls the entire workflow of the tool.
Project Links
- Servo Dependency Tool (Our GitHub): https://github.com/chbrown13/servo-dependency-tool
- Project Request Link: https://github.com/servo/servo/wiki/Cargo-upgrade-service-project
- GitHub Issue Tracker: https://github.com/servo/servo/issues/15600