CSC/ECE 517 Spring 2024 - G2400 DevOp for GitHub Miner app

GitHub Miner

GitHub Miner is a tool for querying GitHub APIs and returning useful information such as a user's contributions, the comments made by a user, the repositories owned by the user, etc.

Documentation on the available methods and functionality is available on the GitHub repo.

Installation

GitHub Miner is written in Python and can be installed locally using a virtual environment (venv) or using Docker.

To use Docker: Ensure that you have Docker installed Update the GITHUB_OAUTH_CLIENT_ID and GITHUB_OAUTH_CLIENT_SECRET values in the docker-compose.yml file based on the instructions here.

Optionally - add a value for the SECRET_KEY in the docker-compose.yml file. See the explanation here on secret key values. Run docker compose up.

Testing

The scope of our project was to create a Docker container for the main GitHub miner app and a docker-compose file to run the app and a MySQL container. As such, creating automated tests was not within the scope of our work. Manual tests were completed by:

1. Running

   docker compose up 

2. Verifying that an application container and a mysql container were both created.
3. Running

   docker exec ghminer flask db upgrade 

   to migrate the database.
4. Running

   docker exec ghminer python -m backend.scripts.seed_db 

   to seed the database.
5. Opening a browser and navigating to http://localhost:5000/test-insert/2/john_doe
6. Navigating to http://localhost:5000/test-retrive/2

Final Project Design

Problem: The project has no linters to enforce stylistic guidelines and users must remember to manually trigger unit tests after code updates and before code changes are merged in GitHub.

Solution: For the final project, we will be creating a CI/CD process that will lint the project code and run the unit tests.

To accomplish this, we will be using GitHub Actions. Information on Github Actions can be found here. This will require us to:

1. Create a workflow configuration .yaml file that lays out the process steps and their order
   1. Identify the correct runner (environment) to run the CI/CD process:
   2. Identify the correct container to run the steps: ubuntu-latest
   3. Identify the correct actions, one for linting and one for testing, to make up the workflow
      1. For linting we use Ruff through the jpetrucciani/ruff-check plugin. We chose this plugin because Ruff is a popular and performant linter used by many large projects. We opted for Ruff over alternatives such as Pylint because we want to keep build times minimal.
      2. For testing we use Pytest as is typically done in Python projects.
      3. Coverage is gathered by passing the --cov argument to Pytest.
   4. Identify the correct outcomes for failing each stage and how to configure them
      1. Linting does not fail the pipeline, but displays any linting warnings in both the PR associated with the build and the build pipeline.
      2. Failing tests cause the build to fail. There is no minimum coverage requirement currently configured; this would be a good area for future improvement.
2. Place the file into a .github/workflows/ folder in the project root
3. Commit the change

Due to the nature of the project, there is no way to run automated tests against the .yaml file outside of linting. Manual verification will be completed by:

1. Navigating to the Actions tab in the project repository in GitHub
2. Verifying using the GUI that there are no errors noted
3. Verify that the unit tests for the project are ran and a coverage report is created

This is the demo: https://www.youtube.com/watch?v=7w12ucodV1k

Below we provide some screenshots demonstrating how the build pipeline can be monitored from the Github webpage.

A PR displaying the pipeline status:

The same PR displaying linter errors:

You will notice that the build pipeline is successful and the PR can be merged despite the presence of linter errors. This is the desired behavior at current so developers are not forcd to resolve all tech debt before they are able to commit new changes. We recommend that debt should be addressed as part of updates to the project. In the future, it may be worth revisiting this to enforce linting rules.

The Actions tab linter errors:

The Actions tab test coverage report:

Team Members

Tanner Neff

Andrew Robie

Srinish Pellakur

Mentor: Jialin Cui