CSC/ECE 517 Spring 2024 - G2402 Implement REST client, REST API, and Graphql API endpoint for repositories - Part 2

GitHub Miner

This is a convenient tool to query a user's GitHub metrics. The project aims to develop API endpoints for GitHub GraphQL queries and GitHub REST queries using Python Flask. It involves integrating existing code with Flask to expose GraphQL queries as API endpoints, developing a REST client, and creating REST endpoints for querying the same data sets. The project also includes thorough testing and documentation of the endpoints.

Live Demo & Source Code

http://152.7.177.239:5000/auth/login Link] (Please login using your personal GitHub accounts, not NCSU accounts)

Source Code

Installation

We recommend using virtual environment. Steps to set up virtual environment:

cd path/to/your/project/directory
python -m venv venv

On macOS and Linux:

source venv/bin/activate

On Windows (Command Prompt):

.\venv\Scripts\activate

On Windows (PowerShell):

.\venv\Scripts\Activate.ps1

Next, install all the necessary libraries:

pip -r requirements.txt

Next, set the PYTHONPATH to

On Windows

set PYTHONPATH=%PYTHONPATH%;path/to/your/project
set PYTHONPATH=%PYTHONPATH%;path/to/your/project/backend

On Unix or MacOS

export PYTHONPATH=$PYTHONPATH:/path/to/your/project
export PYTHONPATH=%PYTHONPATH%;path/to/your/project/backend

You can run the app from your terminal by executing the following command:

python backend\run.py

Below is the MVC architecture of our GitHub project:

GraphQL Endpoints

Get current user login:

/api/graphql/current-user-login

Sample Output:

{
  "viewer": {
    "login": "<your-username>"
  }
}

Get specific user login:

/api/graphql/user-login/<username>

Get list of all commits in a repo:

/api/graphql/specific-user-commits/<owner>/<repo_name>

Sample Output: {

   "repository": {
     "defaultBranchRef": {
       "target": {
         "history": {
           "nodes": [
             {
               "additions": 0,
               "author": {
                 "email": "61797592+Atharva7007@users.noreply.github.com",
                 "name": "Atharva Pansare",
                 "user": {
                   "login": "Atharva7007"
                 }
               },
               "authoredDate": "2020-04-03T09:30:17Z",
               "changedFilesIfAvailable": 1,
               "deletions": 0,
               "message": "Add files via upload",
               "parents": {
                 "totalCount": 1
               }
             }
           ],
           "pageInfo": {
             "endCursor": "98ba34a6c62ff6fe7c4d4de5c342a194f72d66e4 0",
             "hasNextPage": true
           },
           "totalCount": 6
         }
       }
     }
   }
 },

Get details of all contributors in a repo:

/api/graphql/repository-contributors/<owner>/<repo_name>

Sample Output: {

   "repository": {
     "defaultBranchRef": {
       "target": {
         "history": {
           "nodes": [
             {
               "author": {
                 "email": "61797592+Atharva7007@users.noreply.github.com",
                 "name": "Atharva Pansare",
                 "user": {
                   "login": "Atharva7007"
                 }
               }
             }
           ],
           "pageInfo": {
             "endCursor": "98ba34a6c62ff6fe7c4d4de5c342a194f72d66e4 0",
             "hasNextPage": true
           },
           "totalCount": 6
         }
       }
     }
   }
 }

REST API Endpoints

Get current user login:

/api/rest/current-user-login-rest

This endpoint retrieves the login information of the currently authenticated user. It fetches the user's profile data from the GitHub API and returns details such as their username, avatar URL, and other relevant information.

Sample Output:

{
  "avatar_url": "https://avatars.githubusercontent.com/u/61797592?v=4",
  "bio": null,
  "blog": "",
  "company": null,
  "created_at": "2020-03-04T16:49:06Z",
  "email": null,
  "events_url": "https://api.github.com/users/Atharva7007/events{/privacy}",
  "followers": 1,
  "followers_url": "https://api.github.com/users/Atharva7007/followers",
  "following": 3,
  "following_url": "https://api.github.com/users/Atharva7007/following{/other_user}",
  "gists_url": "https://api.github.com/users/Atharva7007/gists{/gist_id}",
  "gravatar_id": "",
  "hireable": null,
  "html_url": "https://github.com/Atharva7007",
  "id": 61797592,
  "location": null,
  "login": "Atharva7007",
  "name": "Atharva Pansare",
  "node_id": "MDQ6VXNlcjYxNzk3NTky",
  "organizations_url": "https://api.github.com/users/Atharva7007/orgs",
  "public_gists": 0,
  "public_repos": 11,
  "received_events_url": "https://api.github.com/users/Atharva7007/received_events",
  "repos_url": "https://api.github.com/users/Atharva7007/repos",
  "site_admin": false,
  "starred_url": "https://api.github.com/users/Atharva7007/starred{/owner}{/repo}",
  "subscriptions_url": "https://api.github.com/users/Atharva7007/subscriptions",
  "twitter_username": null,
  "type": "User",
  "updated_at": "2024-03-14T19:03:46Z",
  "url": "https://api.github.com/users/Atharva7007"
}

Get list of all commits in a repo:

/api/rest/specific-user-commits/<owner>/<repo_name>

This endpoint retrieves a list of all commits made to a specific repository. It takes the repository owner and repository name as parameters. The endpoint fetches the commit history from the GitHub API and returns details about each commit, such as the author, commit message, and changes made.

Sample Output:

[
  {
    "repository": {
      "defaultBranchRef": {
        "target": {
          "history": {
            "nodes": [
              {
                "additions": 0,
                "author": {
                  "email": "61797592+Atharva7007@users.noreply.github.com",
                  "name": "Atharva Pansare",
                  "user": {
                    "login": "Atharva7007"
                  }
                },
                "authoredDate": "2020-04-03T09:30:17Z",
                "changedFilesIfAvailable": 1,
                "deletions": 0,
                "message": "Add files via upload",
                "parents": {
                  "totalCount": 1
                }
              }
            ],
            "pageInfo": {
              "endCursor": "98ba34a6c62ff6fe7c4d4de5c342a194f72d66e4 0",
              "hasNextPage": true
            },
            "totalCount": 6
          }
        }
      }
    }
  }
]

Get details of all contributors in a repo:

/api/rest/repository-contributors/<owner>/<repo_name>

This endpoint fetches the details of all contributors to a specific repository. It takes the repository owner and repository name as parameters. The endpoint retrieves the list of contributors from the GitHub API and returns information about each contributor, such as their username, avatar URL, and permissions within the repository.

Sample Output: [

 {
   "avatar_url": "https://avatars.githubusercontent.com/u/61797592?v=4",
   "events_url": "https://api.github.com/users/Atharva7007/events{/privacy}",
   "followers_url": "https://api.github.com/users/Atharva7007/followers",
   "following_url": "https://api.github.com/users/Atharva7007/following{/other_user}",
   "gists_url": "https://api.github.com/users/Atharva7007/gists{/gist_id}",
   "gravatar_id": "",
   "html_url": "https://github.com/Atharva7007",
   "id": 61797592,
   "login": "Atharva7007",
   "node_id": "MDQ6VXNlcjYxNzk3NTky",
   "organizations_url": "https://api.github.com/users/Atharva7007/orgs",
   "permissions": {
     "admin": true,
     "maintain": true,
     "pull": true,
     "push": true,
     "triage": true
   },
   "received_events_url": "https://api.github.com/users/Atharva7007/received_events",
   "repos_url": "https://api.github.com/users/Atharva7007/repos",
   "role_name": "admin",
   "site_admin": false,
   "starred_url": "https://api.github.com/users/Atharva7007/starred{/owner}{/repo}",
   "subscriptions_url": "https://api.github.com/users/Atharva7007/subscriptions",
   "type": "User",
   "url": "https://api.github.com/users/Atharva7007"
 }

]

Plan of Work

In the second phase of the project, we will focus on developing a robust and user-friendly frontend using React. The frontend will serve as an intuitive interface for displaying and interacting with the data retrieved from the GitHub API endpoints implemented in the first phase.

The plan includes the following key steps:

1. **Setup React Environment**: We will set up a modern React development environment, including the necessary tooling, dependencies, and libraries required for building a performant and scalable frontend application.

2. **Design UI Components**: We will design and develop reusable UI components to present the data from the API endpoints in a visually appealing and organized manner. These components will follow best practices for React component development, ensuring modularity, reusability, and maintainability.

3. **Integrate with API Endpoints**: The React frontend will integrate seamlessly with the existing API endpoints. We will implement efficient HTTP request handling mechanisms to fetch data from the endpoints and update the UI components accordingly.

4. **Implement Pagination**: For endpoints that return paginated data, we will implement intuitive pagination functionality in the React UI. This will allow users to navigate through the results smoothly, enhancing the overall user experience.

5. **Enhance User Experience**: We will focus on improving the user experience by adding features such as search functionality, filtering, sorting, and other relevant enhancements based on the data being displayed. These enhancements will empower users to interact with the data more effectively and derive valuable insights.

6. **Test and Debug**: Thorough testing and debugging will be a critical part of the development process. We will implement unit tests and end-to-end tests to ensure the React frontend functions correctly and consistently across different browsers and devices.

7. **Deploy and Document**: Finally, we will deploy the React frontend to a hosting platform, ensuring it is accessible and easily usable by end-users. Comprehensive documentation will be provided, including instructions for setup, usage, and any relevant information for future maintenance and development.

Implementation Details

The GitHub Miner project consists of two main components: the backend API and the frontend user interface.

The backend API is implemented using Flask, a lightweight Python web framework. It exposes two sets of endpoints: one for GraphQL queries and another for REST API queries. Flask Blueprints are used to route requests to the appropriate endpoints based on the URL path.

The GraphQL implementation utilizes the `graphene` library, which provides a way to define GraphQL schemas and resolvers. The resolvers are responsible for fetching data from the GitHub API using the `pygithub` library.

The REST implementation uses the `requests` library to make HTTP requests to the GitHub API and process the responses.

To ensure reliability and robustness, comprehensive testing strategies, including unit tests and integration tests, have been implemented. These tests validate the functionality of the API endpoints across different scenarios, ensuring the system's correctness and stability.

Implementation Details

Implementation

In the API, we have 2 separate endpoints to retrieve the same data: one for REST and the other using GraphQL queries. To manage this, we have made use of Flask Blueprints where all the "api/graphql" requests get routed to the graphql variants and the "api/rest" requests get routed to the REST variants.

Design Patterns used

1. The REST API Client /backend/app/services/github_query/github_rest/client.py is a Singleton.

Testing

To ensure the reliability and robustness of the GitHub Miner, comprehensive testing strategies, including unit tests and integration tests, have been implemented. These tests validate the functionality of the API endpoints across different scenarios, ensuring the system's correctness and stability. The testing suite covers the following areas:

Unit Tests

Unit tests have been developed to test individual components in isolation, ensuring that each part functions correctly on its own. This includes testing the initialization, argument formatting, field formatting, string representation, and equality of query nodes, as well as the correct behavior of the query builders and authenticators. Examples of unit tests include:

- **TestQueryNode**: Validates the initialization, argument formatting, and field formatting of query nodes.

- **TestQuery**: Ensures correct query initialization, argument substitution, and time formatting.

- **TestQueryNodePaginator**: Tests the functionality of the paginator, including initialization, updating, and resetting.

- **TestPaginatedQuery**: Validates the initialization and execution of paginated queries.

Integration Tests

Integration tests verify the interaction between different components of the system, ensuring that they work together as expected. This includes testing the behavior of the client when making actual requests to the GitHub API, handling authentication, executing queries, and processing paginated responses. Examples of integration tests include:

- **TestClient**: Tests the GitHub client's initialization, header generation, retry logic, query execution, and handling of paginated queries. - **Authentication Tests**: Validate the generation of correct authorization headers and the handling of personal access tokens.

These tests collectively ensure that the GitHub Miner operates reliably, providing accurate and timely data from GitHub's APIs. By covering a wide range of scenarios, from successful queries to error handling and rate limiting, the tests ensure that the application can be used confidently in production environments.

Team Members

Atharva Pansare

Sumedh Limburkar

Viraj Sanap

Mengning Li

Mentor: Jialin Cui

References

GitHub REST API documentation - https://docs.github.com/en/rest?apiVersion=2022-11-28