CSC/ECE 517-GH-2401-Rest-GraphQL-Endpoints

Demo Video & Source Code

Youtube Link

Github Link (Requires NCSU email id)

Objective

The objective of the project is to be able to query and view statistics of a github user by the username and eventually export it so that the data can be used for other mining or analysis purposes

About

The project started with developing API endpoints for invoking GitHub GraphQL queries and GitHub REST queries and in the latest release, we have developed a User Interface to do the same. The tech stack used is Flask for backend and React+Typescript for frontend. We have implemented endpoints for 4 query elements: comment, contributions, profiles and time_range_contributions. Each element contains multiple queries and each query has a separate endpoints for both GraphQL and REST API implemented. The data can also be queried by simply hitting the REST API i.e. without using the UI

Setup

Software Requirements

• Python 3.9 or above (Required for development only)
• node and npm (Required for development only)
• Git
• VSCode
• formatter plugin (autopep8)
• Docker

Setup and Running Guide

• Clone the project repository "https://github.ncsu.edu/opjain/GH_Miner"
• Add your github client secret and client id in a .env file in the backend folder

GITHUB_OAUTH_CLIENT_ID=<your client id>

GITHUB_OAUTH_CLIENT_SECRET=<your client secret>

SECRET_KEY=<any string but empty not allowed>

MYSQL_DATABASE_PASSWORD=<your password>

Please note that the redirect URL in the Github OAuth app should point to {frontend_url:port}/login for a frontend to backend OAuth flow. If you only plan to use the backend API and not the frontend client, the redirect URL should be {backend_url:port}/auth/login/authorize

Running with docker compose

• Open the project folder and start the whole application using docker compose

docker compose build

• To run the docker image

docker compose up

You can find the port forwarding-related configuration in the docker-compose.yml file. You can also use "-d" argument in the command to run the image in background

Running individual docker containers

To build and run the backend

cd GH_Miner docker build -t ghmb .

docker run -p 5000:5000 ghmb

• To build and run the frontend

cd GH_Miner/frontend

docker build -t ghmf .

docker run -p 8080:8080 ghmf

The "ghmb" and "ghmf" are just arbitrary names for the docker images, you can use any name of your choice

Running without docker (Only recommended during development)

• Open up terminal in the project folder "GH_Miner"

Backend

• Create a python virtual environment and activate it

python -m venv venv

1. If you're on windows

.\venv\Scripts\Activate

1. If you're on linux / Mac

source ./venv/bin/activate

• Install all the dependencies

pip install -r requirements.txt

• Run the backend flask app

python -m backend.run

Frontend

• Given that you're already into the GH_Miner folder, go into the frontend folder and install dependencies

cd frontend

npm install

• Run the react app

npm run dev

Additionally, you can refer to the scripts section in package.json file

• build - "npm run build" This will generate a production build of the React.js application in a folder "dist" in the frontend directory
• preview - "npm run preview" This will run the production build of the React.js application

System Architecture

Below is the system architecture of our project:

Our system follows the Model-View-Controller architecture. Frontend is the View with which user will interact. Once user invokes an endpoint, we get to the github_routes file which acts as the controller of the system, which find the appropriate route for the method from the graphql_services which is the model of the system. Once the relevant method is called, model returns the API response to the controller and then this response is finally sent to the view which is our frontend.

The following is the OAuth Authentication flow

Frontend

Below is our layout of our application, which is similar to the GitHub UI:

This is the welcome page of our application. This has a login button.

Once you click on the login button, this page is displayed, where the application would ask for the required permissions

Once the user clicks on login, below page is displayed where username and user's top repositories are displayed.

Once the user clicks on Repositories button on the nav bar, below page is displayed where all the user's repositories are displayed.

Once the user clicks on Time Range contributions button on the nav bar, below page is displayed where for a particular start and end time all the user's contributions are displayed once the user selects the start and end date. To download data of multiple users, the user can upload a file with the format of each line as <github_username,starttime,endtime> (example: jashgopani,2024-01-01T00:09,2024-05-01T00:10)

Once the user searches for someone else in the search box on the nav bar, below page is displayed where username and user's top repositories are displayed.

The user interface includes the following pages -

• Login page - GitHub Oauth is used to authenticate users with GitHub. The user is asked to enter the GitHub username and will be redirected to GitHub for the required permissions to access the data. Once authenticated, the user will be redirected to the profile page.

• Profile page - This page contains the user statistics - profile image, bio, number of followers, popular repositories, contribution data, etc. The user can view the statistics of each of the popular repos - number of forks, number of watchers, repository access status, etc. The contribution graph will give insights into the number of contributions on a particular day.

• Repositories page - This page will list down all the repositories of an authenticated user.

• Time-range contributions - This page will display the statistics of the required user within the desired period of time. The user has to provide the GitHub username, start date, and end date of the user.

• Logout - This will log out the user from our application.

File Structure

Components

• RepositoryCard: This adaptable element is crafted to showcase information in a rectangular box layout, catering to a range of functions like presenting statistics, repositories, and item lists across different sections of the application. Each card features a title, description, star and fork counts for the repository, as well as indicating its accessibility status (Private or Public).

• SearchBox: A Higher Order Component (HOC) tailored to accommodate relevant input fields and a submit button component. It facilitates search operations by allowing users to input parameters such as usernames, start dates, and end dates for filtering contributions data within a specified time range.

• Navbar: This versatile component acts as the application's navigation bar, providing convenient links to all sections/pages within the interface. It utilizes react-router for seamless state management, ensuring smooth transitions between different sections/pages of the app. Users can effortlessly navigate to the Profile page, Repository page, and Time-range contributions page. Moreover, it features a sign-out button for logging out from the application, along with a search box allowing users to search for other users' profiles using their GitHub usernames.

Pages

• LoginPage: This page provides a summary of the user's profile statistics, displaying their GitHub username, follower count, following count, and six highlighted repositories.

• ProfilePage: This page presents an overview of the user's profile statistics, including their GitHub username, follower count, following count, and six highlighted repositories.

• RepositoryPage: This page will display a list of all repositories belonging to the user. The user will have the ability to view the repository's name, the number of stars it has received, the number of times it has been forked, and the date on which the repository was created.

• UserContributionPage: This page presents comprehensive metrics within the specified date and time parameters. It facilitates a comprehensive understanding of various aspects, including the cumulative number of commits, open issues, pull requests, and total pull request reviews, among others.

Utils

• AxiosInstance: Utilizing the "axios" library, AxiosInstance is a dedicated file within the project aimed at creating and configuring an instance of axios for handling HTTP requests. This file establishes default headers, including authentication tokens and request types, along with a base URL. Adopting the singleton pattern, this single instance ensures consistency and efficiency throughout the project.

• Constants: To maintain consistency and minimize naming errors, the Constants file serves as a repository for frequently used terms within the project. By storing and exporting these terms, the code becomes standardized, promoting clarity and ease of maintenance.

Endpoints

Below are the endpoints implemented in the flask backend

All the endpoints except the home and login are protected

Authentication

Login

'/auth/login'

Visiting this endpoint will take the user to github login page, once authenticated the user will be redirected to /index page

Logout

'/auth/logout'

To terminate the current session, user can visit this endpoint

Login Client

'/auth/login/client'

Only to be called by frontend with a query param "code" received from github

View Current User

The below endpoint returns the login name of the currently authenticated user

'/api/graphql/current-user-login'

{
  "viewer": {
    "login": "omjain2001"
  }
}

Comments

User Gists

This query returns the comments in gists of the authenticated user.

GraphQL Endpoint

'/api/graphql/user-gist-comments/'

{
[
  {
    "user": {
      "gistComments": {
        "nodes": [],
        "pageInfo": {
          "endCursor": null,
          "hasNextPage": false
        },
        "totalCount": 0
      },
      "login": "omjain2001"
    }
  }
]
}

REST Endpoint

'/api/rest/user-gist-comments/'

{
[
  {
    "login": "omjain2001",
    "pageInfo": {
      "hasNextPage": false,
      "totalCount": 0
    },
    "user": {
      "gistComments": {
        "nodes": []
      }
    }
  }
]
}

User Commits

This query returns the comments in all commits of the authenticated user.

GraphQL Endpoint

'/api/graphql/user-commit-comments/'

{
[
  {
    "user": {
      "commitComments": {
        "nodes": [],
        "pageInfo": {
          "endCursor": null,
          "hasNextPage": false
        },
        "totalCount": 0
      },
      "login": "omjain2001"
    }
  }
]

}

REST Endpoint

'/api/rest/user-commit-comments/'

{
[
  {
    "login": "omjain2001",
    "pageInfo": {
      "hasNextPage": false,
      "totalCount": 0
    },
    "user": {
      "commitComments": {
        "nodes": []
      }
    }
  }
]
}

User Issues

This query returns the comments in issues and pull requests of all public repositories of the authenticated user.

GraphQL Endpoint

'/api/graphql/user-issue-comments'

{
[
  {
    "user": {
      "issueComments": {
        "nodes": [],
        "pageInfo": {
          "endCursor": null,
          "hasNextPage": false
        },
        "totalCount": 0
      },
      "login": "omjain2001"
    }
  }
]

}

REST Endpoint

'/api/rest/user-issue-comments'

{
[
  {
    "login": "omjain2001",
    "pageInfo": {
      "hasNextPage": false,
      "totalCount": 7
    },
    "user": {
      "issueComments": {
        "nodes": [
          {
            "created_at": "2021-08-31T12:46:01Z"
          },
          {
            "created_at": "2021-08-31T12:46:02Z"
          },
          {
            "created_at": "2021-08-31T13:38:09Z"
          }
        ]
      }
    }
  }
]
}

Contributions

User Gists

This query returns the gists of the authenticated user.

GraphQL Endpoint

'/api/graphql/user-gists/<username>'

REST Endpoint

'/api/rest/user-gists/<username>'

{
[
  {
    "comments": 0,
    "comments_url": "https://api.github.com/gists/c3432f34dc2917bd03aa5a0b2a7e0816/comments",
    "commits_url": "https://api.github.com/gists/c3432f34dc2917bd03aa5a0b2a7e0816/commits",
    "created_at": "2022-12-07T15:11:12Z",
    "description": "Advent of Code (Day 6)",
    "files": {
      "main.go": {
        "filename": "main.go",
        "language": "Go",
        "raw_url": "https://gist.githubusercontent.com/kenoir/c3432f34dc2917bd03aa5a0b2a7e0816/raw/4e167f92cf844dfadf95d9a7aa4ab596d102b3af/main.go",
        "size": 1251,
        "type": "text/plain"
      }
    }
.......
]
}

User Issues

This query returns the issues of the authenticated user.

GraphQL Endpoint

'/api/graphql/user-issues/<username>'

{
[
  {
    "createdAt": "2019-09-07T11:39:21Z"
  },
  {
    "createdAt": "2020-06-08T12:25:17Z"
  },
  {
    "createdAt": "2020-06-12T12:05:17Z"
  },
  {
    "createdAt": "2020-06-23T08:13:02Z"
  },
  {
    "createdAt": "2020-06-28T10:27:58Z"
  }
  
]
}

REST Endpoint

'/api/rest/user-issues/<username>'

User Pull Requests

This query returns the pull requests of the authenticated user. For the REST query, we can only fetch pull requests for a particular repository for the authenticated user.

GraphQL Endpoint

'/api/graphql/user-pull-requests/<username>'

REST Endpoint

'/api/rest/user-pull-requests/<username>/<repo>'

User Repositories

This query returns the repositories of the authenticated user.

GraphQL Endpoint

'/api/graphql/user-repositories/<username>'

REST Endpoint

'/api/rest/user-repositories/<username>'

{
    "allow_forking": true,
    "archive_url": "https://api.github.com/repos/omjain2001/Covid-19/{archive_format}{/ref}",
    "archived": false,
    "assignees_url": "https://api.github.com/repos/omjain2001/Covid-19/assignees{/user}",
    "blobs_url": "https://api.github.com/repos/omjain2001/Covid-19/git/blobs{/sha}",
    "branches_url": "https://api.github.com/repos/omjain2001/Covid-19/branches{/branch}",
    "clone_url": "https://github.com/omjain2001/Covid-19.git",
    "collaborators_url": "https://api.github.com/repos/omjain2001/Covid-19/collaborators{/collaborator}",
    "comments_url": "https://api.github.com/repos/omjain2001/Covid-19/comments{/number}",
    "commits_url": "https://api.github.com/repos/omjain2001/Covid-19/commits{/sha}",
    "compare_url": "https://api.github.com/repos/omjain2001/Covid-19/compare/{base}...{head}",
    "contents_url": "https://api.github.com/repos/omjain2001/Covid-19/contents/{+path}",
    "contributors_url": "https://api.github.com/repos/omjain2001/Covid-19/contributors",
    "created_at": "2020-05-08T16:12:28Z",
    "default_branch": "master",
    "deployments_url": "https://api.github.com/repos/omjain2001/Covid-19/deployments",
    "description": "Website For COVID-19",
    "disabled": false,
    "downloads_url": "https://api.github.com/repos/omjain2001/Covid-19/downloads",
    "events_url": "https://api.github.com/repos/omjain2001/Covid-19/events",
    "fork": false,
    "forks": 1,
    "forks_count": 1,
    "forks_url": "https://api.github.com/repos/omjain2001/Covid-19/forks",
    "full_name": "omjain2001/Covid-19",
    "git_commits_url": "https://api.github.com/repos/omjain2001/Covid-19/git/commits{/sha}",
    "git_refs_url": "https://api.github.com/repos/omjain2001/Covid-19/git/refs{/sha}",
    "git_tags_url": "https://api.github.com/repos/omjain2001/Covid-19/git/tags{/sha}",
    "git_url": "git://github.com/omjain2001/Covid-19.git",
    "has_discussions": false,
    "has_downloads": true,
    "has_issues": true,
    "has_pages": true,
    "has_projects": true,
    "has_wiki": true,
    "homepage": null,
    "hooks_url": "https://api.github.com/repos/omjain2001/Covid-19/hooks",
    "html_url": "https://github.com/omjain2001/Covid-19",
    "id": 262369559,
    "is_template": false,
    "issue_comment_url": "https://api.github.com/repos/omjain2001/Covid-19/issues/comments{/number}",
    "issue_events_url": "https://api.github.com/repos/omjain2001/Covid-19/issues/events{/number}",
    "issues_url": "https://api.github.com/repos/omjain2001/Covid-19/issues{/number}",
    "keys_url": "https://api.github.com/repos/omjain2001/Covid-19/keys{/key_id}",
    "labels_url": "https://api.github.com/repos/omjain2001/Covid-19/labels{/name}",
    "language": "HTML",
    "languages_url": "https://api.github.com/repos/omjain2001/Covid-19/languages",
    "license": null,
    "merges_url": "https://api.github.com/repos/omjain2001/Covid-19/merges",
    "milestones_url": "https://api.github.com/repos/omjain2001/Covid-19/milestones{/number}",
    "mirror_url": null,
    "name": "Covid-19",
    "node_id": "MDEwOlJlcG9zaXRvcnkyNjIzNjk1NTk=",
    "notifications_url": "https://api.github.com/repos/omjain2001/Covid-19/notifications{?since,all,participating}",
    "open_issues": 0,
    "open_issues_count": 0,
    "owner": {
      "avatar_url": "https://avatars.githubusercontent.com/u/60605251?v=4",
      "events_url": "https://api.github.com/users/omjain2001/events{/privacy}",
      "followers_url": "https://api.github.com/users/omjain2001/followers",
      "following_url": "https://api.github.com/users/omjain2001/following{/other_user}",
      "gists_url": "https://api.github.com/users/omjain2001/gists{/gist_id}",
      "gravatar_id": "",
      "html_url": "https://github.com/omjain2001",
      "id": 60605251,
      "login": "omjain2001",
      "node_id": "MDQ6VXNlcjYwNjA1MjUx",
      "organizations_url": "https://api.github.com/users/omjain2001/orgs",
      "received_events_url": "https://api.github.com/users/omjain2001/received_events",
      "repos_url": "https://api.github.com/users/omjain2001/repos",
      "site_admin": false,
      "starred_url": "https://api.github.com/users/omjain2001/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/omjain2001/subscriptions",
      "type": "User",
      "url": "https://api.github.com/users/omjain2001"
    },
    "permissions": {
      "admin": true,
      "maintain": true,
      "pull": true,
      "push": true,
      "triage": true
    },
    "private": false,
    "pulls_url": "https://api.github.com/repos/omjain2001/Covid-19/pulls{/number}",
    "pushed_at": "2021-07-14T05:08:23Z",
    "releases_url": "https://api.github.com/repos/omjain2001/Covid-19/releases{/id}",
    "size": 7759,
    "ssh_url": "git@github.com:omjain2001/Covid-19.git",
    "stargazers_count": 0,
    "stargazers_url": "https://api.github.com/repos/omjain2001/Covid-19/stargazers",
    "statuses_url": "https://api.github.com/repos/omjain2001/Covid-19/statuses/{sha}",
    "subscribers_url": "https://api.github.com/repos/omjain2001/Covid-19/subscribers",
    "subscription_url": "https://api.github.com/repos/omjain2001/Covid-19/subscription",
    "svn_url": "https://github.com/omjain2001/Covid-19",
    "tags_url": "https://api.github.com/repos/omjain2001/Covid-19/tags",
    "teams_url": "https://api.github.com/repos/omjain2001/Covid-19/teams",
    "topics": [],
    "trees_url": "https://api.github.com/repos/omjain2001/Covid-19/git/trees{/sha}",
    "updated_at": "2021-07-15T06:33:18Z",
    "url": "https://api.github.com/repos/omjain2001/Covid-19",
    "visibility": "public",
    "watchers": 0,
    "watchers_count": 0,
    "web_commit_signoff_required": false
}

User Repository Discussions

This query returns the repository discussions of the authenticated user. For the REST query, there is no REST query to fetch repository discussions, hence this endpoint does not return any results.

GraphQL Endpoint

'/api/graphql/user-repository-discussions/<username>'

Profiles

This endpoint returns all the profile statistics (followers, gists, issues, pull requests, repositories, etc.) of the required user. The user must specify the username in this query.

GraphQL Endpoint

'/api/graphql/user-stats/<username>'

REST Endpoint

'/api/rest/user-stats/<username>'

{

  "commit_comments": 0,
  "company": "omjain2001",
  "created_at": "2020-02-03T10:55:04Z",
  "followers": 11,
  "following": 0,
  "gist_comments": 0,
  "gists": 0,
  "github": "omjain2001",
  "issue_comments": 0,
  "issues": 0,
  "projects": 1,
  "pull_requests": 8,
  "repositories": 26,
  "starred_repositories": 1,
  "watching": 21
}

Time Range Contributions

This endpoint returns all the details of the user contributions within a specified duration. The user must specify the username, start date, and end date in this query.

GraphQL Endpoint

'/api/graphql/user-contributions/<username>'

{
  "commit": 37,
  "issue": 0,
  "pr": 8,
  "pr_review": 1,
  "repository": 4,
  "res_con": 0
}

REST Endpoint

'/api/rest/user-contributions/<username>'

Query Params

• start: DateTime (ISO 8601 format)
• end: DateTime (ISO 8601 format)

{
  "commit": 0,
  "issue": 0,
  "pr": 0,
  "pr_review": 6,
  "repository": 30,
  "res_con": 0
}

Implementation Details and Use of LLM

Implementation

Since there were two types of endpoints for each API, i.e. graphql and rest variant, we unified the endpoints to have the api_type as a dynamic parameter. Furthermore, we route the request to the respective service based on the api_type value we get from the URL.

Design Patterns used

1. The REST API Client /backend/app/services/github_query/github_rest/client.py is a Singleton since we only needed a single instance of github object from the pyGithub library throughout the lifecycle of the application

1. The paginate_and_format_comments method declared in the github_rest_services.py is a very well thought and formulated method for performing pagination and transforming and formatting the REST api output like graphQL output. It uses the dependency inversion principle i.e. it has a single responsibility of performing the task based on the given parameters, it doesn't hold any special processing logic of its own which makes it very scalable.

Testing

As of now, testing was performed manually as there was a dependency on Github Rest API and the output would vary for each input and all the routes are protected. The UI Testing was carried out manually as well by verifying the Login flow, Logout flow and by visually checking if the response data is rendered correctly on the screen or not

Login flow steps:

1. Run the whole application and Open frontend App URL in browser
2. Login page should be loaded and a "Login with Github" button should be visible
3. Clicking the Login button should redirect the user to github login page (if user is not already signed in into github)
4. After successful login, user is presented with an Authorization page
5. On the Authorization page, when user clicks "Authorize", github redirects to the /login endpoint of the frontend with an authorization code in the "code" query param
6. Within few seconds, the user is redirected to "/profile" page where all the statistics are displayed and "Signout" button is also visible

Logout flow steps:

1. Once the user is logged in and a "Sign out" button is visible on the screen, clicking the signout button should clear the localStorge data and redirect the user to "/login" page
2. The user should not be able to visit any other links of the application like "/profile", "/repositories" etc. If the user tries to visit any URL, he/she should be redirected to "/login"

Use of LLM

We used CoPilot extension integrated in VS Code editor for code autocompletion, explanation and debugging throughout the project. Generating the dockerfile, boiler plate and even main logic for the API services, all of them were carried out with the help of CoPilot which made the development experience extremely smooth and helped us stay productive.

Team

Mentor

• Jialin Cui <jcui9@ncsu.edu>

Members

• Om Jain <opjain@ncsu.edu>
• Jash Gopani <jbgopani@ncsu.edu>
• Anshul Khairnar <akhairn@ncsu.edu>