CSC/ECE 517-GH-2401-Rest-GraphQL-Endpoints
Live Demo & Source Code
About
The project involves developing API endpoints for GitHub GraphQL queries and GitHub REST queries and to integrate these endpoints into the Python Flask framework, enabling the publication of these queries as accessible API endpoints via URLs. We have implemented endpoints for 4 query elements: comment, contributions, profiles and time_range_contributions. Each element contains multiple queries and each query has a separate endpoints for both GraphQL and REST API implemented.
Setup
Software Requirements
- Python 3.9 or above
- Git
- VSCode
- formatter plugin (autopep8)
- Docker
Steps
- Clone the project repository "https://github.ncsu.edu/opjain/GH_Miner"
- Add your github client secret and client id in a .env file in the backend folder
GITHUB_OAUTH_CLIENT_ID=<your client id>
GITHUB_OAUTH_CLIENT_SECRET=<your client secret>
SECRET_KEY=<any string but empty not allowed>
MYSQL_DATABASE_PASSWORD=<your password>
- Open the project folder and build the docker image using the following command
docker build -t gh_miner .
- To run the docker image
docker run -p 5000:5000 gh_miner
The "-p 5000:5000" is for port forwarding the 5000 port. You can also use "-d" argument in the command to run the image in background
Optionally, if you want to run without docker, you can follow the below steps
- Open up terminal in the project folder "GH_Miner"
- Create a python virtual environment and activate it
python -m venv venv
- If you're on windows
.\venv\Scripts\Activate
- If you're on linux / Mac
source ./venv/bin/activate
- Install all the dependencies
pip install -r requirements.txt
- Run the backend flask app
python -m backend.run
Frontend
Below is our planned layout, which is similar to the GitHub frontend:
Imageee
We are planning to create a login page and only authenticated user can access the home page. Once logged in, user can see 4 options on the navigation bar in the home page: Profile, Comments, Contributions and Time range contributions.
Endpoints
Below are the endpoints implemented in the flask backend
Firstly The user needs to authenticate inorder to access the API
Authentication
Login
'/auth/login'
Visiting this endpoint will take the user to github login page, once authenticated the user will be redirected to /index page
Logout
'/auth/logout'
To terminate the current session, user can visit this endpoint
View Current User
The below endpoint returns the login name of the currently authenticated user
'/api/graphql/current-user-login'
{
"viewer": {
"login": "omjain2001"
}
}
Comments
User Gists
This query returns the comments in gists of the authenticated user.
GraphQL Endpoint
'/api/graphql/user-gist-comments/'
{
[
{
"user": {
"gistComments": {
"nodes": [],
"pageInfo": {
"endCursor": null,
"hasNextPage": false
},
"totalCount": 0
},
"login": "omjain2001"
}
}
]
}
REST Endpoint
'/api/rest/user-gist-comments/'
{
[
{
"login": "omjain2001",
"pageInfo": {
"hasNextPage": false,
"totalCount": 0
},
"user": {
"gistComments": {
"nodes": []
}
}
}
]
}
User Commits
This query returns the comments in all commits of the authenticated user.
GraphQL Endpoint
'/api/graphql/user-commit-comments/'
{
[
{
"user": {
"commitComments": {
"nodes": [],
"pageInfo": {
"endCursor": null,
"hasNextPage": false
},
"totalCount": 0
},
"login": "omjain2001"
}
}
]
}
REST Endpoint
'/api/rest/user-commit-comments/'
{
[
{
"login": "omjain2001",
"pageInfo": {
"hasNextPage": false,
"totalCount": 0
},
"user": {
"commitComments": {
"nodes": []
}
}
}
]
}
User Issues
This query returns the comments in issues and pull requests of all public repositories of the authenticated user.
GraphQL Endpoint
'/api/graphql/user-issue-comments'
{
[
{
"user": {
"issueComments": {
"nodes": [],
"pageInfo": {
"endCursor": null,
"hasNextPage": false
},
"totalCount": 0
},
"login": "omjain2001"
}
}
]
}
REST Endpoint
'/api/rest/user-issue-comments'
{
[
{
"login": "omjain2001",
"pageInfo": {
"hasNextPage": false,
"totalCount": 7
},
"user": {
"issueComments": {
"nodes": [
{
"created_at": "2021-08-31T12:46:01Z"
},
{
"created_at": "2021-08-31T12:46:02Z"
},
{
"created_at": "2021-08-31T13:38:09Z"
}
]
}
}
}
]
}
Contributions
User Gists
This query returns the gists of the authenticated user.
GraphQL Endpoint
'/api/graphql/user-gists/<username>'
REST Endpoint
'/api/rest/user-gists/<username>'
{
[
{
"comments": 0,
"comments_url": "https://api.github.com/gists/c3432f34dc2917bd03aa5a0b2a7e0816/comments",
"commits_url": "https://api.github.com/gists/c3432f34dc2917bd03aa5a0b2a7e0816/commits",
"created_at": "2022-12-07T15:11:12Z",
"description": "Advent of Code (Day 6)",
"files": {
"main.go": {
"filename": "main.go",
"language": "Go",
"raw_url": "https://gist.githubusercontent.com/kenoir/c3432f34dc2917bd03aa5a0b2a7e0816/raw/4e167f92cf844dfadf95d9a7aa4ab596d102b3af/main.go",
"size": 1251,
"type": "text/plain"
}
}
.......
]
}
User Issues
This query returns the issues of the authenticated user.
GraphQL Endpoint
'/api/graphql/user-issues/<username>'
{
[
{
"createdAt": "2019-09-07T11:39:21Z"
},
{
"createdAt": "2020-06-08T12:25:17Z"
},
{
"createdAt": "2020-06-12T12:05:17Z"
},
{
"createdAt": "2020-06-23T08:13:02Z"
},
{
"createdAt": "2020-06-28T10:27:58Z"
}
]
}
REST Endpoint
'/api/rest/user-issues/<username>'
User Pull Requests
This query returns the pull requests of the authenticated user. For the REST query, we can only fetch pull requests for a particular repository for the authenticated user.
GraphQL Endpoint
'/api/graphql/user-pull-requests/<username>'
REST Endpoint
'/api/rest/user-pull-requests/<username>/<repo>'
User Repositories
This query returns the repositories of the authenticated user.
GraphQL Endpoint
'/api/graphql/user-repositories/<username>'
REST Endpoint
'/api/rest/user-repositories/<username>'
{
"allow_forking": true,
"archive_url": "https://api.github.com/repos/omjain2001/Covid-19/{archive_format}{/ref}",
"archived": false,
"assignees_url": "https://api.github.com/repos/omjain2001/Covid-19/assignees{/user}",
"blobs_url": "https://api.github.com/repos/omjain2001/Covid-19/git/blobs{/sha}",
"branches_url": "https://api.github.com/repos/omjain2001/Covid-19/branches{/branch}",
"clone_url": "https://github.com/omjain2001/Covid-19.git",
"collaborators_url": "https://api.github.com/repos/omjain2001/Covid-19/collaborators{/collaborator}",
"comments_url": "https://api.github.com/repos/omjain2001/Covid-19/comments{/number}",
"commits_url": "https://api.github.com/repos/omjain2001/Covid-19/commits{/sha}",
"compare_url": "https://api.github.com/repos/omjain2001/Covid-19/compare/{base}...{head}",
"contents_url": "https://api.github.com/repos/omjain2001/Covid-19/contents/{+path}",
"contributors_url": "https://api.github.com/repos/omjain2001/Covid-19/contributors",
"created_at": "2020-05-08T16:12:28Z",
"default_branch": "master",
"deployments_url": "https://api.github.com/repos/omjain2001/Covid-19/deployments",
"description": "Website For COVID-19",
"disabled": false,
"downloads_url": "https://api.github.com/repos/omjain2001/Covid-19/downloads",
"events_url": "https://api.github.com/repos/omjain2001/Covid-19/events",
"fork": false,
"forks": 1,
"forks_count": 1,
"forks_url": "https://api.github.com/repos/omjain2001/Covid-19/forks",
"full_name": "omjain2001/Covid-19",
"git_commits_url": "https://api.github.com/repos/omjain2001/Covid-19/git/commits{/sha}",
"git_refs_url": "https://api.github.com/repos/omjain2001/Covid-19/git/refs{/sha}",
"git_tags_url": "https://api.github.com/repos/omjain2001/Covid-19/git/tags{/sha}",
"git_url": "git://github.com/omjain2001/Covid-19.git",
"has_discussions": false,
"has_downloads": true,
"has_issues": true,
"has_pages": true,
"has_projects": true,
"has_wiki": true,
"homepage": null,
"hooks_url": "https://api.github.com/repos/omjain2001/Covid-19/hooks",
"html_url": "https://github.com/omjain2001/Covid-19",
"id": 262369559,
"is_template": false,
"issue_comment_url": "https://api.github.com/repos/omjain2001/Covid-19/issues/comments{/number}",
"issue_events_url": "https://api.github.com/repos/omjain2001/Covid-19/issues/events{/number}",
"issues_url": "https://api.github.com/repos/omjain2001/Covid-19/issues{/number}",
"keys_url": "https://api.github.com/repos/omjain2001/Covid-19/keys{/key_id}",
"labels_url": "https://api.github.com/repos/omjain2001/Covid-19/labels{/name}",
"language": "HTML",
"languages_url": "https://api.github.com/repos/omjain2001/Covid-19/languages",
"license": null,
"merges_url": "https://api.github.com/repos/omjain2001/Covid-19/merges",
"milestones_url": "https://api.github.com/repos/omjain2001/Covid-19/milestones{/number}",
"mirror_url": null,
"name": "Covid-19",
"node_id": "MDEwOlJlcG9zaXRvcnkyNjIzNjk1NTk=",
"notifications_url": "https://api.github.com/repos/omjain2001/Covid-19/notifications{?since,all,participating}",
"open_issues": 0,
"open_issues_count": 0,
"owner": {
"avatar_url": "https://avatars.githubusercontent.com/u/60605251?v=4",
"events_url": "https://api.github.com/users/omjain2001/events{/privacy}",
"followers_url": "https://api.github.com/users/omjain2001/followers",
"following_url": "https://api.github.com/users/omjain2001/following{/other_user}",
"gists_url": "https://api.github.com/users/omjain2001/gists{/gist_id}",
"gravatar_id": "",
"html_url": "https://github.com/omjain2001",
"id": 60605251,
"login": "omjain2001",
"node_id": "MDQ6VXNlcjYwNjA1MjUx",
"organizations_url": "https://api.github.com/users/omjain2001/orgs",
"received_events_url": "https://api.github.com/users/omjain2001/received_events",
"repos_url": "https://api.github.com/users/omjain2001/repos",
"site_admin": false,
"starred_url": "https://api.github.com/users/omjain2001/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/omjain2001/subscriptions",
"type": "User",
"url": "https://api.github.com/users/omjain2001"
},
"permissions": {
"admin": true,
"maintain": true,
"pull": true,
"push": true,
"triage": true
},
"private": false,
"pulls_url": "https://api.github.com/repos/omjain2001/Covid-19/pulls{/number}",
"pushed_at": "2021-07-14T05:08:23Z",
"releases_url": "https://api.github.com/repos/omjain2001/Covid-19/releases{/id}",
"size": 7759,
"ssh_url": "git@github.com:omjain2001/Covid-19.git",
"stargazers_count": 0,
"stargazers_url": "https://api.github.com/repos/omjain2001/Covid-19/stargazers",
"statuses_url": "https://api.github.com/repos/omjain2001/Covid-19/statuses/{sha}",
"subscribers_url": "https://api.github.com/repos/omjain2001/Covid-19/subscribers",
"subscription_url": "https://api.github.com/repos/omjain2001/Covid-19/subscription",
"svn_url": "https://github.com/omjain2001/Covid-19",
"tags_url": "https://api.github.com/repos/omjain2001/Covid-19/tags",
"teams_url": "https://api.github.com/repos/omjain2001/Covid-19/teams",
"topics": [],
"trees_url": "https://api.github.com/repos/omjain2001/Covid-19/git/trees{/sha}",
"updated_at": "2021-07-15T06:33:18Z",
"url": "https://api.github.com/repos/omjain2001/Covid-19",
"visibility": "public",
"watchers": 0,
"watchers_count": 0,
"web_commit_signoff_required": false
}
User Repository Discussions
This query returns the repository discussions of the authenticated user. For the REST query, there is no REST query to fetch repository discussions, hence this endpoint does not return any results.
GraphQL Endpoint
'/api/graphql/user-repository-discussions/<username>'
Profiles
This endpoint returns all the profile statistics (followers, gists, issues, pull requests, repositories, etc.) of the required user. The user must specify the username in this query.
GraphQL Endpoint
'/api/graphql/user-stats/<username>'
REST Endpoint
'/api/rest/user-stats/<username>'
{
"commit_comments": 0,
"company": "omjain2001",
"created_at": "2020-02-03T10:55:04Z",
"followers": 11,
"following": 0,
"gist_comments": 0,
"gists": 0,
"github": "omjain2001",
"issue_comments": 0,
"issues": 0,
"projects": 1,
"pull_requests": 8,
"repositories": 26,
"starred_repositories": 1,
"watching": 21
}
Time Range Contributions
This endpoint returns all the details of the user contributions within a specified duration. The user must specify the username, start date, and end date in this query.
GraphQL Endpoint
'/api/graphql/user-contributions/<username>'
{
"commit": 37,
"issue": 0,
"pr": 8,
"pr_review": 1,
"repository": 4,
"res_con": 0
}
REST Endpoint
'/api/rest/user-contributions/<username>'
Query Params
- start: DateTime (ISO 8601 format)
- end: DateTime (ISO 8601 format)
{
"commit": 0,
"issue": 0,
"pr": 0,
"pr_review": 6,
"repository": 30,
"res_con": 0
}
Implementation Details and Use of LLM
Implementation
Since there were two types of endpoints for each API, i.e. graphql and rest variant, we unified the endpoints to have the api_type as a dynamic parameter. Furthermore, we route the request to the respective service based on the api_type value we get from the URL.
Design Patterns used
- The REST API Client /backend/app/services/github_query/github_rest/client.py is a Singleton since we only needed a single instance of github object from the pyGithub library throughout the lifecycle of the application
- The paginate_and_format_comments method declared in the github_rest_services.py is a very well thought and formulated method for performing pagination and transforming and formatting the REST api output like graphQL output. It uses the dependency inversion principle i.e. it has a single responsibility of performing the task based on the given parameters, it doesn't hold any special processing logic of its own which makes it very scalable.
Testing
As of now, testing was performed manually as there was a dependency on Github Rest API and the output would vary for each input. We might need to implement a mock service to unit test the endpoints automatically in the next phase.
Use of LLM
We used CoPilot extension integrated in VS Code editor for code autocompletion, explanation and debugging throughout the project. Generating the dockerfile, boiler plate and even main logic for the API services, all of them were carried out with the help of CoPilot which made the development experience extremely smooth and helped us stay productive.
Team
Mentor
- Jialin Cui <jcui9@ncsu.edu>
Members
- Om Jain <opjain@ncsu.edu>
- Jash Gopani <jbgopani@ncsu.edu>
- Anshul Khairnar <akhairn@ncsu.edu>