CSC/ECE 517 Spring 2024 - G2402 Implement REST client, REST API, and Graphql API endpoint for repositories: Difference between revisions
(Created page with "Placeholder") |
|||
(40 intermediate revisions by 4 users not shown) | |||
Line 1: | Line 1: | ||
__TOC__ | |||
== GitHub Miner == | |||
This is a convenient tool to query a user's GitHub metrics. The project aims to develop API endpoints for GitHub GraphQL queries and GitHub REST queries using Python Flask. It involves integrating existing code with Flask to expose GraphQL queries as API endpoints, developing a REST client, and creating REST endpoints for querying the same data sets. The project also includes thorough testing and documentation of the endpoints. | |||
==Live Demo & Source Code== | |||
http://152.7.177.239:5000/auth/login Link] | |||
(Please login using your personal GitHub accounts, not NCSU accounts) | |||
[https://github.ncsu.edu/slimbur/GH_Miner Source Code] | |||
==Installation== | |||
We recommend using virtual environment. Steps to set up virtual environment: | |||
cd path/to/your/project/directory | |||
python -m venv venv | |||
On macOS and Linux: | |||
source venv/bin/activate | |||
On Windows (Command Prompt): | |||
.\venv\Scripts\activate | |||
On Windows (PowerShell): | |||
.\venv\Scripts\Activate.ps1 | |||
Next, install all the necessary libraries: | |||
pip -r requirements.txt | |||
Next, set the PYTHONPATH to | |||
On Windows | |||
set PYTHONPATH=%PYTHONPATH%;path/to/your/project | |||
set PYTHONPATH=%PYTHONPATH%;path/to/your/project/backend | |||
On Unix or MacOS | |||
export PYTHONPATH=$PYTHONPATH:/path/to/your/project | |||
export PYTHONPATH=%PYTHONPATH%;path/to/your/project/backend | |||
You can run the app from your terminal by executing the following command: | |||
python backend\run.py | |||
==GraphQL Endpoints== | |||
Get current user login: | |||
/api/graphql/current-user-login | |||
Sample Output: | |||
{ | |||
"viewer": { | |||
"login": "<your-username>" | |||
} | |||
} | |||
Get specific user login: | |||
/api/graphql/user-login/<username> | |||
Get list of all commits in a repo: | |||
/api/graphql/specific-user-commits/<owner>/<repo_name> | |||
Sample Output: | |||
{ | |||
"repository": { | |||
"defaultBranchRef": { | |||
"target": { | |||
"history": { | |||
"nodes": [ | |||
{ | |||
"additions": 0, | |||
"author": { | |||
"email": "61797592+Atharva7007@users.noreply.github.com", | |||
"name": "Atharva Pansare", | |||
"user": { | |||
"login": "Atharva7007" | |||
} | |||
}, | |||
"authoredDate": "2020-04-03T09:30:17Z", | |||
"changedFilesIfAvailable": 1, | |||
"deletions": 0, | |||
"message": "Add files via upload", | |||
"parents": { | |||
"totalCount": 1 | |||
} | |||
} | |||
], | |||
"pageInfo": { | |||
"endCursor": "98ba34a6c62ff6fe7c4d4de5c342a194f72d66e4 0", | |||
"hasNextPage": true | |||
}, | |||
"totalCount": 6 | |||
} | |||
} | |||
} | |||
} | |||
}, | |||
Get details of all contributors in a repo: | |||
/api/graphql/repository-contributors/<owner>/<repo_name> | |||
Sample Output: | |||
{ | |||
"repository": { | |||
"defaultBranchRef": { | |||
"target": { | |||
"history": { | |||
"nodes": [ | |||
{ | |||
"author": { | |||
"email": "61797592+Atharva7007@users.noreply.github.com", | |||
"name": "Atharva Pansare", | |||
"user": { | |||
"login": "Atharva7007" | |||
} | |||
} | |||
} | |||
], | |||
"pageInfo": { | |||
"endCursor": "98ba34a6c62ff6fe7c4d4de5c342a194f72d66e4 0", | |||
"hasNextPage": true | |||
}, | |||
"totalCount": 6 | |||
} | |||
} | |||
} | |||
} | |||
} | |||
==REST API Endpoints== | |||
Get current user login: | |||
/api/rest/current-user-login-rest | |||
Sample Output: | |||
{ | |||
"avatar_url": "https://avatars.githubusercontent.com/u/61797592?v=4", | |||
"bio": null, | |||
"blog": "", | |||
"company": null, | |||
"created_at": "2020-03-04T16:49:06Z", | |||
"email": null, | |||
"events_url": "https://api.github.com/users/Atharva7007/events{/privacy}", | |||
"followers": 1, | |||
"followers_url": "https://api.github.com/users/Atharva7007/followers", | |||
"following": 3, | |||
"following_url": "https://api.github.com/users/Atharva7007/following{/other_user}", | |||
"gists_url": "https://api.github.com/users/Atharva7007/gists{/gist_id}", | |||
"gravatar_id": "", | |||
"hireable": null, | |||
"html_url": "https://github.com/Atharva7007", | |||
"id": 61797592, | |||
"location": null, | |||
"login": "Atharva7007", | |||
"name": "Atharva Pansare", | |||
"node_id": "MDQ6VXNlcjYxNzk3NTky", | |||
"organizations_url": "https://api.github.com/users/Atharva7007/orgs", | |||
"public_gists": 0, | |||
"public_repos": 11, | |||
"received_events_url": "https://api.github.com/users/Atharva7007/received_events", | |||
"repos_url": "https://api.github.com/users/Atharva7007/repos", | |||
"site_admin": false, | |||
"starred_url": "https://api.github.com/users/Atharva7007/starred{/owner}{/repo}", | |||
"subscriptions_url": "https://api.github.com/users/Atharva7007/subscriptions", | |||
"twitter_username": null, | |||
"type": "User", | |||
"updated_at": "2024-03-14T19:03:46Z", | |||
"url": "https://api.github.com/users/Atharva7007" | |||
} | |||
Get list of all commits in a repo: | |||
/api/rest/specific-user-commits/<owner>/<repo_name> | |||
Sample Output: | |||
[ | |||
{ | |||
"repository": { | |||
"defaultBranchRef": { | |||
"target": { | |||
"history": { | |||
"nodes": [ | |||
{ | |||
"additions": 0, | |||
"author": { | |||
"email": "61797592+Atharva7007@users.noreply.github.com", | |||
"name": "Atharva Pansare", | |||
"user": { | |||
"login": "Atharva7007" | |||
} | |||
}, | |||
"authoredDate": "2020-04-03T09:30:17Z", | |||
"changedFilesIfAvailable": 1, | |||
"deletions": 0, | |||
"message": "Add files via upload", | |||
"parents": { | |||
"totalCount": 1 | |||
} | |||
} | |||
], | |||
"pageInfo": { | |||
"endCursor": "98ba34a6c62ff6fe7c4d4de5c342a194f72d66e4 0", | |||
"hasNextPage": true | |||
}, | |||
"totalCount": 6 | |||
} | |||
} | |||
} | |||
} | |||
} | |||
] | |||
Get details of all contributors in a repo: | |||
/api/rest/repository-contributors/<owner>/<repo_name> | |||
Sample Output: | |||
[ | |||
{ | |||
"avatar_url": "https://avatars.githubusercontent.com/u/61797592?v=4", | |||
"events_url": "https://api.github.com/users/Atharva7007/events{/privacy}", | |||
"followers_url": "https://api.github.com/users/Atharva7007/followers", | |||
"following_url": "https://api.github.com/users/Atharva7007/following{/other_user}", | |||
"gists_url": "https://api.github.com/users/Atharva7007/gists{/gist_id}", | |||
"gravatar_id": "", | |||
"html_url": "https://github.com/Atharva7007", | |||
"id": 61797592, | |||
"login": "Atharva7007", | |||
"node_id": "MDQ6VXNlcjYxNzk3NTky", | |||
"organizations_url": "https://api.github.com/users/Atharva7007/orgs", | |||
"permissions": { | |||
"admin": true, | |||
"maintain": true, | |||
"pull": true, | |||
"push": true, | |||
"triage": true | |||
}, | |||
"received_events_url": "https://api.github.com/users/Atharva7007/received_events", | |||
"repos_url": "https://api.github.com/users/Atharva7007/repos", | |||
"role_name": "admin", | |||
"site_admin": false, | |||
"starred_url": "https://api.github.com/users/Atharva7007/starred{/owner}{/repo}", | |||
"subscriptions_url": "https://api.github.com/users/Atharva7007/subscriptions", | |||
"type": "User", | |||
"url": "https://api.github.com/users/Atharva7007" | |||
} | |||
] | |||
== Implementation Details == | |||
=== Implementation === | |||
In the API, we have 2 separate endpoints to retrieve the same data: one for REST and the other using GraphQL queries. To manage this, we have made use of Flask Blueprints where all the "api/graphql" requests get routed to the graphql variants and the "api/rest" requests get routed to the REST variants. | |||
=== Design Patterns used === | |||
# The REST API Client '''/backend/app/services/github_query/github_rest/client.py''' is a Singleton. | |||
== Testing == | |||
To ensure the reliability and robustness of the GitHub Miner, comprehensive testing strategies, including unit tests and integration tests, have been implemented. These tests validate the functionality of the API endpoints across different scenarios, ensuring the system's correctness and stability. The testing suite covers the following areas: | |||
=== Unit Tests === | |||
Unit tests have been developed to test individual components in isolation, ensuring that each part functions correctly on its own. This includes testing the initialization, argument formatting, field formatting, string representation, and equality of query nodes, as well as the correct behavior of the query builders and authenticators. Examples of unit tests include: | |||
- **TestQueryNode**: Validates the initialization, argument formatting, and field formatting of query nodes. | |||
- **TestQuery**: Ensures correct query initialization, argument substitution, and time formatting. | |||
- **TestQueryNodePaginator**: Tests the functionality of the paginator, including initialization, updating, and resetting. | |||
- **TestPaginatedQuery**: Validates the initialization and execution of paginated queries. | |||
=== Integration Tests === | |||
Integration tests verify the interaction between different components of the system, ensuring that they work together as expected. This includes testing the behavior of the client when making actual requests to the GitHub API, handling authentication, executing queries, and processing paginated responses. Examples of integration tests include: | |||
- **TestClient**: Tests the GitHub client's initialization, header generation, retry logic, query execution, and handling of paginated queries. | |||
- **Authentication Tests**: Validate the generation of correct authorization headers and the handling of personal access tokens. | |||
These tests collectively ensure that the GitHub Miner operates reliably, providing accurate and timely data from GitHub's APIs. By covering a wide range of scenarios, from successful queries to error handling and rate limiting, the tests ensure that the application can be used confidently in production environments. | |||
== Team Members == | |||
Atharva Pansare | |||
Sumedh Limburkar | |||
Viraj Sanap | |||
Mengning Li | |||
Mentor: Jialin Cui | |||
== References == | |||
GitHub REST API documentation - https://docs.github.com/en/rest?apiVersion=2022-11-28 |
Latest revision as of 20:48, 7 April 2024
GitHub Miner
This is a convenient tool to query a user's GitHub metrics. The project aims to develop API endpoints for GitHub GraphQL queries and GitHub REST queries using Python Flask. It involves integrating existing code with Flask to expose GraphQL queries as API endpoints, developing a REST client, and creating REST endpoints for querying the same data sets. The project also includes thorough testing and documentation of the endpoints.
Live Demo & Source Code
http://152.7.177.239:5000/auth/login Link] (Please login using your personal GitHub accounts, not NCSU accounts)
Installation
We recommend using virtual environment. Steps to set up virtual environment:
cd path/to/your/project/directory python -m venv venv
On macOS and Linux:
source venv/bin/activate
On Windows (Command Prompt):
.\venv\Scripts\activate
On Windows (PowerShell):
.\venv\Scripts\Activate.ps1
Next, install all the necessary libraries:
pip -r requirements.txt
Next, set the PYTHONPATH to
On Windows
set PYTHONPATH=%PYTHONPATH%;path/to/your/project set PYTHONPATH=%PYTHONPATH%;path/to/your/project/backend
On Unix or MacOS
export PYTHONPATH=$PYTHONPATH:/path/to/your/project export PYTHONPATH=%PYTHONPATH%;path/to/your/project/backend
You can run the app from your terminal by executing the following command:
python backend\run.py
GraphQL Endpoints
Get current user login:
/api/graphql/current-user-login
Sample Output:
{ "viewer": { "login": "<your-username>" } }
Get specific user login:
/api/graphql/user-login/<username>
Get list of all commits in a repo:
/api/graphql/specific-user-commits/<owner>/<repo_name>
Sample Output: {
"repository": { "defaultBranchRef": { "target": { "history": { "nodes": [ { "additions": 0, "author": { "email": "61797592+Atharva7007@users.noreply.github.com", "name": "Atharva Pansare", "user": { "login": "Atharva7007" } }, "authoredDate": "2020-04-03T09:30:17Z", "changedFilesIfAvailable": 1, "deletions": 0, "message": "Add files via upload", "parents": { "totalCount": 1 } } ], "pageInfo": { "endCursor": "98ba34a6c62ff6fe7c4d4de5c342a194f72d66e4 0", "hasNextPage": true }, "totalCount": 6 } } } } },
Get details of all contributors in a repo:
/api/graphql/repository-contributors/<owner>/<repo_name>
Sample Output: {
"repository": { "defaultBranchRef": { "target": { "history": { "nodes": [ { "author": { "email": "61797592+Atharva7007@users.noreply.github.com", "name": "Atharva Pansare", "user": { "login": "Atharva7007" } } } ], "pageInfo": { "endCursor": "98ba34a6c62ff6fe7c4d4de5c342a194f72d66e4 0", "hasNextPage": true }, "totalCount": 6 } } } } }
REST API Endpoints
Get current user login:
/api/rest/current-user-login-rest
Sample Output:
{ "avatar_url": "https://avatars.githubusercontent.com/u/61797592?v=4", "bio": null, "blog": "", "company": null, "created_at": "2020-03-04T16:49:06Z", "email": null, "events_url": "https://api.github.com/users/Atharva7007/events{/privacy}", "followers": 1, "followers_url": "https://api.github.com/users/Atharva7007/followers", "following": 3, "following_url": "https://api.github.com/users/Atharva7007/following{/other_user}", "gists_url": "https://api.github.com/users/Atharva7007/gists{/gist_id}", "gravatar_id": "", "hireable": null, "html_url": "https://github.com/Atharva7007", "id": 61797592, "location": null, "login": "Atharva7007", "name": "Atharva Pansare", "node_id": "MDQ6VXNlcjYxNzk3NTky", "organizations_url": "https://api.github.com/users/Atharva7007/orgs", "public_gists": 0, "public_repos": 11, "received_events_url": "https://api.github.com/users/Atharva7007/received_events", "repos_url": "https://api.github.com/users/Atharva7007/repos", "site_admin": false, "starred_url": "https://api.github.com/users/Atharva7007/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/Atharva7007/subscriptions", "twitter_username": null, "type": "User", "updated_at": "2024-03-14T19:03:46Z", "url": "https://api.github.com/users/Atharva7007" }
Get list of all commits in a repo:
/api/rest/specific-user-commits/<owner>/<repo_name>
Sample Output:
[ { "repository": { "defaultBranchRef": { "target": { "history": { "nodes": [ { "additions": 0, "author": { "email": "61797592+Atharva7007@users.noreply.github.com", "name": "Atharva Pansare", "user": { "login": "Atharva7007" } }, "authoredDate": "2020-04-03T09:30:17Z", "changedFilesIfAvailable": 1, "deletions": 0, "message": "Add files via upload", "parents": { "totalCount": 1 } } ], "pageInfo": { "endCursor": "98ba34a6c62ff6fe7c4d4de5c342a194f72d66e4 0", "hasNextPage": true }, "totalCount": 6 } } } } } ]
Get details of all contributors in a repo:
/api/rest/repository-contributors/<owner>/<repo_name>
Sample Output: [
{ "avatar_url": "https://avatars.githubusercontent.com/u/61797592?v=4", "events_url": "https://api.github.com/users/Atharva7007/events{/privacy}", "followers_url": "https://api.github.com/users/Atharva7007/followers", "following_url": "https://api.github.com/users/Atharva7007/following{/other_user}", "gists_url": "https://api.github.com/users/Atharva7007/gists{/gist_id}", "gravatar_id": "", "html_url": "https://github.com/Atharva7007", "id": 61797592, "login": "Atharva7007", "node_id": "MDQ6VXNlcjYxNzk3NTky", "organizations_url": "https://api.github.com/users/Atharva7007/orgs", "permissions": { "admin": true, "maintain": true, "pull": true, "push": true, "triage": true }, "received_events_url": "https://api.github.com/users/Atharva7007/received_events", "repos_url": "https://api.github.com/users/Atharva7007/repos", "role_name": "admin", "site_admin": false, "starred_url": "https://api.github.com/users/Atharva7007/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/Atharva7007/subscriptions", "type": "User", "url": "https://api.github.com/users/Atharva7007" }
]
Implementation Details
Implementation
In the API, we have 2 separate endpoints to retrieve the same data: one for REST and the other using GraphQL queries. To manage this, we have made use of Flask Blueprints where all the "api/graphql" requests get routed to the graphql variants and the "api/rest" requests get routed to the REST variants.
Design Patterns used
- The REST API Client /backend/app/services/github_query/github_rest/client.py is a Singleton.
Testing
To ensure the reliability and robustness of the GitHub Miner, comprehensive testing strategies, including unit tests and integration tests, have been implemented. These tests validate the functionality of the API endpoints across different scenarios, ensuring the system's correctness and stability. The testing suite covers the following areas:
Unit Tests
Unit tests have been developed to test individual components in isolation, ensuring that each part functions correctly on its own. This includes testing the initialization, argument formatting, field formatting, string representation, and equality of query nodes, as well as the correct behavior of the query builders and authenticators. Examples of unit tests include:
- **TestQueryNode**: Validates the initialization, argument formatting, and field formatting of query nodes.
- **TestQuery**: Ensures correct query initialization, argument substitution, and time formatting.
- **TestQueryNodePaginator**: Tests the functionality of the paginator, including initialization, updating, and resetting.
- **TestPaginatedQuery**: Validates the initialization and execution of paginated queries.
Integration Tests
Integration tests verify the interaction between different components of the system, ensuring that they work together as expected. This includes testing the behavior of the client when making actual requests to the GitHub API, handling authentication, executing queries, and processing paginated responses. Examples of integration tests include:
- **TestClient**: Tests the GitHub client's initialization, header generation, retry logic, query execution, and handling of paginated queries. - **Authentication Tests**: Validate the generation of correct authorization headers and the handling of personal access tokens.
These tests collectively ensure that the GitHub Miner operates reliably, providing accurate and timely data from GitHub's APIs. By covering a wide range of scenarios, from successful queries to error handling and rate limiting, the tests ensure that the application can be used confidently in production environments.
Team Members
Atharva Pansare
Sumedh Limburkar
Viraj Sanap
Mengning Li
Mentor: Jialin Cui
References
GitHub REST API documentation - https://docs.github.com/en/rest?apiVersion=2022-11-28