Jjshah: Created page with "= Evaluate Using LLMs Integration = == Team Information == '''Mentor:''' * Ed Gehringer '''Team Members:''' * Jay Shah (jjshah) * Rushabh Shah (rsshah7) * Uchswas Paul (upaul) '''Relevant Links:''' [https://github.com/expertiza/expertiza Link to Expertiza Repository] [https://github.com/jayjshah2000/expertiza Link to Forked Repository] [https://github.com/expertiza/expertiza/pull/2958 Link to Pull Request] == Expertiza Background == Expertiza is an open-sourc..."

2025-04-28T19:18:03Z

Created page with "= Evaluate Using LLMs Integration = == Team Information == '''Mentor:''' * Ed Gehringer '''Team Members:''' * Jay Shah (jjshah) * Rushabh Shah (rsshah7) * Uchswas Paul (upaul) '''Relevant Links:''' [https://github.com/expertiza/expertiza Link to Expertiza Repository] [https://github.com/jayjshah2000/expertiza Link to Forked Repository] [https://github.com/expertiza/expertiza/pull/2958 Link to Pull Request] == Expertiza Background == Expertiza is an open-sourc..."

New page

= Evaluate Using LLMs Integration =

== Team Information ==

'''Mentor:'''
* Ed Gehringer

'''Team Members:'''
* Jay Shah (jjshah)
* Rushabh Shah (rsshah7)
* Uchswas Paul (upaul)

'''Relevant Links:'''

[https://github.com/expertiza/expertiza Link to Expertiza Repository]

[https://github.com/jayjshah2000/expertiza Link to Forked Repository]

[https://github.com/expertiza/expertiza/pull/2958 Link to Pull Request]

== Expertiza Background ==

Expertiza is an open-source project built with the Ruby on Rails MVC framework.
It facilitates peer reviews and collaborative learning activities in educational environments.

== About "Evaluate Using LLMs" Feature ==

* This project adds a new "Evaluate using LLM" feature to Expertiza.
* It enables instructors and TAs to automate the evaluation of peer reviews using an external LLM (Language Model) API.
* The evaluated scores and comments are editable, similar to the manual review report.

== Problem Statement ==

'''Prior to this work:'''

*Review quality was graded manually by instructors/TAs.

*No automation or AI-assistance existed to assess reviews.

'''The goal:'''

* Send all review data to an external LLM.

* Retrieve scores and comments.

* Populate the review report dynamically.

* Allow instructor/TAs to edit or overwrite the LLM evaluation.

== Tasks ==

* Add "Evaluate using LLM" option in report dropdown.

* Create service object ('''LlmEvaluationService''') for API communication.

* Build new partial view ('''_llm_evaluation_report.html.erb''').

* Allow editing of scores and comments returned by LLM.

* Integrate LLM report with the existing report framework.

* Use a stubbed (fake) API response for now.

== Plan of Work ==

* Study how Review Report (ReviewResponseMap) is generated.
* Create service to send review data to API and receive evaluation.
* Develop view partial for LLM Evaluation Report.
* Modify report dropdown to include "Evaluate using LLM" option.
* Update ReportsController to support LLM Evaluation.

== Implementation ==

=== Service Object ===

'''File:''' app/services/llm_evaluation_service.rb

Collects review data, calls dummy API, returns structured evaluation.

=== New View ===

'''File:''' app/views/reports/_llm_evaluation_report.html.erb

Displays LLM-evaluated results.

Editable fields (Score, Comment).

Exportable to CSV.

=== Controller Modification ===

'''File:''' app/controllers/reports_controller.rb

'''Method Added:''' llm_evaluation_report

=== Dropdown Modification ===

'''File:''' app/views/reports/_searchbox.html.erb

Added "Evaluate using LLM" to the report selection.

=== API Stub ===

Fake JSON response mimicking future LLM output.

== Fake API Response ==

<pre> [ { "reviewer_name": "John Doe", "reviewer_id": 101, "team_reviewed": "Team Alpha", "reviews_done": 3, "score_awarded": 90, "avg_score": 88, "metrics": "Volume: 45 words", "grade_for_reviewer": 95, "comment_for_reviewer": "Great feedback with actionable suggestions." } ] </pre>

== Test Plan ==

* Verified dropdown functionality.

* Verified table rendering and editable fields.

* Confirmed dummy API data is properly injected.

* Ensured no effect on existing reports (review, feedback, calibration, etc.)

== Future Work ==

* Replace dummy API with real LLM API integration.

* Allow asynchronous API calls for large assignments.

* Save updated instructor edits after LLM evaluation.

CSC/Independent Study Automatic Evaluation of Peer Reviews - Revision history