CSC/ECE 517 Spring 2024 - E2414 Grading Audit Trail

Problem Statement

NC State faculty members and students use Expertiza Open-Source Website to submit assignments, grade them, form teams, and among other features. In this project (E2414), we address the issue of instructors being able to freely edit a grade with no tracking of what changes are made by whom. There is a need for a way to view all of the changes made to a grade in order to provide accountability

Program Objectives

Grading Audit Trail must be implemented with the following information to be stored:

1. When an instructor assigns a grade, there needs to be an indication of who did it and when it was done.

2. Comments previously provided by other instructors must also be preserved.

This information needs to be stored every time an instructor edits a grade / comments and clicks the save button. The grading audit trail can probably pattern off the submission records history on Expertiza. The submission record page can be reached by logging in as instructor -> Manage -> Assignments -> View Submissions -> History. At the minimum, a grading log entry must include the instructor id, assignment id, student id, grade comment and timestamp.

To effectively track and audit grading activities, the Grading Audit Trail system will integrate with the ElasticSearch, Logstash, and Kibana (ELK) stack. This system will ensure:

1. The minimum data logged will include instructor ID, assignment ID, student ID, grade comment, and timestamp, aligning with the submission records history in Expertiza.

2. Kibana dashboards will be created for instructors and administrators to view and analyze grading history.

Previous work from other teams

Project E2383 tackled the same issue of creating a grading audit trail. Our work was based off of this implementation, which provided much of the core functionality of creating a history record for each grade change, as well as creating a page for displaying grading history. However, there were some shortcomings of this implementation which we attempt to address in our implementation

Issues:

1. Poor link placement and naming

2. Minimal testing of controller for grading history

3. Grading history table used equal-width columns which caused long comments to have large vertical span

4. Several instances of controller code that could be refactored into model

Documentation containing files with logging functionality:

https://docs.google.com/document/d/1oMqNSyvJ_vKEEYstYqlTwxNUceXYXxeYUVL9STPIrR8/edit

Program 4 Design Document

Task Overview

In program 3, we implemented functionality to maintain an audit trail of all changes made to grades by instructors
The work to be completed for program 4 consists of 2 major parts: Refactoring changes from program 3 and Implementing logging support for grade history

Refactoring Changes from Program 3

• The table on the grading history page is too large and there is large amounts of whitespace between columns
• The method assignment_for_history in grading_history.rb is an unnecessary class method, and should be refactored as an instance method utilizing polymorphism
• Creation of grading history records should be done using filters, adhering to single responsibility principle in non-grading-related controllers

Logging Support for Grade History

• Implement logging using ElasticSearch Logstash Kibana (ELK), which is currently in use for logging-related tasks in Expertiza
• Logging should be done for all grade changes made in Expertiza, including both for assignments and reviews

UML Diagram

The UML Diagram includes the GradingHistory object which is currently used to create Grading History for submissions and reviews. This design highlights how the Review_Mapping_Controller and the Grades_Controller will use the Grading_Histories_Controller to create a Grading History. The Grading_Histories_Controller will create a Grading History and will use the log instance method of the controller to log the history that has been created. The log instance method in the Grading_Histories_Controller uses the ExpertizaLogger and LoggerMessage to log the Grading History. The methods that are used from those classes are the info and initialize methods, respectively.

Design Patterns

We will apply several design patterns and principles in our implementation to allow us to write simpler and more maintainable code.

Many of our refactoring changes from program 3 apply the single responsibility principle. By refactoring the class method assignment_for_history into a new class, we decouple the GradingHistory class from Assignments which removes an unnecessary responsibility. Additionally, implementing creation of grading history records as a filter extracts this code from controller methods and into its own function, creating a better separation of responsibilities.

Our implementation of the grading history functionality can use the Memento design pattern. Our implementation keeps track of the state of a grade for an assignment or review anytime the grade is changed. Storing the state after every change provides the ability to identify potential malicious changes to a grade and revert the grade back to what it previously was. Applying the memento design pattern can make our code more readable and easier to understand.

Lastly, our implementation for logging functionality can follow the Mediator design pattern. The ExpertizaLogger and LoggerMessage classes provide a consistent interface for creating log messages, regardless of what application component is creating the log message. By utilizing these classes in our implementation, we avoid the need to directly interact with the logging system, instead relying on the abstraction of a log message. Using this pattern can greatly simplify our implementation.

Plan of Work

Grading History Table Changes

Problem: The above image shows the current layout of the Grading History page. We can see that the table takes up the entire width of the screen, and there is a large amount of unnecessary whitespace in the comments section when the comment is relatively short. Adjusting the column widths would provide a more compact view that is easier for users to read

Solution: We will make the following changes to app/views/grading_histories/index.html.erb to make the table more readable:

1. Resize existing columns with fixed widths to minimize whitespace

2. Dynamically size the 'Comments' column to reduce unnecessary whitespace, up to a fixed maximum width

3. The resulting table will be centered in the middle of the screen

Refactor class method assignment_for_history in grading_history.rb

Problem: The project focused on improving the grading audit trail for the Expertiza Website. The problem faced was the need for a method that could reliably identify the relevant assignment context, whether a submission or a review, so as to obtain the grading history accurately. The previous implementation may have been not sufficiently encapsulated, leading to potential issues when trying to extend functionality or adapt to new requirements.

Solution: We will make the following changes to app/models/grading_history.rb:

1. Transition to Instance Method: Shift from a class method to an instance method within the GradingHistory model, enabling the use of object state and behavior, which is more object-oriented.

2. Utilize Polymorphism: Implement polymorphic associations that will allow GradingHistory to belong to either a Submission or a Review.

3. Simplify the Interface: By using polymorphism, we can simplify the interface of the method, allowing it to behave differently based on the associated object, hence eliminating the need for type checks within the method.

4. Enhance Maintainability: Refactored code with polymorphism will be easier to extend and maintain, as adding new gradable types would not require changes to the assignment_for_history method.

Refactor Grading History Creation using filters

Problem: The creation of grading history records is currently mixed within controllers that handle various responsibilities beyond grading. This violates the single responsibility principle and makes the controllers bulky and less maintainable.

Solution: The refactoring changes will revolve around the save_grade_and_comment and index methods, particularly how the @assignment variable is populated. The goal is to use the Single Responsibility Principle and use filters to handle creation and fetching of grading history records. For the save_grade_and_comment method, the refactoring will aim to simplify the permission checks and ensure that they are clear and concise. For the index method, the retrieval and assignment of @receiver and @assignment based on the record.graded_item_type may be refactored to use a more polymorphic approach. We will make the following changes to app/controllers/review_mapping_controller.rb:

1. GradingHistoriesController: This controller would be refactored to remove the direct creation of grading history records and instead, use filters to handle these actions.

2. GradesController or ReviewMappingControllers: Filters such as save_grade_and_comment and save_grade_and_comment would be added to the GradesController and ReviewMappingController, to trigger grading history record creation at the appropriate points in the request-response cycle.

3. GradingHistory Model: Modifiying the GradingHistory model to support the new way records are created, such as adding or altering methods to be called by the new filters.

4. Routes and Views: Depending on how grading history creation is triggered, there may be minor adjustments needed in the routes and views to ensure that the correct actions are called and that users can use the website properly.

5. Spec Files (Tests): Tests would need to be updated to cover the new filtering logic and to ensure that grading history records are created as expected.

Logging Support for Grade History

Problem: The problem revolves around the need for a comprehensive logging system in the Expertiza platform to ensure transparency and accountability in grade modifications. Currently, the system records grade changes but lacks an integrated logging support for comprehensive tracking of all grade-related activities. An example of the structure of logs in the system is shown below

Solution: We will use the already implemented logging system and integrate it with Grading History to solve the problem. Since the functionalities are similar we will avoid repeating a system that is already in place. Instead we will explore how this system can be extended to another part of the Software(Grading History).

1. Grading_Histories_Controller: ExpertizaLogger and LoggerMessage classes provide the current functionality for logging and will be used in unison in Grading Histories Controller

2. Review_Mapping_Controller: This controller will use Grading Histories Controller to create Grading History and logs for reviews

3. Grades_Controller: This controller will also use Grading Histories Controller to create Grading History and logs for submissions

4. GradingHistory Model: This model will be used to create GradingHistory objects that will be logged

Implementation (Program 3)

Our project's implementation hinged on a blend of time-tested techniques and innovative practices. We began by adopting Test-Driven Development (TDD), writing comprehensive test cases using the TA-provided skeletons as a framework. This ensured that every piece of functionality was verified. Pair programming sessions enhanced our collaborative efforts, enabling real-time code reviews and knowledge sharing, which fostered a high level of code quality and team cohesion. We also leveraged Continuous Integration (CI) and Continuous Deployment (CD) pipelines to streamline our development process. This allowed for automated testing and deployment of our code, ensuring that changes were tested and integrated frequently. The result was a reduction in integration problems and more adaptive and faster release cycles. Moreover, refactoring was an integral part of our workflow. It was not just a one-time event but a continual process of improving the code structure without altering its external behavior. This was crucial for keeping the codebase clean, efficient, and ready for future extensions or changes. By utilizing version control best practices, we ensured that our codebase remained manageable and navigable. Feature branching allowed us to work on different aspects of the project simultaneously without disrupting the main codebase, and pull requests facilitated peer reviews before code was merged.

Creating GradingHistoriesController

The grading_histories_controller.rb functionality created on Ruby on Rails application serves as a critical component for managing and displaying the grading history of reviews in an educational setting. Rails ensures the integration with the model and view layers, facilitating a robust and intuitive interface for users to interact with grading data. Through the implementation of callbacks for pre-action configurations and the inclusion of authorization logic via concerns, the application effectively enforces security and role-based access control. This ensures that only authorized users such as administrators, instructors, and teaching assistants can access sensitive grading information based on their association with specific courses or assignments. The controller is adept at handling various user permissions, distinguishing between submission and review grading histories, and organizing all related entries in a user friendly manner by displaying them in reverse chronological order. This organization enhances the application's usability by allowing users to easily navigate through the grading history.

Creating Model GradingHistory

The grading_history.rb model,has been created and enhanced with a refactoring that introduces a class method assignment_for_history. This method significantly extends the model's functionality by determining the context of grading—whether it pertains to a submission or a review based on the type of grading activity. For submissions, it identifies the associated AssignmentTeam using the graded_member_id to retrieve the corresponding assignment. In the case of reviews, it uses the participant_id to find the specific AssignmentParticipant, facilitating a direct link to the reviewed assignment or activity through the ReviewGrade model. This addition underscores the model's pivotal role in not only linking grading records to instructors and assignments via belongs_to associations but also in dynamically determining the relevant assignment context based on grading type. This enhancement bolsters the model's capacity to provide a comprehensive and nuanced tracking mechanism, offering deeper insights into the historical analysis of grading actions across the platform.

Creating Model ReviewGradingHistory

The review_grading_history.rb model, inheriting from the controller, is tailored for peer review grading events, featuring protection against mass-assignment vulnerabilities and establishing a relationship with the Participant class to track review grades within the system.

Creating Model SubmissionGradingHistory

The submission_grading_history.rb model extends the controller to focus on submission-related grading records, leveraging attr_protected to enhance security and associating with the Team class to directly relate grades to submitting teams within the platform.

Creating View grading_histories

The index.html view for grading_histories has been created to enhance performance and readability. It presents a grade history in a striped table format, centered around a dynamic header that adapts to display grading information pertinent to submissions or reviews. The created and refactored code uses conditional logic to efficiently handle potential nil records, ensuring that the display adapts gracefully in the absence of grading data. When available, the table populates with comprehensive details such as the instructor's full name, grade, comments, and the precise timestamp of grading—presented in an orderly fashion.

Updating Grades Controller

We introduced important functionalities to improve grading transparency and history tracking on the grades_controller.rb. An addition to the save_grade_and_comment_for_submission method is the creation of a GradingHistory record each time a submission is graded. This record stores vital information, including the instructor's ID, assignment ID, the type of graded item ('Submission'), the graded team's ID, along with the given grade and accompanying comment. This development is a significant stride in grading auditability, as it ensures every grading action is logged and traceable within the system. The method also encapsulates error handling to alert users in the event of a failure in saving the grade and comment, thereby maintaining a user-friendly interface.

Updating Review Mapping Controller

We integrated a robust suite of methods to facilitate peer review assignments and management within our educational platform on the review_mapping_controller.rb. The introduction of dynamic reviewer assignments allows for flexible peer evaluation processes, aligning with the evolving needs of academic settings. The calibration feature provides a mechanism for instructors to gauge the quality of reviews, ensuring that the assessment process is both fair and consistent. One the most important improvements is the inclusion of grading history records for reviews. With this feature, the grading process for reviewers is not just a transitory evaluation but a documented history, contributing to transparent and traceable assessment practices. This aspect of the enhancement not only adds depth to the instructor's oversight capabilities but also enriches the feedback loop for students. The controller now supports more organized review policies, checking for outstanding reviews and allowing assignments to adhere strictly to the configured review limits. These checks prevent over-assignment of reviews, thereby maintaining a balanced workload among participants and ensuring the quality of peer assessments. The enhanced error handling and user feedback mechanisms contribute to a smoother user experience by providing clearer guidance and actionable insights during exception scenarios, such as attempting to delete a review mapping in progress or assigning a self-review.

Updating Review Grade Model

The review_grade.rb model was updated with the find_graded_member method contribute to deepening the contextual understanding of review grades within the system. This method introduces the ability to directly associate a graded review with its corresponding assignment through the participant's parent relationship. By encapsulating this logic within the model, the system design adheres more closely to the principles of object-oriented programming, promoting encapsulation and cohesion. This change has a more intuitive and maintainable approach to accessing assignment details from a review grade instance. The find_graded_member method abstracts the complexity of traversing associated records, thereby simplifying external calls and enhancing code readability.

Update Views Reports ReviewReport HTML

The new code added for the views/reports/_review_report.html.erb moves the location of the Grading History link. The link had previously been placed directly adjacent to the grade entry form, cluttering the view as this link unnecessarily extended the width of the column including the grade entry form. With this change we move the Grading History link under the grade entry form, making this column more compact and allowing additional space for other columns

Update Spec Factories rb file

The code we added in the spec/factories/factories.rb file introduces a new factory for GradingHistory, which is designed to create test data for grading history records within the application. This factory sets up a mock GradingHistory object with predefined attributes such as instructor_id, assignment_id, graded_item_type, graded_member_id, grade, and comment. This addition allows for the testing of features related to the tracking and displaying of grading histories in the application, facilitating the assessment of functionalities that involve historical grading data for assignments, submissions, or participants. The inclusion of this factory enhances the test suite by providing a means to easily generate and manipulate grading history data for various testing scenarios.

Implementation (Program 4)

Update Grades Controller rb file

The changes made to the GradesController as displayed in the image focus on the automatic creation of a grading history record every time a grade and comment for a submission is saved. The previously existing manual call to create a GradingHistory record within the save_grade_and_comment_for_submission method has been removed. The creation of the grading history was initially invoked within the method that handles the saving of grades and comments, which could have led to duplicated code and potential oversight if the method was called from multiple points in the application. Instead, an after_action callback has been introduced at the controller level. The create_grading_history method is designated to run automatically after the save_grade_and_comment_for_submission action. This refactoring ensures that the grading history is consistently recorded whenever a submission's grade or comment is updated, without the need to explicitly invoke the history creation within various parts of the code. The introduction of the after_action filter aligns with the principles of DRY and aspect-oriented programming, separating concerns by ensuring that the logging of grading history is a cross-cutting concern handled independently from the business logic of saving grades and comments.

We refactored the code with a dedicated method, add_grading_history, within the GradingHistoriesController. The pull request included a shift from direct history creation to invoking a method that clearly articulated its purpose maintaining a log of grading actions. This not only made the code more readable but also centralized the logging functionality, streamlining future updates and ensuring consistency across the application.

Update Review Mapping Controller rb file

The ReviewMappingController depicted in the image show the application of the same principles to ensure that each review's grade and comment are automatically logged upon submission.An after_action callback named create_grading_history is added to run after the save_grade_and_comment_for_reviewer action. The code indicates that now, every time a grade for a review is saved, a corresponding record in the GradingHistory is created. This is done by capturing the necessary data such as the instructor's ID, assignment ID, the item type (which in this case is a 'Review'), the ID of the member being graded, the grade itself, and any associated comments. This ensures that every action related to the review grading process is automatically followed by the creation of a grading history record, providing a transparent audit trail for review grades as well. This refactoring simplifies the process of creating historical records for grades, making the process consistent and reducing the chance for errors or omissions in the audit trail. By doing so, it extends the accountability features of Expertiza to encompass the review process, fostering a comprehensive and traceable grading system.

We extracted the grading history creation into a separate, dedicated method within the GradingHistoriesController, named add_grading_history. This method centralized the process of logging the grading actions, and the pull request encapsulated these changes, shifting the responsibility from the ReviewMappingController to the GradingHistoriesController.

Update Grade Histories Controller rb file

The GradingHistoriesController controller is pivotal as it manages the history of grading changes, a feature that serves the transparency and accountability needs of our application. Initially, we had a basic mechanism in place to handle grading histories. However, to provide a more robust solution, we added new code that ensures a log message is created alongside every grading history record. This development means that any change made to a grade is now accompanied by an entry in our logging system, which captures the nature of the change, the identity of the grader, and the precise moment the change was made.

Update Index HTML erb file for Grading History

The grading histories index view got simplified. The refactor removes these static width attributes by specifying minimum widths and allowing table cells to expand naturally, the interface becomes more flexible. This ensures that regardless of the length of the content the table can accommodate without awkward text wrapping or truncation. This change significantly improves the readability of grading histories.

Reduced the overall table width to 75%, allowing for better alignment within the page and more whitespace, which is known to improve content focus and readability. Setting a minimum width for the 'Instructor' column, ensuring that even with the reduced table width, the display of the instructor names remains uncluttered. Adjusted the widths of the 'Grade' and 'Graded At' columns to accommodate the typical content size.

Testing Grading Audit Trail

We implemented the test skeletons provided by our TA to enhance our testing framework. By integrating these pre-constructed scenarios into our test suite, we have strengthened the reliability of our application. These skeletons served as a starting point, which we tailored to align with the unique requirements and functionalities of our system. Through a process of refactoring, we eliminated redundant tests, introduced additional scenarios to cover new functionalities, and modified existing tests to better reflect the behaviors and permissions associated with the GradingHistoriesController. This approach has not only expedited our testing phase but has also granted us deeper insights into the intricate permissions and workflows within our application. By verifying each role's access and the corresponding outcomes, we ensured that our application upheld its stringent security and functionality standards.

Test code for Grading History Controller

The grading_histories_controller_spec.rb spec tests various scenarios to ensure proper access control and functionality of the grading history feature within an educational app. It uses fabricated data for users and assignments to check permissions, verifying that only administrators, instructors, and TAs associated with an assignment can view its grading history. The tests confirm that unauthorized roles like students are denied access. Additionally, it tests that the controller sets up necessary variables correctly, depending on whether the context is a submission or review. These tests safeguard against unauthorized access and ensure the reliability of the grading history tracking system.

Updated views for Grading Audit Trail

Submission Grading History Views

The following views demonstrate UI for viewing submission history. The view below can be accessed through the following steps when logged in as an instructor

1. Click the "Manage..." button in the navigation bar

2. For the desired assignment, click the "View Submissions" icon from the list of actions

The view below displays a list of submissions for the given assignment. From this page, instructors are able to assign grades to submissions, and also can see the grading history for the submission by following the "Grading history" link.

After clicking on "Grading history", you are taken to the view below. The instructor is displayed with a listing of changes made to the grade, with the most recent changes appearing first in the list. A complete history of all grades and comments for this submission is displayed.

Review Grading History Views

Similar views exist for viewing the grading history of Reviews. The view below can be accessed through the following steps when logged in as an instructor

1. Click the "Manage..." button in the navigation bar

2. For the desired assignment, click the "View reports" icon from the list of actions

3. Select "Review report" from the dropdown and click View

From this view, an instructor can assign a grade for a review by another student. Below the grade assignment form, there is a link to view the grading history of the review.

Following the "Grading history" link brings the user to the view below. Similar to assignments this provides a complete listing of all changes made to the review grade in tabular format. The page has a header that dynamically changes to specify the graded item type and graded student/team.

In our project, we implemented an efficient and user-friendly interface for managing grading histories within the Expertiza, enhancing both the submission and review processes. The screenshots provided demonstrate a seamless integration of a "Grading History" feature, accessible via links next to each team and individual's submission record. This advancement not only streamlines the grading workflow but also fosters transparency, allowing instructors to track grading progress and view detailed feedback over time. This gives traceability of what TA graded what submission, hence meeting our project objectives. We focused on providing clear and concise information, with each page displaying a log of grades and comments, including timestamps for each entry.

Results

The implementation of the Grading Audit Trail in the Expertiza platform has significantly enhanced the accountability and transparency of the grading process. The project E2414 has successfully addressed the critical need for an auditable grading system. This new feature reliably records the identity of the instructor making the change, the time of the modification, and preserves the lineage of feedback provided by multiple evaluators. Throughout the project, our team learned the importance of robust record-keeping within educational software. The development emphasized the need for precision in logging activities to ensure that all stakeholders, including students, instructors, and administrative staff, can trust the grading system's fairness and accuracy. The project underscored the complexity of implementing new features in a live environment, especially regarding data integrity and user interface design. The Grading Audit Trail now captures and displays the instructor's identity and the exact timestamp when a grade is assigned or edited, fulfilling the requirement for detailed traceability. The system retains and displays all previous comments from different instructors, ensuring that the entire feedback history is available for review. These improvements meets the project's initial objectives but also add value to the educational experience on the Expertiza platform by bolstering the credibility of the grading process.

ELK Logging System References

The ELK (ElasticSearch Logstash Kibana) logging system handles all logging-related tasks in Expertiza. This system provides a user-friendly frontend for viewing the contents of the log file and querying the logs. Information about how to run this system and implement logging in Expertiza can be found at the link below:

Support for logging & data-mining peer-assessment behavior

Setting up on NCSU VCL

Instructions for installing the ELK logging system can be found at the following wiki page: Independent Study Spring 2019/Logging

This document contains instructions for installing on MacOS. Additional steps must be taken to install on VCL. These are not perfect installation instructions, but cover many of the issues we faced when installing

1. Follow the instructions in the above link to install ElasticSearch, Kibana, Logstash, and Filebeat. On each site, select the option to install using a package manager (yum). Do not install these applications as root, as this will create permissions issues since the applications cannot be run as root.
2. Set up expertiza.conf and filebeat.yml as described in the above file. These can be placed in any easily accessible directory
3. The Kibana config file is typically installed in /etc/kibana/. If this is the case, run export KBD_PATH_CONF=/etc/kibana
4. Disable authentication for ElasticSearch, as it is unnecessary and complicated to set up. Set the following in Elasticsearch.yml:
   1. xpack.security.enabled: false
   2. xpack.security.enrollment.enabled: false
5. Run each application in the order described Here. The path to the binaries for each application is typically /usr/share/<application>/bin. If any of the applications fail due to permissions issues, run: sudo chown -R <unityid> <failing_path>, where failing_path is the path to the directory with permissions issues. For filenames of expertiza.conf and filebeat.yml, use absolute paths.
6. Create a firewall rule to allow access to port 5601 from external devices:
   1. iptables -I INPUT -p tcp -s 0.0.0.0/0 --dport 5601 -j ACCEPT
   2. ufw allow 5601
   3. ufw reload
7. If all 4 applications are running without errors, you should now be able to access http://<vcl_ip>:5601 from your browser

Contributors and Relevant Links

Javier Badillo (unityid: jebadill, github: jebadillo69)
Tyler Enck (unityid: tsenck, github: TylerEnck)
Biniyam Sima (unityid: btsima, github: biniyamsima)
Yucheng Zhu (unityid: yzhu67, github: YuchengZhu27)

Github repo https://github.com/TylerEnck/expertiza
Link to PR https://github.com/expertiza/expertiza/pull/2774
Demo Video https://youtu.be/JIxUwI_sK9E