CSC/ECE 517 Fall 2019 - E1990. Integrate suggestion detection algorithm

Introduction

• On Expertiza, students receive review comments for their work from their peers. This review mechanism provides the students a chance to correct/modify their work, based on the reviews they receive. It is expected that the reviewers identify problems and suggest solutions, so that the students can improve their projects.
• It is observed that the students learn more by reviewing others' work than working on the project as it gives them perspective about alternative approaches to solve a problem.
• The Instructor is facilitated with metrics such as average volume and total volume of the content of the reviews provided by a student.

Problem Statement

• The reviewers can fill in the review comments on others' work, however they do not receive feedback on how effective their reviews are for the receiver. It would thus make sense to have a feedback mechanism in place, which can identify whether a reviewer has identified problems and provided suggestions for a student or team's project. In order to achieve this, we need to identify the suggestions in the review comments as a way of determining how useful a review would be. This would motivate the reviewers to give better and constructive reviews.
• We would want the instructor of the course to be able to view how many constructive reviews were provided by a reviewer in comparison to the average number of constructive reviews provided by the other reviewers of the course. The reviews could be graded on this basis.
• To detect the number of suggestions in a text, there is a web service in place, developed previously by some contributors. We wish to integrate this suggestion detection algorithm and use its results to show the suggestion metrics to the reviewer as well as the instructor.
• Please note that the web service mentioned above is currently not working. Thus we will use simulations to mock the response from the web service for this project.

Implementation Details

Part 1 : Mocking the response which is to be received from the suggestion detection web service

• The suggestion detection algorithm was expected to identify the number of suggestions in the review comments given by a reviewer. To mock this number, we used a random number generator. Now we expect to receive a value between 0-10 as the number of suggestions provided by the student in one round of review. (Refer to method: num_suggestions_for_responses_by_a_reviewer in review_mapping_helper.rb)
• In order to be able to use integrate the suggestion detection algorithm in the future, we have included a function with the API calls to the service. This function and the corresponding call to this function is currently commented. (Refer to methods: num_suggestions_for_responses_by_a_reviewer and retrieve_review_suggestion_metrics in review_mapping_helper.rb.)

Part 2 : Suggestion metrics on saving / submitting review

• We wish to present the reviewer(student) with an idea of how useful their reviews are, in comparison to that of the class.
• To achieve this, we have included the following in an alert when the reviewer clicks on "Save" or "Submit":
  • 1. The number of suggestions in the review comments provided by the reviewer for that round
  • 2. The average number of suggestions in the review comments provided by all reviewers of the class

(Again, for the purpose of demonstration, we have used values from the random number generator mentioned above.)

Part 3 : Suggestion metrics depicted as bar graphs in "Review Reports" page

• We wish to present the instructor with comparison of the number of suggestions present in the review comments provided by a student and the average number of suggestions typically provided by the class. We have represented this data in the form of a bar graph showing the round-wise comparison for each assignment. This visualization adheres to the pattern of visualization which was present on this page for volume of review comments and average volume of review comments.

Flowchart

Given below is the design flowchart of our proposed solution:

Once the student finishes (partly or completely) giving his/her review and clicks on the "Save" or "Submit" button, the web service API will be called and the review's text will be sent as JSON. The PeerLogic web service will send back the output(number of suggestions in the review comments) which will then be displayed in an alert to the student.

For all the review scores received by the students, the aggregate data will be displayed to the instructor whenever he/she will view the Review Report for any assignment/project. This will be visible in the metrics column of the review report and will be displayed student-wise, i.e., for each student participating in the assignment.

Overview of code changes

Please refer to Code changes for the exact changes.

We have modified the following files (added methods/ added code to methods/ replace code in the methods):

• review_mapping_helper.rb:
  • display_volume_metric_chart(reviewer) - changes made to resize the bar graph for volume metric and removal of legend for every graph.
  • avg_num_suggestions_per_round(assignment_id, round_id, type) - as the name suggests, the goal is to obtain the average number of suggestions provided by the class in each review round of the assignment.
  • num_suggestions_per_student_per_round(response_maps, round_id) - the goal is to obtain the average number of suggestions provided by the student in each review round of the assignment.
  • comments_in_current_response(response_id) - provides a concatenated text consisting of all review comments in the current response.
  • retrieve_review_suggestion_metrics(comments) - this function has been added to be able to make calls to the API in the future, when it starts working.
  • num_suggestions_for_responses_by_a_reviewer(comments) - this function should make a call to the function mentioned above. Currently, we use this function to simulate the number of suggestions in each comment, using a random number generator.
  • num_suggestions_reviewer(responses) - this function provides the number of suggestion given by a reviewer to a student/team's work.
  • display_suggestion_metric_chart(reviewer) - this method sets up the colors, labels and data to generate the bar graph for the suggestion metrics.
  • initialize_suggestion_chart_elements(reviewer) - this method computes the number of suggestions provided by the student in each round, the average number of suggestions provided by the student in all rounds, the average number of suggestions provided by the class in each round and the average number of suggestions provided by the class in all rounds. This data is generated as a collection of lists.

• _review_report.html.erb: changes are made to restructure the legend and add labels corresponding to suggestion metrics.

• response.html.erb: jQuery changes have been made to get the current response stored in the params and calculate average of the same instantly, when the student clicks on "Save" or "Submit".

Additions & Modifications

Listed below are the major changes along with the screenshots of git diffs of those file/functions.

1. Changes made to display the suggestions metrics bar chart to the instructor in review_mapping_helper.rb file.

2. Changes made to initialize all the metrics and get all their values from the web service / random number generator when the instructor views the Review Report.

3. Changes made to call the web service (currently commented out since the service is not up) / random number generator. We call the function to get number of suggestions for the reviewer per round per team and for the class as a whole to calculate the average. Below is the git diff:

4. Changes made to the response.html.erb where these functions are actually called from.

Automated Tests

• review_assignment_spec.rb: Tests have been added to test the feature of identifying the number of suggestions in a given review.
  • Check whether the student is able to see a suggestion score for his/her review - on clicking save
  • Check whether the student is able to see the average number of suggestions given by the class - on clicking save
  • Check whether the student is able to see a suggestion score for his/her review - on clicking submit
  • Check whether the student is able to see the average number of suggestions given by the class - on clicking submit
  • Check whether the number of suggestions displayed is in the valid range of numbers - on clicking save and submit
  • Check whether the the average number of suggestions given by the class is in the valid range of numbers - on clicking save and submit

• review_mapping_helper_spec.rb : Test have been added to check whether the new legends for metric is now part of the "Review reports" page.

Screenshots

Review reports page as visible to the Instructor, after the above mentioned changes.

Suggestion metrics as visible to the reviewer when saving the review.

Suggestion metrics as visible to the reviewer when submitting the review.

Steps to follow for UI testing

For student

   Steps I:
   1. Login to student profile.
   2. Go to your open assignments and submit your work.
   3. Go to that particular assignment and request a review.
   4. Begin writing a review. 
   5. Click "Save" button to save the review.
   6. An alert should be displayed that shows the review comment analysis for the student.

   Steps II:
   1. Login to student profile.
   2. Go to your open assignments and submit your work.
   3. Go to that particular assignment and request a review.
   4. Begin writing a review. 
   5. Click "Submit Review" button to submit the review.
   6. An alert should be displayed that shows the review comment analysis for the student.

For an instructor

   1. Login as an instructor.
   2. Navigate to all assignments list.
   3. Click on View Reports button and open Review Report for that assignment.
   4. The instructor should be able to see each student's review comment analysis for the submitted reviews under the metric column.

Testing

Following test cases were added to see whether the functionality works as intended.

1. Whether the alert renders the number of suggestions for a reviewee on save/submit.
2. Whether the alert renders the average number of suggestions in a class for a reviewee on save/submit.
3. Whether the function returns the number of suggestions in correct bounds and returns an Integer value.
4. Whether the function returns the average number of suggestions in correct bounds and returns an Integer value.

1. . Whether the review report has the modified legend metrics.

Team Information

Mentor

Ed Gehringer (efg@ncsu.edu)

Team Name

Pizzamas

Members

• Maharshi Parekh (mgparekh)
• Pururaj Dave (pdave2)
• Roshani Narasimhan (rnarasi2)
• Yash Thakkar (yrthakka)

References

1. Expertiza
2. Pull request for a Metrics Legend change
3. Expertiza Github Repo
4. Our project's Github pull request
5. Project Demo