CSC/ECE 517 Spring 2017 E1733 Text metrics

This is the design review of our final project as part of CSC 517 in Spring 2017. The Project is titled Text Metrics.

Description

This project will facilitate few text metrics to evaluate student's peer reviews. The text metrics will provide a result about the amount of text written in peer-reviews, the number of suggestions given, the number of errors pointed out by the reviewer, and whether the peer-review contains any sort of offensive or improper language.

Tasks to be completed

The tasks that have to be completed as part of this project can be identified as follows:

1. Create DB table [review_metrics] to record all the metrics to include the number of: different words, suggestions, errors/problems pointed out by the reviewer and offensive words used in the review text.
2. Create DB table [review_metrics_mapping] to record all the response_id, the metric_id and the value.
3. Create a YAML file under config folder named text_metrics_keywords.yml and put all keywords in it. DO NOT hard-code them.
4. Add a pop-up dialog or a label for both students student_review/list and for instructors add content in review_mapping/response_report page metrics column (volume is already there), for each piece of peer-review, show all metrics.
5. Make sure the code works for both assignments with and without "varying rubric by rounds" selected.
6. Make sure the code updates the review text metrics table when the peer reviews are updated.
7. Sort the reviews based on the text metrics on the “Your scores” pages of students’ view
8. Create tests to make sure the test coverage increases

Current Implementation

In the current implementation, reviewers can view the reviews they have given via use of a view link in the Other's work section of each project as shown in the screenshot below:

Instructors can see the response of the reviewers. However, in order to analyze the quality of review given by the students, the instructors have to manually go through all the reviews given by every student one at a time. This could be a time-consuming process, and the reviewers may not be properly assessed or assessments may be inconsistent between reviews.

In order to ease the process of evaluating the reviewers, metrics which can analyze the text written by the reviewers can be added. This will give more information about the reviews given by the reviewers to the instructors evaluating them.

UML Diagram

The UML diagram below shows the MVC design proposed as part of the project and the relationships between the different elements. The users being students and instructors, two models (review_metrics and review_metrics_mapping) need to be defined with their own controllers review_metrics_controller which will have all the metric-based methods defined in it. The controllers will update the attribute values in the review_metrics and review_metrics_mapping tables respectively when a review is created or updated. This will then be visible in the views.

Proposed Design

TASK 1 - we will be adding a new table, controller and model for Review_metrics. This model will include all the metrics as follows:

1. metric_id → int(11) → primary key
2. metric → int(11) → # initial values in database would be {volume, suggestions, problem_identification, offensive_term}, but more could be added
3. suggestion → int(11) → if suggestion is given in the peer-review

There will also be a review_metric_controller which will include all the necessary CRUD operations for review_metrics. (i.e... Create, Remove, Update and Delete)

We will utilize the following View files for management:

1. review_metrics/edit.html.erb
2. review_metrics/new.html.erb
3. review_metrics/list.html.erb
4. review_metrics/show.html.erb

TASK 2 - we will be adding a new table, controller and model for Review_metrics_mapping. This model will include all the metrics as follows:

1. response_id → int(11) → foreign key
2. metric_id → int(11) → foreign key
3. value → int(11) → metric value

There will also be a review_metric_mapping_controller which will include all the necessary CRUD operations for review_metrics_mapping. (i.e... Create, Remove, Update and Delete)

We will utilize the following View files:

1. student_review/list.html.erb - for the student
2. review_mapping/response_report.html.erb - for the instructor

TASK 3 - we will create a YAML file under config folder named text_metrics_keywords.yml

1. We will put all keywords in it
2. We will create a ruby script under config/intilizers folder named load_data.rb
3. Contents of Load_data.rb: TEXT_METRICS_KEYWORDS = YAML.load_file("#{Rails.root}/config/text_metrics_keywords.yml#)[Rails.env]
4. Code:

 suggestion_words = TEXT_METRICS_KEYWORDS["suggestion"]
 error_words = TEXT_METRICS_KEYWORDS["error"]

TASK 4 - For each peer review , we will be defining the following metrics, which the reviewer/instructor will be able to view after hovering the mouse over over the respective icons (it will either be a pop-up or a label text).

1. Number of words average - Integer
2. Number of words for all the reviews in this assignment in this round if suggestion is given- Integer
3. The percentage of the peer-reviews which offer suggestions in this assignment in this round - Integer
4. If problems or errors are pointed out - Boolean
5. The percentage of the peer-reviews which point out problems in this assignment in this round - Boolean
6. If the peer-review contains offensive language - Boolean
7. The percentages of the peer-reviews which contain offensive language - Integer

The picture below shows how the proposed design will look to the user. The icons from left to right, will be indicative of the following metrics. (NOTE: The icon images are not subject to any copyright rules)

1. Icon 1 -> Number of reviews
2. Icon 2 -> Indicative of Vulgar content
3. Icon 3 -> Indicative of Similarity of reviews
4. Icon 4 -> Indicative of Problems or Errors in the peer review
5. Icon 5 -> Indicative of Suggestions made

The code snippet below shows where the view will be updated when the icons will be added - review_mapping/list.html.erb

                              <%= create_report_table_header("Reviewer" => "16%">
                              "# reviews done per round" => "14%",
                              "Team reviewed" => "20%",
                              "Score awarded / Avg. score" => "14%",
                              "Metrics" => "10%",
                              "Assign grade and write comments" => "36%") %>
     <%@l = -1 %>
     <% sort_reviewer_by_review_volume_desc.each do |r| %>
       <% @response_maps, @bgcolor, @rspan, @l = get_data_for_review_report(@id, r.id, @type, @l) %>

<%if r.id != -1 %> <% username = Participant.find(r.id).user.fullname %> <td bgcolor=<%= @bgcolor %> rowspan= <%= @rspan%>> <%= link_to username, :controller => 'popup', :action => 'reviewer_details_popup', :id => r.id, :assignment_id=>@id%> <td bgcolor=<%= @bgcolor %> rowspan= <%= @rspan%> align = 'left'> <% (1..@assignment.num_review_rounds).each do |round| %> <%= instance_variable_get("@review_in_round_" + round.to_s) %>/<%= @rspan %> <%= ', ' unless round == @assignment.num_review_rounds %> <% end %> <%= link_to "reviewer summary", :controller => 'popup', :action => 'view_review_scores_popup', :reviewer_id => r.id, :assignment_id=>@id %> TASK 5 - When the instructor checks the condition 'Vary rubric by rounds' the new code should work either way. This will be done checking the boolean method is_varying_rubric_by_rounds and then proceeding to update the review_metrics table. TASK 6 - We will be defining an RSpec file in review_metrics_controller which will check that the metrics table is updated when a review is updated. TASK 7 - We will be sorting and displaying the reviews in the file list.html.erb which is part of views/review_mapping based on the number of words metric received from the review_metrics table. Updating a review will effect a change in the review metrics table, therefore this will dynamically change the order if the metric has changed. TASK 8 - We will create new test in order to increase the overall test coverage. NOTE: Code snippets have not been shown for Task 5, 6, 7 and 8 because present code does not exist. We need to create new view and new rspec test cases for the same.

Design Pattern Used

Iterator pattern is used as the design pattern. This pattern is used in order to allow us to iterate over the elements of a collection regardless of their implementation. In our case, the collection is the different responses given by a reviewer in one single feedback. Since we will be iterating over the different responses in a given feedback by a reviewer (where the different responses in a given feedback are the elements of the collection) in order to collect the text metrics, the design pattern used will be an iterator pattern. This will provide a standard interface for starting an iteration and moving to the next element, regardless of the implementation of the collection.

Features to be added

Added label or pop up which will indicate the metrics as icons, as shown below. These icons will directly summarize the review in terms of predefined metrics. The reviews will be sorted according to the text metrics and will be displayed to the student on the 'Your Scores' page.

Gems to be used

This project provides us the opportunity to use the Ruby-NLP gem which has specific methods for analyzing language with predefined patterns. We will be entering a filter for a tentative list of foul words, which when detect will populate as a metric.

Gem - ruby-nlp Gem - Automated Metareview

The Automated Metareview has been used in parts in the present expertiza codebase but has not been implemented. We have take inspiration from the set of phrases such as NEGATIVE_DESCRIPTORS< SUGGESTIVE_WORDS which are some of the constants as part of the gem documentation.

Features like Word Count, Text summarization, Language detection, Text similarity, parsers are all part of this gem file which has been taken as baseline while implemneting few of the features.

Metrics View

Instructor View

1. When an instructor views all Assignments and clicks on the Review Report, this is the view that pops up

2. When an instructor clicks on View Review Metrics, the popup containing the Metrics for all the responses done by the particular reviewer are displayed in a tabular format.

Student View

1. When a student clicks on Other's Work after clicking on a specific Assignment, this is the view that pops up, containing a View Text Metrics for each review that the student has done.

2. When the student clicks on View Review Metrics, the popup opens which displays the metrics for the specific review.

Testing Plan

As verified with the instructor, the scope for testing in this project will be limited to majorly UI testing. For user/reviewer, the steps will be as follows

1. Login to Expertiza with your credentials
2. Open the Assignments, click on Other Work
3. For every review entered, hover the mouse on the icons beside the review
4. The review will be summarised by caption texts on the icon like "Good Content" or "Obscene Word usage" or "200 words reveiw"
5. View the metrics for each review.

Few of the Test cases added to the test file for Review Metric.

For the instructor, the steps will be as follows

1. Login to Expertiza with instructor credentials
2. Open Manage Assignments
3. Select the Assignment. Click on the icon 'View Reviews'
4. Depending on the stage of the assignment , the reviews will be listed in the order they were filled.
5. Metrics will be displayed on hovering the mouse over each icon.

Models

1. ReviewMetric.rb
2. ReviewMetricMapping.rb
3. Response.rb

Controllers

1. popup_controller.rb
2. review_metric_controller.rb
3. review_metric_mapping_controller.rb

Views

1. review_metrics/edit.html.erb
2. review_metrics/new.html.erb
3. review_metrics/list.html.erb
4. review_metrics/show.html.erb
5. student_review/list.html
6. review_mapping/review_report.html.erb
7. popup/view_review_metrics
8. popup/view_student_review_metrics

Important Links

Link to Github repository : https://github.com/rmmaily/expertiza

Link to Pull request : https://github.com/expertiza/expertiza/pull/937

References

1. https://sourcemaking.com/design_patterns
2. http://wiki.expertiza.ncsu.edu/index.php/Main_Page
3. https://github.com/diasks2/ruby-nlp
4. http://www.ibm.com/developerworks/rational/library/769.html