CSC/ECE 517 Spring 2017/E1730. Override-review-grader pre-sort
E1730
Please note that our problem statement has changed! The 'Problem Statement' section in this article describes the new one.
Deployment Link
About Expertiza
Expertiza is an open source peer review system.
It allows students to upload their work which can then be reviewed by other students using a rubric.
Instructors and graders can view the reviews and they can then assign grades to the reviewers as well the reviewee.
Problem Statement
While grading students, a grader might want to organize reviews by sorting them by a particular metric.
The only available metric was ‘Average Volume’ (number of words in the review).
A new metric called ‘Overall Sentiment’ is added which is a gauge of the sentiment of the review, that is, whether the review was positive or negative.
The task was to allow a grader to select a particular metric, and the view displays that metric for each review. In addition to that, the grader should be able to sort the reviews by that selected metric.
Design
1) For each review, send a POST request to the Sentiment Analysis URL for sentiment analysis with the review as parameters.
2) Store the sentiment values in a list.
3) In the view, allow the grader to select the metric(average volume or overall sentiment) from the drop down menu.
4) The view is rendered accordingly to either show overall sentiment in the metric or the average volume
5) Using the tablesorter (jQuery) addParser method, the Metric column is sorted by the appropriate metric.
Current Implementation
Only a single metric for sorting reviews called ‘Average Volume’ was present. The table was sortable only by that metric.
New Implementation
The ‘Overall Sentiment’ metric is added to the system. The grader can choose between two metrics (Average volume, Overall Sentiment) and sort reviews by that metric.
Gem
A gem called HTTParty is added which facilitates the sending of HTTP requests (GET, POST, etc.) to the web service using Ruby code.
Files Modified
app/controllers/review_mapping_controller.rb
app/helpers/review_mapping_helper.rb
app/views/review_mapping/_review_report.html.erb
app/views/review_mapping/_searchbox.html.erb
spec/helpers/review_mapping_helper_spec.rb
Code Snippets
Method description:
This method constructs the query to be sent to the sentiment generation service.
It takes the review id and the text in the review and constructs the query in the format required by the sentiment generation server.
Code:
def construct_sentiment_query(id, review_text)
review = {}
review["id"] = id.to_s
review["text"] = review_text
review
end
Method description:
This method retrieves the response from the sentiment generation service. It makes a JSON request and expects a JSON response.
It takes two parameters, the review for which the sentiment is to be obtained and a boolean to check whether it's the first try in obtaining the sentiment value.
In the cases where the Sentiment generation server fails to calculate a proper sentiment value, we will retry our request with only a single sentence of the review.
It was decided after a meeting with professor to simply use the first sentence of the review text as the single sentence that would be used during the retry, to try and generate a sentiment value.
Code:
def retrieve_sentiment_response(review, first_try)
if first_try
response = HTTParty.post(
'http://peerlogic.csc.ncsu.edu/sentiment/analyze_reviews_bulk',
body: {"reviews" => [review]}.to_json,
headers: {'Content-Type' => 'application/json'}
)
else
# Send only the first sentence of the review for sentiment analysis in case of a failure
first_sentence = review["text"].split('.')[0]
reconstructed_review = construct_sentiment_query(review["id"], first_sentence)
response = HTTParty.post(
'http://peerlogic.csc.ncsu.edu/sentiment/analyze_reviews_bulk',
body: {"reviews" => [reconstructed_review]}.to_json,
headers: {'Content-Type' => 'application/json'}
)
end
response
end
Method description:
This method creates a sentiment hash with the id of the review and the results from the sentiment generation service.
Code:
def create_sentiment(id, sentiment_value)
sentiment = {}
sentiment["id"] = id
sentiment["sentiment"] = sentiment_value
sentiment
end
Method description:
This method handles the case when a sentiment generation query has been retried.
This is the last effort to retrieve a sentiment from the server. If it succeeds and a 200 response code is received, we parse the JSON response to extract the sentiment value.
If unsuccessful, we assign the sentiment a -500 value to represent an error condition where even the retry had failed to get a proper response from the server.
This case could be due to a variety of reasons like an overloaded server, a malformed query, etc.
Code:
def handle_sentiment_generation_retry(response, review)
sentiment = case response.code
when 200
create_sentiment(response.parsed_response["sentiments"][0]["id"], response.parsed_response["sentiments"][0]["sentiment"])
else
# Instead of checking for individual error response codes, have a generic code set for any server related error
# For now the value representing any server error will be -500
create_sentiment(review["id"], "-500")
end
sentiment
end
Method description:
This method generates the sentiment list for all the reviews.
It gets the sentiments created from parsing the responses from Sentiment generation server and inserts them into a list.
Code:
def generate_sentiment_list
@sentiment_list = []
@reviewers.each do |r|
review = construct_sentiment_query(r.id, Response.concatenate_all_review_comments(@id, r).join(" "))
response = retrieve_sentiment_response(review, true)
# Insert the extracted sentiment into the sentiment list
@sentiment_list << case response.code
when 200
# Extract the sentiment from the response in case of a successful response
create_sentiment(response.parsed_response["sentiments"][0]["id"], response.parsed_response["sentiments"][0]["sentiment"])
else
# Retry to get a sentiment response once again in case of a failure
handle_sentiment_generation_retry(retrieve_sentiment_response(review, false), review)
end
end
@sentiment_list
end
Method description:
This method creates the correct format i.e. Metric Name: Metric Value, so that the view can easily render it.
Code:
def display_sentiment_metric(id)
hashed_sentiment = @sentiment_list.select {|sentiment| sentiment["id"] == id.to_s }
sentiment_value = hashed_sentiment[0]["sentiment"].to_f.round(2)
metric = "Overall Sentiment: #{sentiment_value}
"
metric.html_safe
end
Test Plan
Scenario 1:
Testing Tool:
RSpec
Description:
Check whether the sentiment query is constructed.
require 'rails_helper'
describe 'ReviewMappingHelper', :type => :helper do
describe "#construct_sentiment_query" do
it "should not return nil" do
expect(helper.construct_sentiment_query(1,"Text")).not_to eq(nil)
end
end
Scenario 2:
Testing Tool:
RSpec
Description:
Test whether the webservice returns a response
describe "#retrieve_sentiment_response" do
it "should not return nil" do
review = helper.construct_sentiment_query(1, "Test Review")
# Test first try to get sentiment from the sentiment analysis web service
expect(helper.retrieve_sentiment_response(review, true)).not_to eq(nil)
# Test a retry to get sentiment from the sentiment analysis web service
expect(helper.retrieve_sentiment_response(review, false)).not_to eq(nil)
end
Scenario 3:
Testing tool:
RSpec
Description:
Check whether the web service does not respond with a 404 error or a 500 error
it "should not get web response with 404 error or 500 error" do
review = helper.construct_sentiment_query(1, "Test Reviews to check if our Rest Client is able to reach the sentiment analysis service.")
# Test a first try to get sentiment from the sentiment analysis web service does not return a 404 or 500
expect(helper.retrieve_sentiment_response(review, true).code).not_to eq(404) and
expect(helper.retrieve_sentiment_response(review, true).code).not_to eq(500)
# Test a retry to get sentiment from the sentiment analysis web service does not return a 404 or 500
expect(helper.retrieve_sentiment_response(review, false).code).not_to eq(404) and
expect(helper.retrieve_sentiment_response(review, false).code).not_to eq(500)
end
Scenario 4:
Testing tool:
RSpec
Description:
Check whether the web service returns a response in JSON format
it "should return json response" do
review = helper.construct_sentiment_query(1, "Test Review")
# Test first try to get sentiment from the sentiment analysis web service
expect(helper.retrieve_sentiment_response(review, true).header['Content-Type']).to include 'application/json'
# Test a retry to get sentiment from the sentiment analysis web service
expect(helper.retrieve_sentiment_response(review, false).header['Content-Type']).to include 'application/json'
end
end
Scenario 5:
Testing tool:
RSpec
Description:
Check whether the retry returns a valid response. It may be a -500 which would denote that the retry failed, however it should never be nil.
describe "#handle_sentiment_generation_retry" do
it "should not return nil" do
review = helper.construct_sentiment_query(1, "Test Reviews to check if our client can handle error scenarios.")
# Test for a response generated by a retry
response = helper.retrieve_sentiment_response(review, false)
expect(helper.handle_sentiment_generation_retry(response, review)).not_to eq(nil)
end
end
Scenario 6:
Testing tool:
RSpec
Description:
Check whether the sentiment list does not return a nil value
describe "#generate_sentiment_list" do
it "should not return nil" do
@id = 1
@assignment = Assignment.where(id: @id)
@reviewers = ReviewResponseMap.review_response_report(@id, @assignment, "ReviewResponseMap", nil)
expect(helper.generate_sentiment_list).not_to eq(nil)
end
end
end
Manual Testing
Steps to run the RSpec test plan mentioned above:
1) cd into the expertiza directory
2) Run the command: rspec spec/helpers/review_mapping_helper_spec.rb
Steps to test the implemented feature:
1) Login to Expertiza as an instructor, with the username - instructor6 and the password - password
2) Under the Manage.. tab on the header, click Assignments.
3) Click the Assignments tab on the page, in case the Courses are loaded(which happens by default sometimes)
4) For any assignment, click View Review Report icon
5) That will take you to the review report page
6) To choose a metric (Average Volume (default), Sentiment), choose from the Metric dropdown and click Select
7) The page will reload with the appropriate metric
8) You can sort using the arrows in the column header.
If you have any issue with testing the feature, check out the Youtube video: