CSC/ECE 517 Fall 2018/E1878 Integrate Suggestion Detection Algorithm: Difference between revisions

From Expertiza_Wiki
Jump to navigation Jump to search
 
(27 intermediate revisions by 2 users not shown)
Line 1: Line 1:
CSC-517 Fall 2018 - E1878 Integrate suggestion detection algorithm in Expertiza Project. This Wiki page explains how we are planning to integrate the suggestion detection algorithm in Expertiza project.
CSC-517 Fall 2018 - E1878 Integrate suggestion detection algorithm in Expertiza Project. This Wiki page explains the integration of the suggestion detection algorithm in Expertiza project.


==Introduction==
==Introduction==
Line 11: Line 11:
====Expertiza====
====Expertiza====
[http://expertiza.ncsu.edu/ Expertiza] is an Open Source project based on the [http://rubyonrails.org/ Ruby on Rails] framework, supported by National Science Foundation. It is the software to create reusable learning objects through peer review. It is a project where students can submit and peer review learning objects(articles, code, websites, etc). The users of this software include students and professors. Expertiza is used by select professors and students in North Carolina State University, for example. It supports team projects, reviews of projects/teammates, submission URLs, Wiki pages and certain document types. Instructors can create projects and the students can bid for the projects. The students can be assigned teams for a particular project or they may form their own team with fellow classmates.
[http://expertiza.ncsu.edu/ Expertiza] is an Open Source project based on the [http://rubyonrails.org/ Ruby on Rails] framework, supported by National Science Foundation. It is the software to create reusable learning objects through peer review. It is a project where students can submit and peer review learning objects(articles, code, websites, etc). The users of this software include students and professors. Expertiza is used by select professors and students in North Carolina State University, for example. It supports team projects, reviews of projects/teammates, submission URLs, Wiki pages and certain document types. Instructors can create projects and the students can bid for the projects. The students can be assigned teams for a particular project or they may form their own team with fellow classmates.
====Suggestion detection algorithm====
====Suggestion detection algorithm====
Peer-review systems like Expertiza utilize a lot of students’ input to determine each other’s performance. In the same time, we hope students could also gain knowledge from the reviews received thus improve their own performance. In order to make this happen, we would like to have everyone give quality reviews instead of generic ones.
Peer-review systems like Expertiza utilize a lot of students’ input to determine each other’s performance. In the same time, we hope students could also gain knowledge from the reviews received thus improve their own performance.  
'''Currently, we have a few classifiers that could catch useful components of review comments, such as if it contains suggestions, etc. These classifiers are already ported into web services that we’d like to be integrated into Expertiza.''' As stated in the Problem statement, we will be getting the response from the REST endpoints as given and integrate it with Expertiza. '''According to the problem statement, we will be considering the "Suggestion Detection Algorithm" to be a black box, which take a json with text as input and returns a json with few metrics as given below.'''


''Input to the "Suggestion Detection Algorithm"''
'''Currently, we have a few classifiers that could catch useful components of review comments, such as if it contains suggestions, etc. These classifiers are already ported into web services that we’d like to be integrated into Expertiza.''' As stated in the Problem statement, we get the response from the REST endpoints as given and integrate it with Expertiza. The suggestion detection algorithm from which we are getting the metrics is added in the "References" section in case more details are needed on it. '''According to the problem statement, we will be considering the "Suggestion Detection Algorithm" to be a black box, which take a json with text as input and returns a json with few metrics as given below.'''
Input text is to be passed in the following JSON format to the "Suggestion Detection Algorithm"
{ "text" : "This is an excellent project. Keep up the great work" }


''Input JSON to "Suggestion Detection Algorithm"''
  { "text" : "This is an excellent project. Keep up the great work" }
''Output JSON the "Suggestion Detection Algorithm"''


''Output to the "Suggestion Detection Algorithm"''
Output is returned in the following JSON format:
   {"sentiment_score": 0.9,
   {"sentiment_score": 0.9,
   "sentiment_tone": "Positive",
   "sentiment_tone": "Positive",
Line 31: Line 33:


====Problem statement====
====Problem statement====
#When a student submits a review, we would like to call this web service with the student’s review as the input. We would then want to tell the student whether their reviews contain suggestions or not, so they can make improvements based on the results of the webservice.  
#When a student submits a review, we call this web service with the student’s review as the input. We then tell the student whether their reviews contain suggestions or not, so they can make improvements based on the results of the web service.  
#We would also like to evaluate how much time this API is taking and if possible work a way out to improve it. We don’t want the system to be terribly slow.
#We evaluate how much time this API is taking. We don’t want the system to be terribly slow.
 
==Files modified==
==Files modified==
# app/views/response/response.html.erb
# app/views/response/response.html.erb
# app/views/submitted_content/_self_review.html.erb
 
==Design document==
==Design document==
==Wireframes==
==Wireframes==
====Current State====
====Current State====
In its current state, the student directly writes their review in  segmented text boxes under instructor specified questions with no further checks about the relevance of the review they have written as shown below:
In its current state, the student directly writes their review in  segmented text boxes under instructor specified questions with no further checks about the relevance of the review they have written as shown below:


[[File:Before.jpg]]
====UML Diagrams====
We are not modifying any models / adding any logic that requires an UML diagram. Most of our changes are on HTML. Our logic to be implemented is pre-processing the review and sending it as JSON to the Suggestion Detection API, getting the JSON response from API, post-processing the metrics to output the Suggestion Analysis.
The overall flow of the logic is explained as follows:
[[File:Latest_Prash_screenshot.png]]


[[File:Before.jpg]]
==Implementation==
====Expected State====
 
We plan to implement buttons next to each text box so that the student can check their review for volume, sentiment, suggestion, praise and criticism metrics. The rationale for including buttons right next to each text box, is so that the student gets to test only the written matter that they wish to check, and not the entirety of their review.
===Pre-requisites===
Following are the necessary steps needed to be set up in order to have the feature running :
# Browser: Google Chrome
# Cores plug-in (link in references)
 
Before the JSON response is displayed, certain fields are first filtered out or formatted. For instance, the field "text" is filtered out as it is redundant to display it again. The field "sentiment_tone" is formatted with the help of another font color JSON depending on its value.
Following is the code snippet of our implementation :
 
<pre>
//Suggestion detection API call
      tinyMCE.on('AddEditor', function(e) {
      e.editor.on('blur', function (e) {
        ed = e.target;
        if (ed.id.includes("comments")){
          json_review = JSON.stringify({ "text": ed.getContent({format : 'text'}) })
          $.ajax({
            url: "https://peer-reviews-nlp.herokuapp.com/all",
            type: "POST",
            crossDomain: true,
            contentHeader: "application/javascript",
            contentType: "application/json",
            data: json_review,
            dataType: "json",
            //Response JSON fields are filtered and then displayed
           
            success: function (response) {
    console.log("success");
var APIresultbox ="";
var colors = {"Negative": ":</b> <font style=\"color:red;\">",
"Positive": ":</b> <font style=\"color:green;\">",
"Neutral": ":</b> <font style=\"color:saddlebrown;\">",  
"Mixed": ":</b> <font style=\"color:saddlebrown;\">"};
APIresultbox += "<table border='1'>"
                //Add the response to the API result box
        for (x in response) {
                        if(x == "text"){
                                continue;
                        }
if (x == "sentiment_tone")
APIresultbox += "<tr><td><b> " + x + colors[response[x]] + response[x] + "</font></td></tr>";
else
          APIresultbox += "<tr><td><b> " + x + ":</b> " +response[x] + "</td></tr>";
      }
        APIresultbox += "</table>"
console.log(APIresultbox);
              console.log(response);
            if(ed.id.includes("responses"))
                  $('#' + ed.id.replace("responses", "analysis")).html(APIresultbox);
              else if(ed.id.includes('review_comments'))
                $('#review_comments_analysis').html(APIresultbox);
            },
            error: function (xhr, status) {
      console.log(status)
              $('#save_progress').html('Unable to reach suggestion detection service at the moment')
            }
          });
        }
      });
    }
</pre>


==Screenshot with integrated API==
[[File:E1878Final.jpg]]


[[File:After.jpg]]
===Test plan===
===Test plan===


Line 54: Line 126:
           I want to view a popup displaying the suggestion metrics for the segment of the review I am writing
           I want to view a popup displaying the suggestion metrics for the segment of the review I am writing
           So that I can use it to improve the relevance of the review segment provided
           So that I can use it to improve the relevance of the review segment provided
  Scenario: Display Suggestion Metrics on Empty Review Segment
  Scenario: Display Suggestion Metrics on Empty Review Segment
           Given the Student has not entered any text in the review segment
           Given the Student has not entered any text in the review segment
           When the Student clicks the Analyze button for the segment
           When the Student clicks outside the textbox
           Then a popup will show that the user has not entered any text for suggestion metrics analysis.
           No suggestions are displayed


  Scenario: Display Suggestion Metrics on Review Segment with Text
  Scenario: Display Suggestion Metrics on Review Segment with Text
           Given the Student has entered any form of text in the review segment
           Given the Student has entered any form of text in the review segment
           When the Student clicks the Analyze button for the segment
           When the Student clicks outside the text box for the segment
           Then a popup will show the user the results of the suggestion metrics analysis
           Then a table will show the user the results of the suggestion metrics analysis
 
Scenario: Display Suggestion Metrics on Review Segment with Text
          Given the Student has entered positive text and suggestions in the review segment
          When the Student clicks outside the text box for the segment
          Then a table will show the user the results of the suggestion metrics analysis
 
Scenario: Display Suggestion Metrics on Review Segment with Text
          Given the Student has entered negative text and suggestions in the review segment
          When the Student clicks outside the text box for the segment
          Then a table will show the user the results of the suggestion metrics analysis


==References==
==References==
Line 69: Line 152:
#[http://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://wikis.lib.ncsu.edu/index.php/Expertiza Expertiza project documentation wiki]
#[http://wikis.lib.ncsu.edu/index.php/Expertiza Expertiza project documentation wiki]
#[http://chrome.google.com/webstore/detail/allow-control-allow-origi/nlfbmbojpeacfghkpbjhddihlkkiljbi?hl=en Plug-In: Cores]

Latest revision as of 17:36, 12 December 2018

CSC-517 Fall 2018 - E1878 Integrate suggestion detection algorithm in Expertiza Project. This Wiki page explains the integration of the suggestion detection algorithm in Expertiza project.

Introduction

Team

  1. Ameya Dhavalikar
  2. Komal Kangutkar
  3. Prashanthi Kanniappan Murthy
  4. Vibhav Nandavaram Abbai Srivaths

Background information

Expertiza

Expertiza is an Open Source project based on the Ruby on Rails framework, supported by National Science Foundation. It is the software to create reusable learning objects through peer review. It is a project where students can submit and peer review learning objects(articles, code, websites, etc). The users of this software include students and professors. Expertiza is used by select professors and students in North Carolina State University, for example. It supports team projects, reviews of projects/teammates, submission URLs, Wiki pages and certain document types. Instructors can create projects and the students can bid for the projects. The students can be assigned teams for a particular project or they may form their own team with fellow classmates.

Suggestion detection algorithm

Peer-review systems like Expertiza utilize a lot of students’ input to determine each other’s performance. In the same time, we hope students could also gain knowledge from the reviews received thus improve their own performance.

Currently, we have a few classifiers that could catch useful components of review comments, such as if it contains suggestions, etc. These classifiers are already ported into web services that we’d like to be integrated into Expertiza. As stated in the Problem statement, we get the response from the REST endpoints as given and integrate it with Expertiza. The suggestion detection algorithm from which we are getting the metrics is added in the "References" section in case more details are needed on it. According to the problem statement, we will be considering the "Suggestion Detection Algorithm" to be a black box, which take a json with text as input and returns a json with few metrics as given below.

Input JSON to "Suggestion Detection Algorithm"

 { "text" : "This is an excellent project. Keep up the great work" }


Output JSON the "Suggestion Detection Algorithm"

  {"sentiment_score": 0.9,
 "sentiment_tone": "Positive",
 "suggestions": "absent",
 "suggestions_chances": 10.17,
 "text": "This is an excellent project. Keep up the great work",
  "total_volume": 10,
  "volume_without_stopwords": 6}

Problem statement

  1. When a student submits a review, we call this web service with the student’s review as the input. We then tell the student whether their reviews contain suggestions or not, so they can make improvements based on the results of the web service.
  2. We evaluate how much time this API is taking. We don’t want the system to be terribly slow.

Files modified

  1. app/views/response/response.html.erb

Design document

Wireframes

Current State

In its current state, the student directly writes their review in segmented text boxes under instructor specified questions with no further checks about the relevance of the review they have written as shown below:

UML Diagrams

We are not modifying any models / adding any logic that requires an UML diagram. Most of our changes are on HTML. Our logic to be implemented is pre-processing the review and sending it as JSON to the Suggestion Detection API, getting the JSON response from API, post-processing the metrics to output the Suggestion Analysis.

The overall flow of the logic is explained as follows:

Implementation

Pre-requisites

Following are the necessary steps needed to be set up in order to have the feature running :

  1. Browser: Google Chrome
  2. Cores plug-in (link in references)

Before the JSON response is displayed, certain fields are first filtered out or formatted. For instance, the field "text" is filtered out as it is redundant to display it again. The field "sentiment_tone" is formatted with the help of another font color JSON depending on its value. Following is the code snippet of our implementation :

//Suggestion detection API call
      tinyMCE.on('AddEditor', function(e) {
      e.editor.on('blur', function (e) {
        ed = e.target;
        if (ed.id.includes("comments")){
          json_review = JSON.stringify({ "text": ed.getContent({format : 'text'}) })
          $.ajax({
            url: "https://peer-reviews-nlp.herokuapp.com/all",
            type: "POST",
            crossDomain: true,
            contentHeader: "application/javascript",
            contentType: "application/json",
            data: json_review,
            dataType: "json",
            //Response JSON fields are filtered and then displayed
            
            success: function (response) {
	    	console.log("success");
		var APIresultbox ="";
		var colors = {"Negative": ":</b> <font style=\"color:red;\">", 
				"Positive": ":</b> <font style=\"color:green;\">", 
				"Neutral": ":</b> <font style=\"color:saddlebrown;\">", 
				"Mixed": ":</b> <font style=\"color:saddlebrown;\">"};
		APIresultbox += "<table border='1'>"
                //Add the response to the API result box
        	for (x in response) {
                        if(x == "text"){
                                continue;
                        }
			if (x == "sentiment_tone")
				APIresultbox += "<tr><td><b> " + x + colors[response[x]] + response[x] + "</font></td></tr>";
			else
           			APIresultbox += "<tr><td><b> " + x + ":</b> " +response[x] + "</td></tr>";		
       		}
        APIresultbox += "</table>"
	console.log(APIresultbox);
              console.log(response);
             if(ed.id.includes("responses"))
                  $('#' + ed.id.replace("responses", "analysis")).html(APIresultbox);
              else if(ed.id.includes('review_comments'))
                 $('#review_comments_analysis').html(APIresultbox);
            },
            error: function (xhr, status) {
	      console.log(status)
              $('#save_progress').html('Unable to reach suggestion detection service at the moment')
            }
          });
        }
      });
    }

Screenshot with integrated API

Test plan

Test: ViewSuggestionMetricsForReviewSegment

Feature: As a Student,
         I want to view a popup displaying the suggestion metrics for the segment of the review I am writing
         So that I can use it to improve the relevance of the review segment provided
Scenario: Display Suggestion Metrics on Empty Review Segment
          Given the Student has not entered any text in the review segment
          When the Student clicks outside the textbox
          No suggestions are displayed
Scenario: Display Suggestion Metrics on Review Segment with Text
          Given the Student has entered any form of text in the review segment
          When the Student clicks outside the text box for the segment
          Then a table will show the user the results of the suggestion metrics analysis
Scenario: Display Suggestion Metrics on Review Segment with Text
          Given the Student has entered positive text and suggestions in the review segment
          When the Student clicks outside the text box for the segment
          Then a table will show the user the results of the suggestion metrics analysis
Scenario: Display Suggestion Metrics on Review Segment with Text
          Given the Student has entered negative text and suggestions in the review segment
          When the Student clicks outside the text box for the segment
          Then a table will show the user the results of the suggestion metrics analysis

References

  1. NLP Project for suggestions algorithm
  2. The live Expertiza website
  3. Expertiza on GitHub
  4. Expertiza project documentation wiki
  5. Plug-In: Cores