PeerLogic Web Services: Summarization Service

Summarization Service

The extent of feedback in peer review applications could easily overwhelm the student, which may cancel out the desired effects of helping students to identify their strengths and weaknesses related to the assignment. Our preliminary study shows that within traditional classes that are enrolled in Expertiza, the students get feedback from 3-5 reviewers. In a few cases, students could even get feedback from more than 10 reviewers. Each feedback could be very extensive depending on the given rubric. In Expertiza, we found that the rubric contains on average, 8-9 criteria, with two courses even having 159 criteria. Each student / team receives reviews from 5 reviewers on average, and the highest number of reviews was from 72 reviewers, within multiple rounds. The number of words in the feedback that each student / team receives from multiple reviewers on average is 175 words, and the most extensive feedback reaching 8500 words. When assuming that a sentence in average consist of 15 words, it means that each reviewee gets approximately 11-12 sentences, but it could also reach 566 sentences, which would clearly be overwhelming. It is even worse when the students have to read feedback from multiple reviewers that sounds repetitive.

One way to improve this situation is to provide a summary of feedback when the amount has grown extensive. There are a few different ways that summaries could be used in peer review systems. First, the instructor could benefit from having a summary of the qualitative feedback that his students get from their peers. It allows the instructor to sense the general of the problems of his class in that particular assignment. On the student side, having a summary of the feedback could also help them to get a quick glimpse of their strength and weaknesses. This paper focuses on studying providing the summaries for the students.

Summarization

A summary for the students could be visualized differently. For instance, when the peer review systems rely on rubrics, a summary could be provided for each piece of feedback given for a criterion such as depicted in figure above. Alternatively, the summary could be presented as a holistic narrative that includes the rubric and the feedback. The summary could also be presented as bullet points grouped under the tone polarity that may resemble the pros and cons of the work. For this study, we choose to show the summary as a narrative since it is the most compact form to show the summary.

To help implement automatic summarization in peer review applications, we provide a web service that can be used to generate summaries automatically. The summarization web service uses python sumy library, which already integrated the following algorithms:

Luhn – heurestic method, reference Edmundson heurestic method with previous statistic research, reference Latent Semantic Analysis, LSA – one of the algorithm from http://scholar.google.com/citations?user=0fTuW_YAAAAJ&hl=enI think the author is using more advanced algorithms now. Steinberger, J. a JeĹľek, K. Using latent semantic an and summary evaluation. In In Proceedings ISIM ‘04. 2004. S. 93-100. LexRank – Unsupervised approach inspired by algorithms PageRank and HITS, reference TextRank – some sort of combination of a few resources that I found on the internet. I really don’t remember the sources. Probably Wikipedia and some papers in 1st page of Google 🙂 SumBasic – Method that is often used as a baseline in the literature. Source: Read about SumBasic KL-Sum – Method that greedily adds sentences to a summary so long as it decreases the KL Divergence. Source: Read about KL-Sum

At the moment, we focus on providing a hosted web service, since we haven’t prepared an easy way for doing a local installation.

To use the web service to summarize reviews, it requires that reviews to an artifact (submission) to be splitted up into sentences and put into an array. The array should be serialized into Json and send to the following URL via POST message: “http://peerlogic.csc.ncsu.edu/sum/[suffix]”. The following is an example of the input and output of the web service:

input JSON : {“sentences”:[“sentence1”, “sentence2”, “sentence3”]}
output JSON :{ “summary“: “summary1” }

the suffix could be one of the following :

/sum/v1.0/summary

   Summarize a given set of sentences

/sum/v1.0/summary/{length}

   Summarize a given set of sentences and length of the summary
   {length} determines the length of the output summary

/sum/v1.0/summary/{length}/{algorithm}

   Summarize a given set of sentences, length of the summary, and type of algorithm
   {length} determines the length of the output summary
   {algorithm} determines the algorithm to be used. please choose on of these: textrank, lexrank, luhn, edmonson, kl, lsa, sumbasic, random

Code Example

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.ProtocolException;
import java.net.URL;

import com.google.gson.Gson;

public class SummarizationClient {
 
 public SummarizationClient(){
 
 }
 
 public static void main(String[] args) {
   Sentences s = new Sentences();
 
   //prepare input array of sentences
   String[] arrayOfSentences = {"I'll start by saying that the rose gold 6S is very much pink color.", 
   "It's not a yellow-pink color as the name rose gold would suggest.", 
   "It's a dusty rose pink.", 
   "Today everyone I've talked to has asked me if the pink 6S was actually a pretty pink color, or a yellowish-pink shade.", 
   "Yep, it's pink!", 
   "The timing couldn't have been more perfect for me this year, with the announcement of the new iPhone 6S.", 
   "I got my 4S a few years back, just months before the iPhone 5 was announced.", 
   "I've loved my little 4S, and up until recently, I had not really been planing on upgrading to a newer model.", 
   "I'm on an endless mission to lower my family's ever growing cell phone bill, not raise it.", 
   "So the added length of the 5 models weren't enough of a size difference to make me want to upgrade.", 
   "Then when the new larger size of the iPhone 6 came out, my first reaction was that I wanted it.", 
   "However there was a bit of a concern last year with how bendable the iPhone 6 was, and also that Apple had not included a more durable glass that some had originally thought they might.", 
   "This stronger glass was a feature I had been looking forward to.", 
   "My husband and teenage son both use ruggedized smartphones by Casio and Kyocera, and I've seen what those phones can survive.", 
   "My husband has dropped his phone in a lake while fishing, tosses his phone anywhere without the slightest concern for the glass, and my son drops his phone on every hard surface possible, and the screens not only stay in one piece, but without a scratch on them and work just fine even after being completely under water.", 
   "While I didn't need the extreme toughness of being waterproof, years of seeing my friend's iPhones with spider web cracks covering their screens made me want to hold out a little longer.", 
   "My 4S was still small enough and thick/bulky enough that I hadn't damaged the screen.", 
   "Rumors of an iPhone with a stronger glass made me want to hold out just a little longer.", 
   "Then last month my old 4S started acting up.", 
   "I stopped getting notifications of any kind, my phone would drop calls and the internet was getting painfully slow.", 
   "I knew it was time.", 
   "I thought maybe when Apple released their new phones, I could pick up last years model a bit cheaper.", 
   "Then I saw it.", 
   "ROSE GOLD.", 
   "That was the shiny bait that made me look closer.", 
   "What a beautiful color.", 
   "And then I saw the words I had been waiting for.", 
   "That the glass on the iPhone 6S is made using a process that makes it stronger and the most durable in the smartphone industry.", 
   "The best of both worlds, large phone/screen and strong glass.", 
   "And of course....pink.", 
   "So I pre-ordered it and from the moment I received it, have been blown away by the features.", 
   "The screen resolution is stunning, the photos it takes are beautiful and the processor works FAST.", 
   "The Live Photos feature surprised me the first time I was looking through my pictures.", 
   "I took a photo of one of our dogs to send to my daughter via a text and when I went to select it, my dog's tail was wagging! Describing it, it sounds like it's just a clip of a short video, but it really is a little different.", 
   "When you're scrolling through photos and each one has a second of movement to it, it's wild.", 
   "It's almost like looking through a living photo album.", 
   "The fingerprint touch ID to unlock the phone and order iTunes music works really well.", 
   "I was a bit concerned at first I was adding even more steps to unlock my phone each time I wanted to do some small task, like check the time.", 
   "But using the fingerprint ID gets me in even faster, since I'm not swiping my finger across the screen to unlock it any longer.", 
   "For me, the most disappointing feature of the phone was putting a case on it! Having to cover the beautiful rose gold color.", 
   "However I've ordered a new clear case to help with that.", 
   "Extra protection and I can still see the gorgeous phone color.", 
   "Upgrading to this new 6S has been worth every penny.", 
   "With the super fast processor, beautiful rose gold color, Live Pictures and stronger glass, I'm so glad I waited the extra year.", 
   "Loving everything about it."};
   
   
   Gson gson = new Gson();   
   // put the array of sentences into the input object
   s.sentences = arrayOfSentences;
   //serialize the input object into json
   String input = gson.toJson(s); 
 
   try {
     //call summarization web service and tell it to produce a summary with a length of 3 sentences (the number in the end of the URL is the expected length)
     URL url = new URL("http://peerlogic.csc.ncsu.edu/sum/v1.0/summary/3"); 
     //call the web service
     String response = invokeSummarizationSvc(input, url);
     //print the json result
     System.out.println("Output from Server : " + response);
 
   } catch (MalformedURLException e) {
     e.printStackTrace();
   } catch (IOException e) {
     e.printStackTrace();
   }
 }

 private static String invokeSummarizationSvc(String input, URL url) throws IOException, ProtocolException {
   HttpURLConnection conn = (HttpURLConnection) url.openConnection();
   conn.setDoOutput(true);
   conn.setRequestMethod("POST");
   //content type must be set to Json, otherwise the server doesn't know how to parse the input
   conn.setRequestProperty("Content-Type", "application/json");
   OutputStream os = conn.getOutputStream();
   os.write(input.getBytes());

   if (conn.getResponseCode() != HttpURLConnection.HTTP_OK) {
     throw new RuntimeException("Failed : HTTP error code : "
       + conn.getResponseCode());
   }
 
   BufferedReader br = new BufferedReader(new InputStreamReader((conn.getInputStream())));

   String line = "", lines = ""; 
   while ((line = br.readLine()) != null){
     lines += line + "\n";
   }
 
   conn.disconnect();
 
   return lines;
 }
}

//class to serialize the text into Json
class Sentences {
 public String[] sentences;
}