CSC/ECE 517 Fall 2019 - E1993 Track Time Between Successive Tag Assignments

Introduction

The Expertiza project takes advantage of peer-review among students to allow them to learn from each other. Tracking the time that a student spends on each submitted resources is meaningful to instructors to study and improve the teaching experience. Unfortunately, most peer assessment systems do not manage the content of students’ submission within the systems. They usually allow the authors submit external links to the submission (e.g. GitHub code / deployed application), which makes it difficult for the system to track the time that the reviewers spend on the submissions.

Project Description

CSC/ECE 517 classes have helped us by “tagging” review comments over the past two years. This is important for us, because it is how we get the “labels” that we need to train our machine-learning models to recognize review comments that detect problems, make suggestions, or that are considered helpful by the authors. Our goal is to help reviewers by telling them how helpful their review will be before they submit it.

Tagging a review comment usually means sliding 4 sliders to either side, depending on which of four attributes it has. But can we trust the tags that students assign? In past semesters, our checks have revealed that some students appear not to be paying much attention to the tags they assign: the tags seem to be unrelated to the characteristic they are supposed to rate, or they follow a set pattern, like repeated patterns of one tag yes, then one tag no. Studies on other kinds of “crowdwork” have shown that the time spent between assigning each label indicates how careful the labeling (“tagging”) has been. We believe that students who tag “too fast” are probably not paying enough attention, and want to set their tags aside to be examined by course staff and researchers..

Proposed Approach

We would like to modify the model reflecting review tagging actions (answer_tag entity), adding new fields to track the time interval between each tagging action, as well as revise the controller to implement this interval tracking functionality. A few things to take into consideration: A user might not tag reviews in sequence, they may jump through them and only tag the one he or she is interested When annotator tags 1 rubric of all reviews then move onto the next, their behavior will be much different compared with those who tags 4 rubrics of each review. Sometimes an annotator could take long breaks to refresh and relax, some even take days off, these irregularities needs to be handled. A user may slide the slider back and forth for a number of times, then go to the next slider; they may also come back and revise a tag they made, these needs to be treated differently. The first step would be to examine the literature on measuring the reliability of crowdwork, and determine appropriate measures to apply to review tagging. The next step would be to code these metrics and call them from report_formatter_helper.rb, so that they will be available to the course staff when it grades answer tagging. There should also be a threshold set so that any record in the answer_tags table created by a user with suspect metrics would be marked as unreliable (you can add a field for that to the answer_tags table).

Behavior Diagram for Instructor/TA Diagram

Flow Chart

Proposed Test Plan

Automated Testing Using Rspec

We have proposed to use Rspec for testing the functionality of our project. Automated tests are carried out to see whether for each tagged review is the time getting saved. The main functionality to be tested is that is the controller saving the time between consecutive review taggings. This time is then saved to the database which is later used for analysis whether the tagging of the review was done properly or not. This is analysis is beyond the scope of our project. We also propose to develop certain use cases which include edge cases like if a user leaves the tagging process for sometime(like half and hour) and comes back to resume the tagging process. This is one of the cases which needs to be tested apart from the main test which is to see whether the time between consecutive reviews is getting saved in the database.

Team Information

1. Anmol Desai(adesai5@ncsu.edu)
2. Dhruvil Shah(dshah4@ncsu.edu)
3. Jeel Sukhadia(jsukhad@ncsu.edu
4. YunKai "Kai" Xiao(yxia28@ncsu.edu)

Mentor: Mohit Jain(mjain6@ncsu.edu)

References

1. Expertiza on GitHub
2. RSpec Documentation