CSC/ECE 517 Fall 2020 - E2080. Track the time students look at other submissions

Introduction

The Expertiza project takes advantage of peer-review among students to allow them to learn from each other. Tracking the time that a student spends on each submitted resources is meaningful to instructors to study and improve the teaching experience. Unfortunately, most peer assessment systems do not manage the content of students’ submission within the systems. They usually allow the authors submit external links to the submission (e.g. GitHub code / deployed application), which makes it difficult for the system to track the time that the reviewers spend on the submissions.

Problem Statement

Expertiza allows students to peer review the work of other students in their course. To ensure the quality of the peer reviews, instructors would like to have the ability to track the time a student spends on a peer-review. These metrics need to be tracked and displayed in a way that the instructor is able to gain valuable insight into the quality of a review or set of reviews.

Various metrics will be tracked including

1. The time spent on the primary review page
2. The time spent on secondary/external links and downloadables

Previous Implementations

1. E1791 were able to implement the tracking mechanism for recording the time spent on looking at submissions that were of the type of links and downloadable files. The time spent on links were tracked using window popups when they are opened and closed. The downloadable files of the type text and images were displayed on a new HTML page to track the time spent on viewing that. Each time a submission link or downloadable file is clicked by the reviewer, a new record is created in the database. This causes a lot of database operations which degrades the performance of the application. Also, the way in which the results are displayed is not user-friendly. Other than these issues, the team provided a good implementation of the feature.
2. E1872 started with E1971's implementation as their base and tried to display the results in a tabular format. The table included useful statistics but it is being displayed outside the review report to the right side which does not blend in with the review report table. Also, the table is hard to read as a lot of information is presented in a cluttered manner. It is hard to map each statistic with its corresponding review. Furthermore, the team did not include any tests.
3. E1989 is the most recent implementation, built from earlier designs, features a solid UI and ample tracking of review time across the board. They started off with project E1791 as their base and focused on displaying the results in a user-friendly manner. For this purpose, the results are displayed in a new window so that it does not look cluttered. The issue of extensive database operations still remains as future work in their project.

Proposed Solution

• From the suggestions of the previous team, E1989, we plan to improve their implementation by reducing the frequency of database queries and insertions. In E1989's current implementation every time a start time is logged for expertiza/link/file, a new entry is created in the database. As a result the submission_viewing_event table increases in size very rapidly as it stores start and end times for each link if a particular event occurs. The solution is to save all entries locally on the users system and once the user presses Submit, save the entry in the database.
• Secondly, E1989's implementation has a decent UI that effectively displays the necessary information to the user. We plan to add minor improvements to their UI to try to improve usability.
• The previous team also mentioned other issues involving the "Save review after 60 seconds" checkbox feature, that may be looked into in the case of extra time.

PStore

PStore is a file based persistent storage method that is based on a hash. Ruby objects (values) can be stored into the data store file by name (keys). Data can therefore easily be read and written to as needed.

To create a PStore file you need the following declaration:

store = PStore.new("filename.pstore")

If the file "filename.pstore" does not exist it will be created with the given name, otherwise the existing data will be read. In this case, "store" will now point to a persistent hash that you can begin to read and write to.

PStore requires all reads and writes to occur within a transaction block. Transaction blocks avoid the problem of multiple processes accessing the same store, since only one transaction can run at a time. Any number of changes can be done in one transaction and a transaction can be ended prematurely using either abort() or commit().

See here for more information on PStore

Design pattern

To preserve the logic implemented by team E1989, our Local Storage class is designed to have the same methods you would use to access an entry in a database, but instead interfaces with a PStore file. This allowed us to refactor E1989's methods in the "submission_viewing_events" controller without potentially breaking their implementation.

For instance, in team E1989's method "record_start_time" the line:

submission_viewing_event_records = SubmissionViewingEvent.where(map_id: param_args[:map_id], round: param_args[:round], link: param_args[:link])

is intended to check if a link is already opened and timed by looking up the link via the "where" method. This line was refactored to:

store = LocalStorage.new()
submission_viewing_event_records = store.where(map_id: param_args[:map_id], round: param_args[:round], link: param_args[:link])

The "where" method is instead an instance method of the LocalStorage class that looks up the link in the same way, but in the PStore file rather than the database.

To solve the issue of too many viewing event entries in the database, two separate kinds of "save" methods were written. "save" saves a viewing event entry into the PStore file, while "hard_save" saves entries stored in the PStore file, in the database. This allows us to save entries into the PStore file and only officially save those entries in the database once the review is complete.

Our method "remove" in the LocalStorage class removes entries from the PStore file. This method is called in the "submission_viewing_events" controller after saving an entry into the database.

Flowchart

The following is the flowchart presented by E1989 that does a good job outlining the logic for tracking the time students spend reviewing:

The following flowchart outlines how PStore and its methods work together to emulate methods used to access a database

Code Changes

Our implementation is centered around two different classes and the default 'pstore' gem

Our two classes work together to limit data written directly to the database.

Local Submitted Content

This class serves as a sort of "instance" or a standard for representing the tracking of the time for a single link which was visited.

class LocalSubmittedContent
    attr_accessor :map_id, :round, :link , :start_at, :end_at, :created_at, :updated_at

    def initialize(**args)
      @map_id = args.fetch(:map_id,nil)
      @round = args.fetch(:round,nil)
      @link = args.fetch(:link,nil)
      @start_at = args.fetch(:start_at,nil)
      @end_at = args.fetch(:end_at,nil)
      @created_at = args.fetch(:created_at,nil)
      @updated_at = args.fetch(:updated_at,nil)
    end

    def initialize(args)
      @map_id = args.fetch(:map_id,nil)
      @round = args.fetch(:round,nil)
      @link = args.fetch(:link,nil)
      @start_at = args.fetch(:start_at,nil)
      @end_at = args.fetch(:end_at,nil)
      @created_at = args.fetch(:created_at,nil)
      @updated_at = args.fetch(:updated_at,nil)
    end
    
    def to_h()
        return {map_id: @map_id, round: @round, link: @link, start_at: @start_at, end_at: @end_at, created_at: @created_at, updated_at: @updated_at}
    end

    def ==(other)
      return @map_id = other.map_id && @round == other.round && @link == other.link &&
      @start_at == other.start_at && @end_at == other.end_at &&
      @created_at == other.created_at && @updated_at == other.updated_at
    end

  end

LocalStorage

This class serves as the medium for transaction of data between the local @registry instance variable, pstore and the database

It manages the pstore "hash" by restricting access and saving instances of LocalSubmittedContent when required.

It has a lookup feature with .where() where the user can search for LocalSubmittedContent given specified parameters

Lastly it provides the functionality to save the entries from pstore into the database

class LocalStorage

    def initialize()
      @registry = []
      @pstore = PStore.new("local_submitted_content.pstore")
      @pstore.transaction do
        @pstore[:registry] ||= []
      end
      @registry = read()
    end

    def save(instance)
        @pstore.transaction do
          @registry << instance
          @pstore[:registry] = @registry
        end
    end

    def sync()
      @pstore.transaction do
        @pstore[:registry] = @registry
      end
    end

    # Find all entries that meet every field in the params hash
    # return list of matching entries 
    def where(params)
        found = []

        @registry.each do |item|
          if item.to_h().values_at(*params.keys) == params.values
            found << item
          end
        end
        return found
    end


    # Reads and returns data from Pstore registry
    def read()
      @pstore.transaction do 
        return @pstore[:registry]
      end
    end

    # Actually saves into the database
    def hard_save(instance)
        return SubmissionViewingEvent.create(instance.to_h())
    end

    def hard_save_all()
      @registry.each do |item|
        SubmissionViewingEvent.create(item.to_h())
      end
    end

    def remove(instance)
      @registry.each_with_index do |item,i|
        if item.to_h() == instance.to_h()
          @registry.delete_at(i)
        end
      end
      sync()
    end

    def remove_all()
      @registry = []
      sync()
    end

  end

Test Plan

Automated Testing Using RSpec

The main feature that we want to test is that the intermediate review timings that we are planning to store on the browser's local storage are being accurately stored or not. There are already tests written to test the total time tracking feature. We will be adding additional tests to test the storage and retrieval of the intermediate timings.

Manual UI Testing

Our major focus for this project is to change the current implementation to use significantly less number of database operations by storing the intermediate timings in the local storage of the browser and write only the final time spent on viewing the submissions. So, the only manual UI testing that can be performed is to check if the total time spent by students is accurately tracked and can be viewed by the instructor.

This can be tested using the following steps:

1. Log in to Expertiza as a student and go to the assignments tab.
2. Click on a particular assignment and review the submission by click on links and spending some time viewing them.
3. After reviewing is done, submit the review.
4. Log out from the student account.
5. To look at the time spent on the reviews, log in as the instructor.
6. Click on the assignments tab.
7. Click on the review report icon beside the specific assignment that you reviewed using the student's account.
8. You should be able to see the total time spent on the submissions by the student.

LocalSubmittedContent RSpec Tests

RSpec tests were written to test the following LocalSubmittedContent methods:

• initialize
• to_h
• ==

LocalStorage RSpec Tests

RSpec tests were written to test the following LocalStorage methods:

• initialize
• save

describe "#save" do
            it "should save a instance to the Pstore registry" do
                storage = LocalStorage.new()
                content = LocalSubmittedContent.new(map_id: 1, round: 2, link: "http://google.com",start_at: "2017-12-05 19:11:52", end_at: 
                "2017-12-05 20:11:52" )
                storage.save(content)
                expect(storage.instance_variable_get(:@registry).include? content).to be
            end
        end

• sync

describe "#sync" do
            it "should fetch the registry from pstore" do
                storage = LocalStorage.new()
                content = LocalSubmittedContent.new(map_id: 1, round: 2, link: "http://google.com",start_at: "2017-12-05 19:11:52", 
                end_at: "2017-12-05 20:11:52" )
                storage.save(content)
                storage = nil
                storage = LocalStorage.new()
                expect(storage.sync()[0] == content).to be(true)
            end
        end

• where

describe "#where" do
            it "should retrieve a single matching instance from Pstore registry" do
                storage = LocalStorage.new()
                content = LocalSubmittedContent.new(map_id: 1, round: 2, link: "http://google.com",start_at: "2017-12-05 19:11:52", 
                end_at: "2017-12-05 20:11:52" )
                content1 = LocalSubmittedContent.new(map_id: 3, round: 2, link: "http://google.com",start_at: "2017-12-05 19:11:52", 
                end_at: "2017-12-05 20:11:52" )
                storage.save(content)
                storage.save(content1)
                expect(storage.where(map_id:1)[0].to_h()).to eq(content.to_h())
            end
        end

• read

describe "#read" do
            it "should pull updated data from pstore" do
                storage = LocalStorage.new()
                pstore = PStore.new("local_submitted_content.pstore")
                registry = nil
                pstore.transaction do 
                    registry = pstore[:registry]
                end 
                expect(storage.read()).to eq(registry)
            end
        end

• hard_save

describe "#hard_save" do
            it "should save a instance to the database" do
                storage = LocalStorage.new()
                content = LocalSubmittedContent.new(map_id: 1, round: 2, link: "http://google.com",start_at: "2017-12-05 19:11:52", end_at: "2017-12-05 20:11:52" )
                expect(storage.hard_save(content)).not_to be_nil
            end
        end

• hard_save_all
• remove

describe "#remove" do 
            it "should remove a instance from pstore" do
                storage = LocalStorage.new()
                content = LocalSubmittedContent.new(map_id: 13, round: 2, link: "http://google.com",start_at: "2017-12-05 19:11:52", end_at: "2017-12-05 20:11:52" )
                storage.save(content)
                storage.remove(content)
                expect(storage.read().include?(content)).to be_falsy
            end
        end

• remove_all

Helpful Links

1. Our Fork
2. Our Pull Request
3. Our Video

Identified Issues

1. The "Save review after every 60 seconds" checkbox does not work correctly, hence we defaulted that to unchecked as opposed to previous implementation where it was checked, because it hampers with our implementation.
2. Because methods in the submission_viewing_event controller were modified to implement LocalStorage, there should be Rspec tests added to test those refactored methods.

Team Information

1. Luke McConnaughey (lcmcconn)
2. Pedro Benitez (pbenite)
3. Rohit Nair (rnair2)
4. Surbhi Jha (sjha6)

Mentor: Yunkai 'Kai' Xiao (yxiao28)
Professor: Dr. Edward F. Gehringer (efg)

References

1. Expertiza on GitHub
2. RSpec Documentation