CSC/ECE 517 Fall 2019 - E1989. Track the time students look at other submissions

From Expertiza_Wiki
Jump to navigation Jump to search

Introduction

The Expertiza project takes advantage of peer-review among students to allow them to learn from each other. Tracking the time that a student spends on each submitted resources is meaningful to instructors to study and improve the teaching experience. Unfortunately, most peer assessment systems do not manage the content of students’ submission within the systems. They usually allow the authors submit external links to the submission (e.g. GitHub code / deployed application), which makes it difficult for the system to track the time that the reviewers spend on the submissions.

Problem Statement

Knowing how much time a student spends on a review is helpful when determining the quality of the review itself. That being said Expertiza needs to be able to track and display how much time a student spends on each review.

The time spent on a review is the sum of multiple sources:

  1. The time spent on the Expertiza assignment review page itself
  2. The time spent looking at external pages linked from the review page
  3. The time spent looking at downloadable files submitted by other students

The purpose of this project three-fold:

  1. Gather the timing data from the sources above.
  2. Display the data on the "Review Report" page (views/review_mapping/_review_report.html.erb) to show the time the student spent per review. Given that previous attempts have been rejected due to UI issues, it is important that this data is displayed cleanly.
  3. Write tests that confirm that the above goals are functioning correctly.

Current Implementation

So far, Expertiza does not have any such feature. However, three teams have already worked on this in the past but their builds were not merged due to some problems.

  1. E1705 identified how to track the active time of windows opened from the submitted links. (Wiki)
  2. E1791 provided detailed insights on how they planned to track time taken by a student in viewing a submission and possible edge cases. Further, they also implemented popups and figured out a way to open downloadable files. However, the details are rendered in a not-so-friendly manner and hence it was not merged. (Wiki)
  3. E1872 tried to solve this by incorporating the statistics in the review reports page, but their UI made the page cluttered and not friendly. Further, it was hard to identify which statistic belonged to which review, and there were almost no tests. (Wiki)

Proposed Solution

After investigating the prior attempts at this task, we have decided it would be best to begin our project by building off of the work done in project E1791. The reason for choosing to build off of this particular project is because they have already put the work into tracking time spent viewing external pages as well as time spent viewing certain types of downloadable files. In order to achieve our goals outlined in the Problem Statement section, the following changes need to be made:

The time spent on the Expertiza assignment review page needs to be tracked.
  • Due to Expertiza generating report text boxes with HTML iFrames, we will track whether or not the document hasFocus() to determine when a student is on the page or not.
  • To prevent the user from cheating the system by just keeping the review page open without doing work, a timeout feature will be implemented. After 5 minutes of mouse/keyboard inactivity, a popup will be displayed asking if the user is still working. At that point, the time contributed towards the total by the Expertiza page will stop being tracked until the user interacts with the popup to indicate they are still working. This is already implemented in project E1791.
The time spent viewing the external links and downloadable files will need to be made more accurate.
  • Currently, if a student has an external link open as well as the Expertiza page, time is being tracked for both. Changes will be made so that when the student is working on the Expertiza assignment review page time is not tracked for the external links or downloadable files.
The overall time spent on the review needs to be displayed in a "user friendly manner" on the "Review Report" page.
  • Due to complaints on a tabular method, citing the review report table becoming too cluttered, we intend to create a pop-up window that will display the results in a table. The entries in "Team Reviewed" will be clickable. When clicked, they will display a popup that contains detailed information on where the time for the review was spent. For example, if a student spent a total of 22 minutes on the review, it will show that the student spent 5 minutes on the Expertiza review page, 10 minutes looking at external links, and 7 minutes looking at downloadable files. It will display these details in text format as well as graphically using a pie chart. The purpose of choosing this design is two-fold:
  1. It will not require the instructor to have to go to a different page every time they want more details on a particular review. Given that the review report page takes a substantial time to load, this is a necessity.
  2. It will prevent the table on the review report page from being cluttered by figures and too much data.

Design pattern

Strategy: In this project, for each review page, we need to deal with different type of links. For online links, such as github repo, youtube link, etc, we use javascript to open a new window detect time when the window is closed. For submitted files, such as txt, jpg, we will open it with views/response/_submitted_files.html.erb. For other files that need to be download and open locally, we can only make approximation for review time.

Flowchart

Reviewer
Every time a start time is logged for expertiza/link/file, a new entry will be created in the database.End time will be updated on the last entry present for the same link/file.



Instructor
Time for individual links will summed and then displayed as bar graph.

Code Changes

1. Created a controller to handle the start/end times of the files viewed by the user. => controllers/submission_viewing_events_controller.rb

2. Created a table in database to log start and end time for each link/file. It's schema is described below.

Schema for table submission_viewing_events
Field Type Null Key
id int(11) NO Primary Key
map_id int(11) YES
round int(11) YES
link varchar(255) YES
start_at datetime YES
end_at datetime YES
created_at datetime NO
updated_at datetime NO

3. Added view that will display with pie chart when instructor wants to see time spent on individual links/files.

Files that have been Added or Changed

submission_viewing_events_controller.rb
 The record_start_time function starts the time for links that have been visited by the user. 

def record_start_time
  param_args = params[:submission_viewing_event]
  # check if this link is already opened and timed
  submission_viewing_event_records = SubmissionViewingEvent.where(map_id: param_args[:map_id], round: param_args[:round], link: param_args[:link])
  # if opened, end these records with current time
  if submission_viewing_event_records
    submission_viewing_event_records.each do |time_record|
	  if time_record.end_at.nil?
	    # time_record.update_attribute('end_at', start_at)
	    time_record.destroy
	  end
    end
  end
  # create new response time record for current link
  submission_viewing_event = SubmissionViewingEvent.new(submission_viewing_event_params)
  submission_viewing_event.save

  #if creating start time for expertiza update end times for all other links.
  if param_args[:link]=='Expertiza Review' 
    params[:submission_viewing_event][:link] = nil
    params[:submission_viewing_event][:end_at] = params[:submission_viewing_event][:start_at]
    record_end_time()
  end
  render nothing: true
end


The record_end_time function records the end time for links that have been visited by the user. 

def record_end_time
  data = params.require(:submission_viewing_event)
  if data[:link].nil?
    submission_viewing_event_records = SubmissionViewingEvent.where(map_id: data[:map_id], round: data[:round], end_at: nil).where.not(link: "Expertiza Review")
  else
    submission_viewing_event_records = SubmissionViewingEvent.where(map_id: data[:map_id], round: data[:round], link: data[:link])
  end
  submission_viewing_event_records.each do |time_record|
    if time_record.end_at.nil?
	time_record.update_attribute('end_at', data[:end_at])
	# break
    end
  end
  respond_to do |format|
    format.json { head :no_content }
  end
end


The mark_end_time function records the end time for links that have no end times. 

def mark_end_time
  data = params.require(:submission_viewing_event)
  @link_array = []
  submission_viewing_event_records = SubmissionViewingEvent.where(map_id: data[:map_id], round: data[:round])
  submission_viewing_event_records.each do |submissionviewingevent_entry|
    if submissionviewingevent_entry.end_at.nil?
	@link_array.push(submissionviewingevent_entry.link)
	submissionviewingevent_entry.update_attribute('end_at', data[:end_at])
    end
  end
  respond_to do |format|
    format.json { render json: @link_array }
  end
end


reviewer_details_popup.html.erb
_review_report.html.erb
_review_submissions_time_spent.html.erb
response_times.coffee
response_times.scss
response_controller.rb
response_times_controller.rb
responses_times_helper.rb
review_mapping_helper.rb
response_time.rb
_submitted_files.html.erb

Algorithm For Precise Timing Accuracy

Previous implementations attempted to record the time users spent viewing each link, but their solution has multiple problems. The primary problem was that when multiple links were open at the same time (thus overlapping with one another) the times at which they overlap are recorded twice. To fix this, we implemented the following algorithm within \app\views\reports\_review_submissions_time_spent.html.erb: 

Consider that each link duration (the amount of time that link is being looked at) has a starting and ending edge (see diagram below). These durations are stored in an array durations_to_consider.

Iterate through every link duration in durations_to_consider:
    If any other link's start and end times both fit within our start and end time:
        Remove from durations_to_consider

For list of durations_to_consider, add those durations' start and end edges to edge_array.

Ensure that entries in edge_array are sorted from earliest to latest.

Create list duration_sections.

Start at the first edge in edge_array. Go until next edge. Record the time difference as a duration and add time to duration_sections.

    If, after we add an edge that corresponds to the ending edge of a link duration, there are no open start edges (a start edge that belongs to a link, that does not yet have an accompanying end edge that belongs to that link): 
        Count the NEXT start edge as the start edge of the next duration section. This will prevent empty space caused by no link being open at that time from being counted towards the total.

Total up the time for all durations to get the total amount of time spent on links.

NOTE: The following diagrams are simply meant to help explain the timing algorithm. They are not indicative of the real system. For example, there will never be a time when an Expertiza duration overlaps with a link duration.

The following diagram shows an example of what links could look like prior to our algorithm running:

After our algorithm runs, one can see that the overlapping segments are eliminated. For convenience, each duration_section is highlighted:

Test Plan

Automated Testing Using RSpec

We have made changes to the controllers/submission_viewing_events_controller.rb and added test cases for the same, which have passed all the test cases. The tests can be executed rpec spec and the results for the files we have modified are shown below: 

Randomized with seed 20933
.....

Finished in 10.31 seconds (files took 7.85 seconds to load)
5 examples, 0 failures

Manual UI Testing

We hope to not need any manual UI testing, though if it is needed, it'll look something like this:

Verifying Time is Tracked Correctly:

  1. Sign in as as a student
  2. Review an assignment
  3. Go through the review as normal
    1. Make sure you open links to github or pull requests
    2. Make sure you download and/or view files that are attatched
  4. Submit your review when you're done
  5. Log out as a student
  6. Log in as an instructor
  7. Navigate to the review report tab
  8. You should see a new column detailing the time the user spent on the review.

Our Work

The code we created can be found below.

The project could be run locally by cloning the GitHub Repository and then running the following commands sequentially. 

bundle install
rake db:create:all
rake db:migrate
rails s

Issues found and fixes

  1. If a student only saves the review and does not submit it before the due date, the response exists but the entry is not updated as submitted. Because of this the response entry is not updated with the end time and hence when we try to calculate the total response object time, it calculates that as zero and hence shows "Review hasn't been submitted". The response table should be updated once it's past due date. This was out of scope for this project hence we leave it as future fix.
  2. The "Save review after every 60 seconds" checkbox does not work correctly, hence we defaulted that to unchecked as opposed to previous implementation where it was checked, because it hampers with our implementation. This is another fix required.
  3. The submission_viewing_event table increases in size very rapidly as it stores start and end times for each link if a particular event occurs. Solution to that would be to save all the entries locally in users system and update the database only with expertiza time and link time after performing the time calculation, once the user clicks Save/Submit. This would prevent the database from saving unnecessary amount of information and also make it work faster.

Team Information

  1. Dylan Spruill (drspruil)
  2. Forrest Devita (fcdevita)
  3. Rohan Pillai (rspillai)
  4. Shalin Rathi (sjrathi)

Mentor: Akanksha Mohan (amohan7)
Professor: Dr. Edward F. Gehringer (efg)

References

  1. Expertiza on GitHub
  2. RSpec Documentation
Retrieved from "https://wiki.expertiza.ncsu.edu/index.php?title=CSC/ECE_517_Fall_2019_-_E1989._Track_the_time_students_look_at_other_submissions&oldid=131193"