CSC/ECE 517 Spring 2015 S1524 FSZZ

E1524. Refactor staggered-deadline assignments

Overview

Introduction to Expertiza

Expertiza is a web application where students can submit and peer-review learning objects (articles, code, web sites, etc). It is used in select courses at NC State and by professors at several other colleges and universities<ref>Expertiza. Github</ref>.

Staggered Deadlines

Staggered deadlines involve planning alternate submission dates for papers, projects, or exams when a student has conflicting due dates for these. The key part of staggered deadlines is the planning. Staggered deadlines are always established well in advance of the scheduled due date. It is the advanced planning of these deadlines that makes them "staggered deadlines" rather than extensions<ref>Elmichaels. Difficulties with Staggered Deadlines. Jan 15, 2013</ref>.

Documentation

• An instructor manual, explaining how to create an deploy an assignment in Expertiza.
• An instructor video, slightly dated, showing how to create and deploy an assignment
• A guide for Creating_Custom_Rubric
• For students, a Powerpoint or PDF presentation explaining how to submit and review an assignment with Expertiza.
• For students, a video showing how to use the system to submit and review an assignment.
• For students, a Powerpoint or PDF presentation explaining how to submit and review wiki pages with Expertiza.
• For students, a Powerpoint or PDF presentation explaining how to form teams and sign up for topics<ref>Expertiza. Wolfwikis</ref>.

Problem Statement

Background

In this semester’s 517 class. Wiki 1a and Wiki 1b are structured as separate assignments, with separate signup sheets, teams, and reviews. But really, since only one of the two was done by any student, it would’ve been better to have a single assignment. Still, some topics could be done soon after the course started, whereas others were better done after we had studied related topics in class.

Staggered-deadline Assignment

This raises the idea of a staggered-deadline assignment, where different topics have different submission and review deadlines, rather than all topics having the same deadline.

Benefit

• Easy to manage. Because in the past, instructor has to build two separate assignments for Wiki 1a and 1b, now one assignment for Wiki is enough.

• Increase flexibility. For instance, when an OSS team facing a much more difficult problem than other teams, instructor may postpone their deadline several days. It will make Expertiza more humanize.

• Take time to do calibration. We find that two Wiki assignments started at Jan. 28th and ended at Feb. 25th, which takes too much time. And with staggered-deadline assignment, we can distribute less time on them. And we can take redundant time to do calibrating. Because we find that many students actually do not know how to evaluate other ones’ work, even there are some rubrics offered. So we recommend to let students grade sample assignments first, then calibrate their grading behaviors. Although it will take some time, we consider it is important to help students distinguish good assignments from common ones.

Product Verion Functionality

• In production version, instructor can set Submission deadline, Review deadline and Metareview deadline for each topic.

Set different deadline for each topic

• Also, instructor can set the dependencies of different topics. And there is a dependency graph generated automatically.

Dependency graph for all topics

Use Case

• Use Case #1: Create staggered-deadline assignment
  • Actor: Instructor
  • Actions:
    • Instructor logs in Expertiza.
    • Instructor creates a new staggered-deadline assignment.

• Use Case #2: Peer review only
  • Actor: Student finish work for first round
  • Actions:
    • Sign up topic.
    • Hand in assignment.
    • Peer review.
    • Write author feedback to reviewer.

• Use Case #3: Peer review and still work on assignments
  • Actor: Student not finish work for first round
  • Actions:
    • Sign up topic.
    • Work on assignment.
    • Peer review.

Use Case Diagram

Error Message Present

• First, login expertiza, in the "Manage Content" page, create a new assignment. Then, as shown below, choose the "pencil" icon to edit the assignment.

Assignment panel 1

• Check"Has topics?" and "Staggered deadline assignment?" two checkbox, and create new topics in "Topics" panel, click "save".

Create Topics

• Back to the "Manage Content" page, choose the "Edit signup sheet" icon.

Assignment panel 2

• Error: can’t write unknown attribute 't_id'.

Error

Initial Analyse

In "sign_up_sheet_controller", there are several errors and confusions in function "add_sign_up_topic". The function is used to display a page that lists all the available topics for a staggered-deadline assignment.
See the function code below:

def add_signup_topic
  load_add_signup_topics(params[:id])

  @review_rounds = Assignment.find(params[:id]).get_review_rounds
  @topics = SignUpTopic.where(assignment_id: params[:id])

  #Use this until you figure out how to initialize this array
  @duedates = SignUpTopic.find_by_sql("SELECT s.id as topic_id FROM sign_up_topics s WHERE s.assignment_id = " + params[:id].to_s)

  unless @topics.nil?
    i=0
    @topics.each { |topic|

      @duedates[i]['t_id'] = topic.id
      @duedates[i]['topic_identifier'] = topic.topic_identifier
      @duedates[i]['topic_name'] = topic.topic_name

      for j in 1..@review_rounds
        duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id:  DeadlineType.find_by_name('submission').id).first
        duedate_rev = TopicDeadline.where(topic_id: topic.id, deadline_type_id:  DeadlineType.find_by_name('review').id).first
        if !duedate_subm.nil? && !duedate_rev.nil?
          @duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
          @duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
        else
          #the topic is new. so copy deadlines from assignment
          set_of_due_dates = DueDate.where(assignment_id: params[:id])
          set_of_due_dates.each { |due_date|
            create_topic_deadline(due_date, 0, topic.id)
          }

          @duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
          @duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
        end

      end
      duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id:  DeadlineType.find_by_name('metareview').id).first
      @duedates[i]['submission_'+ (@review_rounds+1).to_s] = !(duedate_subm.nil?)?(DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")):nil
      i = i + 1
    }
  end

end

The main confusions are:

• In line 8, the variable "duedates" are declared as type "SignUpTopic". The naming is quite confusing.
• In line 14, the SignUpTopic class doesn’t have a column “t_id”, it raises an error. We think it should be "topic_id".
• From the code we can guess the variable "duedates" is used to store the deadline for each topic. The class “DueData” does exist, but it does not have column “topic_identifier” or “topic_name”. It only stores the deadline for a single assignment.

Puzzle

• If there is more than one round, which means students can submit their assignments more than once, so the staggered-deadline should refer to which deadline?

[Professor]In a staggered-deadline assignment, there is one submission deadline and one review deadline per round. These are always set on a per-topic basis. If the view currently works (and I think it might), as soon as you make an assignment a staggered-deadline assignment, the Signup Sheet (Topics) page will have a link at the bottom to show deadlines for each topic. Click it, and you will see a separate text box for each deadline for each topic.

• If staggered-deadline of one assignment is even later than the second-round review, is it means that that assignment do not need peer review and go directly into meta-review stage?

[Professor]No, there are also separate meta-review deadlines for a staggered-deadline assignment. I think in the current implementation, only submission deadlines, review deadlines, and meta-review deadlines vary by topic. In a more complete implementation, there would be a way for ANY deadline to vary by topic. This would include signup deadlines, drop-topic deadlines, team-formation deadlines, and any other kind of deadline that is defined later.

• When to set staggered-deadline assignments? Create assignment or Before submission?

• What is the meaning of "dependencies" between topics?

References

<references/>