CSC/ECE 517 Spring 2015 S1524 FSZZ

The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

E1524. Refactor staggered-deadline assignments

Overview

Introduction to Expertiza

Expertiza is a web application where students can submit and peer-review learning objects (articles, code, web sites, etc). It is used in select courses at NC State and by professors at several other colleges and universities<ref>Expertiza. Github</ref>.

Staggered Deadlines

Staggered deadlines involve planning alternate submission dates for papers, projects, or exams when a student has conflicting due dates for these. The key part of staggered deadlines is the planning. Staggered deadlines are always established well in advance of the scheduled due date. It is the advanced planning of these deadlines that makes them "staggered deadlines" rather than extensions<ref>Elmichaels. Difficulties with Staggered Deadlines. Jan 15, 2013</ref>.

Documentation

Controller

Model

View

Problem Statement

Background

In this semester's 517 class. Wiki 1a and Wiki 1b are structured as separate assignments, with separate signup sheets, teams, and reviews. But really, since only one of the two was done by any student, it would've been better to have a single assignment. Still, some topics could be done soon after the course started, whereas others were better done after we had studied related topics in class.

Staggered-deadline Assignment

This raises the idea of a staggered-deadline assignment, where different topics have different submission and review deadlines, rather than all topics having the same deadline.

Benefit

Easy to manage. Because in the past, instructor has to build two separate assignments for Wiki 1a and 1b, now one assignment for Wiki is enough.

Increase flexibility. For instance, when an OSS team facing a much more difficult problem than other teams, instructor may postpone their deadline several days. It will make Expertiza more humanize.

Take time to do calibration. We find that two Wiki assignments started at Jan. 28th and ended at Feb. 25th, which takes too much time. And with staggered-deadline assignment, we can distribute less time on them. And we can take redundant time to do calibrating. Because we find that many students actually do not know how to evaluate other ones' work, even there are some rubrics offered. So we recommend to let students grade sample assignments first, then calibrate their grading behaviors. Although it will take some time, we consider it is important to help students distinguish good assignments from common ones.

Product Verion Functionality

In production version, instructor can set Submission deadline, Review deadline and Metareview deadline for each topic.

Also, instructor can set the dependencies of different topics. And there is a dependency graph generated automatically.

Design Pattern

MVC
- Model–view–controller (MVC) is a software architectural pattern for implementing user interfaces. It divides a given software application into three interconnected parts, so as to separate internal representations of information from the ways that information is presented to or accepted from the user.<ref>Model–view–controller. Wikipedia</ref>
- We need to modify model, view and controller in order to fix this functionality.

Publish-subscribe
- In software architecture, publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers. Instead, published messages are characterized into classes, without knowledge of what, if any, subscribers there may be. Similarly, subscribers express interest in one or more classes, and only receive messages that are of interest, without knowledge of what, if any, publishers there are.<ref>Publish–subscribe pattern. Wikipedia</ref>
- When instructor set an assignment as staggered-dealine assignment, in theory, all the students will know.

Use Case

Use Case #1: Create staggered-deadline assignment
- Actor: Instructor
- Actions:
  - Instructor logs in Expertiza.
  - Instructor creates a new staggered-deadline assignment.

Use Case #2: Peer review only
- Actor: Student finish work for first round
- Actions:
  - Sign up topic.
  - Hand in assignment.
  - Peer review.
  - Write author feedback to reviewer.

Use Case #3: Peer review and still work on assignments
- Actor: Student not finish work for first round
- Actions:
  - Sign up topic.
  - Work on assignment.
  - Peer review.

Error Message Present

First, login expertiza, in the "Manage Content" page, create a new assignment. Then, as shown below, choose the "pencil" icon to edit the assignment.

Check"Has topics?" and "Staggered deadline assignment?" two checkbox, and create new topics in "Topics" panel, click "save".

Back to the "Manage Content" page, choose the "Edit signup sheet" icon.

Error: can't write unknown attribute 't_id'.

Problem Analysis

After discussing with professor, the "sign up sheet" icon in assignment pop up panel should be moved. Because that icon has the same functionality as "topic" panel in assignment edit window. These work has already done by OSS1509. So we have to move all the staggered-deadline realted code to the assignments_controller.rb and assignments/edit.html.erb.

In "sign_up_sheet_controller", there are several errors and confusions in function "add_sign_up_topic". The function is used to display a page that lists all the available topics for a staggered-deadline assignment.

See the function code below:

def add_signup_topic
  load_add_signup_topics(params[:id])

  @review_rounds = Assignment.find(params[:id]).get_review_rounds
  @topics = SignUpTopic.where(assignment_id: params[:id])

  #Use this until you figure out how to initialize this array
  @duedates = SignUpTopic.find_by_sql("SELECT s.id as topic_id FROM sign_up_topics s WHERE s.assignment_id = " + params[:id].to_s)

  unless @topics.nil?
    i=0
    @topics.each { |topic|

      @duedates[i]['t_id'] = topic.id
      @duedates[i]['topic_identifier'] = topic.topic_identifier
      @duedates[i]['topic_name'] = topic.topic_name

      for j in 1..@review_rounds
        duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id:  DeadlineType.find_by_name('submission').id).first
        duedate_rev = TopicDeadline.where(topic_id: topic.id, deadline_type_id:  DeadlineType.find_by_name('review').id).first
        if !duedate_subm.nil? && !duedate_rev.nil?
          @duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
          @duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
        else
          #the topic is new. so copy deadlines from assignment
          set_of_due_dates = DueDate.where(assignment_id: params[:id])
          set_of_due_dates.each { |due_date|
            create_topic_deadline(due_date, 0, topic.id)
          }

          @duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
          @duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
        end

      end
      duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id:  DeadlineType.find_by_name('metareview').id).first
      @duedates[i]['submission_'+ (@review_rounds+1).to_s] = !(duedate_subm.nil?)?(DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")):nil
      i = i + 1
    }
  end

end

The main confusions are:

In line 8, the variable "duedates" are declared as type "SignUpTopic". The naming is quite confusing.
In line 14, the SignUpTopic class doesn't have a column "t_id", it raises an error. We think it should be "topic_id".
From the code we can guess the variable "duedates" is used to store the deadline for each topic. The class "DueData" does exist, but it does not have column "topic_identifier" or "topic_name". It only stores the deadline for a single assignment.

Puzzle

If there is more than one round, which means students can submit their assignments more than once, so the staggered-deadline should refer to which deadline?

[Professor] In a staggered-deadline assignment, there is one submission deadline and one review deadline per round. These are always set on a per-topic basis. If the view currently works (and I think it might), as soon as you make an assignment a staggered-deadline assignment, the Signup Sheet (Topics) page will have a link at the bottom to show deadlines for each topic. Click it, and you will see a separate text box for each deadline for each topic.

If staggered-deadline of one assignment is even later than the second-round review, is it means that that assignment do not need peer review and go directly into meta-review stage?

[Professor] No, there are also separate meta-review deadlines for a staggered-deadline assignment. I think in the current implementation, only submission deadlines, review deadlines, and meta-review deadlines vary by topic. In a more complete implementation, there would be a way for ANY deadline to vary by topic. This would include signup deadlines, drop-topic deadlines, team-formation deadlines, and any other kind of deadline that is defined later.

When to set staggered-deadline assignments? Create assignment or Before submission?

[Professor] Basically, the staggered-deadline is set at the same time when an assignment is created. But if one team cannot hand in their work with sufficient reasons, instructor may extend the deadline of their topic.

What is the meaning of "dependencies" between topics?

[Professor] The dependencies of topics means one topic cannot start until another topic finishes. For instance, Ruby topic must be finished before Rails topic.

Workflow

Set topics staggered deadline

At very beginning, we find that the type of due_date variable SignUpTopics. However, we found that many attributes used not belonging to SignUpTopics class. So we decide to use hash tables instead. In order to keep update the staggered deadline, we put this variable in session. We also comprehended the relationship of corresponding tables. The original due dates are stored in due_date table. And staggered-deadlines are stored in TopicDeadline table. This table also stores different deadline types.

Below is save_topic_deadlines method after refactoring.

def save_topic_deadlines
 #session[:duedates] stores all original duedates info
 #due_dates stores staggered duedates
 due_dates = params[:due_date]
 topics = SignUpTopic.where(assignment_id: params[:assignment_id])
 review_rounds = Assignment.find(params[:assignment_id]).get_review_rounds
 # j represents the review rounds
 j = 0
 topics.each { |topic|
   for i in 1..review_rounds
     topic_deadline_type_subm = DeadlineType.find_by_name('submission').id
     topic_deadline_subm = TopicDeadline.where(topic_id: session[:duedates][j]['id'].to_i, deadline_type_id: topic_deadline_type_subm, round: i).first
     topic_deadline_subm.update_attributes({'due_at' => due_dates[session[:duedates][j]['id'].to_s + '_submission_' + i.to_s + '_due_date']})
     flash[:error] = "Please enter a valid " + (i > 1 ? "Resubmission deadline " + (i-1).to_s : "Submission deadline") if topic_deadline_subm.errors.length > 0
     topic_deadline_type_rev = DeadlineType.find_by_name('review').id
     topic_deadline_rev = TopicDeadline.where(topic_id: session[:duedates][j]['id'].to_i, deadline_type_id: topic_deadline_type_rev, round:i).first
     topic_deadline_rev.update_attributes({'due_at' => due_dates[session[:duedates][j]['id'].to_s + '_review_' + i.to_s + '_due_date']})
     flash[:error] = "Please enter a valid Review deadline " + (i > 1 ? (i-1).to_s : "") if topic_deadline_rev.errors.length > 0
   end
   topic_deadline_subm = TopicDeadline.where(topic_id: session[:duedates][j]['id'], deadline_type_id:  DeadlineType.find_by_name('metareview').id).first
   topic_deadline_subm.update_attributes({'due_at' => due_dates[session[:duedates][j]['id'].to_s + '_submission_' + (review_rounds+1).to_s + '_due_date']})
   flash[:error] = "Please enter a valid Meta review deadline" if topic_deadline_subm.errors.length > 0
   j = j + 1
 }
 redirect_to_assignment_edit(params[:assignment_id])
end

Save topics dependency and show dependency graph

In edit signup sheet for assignment, we want to save topic dependency and show the depandency graph, as show below:

In order to show the dependency graph, we must first save topics dependency and save the topic dependency graph under public/assets/staggered_deadline_assignment_graph path. We first add new method in 'due_date' model:

  def self.assign_start_due_date(assignment_id, set_of_topics)

    #Remember, in create_common_start_time_topics function we reversed the graph so reverse it back
    set_of_topics = set_of_topics.reverse

    set_of_topics_due_dates = Array.new
    i=0
    days_between_submissions = Assignment.find(assignment_id)['days_between_submissions'].to_i
    set_of_topics.each { |set_of_topic|
      set_of_due_dates = nil
      if i==0
        #take the first set from the table which user stores
        set_of_due_dates = DueDate.where(assignment_id)
        offset = 0
      else
        set_of_due_dates = TopicDeadline.where(set_of_topics[i-1][0])
        set_of_due_dates.sort_by { |a, b| a.due_at <=> b.due_at }
        offset = days_between_submissions
      end

      set_of_topic.each { |topic_id|
        #if the due dates have already been created and the save dependency is being clicked,
        #then delete existing n create again
        prev_saved_due_dates = TopicDeadline.where(topic_id)

        #Only if there is a dependency for the topic
        if !prev_saved_due_dates.nil?
          num_due_dates = prev_saved_due_dates.length
          #for each due date in the current topic he want to compare it to the previous due date
          for x in 0..num_due_dates - 1
            #we don't want the old date to move earlier in time so we save it as the new due date and destroy the old one  
            if DateTime.parse(set_of_due_dates[x].due_at.to_s) + offset.to_i < DateTime.parse(prev_saved_due_dates[x].due_at.to_s)
              set_of_due_dates[x] = prev_saved_due_dates[x]
              offset = 0
            end
            prev_saved_due_dates[x].destroy
          end
        end

        set_of_due_dates.each { |due_date|
          create_topic_deadline(due_date, offset, topic_id)
        }
      }
      i = i+1
    }
  end

Then, change the save_topic_dependencies method in sign_up_sheet_controller.rb :

  - redirect_to_sign_up(params[:assignment_id])
  + redirect_to :action => 'add_signup_topics_staggered', :id => params[:assignment_id]

Since we use RGL:Graph#write_to_graphic_file method to turn the dependency graph into jpg file, we need to install graphviz in our gems. Otherwise, the RGL will uses dot files as an intermediary format to produce image formats. If we download the GraphViz packege, RGL will invoke it to produce a jpg file.

gem 'graphviz'

Then, after we click the Save dependency button, and click show dependency graph button, the dependency graph is shown below:

Test

Since we have done several modifications to the Expertiza program, some new tests are needed to test the features. We use the stub to simulate an authorized login (as instructor).

ApplicationController.any_instance.stub(:current_role_name).and_return('Instructor')
ApplicationController.any_instance.stub(:undo_link).and_return(TRUE)

Each time the program executes the current_role_name function, it gets "instructor". In the test, we mainly test two things: staggered assignments and their corresponding dependency graph.

Test the staggered assignment

it "should be able to create topic for assignment" do
  get :create, id: @assignment.id, topic: {topic_name: "New Topic", max_choosers: 2, topic_identifier: "Ch1", category: "Programming"}
  expect(response).should redirect_to(edit_assignment_path(@assignment.id) + "#tabs-5")
end

it "should be able to edit topic" do
  get :edit, id: @topic.id
  expect(response).to be_success
end

it "should be able to delete topic" do
  delete :destroy, id: @topic.id, assignment_id: @assignment.id
  expect(response).should redirect_to edit_assignment_path(@assignment.id) + "#tabs-5"
end

Test the dependency graph.

it "should be able to generate topic dependency" do
    post :save_topic_dependencies, assignment_id: @assignment.id
    expect(File).to exist("public/assets/staggered_deadline_assignment_graph/graph_#{@assignment.id}.jpg")
  end

it "should be able to detect cycles" do
  post :save_topic_dependencies, assignment_id: @assignment.id,
       ('topic_dependencies_' + @topic1.id.to_s)=>{"dependent_on"=>[@topic2.id.to_s]},
       ('topic_dependencies_' + @topic2.id.to_s)=>{"dependent_on"=>[@topic1.id.to_s]}
  expect(flash[:error]).to eq("There may be one or more cycles in the dependencies. Please correct them")
end

Deploy on VCL

Visit http://vcl.ncsu.edu/ > Make a Reservation
Select Ruby on Rails / Expertiza from the "Please select the environment ..." dropdown, select a duration, and then click Create Reservation
Once the reservation is available, click Connect
Follow the instructions on the connect screen (connect via SSH and then VNC). Connecting to SSH starts up VNC, the graphical environment to which you will connect.
When connecting for the first time, it will prompt you for a password to access your (VNC) desktop. You can use any password, and it will be saved for later sessions.
Follow the instructions given to connect to VNC. It is recommended to use VNC Viewer for Google Chrome.
Once connected to VNC, you should see Rubymine running.
Open a terminal (Applications > Accessories > Terminal), try hitting <ctrl>-C to kill pending task if your terminal does not work properly.
Ensure you have at least 20M disk quota available: fs lq . You can allocate more space at https://sysnews.ncsu.edu/tools-bin/usmdb-quota
Clone the git repository: git clone (expertiza repository URL)
Change to the directory (cd expertiza). You may create a "database.yml" file based on "database.yml.example".
Use rvm use (ruby version) to switch to proper ruby version. Then run bundle install (you may need to downgrade the bundler to avoid bundle install failure) and rake db:migrate to set up development environment.
Use "rails s -b 0.0.0.0" to start server which allows the access outside VCL by using url: VCL_ip:3000.

Related materials

An instructor manual, explaining how to create an deploy an assignment in Expertiza.
An instructor video, slightly dated, showing how to create and deploy an assignment
A guide for Creating_Custom_Rubric
For students, a Powerpoint or PDF presentation explaining how to submit and review an assignment with Expertiza.
For students, a video showing how to use the system to submit and review an assignment.
For students, a Powerpoint or PDF presentation explaining how to submit and review wiki pages with Expertiza.
For students, a Powerpoint or PDF presentation explaining how to form teams and sign up for topics<ref>Expertiza. Wolfwikis</ref>.

References

CSC/ECE 517 Spring 2015 S1524 FSZZ

Overview

Introduction to Expertiza

Staggered Deadlines

Contents

Documentation

Problem Statement

Background

Staggered-deadline Assignment

Benefit

Product Verion Functionality

Design Pattern

Use Case

Error Message Present

Problem Analysis

Puzzle

Workflow

Set topics staggered deadline

Save topics dependency and show dependency graph

Test

Deploy on VCL

Related materials

References

Navigation menu

CSC/ECE 517 Spring 2015 S1524 FSZZ

Overview

Introduction to Expertiza

Staggered Deadlines

Documentation

Problem Statement

Background

Staggered-deadline Assignment

Benefit

Product Verion Functionality

Design Pattern

Use Case

Error Message Present

Problem Analysis

Puzzle

Workflow

Set topics staggered deadline

Save topics dependency and show dependency graph

Test

Deploy on VCL

Related materials

References

Navigation menu

Search