CSC/ECE 517 Fall 2020 - E2083. Revision planning tool E2016
This page provides a description of the Expertiza based OSS project.
Introduction
Rounds of peer reviews may be implemented between submissions for assignments on Expertiza. In order to better track the implementation of reviewer's suggestions, a Revision Planning Tool should be implemented.
Problem Statement
In the first round of Expertiza reviews, we ask reviewers to give authors some guidance on how to improve their work. Then in the second round, reviewers rate how well authors have followed their suggestions. We could carry the interaction one step further if we asked authors to make up a revision plan based on the first-round reviews. That is, authors would say what they were planning to do to improve their work. Then second-round reviewers would assess how well they did it. In essence, this means that authors would be adding criteria to the second-round rubric that applied only to their submission. We are interested in having this implemented and used in a class so that we can study its effect.
Previous Implementations
Revision planning has been implemented twice before, once in E1875 and once in E2016. While the functionality worked and effectively minimized changes to the code, they also had the following problems:
- Hardcoded “round” numbers in many places of the code.
- Documentation does not reflect the new changes they made.
- Revision planning responses and responses to the other items are not distinguished in heatgrid view.
- The idea of adding a
team_id
field to each question is intuitive. However, they failed to come up with a clean implementation of this idea. Specifically, they had passed some trailing parameters several methods down before reaching the place that needs them.
Rationale
To implement revision planning, E2016 added is_revision_planning_enabled
(Boolean) to Assignment
and team_id
to Question
. Adding a Boolean to enable revision planning for an assignment makes sense, but having certain questions belong to a team is not a clean implementation. Instead, we will create a new questionnaire that will belong to each team and keep track of the set of questions. We will use this new questionnaire to separate the revision plan responses and responses to other items in heat-grid view by creating a different table for each.
Files Modified
This is a list of files modified and added. The list will be updated as the implementation progresses.
Changed:
- app/controllers/advice_controller.rb
- app/controllers/grades_controller.rb
- app/controllers/questionnaires_controller.rb
- app/controllers/questions_controller.rb
- app/controllers/response_controller.rb
- app/controllers/student_task_controller.rb
- app/helpers/summary_helper.rb
- app/helpers/grades_helper.rb
- app/models/questionnaire.rb
- app/models/response.rb
- app/models/student_task.rb
- app/models/vm_question_response.rb
- app/models/on_the_fly_calc.rb
- app/views/advice/edit_advice.html.erb
- app/views/assignments/edit/_general.html.erb
- app/views/grades/view_team.html.erb
- app/views/questionnaires/_questionnaire.html.erb
- app/views/response/response.html.erb
- app/views/student_task/view.html.erb
- config/routes.rb
- db/schema.rb
- spec/controllers/response_controller_spec.rb
- spec/models/response_spec.rb
- spec/models/review_response_map_spec.rb
- spec/features/assignment_creation_general_tab_spec.rb
Added:
- app/controllers/revision_plan_questionnaires_controller.rb
- app/models/revision_plan_questionnaire.rb
- app/models/revision_plan_team_map.rb
- app/views/questionnaires/_questions.html.erb
- app/views/revision_plan_questionnaires/edit.html.erb
- db/migrate/20201106020642_create_revision_plan_team_maps.rb /
- db/migrate/20201105023711_add_is_revison_planning_enabled_to_assignments.rb
- spec/controllers/revision_plan_questionnaires_controller_spec.rb
- spec/factories/revision_plan_factory.rb
Design
Database Design
Items in green are additions.
Changes:
- In the assignment table we have added
is_revision_planning_enabled?
column to indicate whether the assignment accepts a revision plan along with review rubric. RevisionPlanTeamMap
maps a questionnaire to an assignment team and round. This will map to a questionnaire of type revision plan that will be created by the revewee.RevisionPlanQuestionnaire
extendsQuestionnaire
using single table inheritance.
User Interface
Enable Revision Planning
In order to enable Revision Planning, the setting must be enabled when creating or editing an assignment under the General tab. The wireframe below demonstrates creating an assignment, and editing the assignment functions similarly. Revision plan? should be checked to enable this option.
Initial Wireframe
Final UI
Implementation
- Added a Boolean field
is_revision_planning_enabled
toAssignment
that saves whether revision planning is enabled.
db/migrate/20201105023711_add_is_revison_planning_enabled_to_assignments.rb
class AddIsRevisonPlanningEnabledToAssignments < ActiveRecord::Migration def change add_column :assignments, :is_revision_planning_enabled, :boolean, default: false end end
- Add checkbox to enable revision planning in edit assignment view.
app/views/assignments/edit/_general.html.erb
<input name="assignment_form[assignment][is_revision_planning_enabled]" type="hidden" value="false"/> <%= check_box_tag('assignment_form[assignment][is_revision_planning_enabled]', 'true', @assignment_form.assignment.is_revision_planning_enabled) %> <%= label_tag('assignment_form[assignment][is_revision_planning_enabled]', 'Revision Planning?') %>
Assignment Overview Page (contains the Link to the revision planning page)
The Revision Planning link is available to students during every submission period (except the first round submission) and not available during every review period. As shown in the wireframe, by clicking Revision Planning students would be redirected to the ‘Revision planning page’.
Initial Wireframe
Final UI
Implementation
- Show link to revision planning page when revision planning is enabled for an assignment, round 1 has passed and current stage is submission.
In order to check if a team is allowed to submit a revision plan @can_submit_revision_plan
field is added to StudentTaskController
's view action. If a revision plan questionnaire (@revision_plan_questionnaire_id
) exists for the current round then clicking the link performs RevisionPlanQuestionnairesController
's edit action else new action is performed.
app/views/student_task/view.html.erb
<% if @team && @assignment.is_revision_planning_enabled %> <li> <% if @can_submit_revision_plan %> <% if @revision_plan_questionnaire_id %> <%= link_to "Revision Planning", controller: 'revision_plan_questionnaires', action: 'edit', id: @revision_plan_questionnaire_id %> <% else %> <%= link_to "Revision Planning", controller: 'revision_plan_questionnaires', action: 'new', team_id: @team %> <% end %> (Add new questions to next-round rubric) <% else %> <font color="gray">Revision Planning</font> (You are not allowed to submit your revison plan right now) <% end %> </li> <%end%>
app/controllers/student_task_controller.rb
def view student_task = StudentTask.from_participant_id params[:id] ... ... # Revision plan feature @can_submit_revision_plan = student_task.can_submit_revision_plan? @revision_plan_questionnaire_id = student_task.revision_plan_questionnaire_id end
app/models/student_task.rb
def current_round assignment.number_of_current_round(@topic.try(:id)) end def can_submit_revision_plan? current_stage == 'submission' && current_round > 1 end def revision_plan_questionnaire_id RevisionPlanTeamMap.find_by(team: @participant.team, used_in_round: current_round).try(:questionnaire_id) end
Editing the Revision Plan Questionnaire
After creating the Revision Plan Questionnaire, it must be edited. Questions can be added by specifying the amount of questions and their type. Questions can be removed by clicking Remove in the leftmost column. Once the questionnaire is complete, it can be saved. This page will be visible during each submission period after the first and will be unavailable during all review periods.
Initial Wireframe
Final UI
Implementation
- Create a model RevisionPlanTeamMap that maps a revision plan to a team and stores information of the round it is used in.
/app/models/revision_plan_team_map.rb
class RevisionPlanTeamMap < ActiveRecord::Base belongs_to :team belongs_to :questionnaire end
/db/migrate/20201106020642_create_revision_plan_team_maps.rb
class CreateRevisionPlanTeamMaps < ActiveRecord::Migration def change create_table :revision_plan_team_maps do |t| t.references :team, index: true, foreign_key: true t.references :questionnaire, index: true, foreign_key: true t.integer :used_in_round t.timestamps null: false end end end
- Create a Revision Plan Questionnaire Model and Controller
app/models/revision_plan_questionnaire.rb
Added a model for revision plan questionnaire. The code provides method to get questionnaire for a current round based on team.
def self.get_questionnaire_for_current_round(team_id) assignment_team = Team.find(team_id) assignment = assignment_team.assignment current_round = assignment.number_of_current_round(assignment_team.topic) questionnaire = RevisionPlanTeamMap.find_by(team: assignment_team, used_in_round: current_round).try(:questionnaire) unless questionnaire questionnaire = RevisionPlanQuestionnaire.new questionnaire.name = 'Revision Plan Questionnaire' questionnaire.instructor_id = assignment.instructor_id questionnaire.max_question_score = 5 questionnaire.save questionnaire_team_map = RevisionPlanTeamMap.create(team_id: assignment_team.id, used_in_round: current_round, questionnaire_id: questionnaire.id) end return questionnaire end
This method returns the team associated to a revision plan questionnaire.
def team revision_plan_team_map.team end
app/models/questionnaire.rb
Added a new type of questionnaire 'RevisionPlanQuestionnaire'
to QUESTIONNAIRE_TYPES
QUESTIONNAIRE_TYPES = ['ReviewQuestionnaire', ... ... 'RevisionPlanQuestionnaire'].freeze
app/controllers/revision_plan_questionnaires_controller.rb
Added a controller RevisionPlanQuestionnairesController
. The controller derives from QuestionnairesController
, it overrides the new
action to allow creation of a RevisionPlanQuestionnaire
for a team
. It also updates permissions for controller.
def action_allowed? case params[:action] when 'edit' questionnaire = Questionnaire.find(params[:id]) questionnaire.team.users.collect { |u| u.id }.include? session[:user].id || super (user_logged_in? && questionnaire.team.users.collect { |u| u.id }.include?(session[:user].id)) || super else super end end
def new begin questionnaire = RevisionPlanQuestionnaire.get_questionnaire_for_current_round(params[:team_id]) redirect_to action: 'edit', id: questionnaire.id rescue StandardError flash[:error] = $ERROR_INFO end end
config/routes.rb
Add routes for RevisionPlanQuestionnairesController
.
resources :revision_plan_questionnaires, controller: :revision_plan_questionnaires, only: %i[new edit update] do collection do post :add_new_questions post :save_all_questions end end
- Add a view to edit Revision Plan
app/views/revision_plan_questionnaires/edit.html.erb
<%= render :partial => 'question_weight' %> <h1>Edit <%= @questionnaire.display_type %></h1> <%= render :partial => 'questions' %> <br /> <a href="javascript:window.history.back()">Back</a>
- Refactor code to display questions into a partial. Allows for code reuse.
app/controllers/questionnaires_controller.rb
Updated redirect_to
links for add_new_questions
and save_all_questions
methods.
redirect_to action: 'edit', id: questionnaire_id.to_sym
app/views/questionnaires/_questionnaire.html.erb
This file generates a view to edit/create a new questionnaire. It has functionality to display questions and some functions that are not needed for revision plan questionnaire. To enable code reuse the functionality to add/edit/remove/save questions was moved to a new partial app/views/questionnaires/_questions.html.erb.
app/views/questionnaires/_questions.html.erb
- Add code to allow users to delete questions from revision plan questionnaire.
app/controllers/questions_controller.rb
Update authorization code to allow a user to remove questions from his teams revision plan.
def action_allowed? if ['destroy'].include?(params[:action]) question = Question.find(params[:id]) if(user_logged_in? && question.questionnaire.owner?(session[:user].id)) return true end end current_user_has_ta_privileges? end
def destroy ... ... if(question.questionnaire.type == 'RevisionPlanQuestionnaire') redirect_to edit_revision_plan_questionnaire_path(questionnaire_id.to_s.to_sym) else redirect_to edit_questionnaire_path(questionnaire_id.to_s.to_sym) end end
app/models/questionnaire.rb
Method returns true if user owns questionnaire.
def owner?(user_id) instructor_id == user_id end
app/models/revision_plan_questionnaire.rb
Overrides implementation for owner?
defined in Questionnaire
to return true if a team owns the questionnaire.
def owner?(user_id) team.users.map{ |u| u.id }.include?(user_id) || super end
- Add code to allow users to access show/edit advice functionality for revision plan questions.
app/controllers/advice_controller.rb
Update authorization code to allow a user to update advice his teams revision plan.
def action_allowed? questionnaire = Questionnaire.find(params[:id]) if(user_logged_in? && questionnaire.owner?(session[:user].id)) return true end current_user_has_ta_privileges? end
app/views/advice/edit_advice.html.erb
Update 'Back to questionnaire' link.
<% if @questionnaire.type == 'RevisionPlanQuestionnaire'%> <%= link_to 'Back to questionnaire', :controller => 'revision_plan_questionnaires', :action => 'edit', :id => @questionnaire.id %> <% else %> <%= link_to 'Back to questionnaire', :controller => 'questionnaires', :action => 'edit', :id => @questionnaire.id %> <% end %>
Reviewing an Assignment
Once the Revision Plan Questionnaire and its resubmission round have been finished, the Revision Plan Questionnaire is appended to the Assignment Questionnaire for that round.
Initial Wireframe
Final UI
Implementation
- Add Revision Plan questions to Response
app/controllers/response_controller.rb
This methods gets questions sorted by sequence from review and revision questionnaire, and merges them in a single list.
def set_questions_for_new_response @questions = sort_questions(@questionnaire.questions) if(@assignment && @assignment.is_revision_planning_enabled) reviewees_topic = SignedUpTeam.topic_id_by_team_id(@contributor.id) current_round = @assignment.number_of_current_round(reviewees_topic) @revision_plan_questionnaire = RevisionPlanTeamMap.find_by(team_id: @map.reviewee_id, used_in_round: current_round).try(:questionnaire) if(@revision_plan_questionnaire) @questions += sort_questions(@revision_plan_questionnaire.questions) end end return @questions end
This method gets questions from a response instead of questionnaire. We use this when response has already been created.
def set_questions @questions = @response.get_questions end
app/models/response.rb
Added a method to get all the questionnaires for a response, response is a collection of answers. Answers contain reference to their question, and question have a reference to questionnaire.
def questionnaires_by_answers(answers) answers_with_questionnaires = answers.select{ |ans| ans && ans.question && ans.question.questionnaire } questionnaires = answers_with_questionnaires.collect{ |ans| ans.question.questionnaire }.uniq unless(questionnaires.any?) questionnaires = [] questionnaires << questionnaire_by_answer(answers.first) end questionnaires end
This method gets all the questions for a response.
def get_questions @questions = [] questionnaires = questionnaires_by_answers(scores) questionnaires.each {|questionnaire| @questions += questionnaire.questions } return @questions end
- Separating questions by their questionnaire type
app/models/response.rb
construct_review_response
method has been updated to create separate tables for review questions, revision questions and additional comment.
app/models/questionnaire.rb
Add a method to check if heading is to be displayed for the questionnaire.
def display_heading? return false end
app/models/revision_plan_questionnaire.rb
Overriding the method defined in questionnaire.rb to display heading for revision plan questionnaire.
def display_heading? return true end
app/views/response/response.html.erb
View groups questions by questionnaire type and then adds a heading for questionnaires that return true for display_heading?
.
Bug fix: The existing implementation has a bug where the first question in a questionnaire is displayed last in response. This was fixed as well.
Summary Report Page
When a project has been reviewed at least once, a participant is able to view their team's score. The UI below shows what this looks like after the second round of reviews. For the second and all subsequent reviews, the results of questions that were created by the instructor are shown under Assignment Questionnaire. The results of the questions created by the team are shown under Improvement Plan.
Initial Wireframe
Final UI
Implementation
- Show revision plan questions in heat grid.
Heat grids for review responses are currently implemented in a view model defined in vm_question_response.rb. All functionality of this view model has remained in place, and additional responses are passed to it to create the additional heat grids for revision plan responses.
app/helpers/grades_helper.rb
The method generate_heatgrid
was created from the original code shared by both view_heatgrid
and view_team
in app/controllers/grades_controller, but with the additional functionality of checking for Revision Plans in responses. When assignment.vary_by_round
, an additional heatgrid is added for each round of Revision Planning. When !assignment.vary_by_round
, only one heatgrid is added for the Revision Plan used in the most recent round of review.
def generate_heatgrid(participant, assignment, team, team_id, questionnaires, vmlist) counter_for_revisions = -1 counter_for_same_rubric = 0 questionnaires.each do |questionnaire| if assignment.vary_by_round? && questionnaire.type == "ReviewQuestionnaire" questionnaires = AssignmentQuestionnaire.where(assignment_id: assignment.id, questionnaire_id: questionnaire.id) if questionnaires.count > 1 @round = questionnaires[counter_for_same_rubric].used_in_round counter_for_same_rubric += 1 else @round = questionnaires[0].used_in_round counter_for_same_rubric = 0 end end add_response_to_vmlist(participant, assignment, team, questionnaire, vmlist, @round) if assignment.vary_by_round? && assignment.is_revision_planning_enabled? rp_questionnaire = RevisionPlanTeamMap.find_by(team: Team.find(team_id), used_in_round: counter_for_revisions).try(:questionnaire) if rp_questionnaire add_response_to_vmlist(participant, assignment, team, rp_questionnaire, vmlist, @round) end counter_for_revisions += 1 elsif assignment.is_revision_planning_enabled? && questionnaire == questionnaires.last if assignment.get_current_stage == "Finished" current_round = assignment.rounds_of_reviews else reviewees_topic = SignedUpTeam.topic_id_by_team_id(participant.id) current_round = assignment.number_of_current_round(reviewees_topic) end rp_questionnaire = RevisionPlanTeamMap.find_by(team: Team.find(team_id), used_in_round: current_round).try(:questionnaire) if rp_questionnaire add_response_to_vmlist(participant, assignment, team, rp_questionnaire, vmlist, @round) end end end end
The method add_response_to_vmlist
was added to refactor the code that is used multiple times to add another review to the view model.
def add_response_to_vmlist(participant, assignment, team, questionnaire, vmlist, round) vm = VmQuestionResponse.new(questionnaire, assignment, round) vmquestions = questionnaire.questions vm.add_questions(vmquestions) vm.add_team_members(team) vm.add_reviews(participant, team, assignment.vary_by_round) vm.number_of_comments_greater_than_10_words vmlist << vm end
app/helpers/summary_helper.rb
Bug Fix: Reverted get_sentences
back to version in master branch to prevent Active Record Error caused by beta version of code.
def get_sentences(ans) unless ans.comments.nil? ans.comments.gsub!(/[.?!]/, '\1|') sentences = ans.comments.split('|') sentences.map!(&:strip) end end
app/models/vm_question_response.rb
Allow RevisionPlanQuestionnaires
to be added to the view model in the same way as ReviewQuestionnaires
.
if (@questionnaire_type == "ReviewQuestionnaire" or @questionnaire_type == "RevisionPlanQuestionnaire")
app/views/grades/view_team.html.erb
Print round number headings for Revision Plan heat grids.
<% if vm.questionnaire_type == "ReviewQuestionnaire" or vm.questionnaire_type == "RevisionPlanQuestionnaire"%>
- Separated Review and Revision Plan questions in response
app/models/response.rb
Created separate arrays for ReviewQuestionnaire questions and RevsionPlanQuestionnaire questions. Added headings for Review Responses and Revision Plan Responses.
review_questions = [] revision_plan_questions = [] #begin table for Review Responses ... map = ResponseMap.find(self.map_id) unless map.is_a? ReviewResponseMap code = add_table_rows questionnaire_max, questions, answers, code, tag_prompt_deployments, current_user else assignment = map.assignment questions.each do |question| if(question.questionnaire.type == 'ReviewQuestionnaire') review_questions.append(question) elsif(question.questionnaire.type == 'RevisionPlanQuestionnaire') revision_plan_questions.append(question) end end code = add_table_rows questionnaire_max, review_questions, answers, code, tag_prompt_deployments, current_user if assignment.is_revision_planning_enabled #end table for Review Responses and begin table for Revision Plan Responses code = add_table_rows questionnaire_max, revision_plan_questions, answers, code, tag_prompt_deployments, current_user end
- Refactored
view_team
andview_heatgrid
app/controllers/grades_controller.rb
The code in the view_team
method was almost identical to the code in the view_heatgrid
method in app/helpers/grades_helper.rb. It was refactored into a new method generate_heatgrid
in app/helpers/grades_helper.rb and used in view_team
.
def view_team @participant = AssignmentParticipant.find(params[:id]) @assignment = @participant.assignment @team = @participant.team @team_id = @team.id questionnaires = @assignment.questionnaires @questions = retrieve_questions questionnaires, @assignment.id @pscore = @participant.scores(@questions) @vmlist = [] generate_heatgrid(@participant, @assignment, @team, @team_id, questionnaires, @vmlist) @current_role_name = current_role_name end
app/helpers/grades_helper.rb
The code in the view_heatgrid
method was almost identical to the code in the view_team
method in app/controllers/grades_controller.rb. It was refactored into a new method generate_heatgrid
and used in view_heatgrid
.
def view_heatgrid(participant_id, type) # get participant, team, questionnaires for assignment. @participant = AssignmentParticipant.find(participant_id) @assignment = @participant.assignment @team = @participant.team @team_id = @team.id @type = type questionnaires = @assignment.questionnaires @vmlist = [] generate_heatgrid(@participant, @assignment, @team, @team_id, questionnaires, @vmlist) render "grades/view_heatgrid.html.erb" end
Control Flow Diagram
Test Plan
RSpec Testing
The RSpec tests are written to test both controller and models.
Controllers
- spec/controllers/response_controller_spec.rb: Updated spec to pass build by stubbing
get_questions
method. - spec/factories/revision_plan_factory.rb: Factories can be used to create objects unique to revision plans, including:
RevisionPlanQuestionnaire
RevisionPlanTeamMap
- spec/controllers/revision_plan_questionnaires_controller_spec.rb: Add four examples to test revision plan questionnaires controller
describe '#action_allowed?' do context 'when params action is edit or update' do before(:each) do controller.params = {id: '1', action: 'edit',team_id: 1} controller.request.session[:user] = student end context 'when the role name of current user is super admin or admin' do ... context 'when current user is the student of current questionnaires' do ... context 'when current user is a student but not of the team' do ... context 'when params action is not edit and update' do ... ...
Models
- spec/models/response_spec.rb: Updated spec to pass build by updating expected html output.
- spec/models/review_response_map_spec.rb: Updated spec to pass build by updating expected html output.
Features
- spec/features/assignment_creation_general_tab_spec.rb
Test if checking box tag Revision Planning?
enables revision planning.
it "should enable revision planning" do fill_assignment_form check("assignment_form_assignment_is_revision_planning_enabled") click_button 'Save' assignment = Assignment.where(name: 'edit assignment for test').first expect(assignment).to have_attributes( name: 'edit assignment for test', course_id: Course.find_by(name: 'Course 2').id, directory_path: 'testDirectory1', spec_location: 'testLocation1', is_revision_planning_enabled: true, ) end
Manual Testing
Manual testing aims to verify the following:
- Create an assignment with revision planning enabled.
- Are participants allowed to create/edit revision plan when round 1+ (1 or greater than 1) reviews have finished.
- Is revision plan editing disabled when assignment is in review stage.
- Are reviewers shown questions created by reviewees.
- Are participants shown summary of score for revision plan after review deadline has expired.
Team Members
Chaitanya Mehta (cmehta)
Darby Madewell (demadewe)
Dongni Yang (dyang23)
Sidharth Mehta (smehta22)
Mentor: Yulin Zhang (yzhan114)
References
- Previous Implementation: E1875 wiki, E2016 wiki
- Forked Repository: E2083 github
- Pull Request: E2083 pull request
- Demo: E2083 demo video
- Deployment: temporary deployment link