Introduction

About our team

Team members:

Tianji Gao (tgao5@ncsu.edu)
Guoyi Wang (gwang25@ncsu.edu)
Yulin Zhang (yzhan114@ncsu.edu)
Boxuan Zhong (bzhong2@ncsu.edu)

Project mentor: Edward Gehringer (efg@ncsu.edu)

What is Revision Planning?

In the first round of the Expertiza reviews, reviewers are asked to give authors some guidance on how to improve their work. Then in the second round, reviewers rate how well authors have followed their suggestions. Revision planning is a mechanism used to carry the interaction one step further by having authors to supply a revision plan based on the previous round reviews. That is, the authors would derive their plan for code improvement from the previous round reviews and reviewers would later assess how well they did it.

Revision planning is helpful because it makes the author think about what's necessary to improve the work before putting forth the effort to improve it. This leads to a more reflective work process and is likely to produce a better-finished product. When reviewers have an opportunity to give feedback to the author, they too will learn what a good revision plan looks like.

According to the given instructions, a revision plan consists of a description of the plan, followed by any number of questions that would later be appended to the future review questionnaire. The revision plan is per AssignmentTeam-based, which means the authors’ questions would only be used to evaluate their submission and not anyone else. By adding the functionality of revision planning, it helps researchers study the effect of the reviewer’s suggestions on the code improvement.

Previous Implementation

This functionality has previously been done by a team of students from the Fall semester of 2018. Their implementation was merged into the master branch but was reverted due to the following design concerns:

The relationship between `Questionnaire` and `SubmissionRecord` is unclear.
Uses a lot of special-purpose code when existing codes may fulfill the same job.
Revision planning cannot be enabled or disabled for an assignment.
Numeric labelings for the revision plan questions begin from 1 again, instead of continuing after the original rubric questions.
Codebase contains commented codes that are no longer wanted.

Check out the wiki page and the pull request on GitHub if you would like to learn more about the previous implementation of this project.

Please note that unlike the other teams we have reviewed, this project is a complete redo rather than modifications built upon the previous team’s codes because our approach to this problem would be different than theirs. Therefore, we will not mention the previous implementation in the later content.

Problem Statement

For this project, we identified 4 major work items that together fulfill the stated requirements.

Sort out the relationship among classes and introduce the new abstraction of the revision plan to the system in a way that it doesn’t interfere with the majority of codes

We decided to relate each Question object with team_id. A ReviewQuestionnaire may have both questions with no team_id and some questions with a team_id. A question with no team_id indicates that it does not belong to any assignment teams so it is a question set up by the instructor, that will appear on review forms for all submissions. A question with a team_id, in contrast, indicates that it belongs to a particular team so it is a revision plan question. Both types of questions will be saved under the same questionnaire used for a given round. In this way, we can maximize the usage of existing codes and the only major change should be contained within the Question class.

Modify the existing views and controllers to accommodate the new functionality which includes

Allowing teaching staff to enable/disable revision planning for an assignment.
Allowing team members to create/edit their revision plan during each submission period after the first round.
Allowing both rubric questions and revision plan questions to appear on the same page and be serialized correctly.
Allowing feedback on the revision plan only to be viewed by the team that creates the plan and that team's instructor.

This will involve some minor changes such as appending an optional trailing parameter to some method signatures, adding interactive elements to the views, and slightly adjusting the structure of certain view templates.

In addition, we planned to:

Provide an adequate amount of tests to improve code coverage.
Do necessary refactoring and resolve any CodeClimate issues.

After communicated with our mentor Dr. Gehringer, we have been clarified with the following two problem statements.

Every new question must be linked to the second-round questionnaire.

This means both questions from the team’s revision plan and questions from the review rubric should be displayed together in the frontend. Since we decided to add revision plan questions to the review rubric of the round, we automatically linked every new question to the questionnaire of that round.

Every new question must be linked to the author’s submission

By saying every new question must be linked to the author’s submission, it means that there should be some relationships between the team and the team’s revision plan questions presented in the database. We addressed this problem by associating them with a team_id field. See Database Design section for more details. This will probably involve a DB migration to add the team_id field to the questions table.

Design

Control Flow Diagram

The below image shows the control flow of the revision planning functionality.

[Better to test if you are in a resubmission period, since there are method calls to do that. If you tested if Round 1 had ended, you'd have to write queries and then do calculations on the data.]

The below image shows the control flow of the revision planning functionality. It involves 3 types of actors, student (reviewee), student (reviewer) and instructor/TA who manages the assignment and review processes. To understand each actor’s responsibility, trace each colored line that arose from each actor in the direction specified by the arrows. The diamond shape represents a decision or precondition, that is, only after the condition meets can the next action proceeds.

Summary of actions

A TA/Instructor can
1. Enable revision planning
2. Impersonate students to perform their responsibility
3. View feedback report of all teams
A student (reviewee) can
1. Make revision during the second round submission period, which includes reading first-round feedback and adding revision plan questions according to that feedback.
2. View feedback report of the team it belongs to
A student (reviewer) can
1. Give feedback on the team’s revised work by answering each question (including the team's revision plan questions) appeared on the review page.
2. View the feedback it wrote to the team.

UI Design

A revision plan should be similar to other review questionnaires. Since functionalities on the review questionnaire have been maturely implemented, we expected to make the least amount of interface changes by utilizing the existing view templates whenever possible. The subsections listed the changes we planned to make.

Enabling revision planning

Implementation of enabling/disabling revision planning for each assignment can be rather straightforward. We looked to add an additional checkbox under the "Review strategy" tab of the assignment’s edit page. This checkbox is labeled as "Enable Revision Planning?" to indicate whether the instructor wants to include this functionality in the newly-created assignment. It is most reasonable to place the checkbox here because it is review related and other similar functionalities like Self Reviews are also implemented in this manner.

Link to the revision planning page

If the instructor decided to include revision planning in this assignment, then the link to “Revision Planning” would appear on the student’s assignment page but would stay disabled during the first round. After that, It would become clickable during every submission period and greyed again during every review period. By clicking it, students would be redirected to a whole new page explained under the ‘Revision planning page’ subsection. [Revision planning should surely come before "Your scores".]

Revision planning page

The revision plan is just like other questionnaires in that it contains a set of questions for reviewers to answer. The only difference is that the revision plan comes with an additional description to help reviewers understand what changes have been made so far. Therefore, it should make use of most existing view templates and controller codes with minimized changes. As the image is shown, the only modification made from the existing questionnaire creation template would be to include a link that redirects students to the submission page, where the uploading of the revision plan will be handled by the existing implementation. The advantage to upload an external link rather than typing everything to the textbox element is that the description can be well-formatted if it displays outside the form and not causing a distraction effect for reviewers during the review. We also decided to leave out (or hide) the place where instructors set the configuration stuff like the range of scores and the questionnaire's visibility. These configurations should use default values defined in the system rather than having students come up with their own.

Review page

The format of the review page remains almost exactly the same. To distinguish between rubric questions set up by the instructor and the revision plan questions created by the team under review, all the revision plan questions are placed after the rubric questions, split by an enlarged “Revision planning” subheader. [Having a "Revision planning" header is reasonable, but if the author changes a sequence number to put one of their questions earlier in the rubric, that should be allowed.]

Feedback report

Teaching staff and students have different windows to access the feedback report.

Teaching staff: Manage->Assignments->Edit Assignment->Other stuff->View scores
Students: Assignments->View Assignment->Alternative View

In addition, either instructor and TA can impersonate students to access the feedback report from their views. We would like to consider both cases and illustrate each of them separately.

TA/Instructor

Scores for the second round review rubric and the author’s revision plan questions will be displayed on the same table and are serialized correctly. See the figure below for an example. Let say the second round rubric has only 5 questions, the remaining questions (6-10) will be revision planning questions written by a particular team.

For reviewer: if you reviewed our first draft design, you should notice that we originally chose to place revision plan scores on a distinct table. After our mentor clarified to us, we realized that there can be possibly more than 2 rounds of submission and review periods for a given assignment. Therefore, scores for revision planning questions can vary round by round. Therefore, our previous solution will not work since it confuses the user of which round the revision plan scores refer to. [It would be nice if there was a way to distinguish instructor questions from questions added by students. This might be done with a special symbol, or by shading the background of the cell with the question number, etc.]

Student

The revision planning section will be added to the students’ view as shown in the snapshot below. It displays in the same order as how the review page does. A “Revision Planning” subheader is also used here to indicate the starting of the revision planning section.

Database Design

Here we present the diagram of our database design. As the yellow borders show, we only plan to modify the structure of the Question table and the Assignment table.

In the Assignment table, the column is_revision_planning_enabled? will be needed to indicate whether the instructor would like to incorporate the revision planning feature.

Additionally, we add team_id to each Question object to distinguish where each question belongs to. A Question object with an empty team_id value will be the question under the original rubric, while the object with a non-empty team_id field will be the question created by the team associated with the team_id. That is, instead of creating a whole new RevisionPlanQuestionnaire class, we decided to dump all the revision plan questions that are created in a given round to the rubric that is used for that round. In this way, we minimize the change to the system to make the original rubric questions and the revision planning questions retrieved together more easily.

Code Modifications

Controllers

controllers/questionnaires_controller.rb

action_allowed?: allow students could edit revision questionnaire.
edit: when params[:team_id] is exist, "edit_revision_plan" will be redirected.
update: extract the duplicate code as a new function update_revision.
add_new_questions: add team_id to question, allow student to add new question for revision plan.
save_all_questions: The duplicate code is extracted as a new function update_revision. The function adjust_question_seq is used to adjust the sequence of the question in the revision plan based on the origin rubric.
edit_revision_plan: a new method that prepares view template and supplies revision planning questions that belong only to the current team
update_revision: a new private method that could update questions that are already created. This method is extracted from original methods update and save_all_questions.
adjust_question_seq: a new private method added to the QuestionnaireController that adjust the questions created by students in the original rubric.
question_params: add team_id

controllers/questions_controller.rb

action_allowed?: allow students to remove questions in revision plan.
destroy: add team_id, so that when student click "remove" buttom, it will redirect to revision plan page.

controllers/response_controller.rb

update/ new/ create/ set_content: use set_questions to get the questions from the original rubric and from revision plan proposed by the team with the corresponding reviewee_id.
set_questions: a new private method that gets a correct set of questions for each team in a given round.

controllers/grades_controller.rb

view_my_scores：add @team_id to retrieve_questions method, so students could see the revision plan question in "Your scores"
view_team: add @team_id to retrieve_questions method, so students could see the revision plan question in "Alternative View". And a new method sort_questions defined in grades_helper is used to sort questions including questions in revision plans.
calculate_all_penalties: AssignmentParticipant is used to looking for the participant because the participant_id as a argument in function calculate_penalty is expected a AssignmentParticipant.

Models

models/due_date.rb

self.get_next_due_date: get the next_due_data in ascending order.

models/question.rb

Form association relationship with Team via team_id.
- belongs_to :team, class_name: ‘AssignmentTeam’, foreign_key: ‘team_id’
- validates :team_id, numericality: true, allow_nil: true

models/questionnaire.rb

questions: a new method could return questions in the original rubric as well as the team's revision planning questions if the team_id is supplied.

models/response.rb

display_as_html: add team_id and round, so the response of revision plan could be display as html.
construct_review_respone: add team_id and round, so the response of revision plan could be return.
Replace all the occurrences of questionnaire.questions with questionnaire.questions(self.response_map.reviewee_id) so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding reviewee_id.

models/assignment_team.rb

Form aggregation relationship with Question via team_id
- has_many :revision_plan_questions, class_name: ‘Question’, foreign_key: ‘team_id’

models/revision_plan_questionnaire.rb

self.questions: a new model file could return the team's revision planning questions

Views

views/questionnaires/_questioinnaire.html.erb

The code is extracted as a partial template named views/questionnaires_questions.html.erb.

views/questionnaires/edit_revision_plan.html.erb

It is a newly-added view file for students to create and edit their revision plan. It use some existing codes from the app/views/questionnaires/_questionnaire.html.erb view file to reduce code redundancy.

views/questionnaires/_questions.html.erb

Extract codes from the app/views/questionnaires/_questionnaire.html.erb view file to make it a standalone partial template that will later be loaded by the app/views/questionnaires/edit_revision_plan.html.erb and app/views/questionnaires/_questioinnaire.html.erb.

views/student_task/view.html.erb

Add a “Revision Planning” link for students to edit their revision plan. The link will lead students to the “Edit Revision Plan” page. If the revision planning feature is enabled, this link will appear disabled at first and only become clickable during each submission period after round 1.

views/assignments/edit/_review_strategy.html.erb

Add “Enable Revision Planning?” checkbox for each assignment because not every assignment needs this feature. TA/instructor have the option to include this feature more flexibly. The “Revision Planning” link will disappear from the assignment page if the checkbox is not checked.

views/response/view.html.erb

Pass the @map.reviewee_id and @response.round to function display_as_html.

views/response/response.html.erb

Change the structure

views/grades/_review.html.erb

Pass the participant.team.id and round to function display_as_html.

Helpers

helpers/grades_helper.rb

sort_questions: a new method that could sorts questions by sequence number
view_heatgrid: use Questionnaire model’s questions method with team_id and round to retrieve a proper question set for each team, and then the method sort_question is used to return a sorted questions.
retrieve_questions: add an extra parameter team_id whose default value is set to be nil. It then invoke Questionnaire model’s questions method with team_id and round, then retrieve a proper question set for each team.

Schema

Assignment table: add one column named is_revision_planning_enabled? to indicate whether this feature has been activated.

Question table: add one column named team_id to distinguish whether the question is from the official review rubric or from a particular team's revision plan.

Testing

RSpec Test Plan

Controllers

spec/controllers/questionnaires_controller_spec.rb：
- describe '#edit_revision_plan': function edit_revision_plan could renders the questionnaires#edit_revision_plan page

Models

spec/models/question_spec.rb
- Describe ‘#Questions’：function Questions will return the questions with or without team_id in ReviewQuestionnaire.
spec/models/questionnaire_spec.rb
- Describe ‘#questions’: functionquestions will return the questions from the original rubric if no team_id and it will return the questions with supplied team_id.

All tests are passed. We delete test of #display_as_html in spec/models/response_spec.rb because fixing them would cause more UI errors.

UI Testing Instructions (For Reviewers)

Setup

Login information

Visit xxx [expertiza deployment link]

   User name: instructor6/student8030/student8031
   Password: password

Enable/Disable Revision Planning

Login as instructor6.
Go to an assignment’s edit page. Under the “Review strategy” tab, check the checkbox labeled “Enable Revision Planning?” to enable the revision planning feature.

Enable/Disable the “Revision Planning” link

After the instructor configures the assignment to include the revision planning feature, the “Revision Planning” link will appear on the student's assignment page but will remain disabled and only be enabled during each submission period after round 1. Therefore, to enable the link:
- Login as instructor6.
- Go to an assignment’s edit page. Under the “Due dates” tab, change the round 2 submission date to whenever date in the future.
To disable the link after round 2 submission period:
- Login as instructor6.
- Go to an assignment’s edit page. Under the “Due dates” tab, change the round 2 submission date to whenever date from the past and change the round 2 review date to whenever date in the future.

Create teams for the assignment

Login as instructor6.
Go to an assignment’s edit page. Under the “Other stuff” tab, click “Add participant”.
In the new page, add two students, student8030 and student8031, so one student can create revision plan questions during the submission period while the other can respond to these questions during the review period.
Go back to the assignment’s edit page. Under the “Other stuff” tab, click “Create teams”.
In the new page, group the two added students to separate teams.

Functionalities

Edit a Revision Plan

Login as student8030.
In the assignment page, click on the “Revision Planning” link. which redirects the user to a new page used to create a revision plan. Fill the blanks and click on the “Save review questionnaire” button, and the revision plan should be saved.

Test retrieval of revision plan questions for a specific team

Login as student8030.
In the assignment page, click on the “Revision Planning” link after steps under Edit a Revision Plan have been done. The “Revision Planning” link should redirect the user to the Revision Planning edit page that is populated with previously saved questions.

Check the Revision Plan questions in the questionnaire.

Login as instructor6.
Adjust the time frame to the second round review period.
Log out and log in again as student8031.
In the assignment page, click on the “Others’ work” link, which takes the user to the review page where one requests a new team’s submission to review. Go to the only other team’s review page and check if the questions are properly displayed under the “Revision Planning” subheader.

Check responses to the Revision Plan questions

Login again as student8030.
In the assignment page, click on the “Alternative View” link, and see if student8030 gets responses for both the original rubric questions as well as its revision plan questions.

CSC/ECE 517 Spring 2020 E2016 Revision planning tool