Introduction

About our team

Team members:

Tianji Gao (tgao5@ncsu.edu)
Guoyi Wang (gwang25@ncsu.edu)
Yulin Zhang (yzhan114@ncsu.edu)
Boxuan Zhong (bzhong2@ncsu.edu)

Project mentor: Edward Gehringer (efg@ncsu.edu)

What is Revision Planning?

In the first round of the Expertiza reviews, reviewers are asked to give authors some guidance on how to improve their work. Then in the second round, reviewers rate how well authors have followed their suggestions. Revision planning is a mechanism used to carry the interaction one step further by having authors to supply a revision plan based on the previous round reviews. That is, the authors would derive their plan for code improvement from the previous round reviews and reviewers would later assess how well they did it.

Revision planning is helpful because it makes the author think about what's necessary to improve the work before putting forth the effort to improve it. This leads to a more reflective work process and is likely to produce a better-finished product. When reviewers have an opportunity to give feedback to the author, they too will learn what a good revision plan looks like.

According to the given instructions, a revision plan consists of a description of the plan, followed by any number of questions that would later be appended to the future review questionnaire. The revision plan is per AssignmentTeam-based, which means the authors’ questions would only be used to evaluate their submission and not anyone else. By adding the functionality of revision planning, it helps researchers study the effect of the reviewer’s suggestions on the code improvement.

Previous Implementation

This functionality has previously been done by a team of students from the Fall semester of 2018. Their implementation was merged into the master branch but was reverted due to the following design concerns:

The relationship between `Questionnaire` and `SubmissionRecord` is unclear.
Uses a lot of special-purpose code when existing codes may fulfill the same job.
Revision planning cannot be enabled or disabled for an assignment.
Numeric labelings for the revision plan questions begin from 1 again, instead of continuing after the original rubric questions.
Codebase contains commented codes that are no longer wanted.

Check out the wiki page and the pull request on GitHub if you would like to learn more about the previous implementation of this project.

Please note that unlike the other teams we have reviewed, this project is a complete redo rather than modifications built upon the previous team’s codes because our approach to this problem would be different than theirs. Therefore, we will not mention the previous implementation in the later content.

Problem Statement

For this project, we identified 4 major work items that together fulfill the stated requirements.

Sort out the relationship among classes and introduce the new abstraction of the revision plan to the system in a way that it doesn’t interfere with the majority of codes

We decided to relate each Question object with team_id. A ReviewQuestionnaire will have either questions with no team_id or with a team_id. A question with no team_id indicates that it does not belong to any assignment teams so it is a question set up by the instructor. A question with a team_id, in contrast, indicates that it belongs to a particular team so it is a revision plan question. Both types of questions will be saved under the same questionnaire used for a given round. In this way, we can maximize the usage of existing codes and the only major change should be contained within the Question class.

Modify the existing views and controllers to accommodate the new functionality which includes

Allowing teaching staff to enable/disable revision planning for an assignment.
Allowing team members to create/edit their revision plan during each submission period after the first round.
Allowing both rubric questions and revision plan questions to appear on the same page and be serialized correctly.
Allowing feedback on the revision plan only to be viewed by the team that creates the plan and that team's instructor.

This will involve some minor changes such as appending some method signatures with an optional trailing parameter, adding interactive elements to the views, and slightly adjusting the structure of certain view templates.

In addition, we planned to:

Provide an adequate amount of tests to improve code coverage.
Do necessary refactoring and resolve any CodeClimate issues.

After communicated with our mentor Dr. Gehringer, we have been clarified with the following two problem statements.

Every new question must be linked to the second-round questionnaire.

This means both questions from the team’s revision plan and questions from the review rubric should be displayed together in the frontend. Since we decided to add revision plan questions to the review rubric of the round, we automatically linked every new question to the questionnaire of that round.

Every new question must be linked to the author’s submission (this will probably involve a DB migration)

By saying every new question must be linked to the author’s submission, it means that there should be some relationships between the team and the team’s revision plan questions presented in the database. We addressed this problem by associating them with a team_id field. See Database Design section for more details.

Design

Control Flow Diagram

The below image shows the control flow of the revision planning functionality.

The below image shows the control flow of the revision planning functionality. It involves 3 types of actors, student(reviewee), student(reviewer) and instructor/TA who manages the assignment and review processes. To understand each actor’s responsibility, trace each colored line that arose from each actor in the direction specified by the arrows. The diamond shape represents a decision or precondition, that is, only after the condition meets can the next action proceeds.

Summary of actions

A TA/Instructor can
1. Enable revision planning
2. Impersonate students to perform their responsibility
3. View feedback report of all teams
A student(reviewee) can
1. Make revision during the second round submission period, which includes reading first-round feedback and adding revision plan questions according to that feedback.
2. View feedback report of the team it belongs to
A student(reviewer) can
1. Give feedback on the team’s revised work by answering each question (including the team's revision plan questions) appeared on the review page.
2. View the feedback it wrote to the team.

UI Design

A revision plan should be similar to other review questionnaires. Since functionalities on the review questionnaire have been maturely implemented, we expected to make the least amount of interface changes by utilizing the existing view templates whenever possible. The subsections listed the changes we planned to make.

Enabling revision planning

Implementation of enabling/disabling revision planning for each assignment can be rather straightforward. We looked to add an additional checkbox under the "Review strategy" tab of the assignment’s edit page. This checkbox is labeled as "Enable Revision Planning?" to indicate whether the instructor wants to include this functionality in the newly-created assignment. It is most reasonable to place the checkbox here because it is review related and other similar functionalities like Self Reviews are also implemented in this manner.

Link to the revision planning page

If the instructor decided to include revision planning in this assignment, then the link to “Revision Planning” would appear on the student’s assignment page but would stay disabled during the first round. After that, It would become clickable during every submission period and greyed again during every review period. By clicking it, students would be redirected to a whole new page explained under the ‘Revision planning page’ subsection.

Revision planning page

The revision plan is just like other questionnaires that it contains a set of questions for reviewers to answer. The only difference is that the revision plan comes with an additional description to help reviewers understand what changes have been made so far.

Therefore, it should make use of most existing view templates and controller codes with minimized changes. As the image is shown, the only modification made from the existing questionnaire creation template would be to include a link that redirects students to the submission page, where the uploading of the revision plan will be handled by the existing implementation. The advantage to upload an external link rather than typing everything to the textbox element is that the description can be well-formatted if it displays outside the form and not causing a distraction effect for reviewers. We also decided to leave out (or hide) the place where instructors set the configuration stuff like the range of scores and the questionnaire's visibility. These configurations should use default values defined in the system rather than having students come up with their own.

Review page

The format of the review page remains almost exactly the same. To distinguish between rubric questions set up by the instructor and the revision plan questions created by the team under review, all the revision plan questions are placed after the rubric questions, split by an enlarged “Revision Planning” subheader.

Feedback report

Teaching staff and students have different windows to access the feedback report.

Teaching staff: Manage->Assignments->Edit Assignment->Other stuff->View scores
Students: Assignments->View Assignment->Alternative View

In addition, either instructor and TA can impersonate students to access the feedback report from their views. We would like to consider both cases and illustrate each of them separately.

TA/Instructor

Scores for the second round review rubric and the author’s revision plan questions will be displayed on the same table and are serialized correctly. See the figure below for an example. Let say the second round rubric has only 5 questions, the remaining questions (6-10) will be revision planning questions written by a particular team.

For reviewer: if you reviewed our first draft design, you should notice that we originally chose to place revision plan scores on a distinct table. After our mentor clarified to us, we realized that there can be possibly more than 2 rounds of submission and review periods for a given assignment. Therefore, scores for revision planning questions can vary round by round. Therefore, our previous solution will not work since it confuses the user of which round the revision plan scores refer to.

Student

The revision planning section will be added to the students’ view as shown in the snapshot below. It displays in the same order as how the review page does. A “Revision Planning” subheader is also used here to indicate the starting of the revision planning section.

Database Design

Here we present the diagram of our database design. As the yellow borders show, we only plan to modify the structure of the Question table and the Assignment table.

In the Assignment table, the column is_revision_planning_enabled? will be needed to indicate whether the instructor would like to incorporate the revision planning feature.

Additionally, we add team_id to each Question object to distinguish whether the question is on the original rubric or is added by students as part of their revision plan. A Question object with an empty team_id value will be the question under the original rubric, while the object with a non-empty team_id field will be regarded as the question created by the team that the team_id refers to. That is, instead of creating a whole new RevisionPlanQuestionnaire class, we decided to dump all the revision plan questions that are created in a given round to the rubric that is used for that round. In this way, we minimize the change to the system to make the original rubric questions and the revision planning questions retrieved together more easily.

Code Modifications

Controllers

app/controllers/questionnaires_controller.rb

edit_revision_plan: a new method added to the QuestionnaireController that prepares view template and supplies revision planning questions that belong only to the current team
update_revision_plan: a new method added to the QuestionnaireController that saves revision plan questions under the current round rubric
Require some refactoring to share some existing functionalities with the two new methods we described above

app/controllers/response_controller.rb

Replace all the occurrences of @questionnaire.questions with @questionnaire.questions(@map.reviewee_id) so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding reviewee_id.

app/controllers/grades_controller.rb

Call the retrieve_questions method every time with an extra parameter “@team_id”.

Models

app/models/question.rb

Form association relationship with Team via team_id
- e.g. belongs_to :team, class_name: ‘AssignmentTeam’, foreign_key: ‘team_id’
questions: change the method signature to questions(team_id=nil) which uses nil as the parameter team_id’s default value, so users can choose to supply a team_id argument or not. When team_id is supplied, it returns both questions with no team_id as well as questions that have this team_id. In addition, it will add to the return list an unsaved “Revision Planning” QuestionnaireHeader so the list can be displayed nicely on the browser with each section separated.

app/models/assignment_team.rb

Form aggregation relationship with Question via team_id
- e.g. has_many :revision_plan_questions, class_name: ‘Question’, foreign_key: ‘team_id’

app/models/response.rb

Replace all the occurrences of questionnaire.questions with questionnaire.questions(self.response_map.reviewee_id) so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding reviewee_id.

Views

app/views/questionnaires/edit_revision_plan.html.erb

It is a newly-added view file for students to create and edit their revision plan. It will utilize some existing codes from the app/views/questionnaires/_questionnaire.html.erb view file to reduce code redundancy.

app/views/questionnaires/_questions.html.erb

Extract codes from the app/views/questionnaires/_questionnaire.html.erb view file to make it a standalone partial template that will later be loaded by the app/views/questionnaires/edit_revision_plan.html.erb view file described above.

app/views/student_task/view.html.erb

Add a “Revision Planning” link for students to edit their revision plan. The link will lead students to the “Edit Revision Plan” page. If the revision planning feature is enabled, this link will appear disabled at first and only become clickable during each submission period after round 1.

app/views/assignments/edit/_review_strategy.html.erb

Add “Enable Revision Planning?” checkbox for each assignment because not every assignment needs this feature. TA/instructor have the option to include this feature more flexibly. The “Revision Planning” link will disappear from the assignment page if the checkbox is not checked.

Helpers

app/helpers/grades_helper.rb

retrieve_questions: add an extra parameter team_id whose default value is set to be nil. It then invoke Questionnaire model’s questions method with this team_id and retrieve a proper question set for each team.

Schema

Assignment table: add one column named is_revision_planning_enabled? to indicate whether this feature has been activated.
Question table: add one column named team_id to distinguish whether the question is from the official review rubric or from a particular team's revision plan.

Testing

RSpec Test Plan

Controllers

spec/controllers/questionnaires_controller_spec.rb
- Describe ‘#edit_revision_plan’
  - Context ‘when params[:id] is valid’
  - Context ‘when params[:id] is not valid'
- Describe ‘#update_revision_plan’
  - Context 'when params[:add_new_questions] is not nil'
  - Context 'when params[:view_advice] is not nil'
  - Context 'when both params[:add_new_questions] and params[:view_advice] are nil'
spec/controllers/grades_controller_spec.rb
- Describe ‘#view’
  - Context ‘When the revision plan is included in one round’
- Describe ‘#view_my_scores’
  - Context ‘When the revision plan is included in one round’
spec/controllers/response_controller_spec.rb
- Fix all failed tests to accommodate modified codes.
- Add more tests regarding revision planning.

Models

spec/models/question_spec.rb
- Describe ‘#questions’
  - Context ‘when team_id is supplied’
  - Context ‘when team_id is not supplied’

UI Testing Instructions (For Reviewers)

Setup

Login information

Visit xxx [expertiza deployment link]

   User name: instructor6/student8030/student8031
   Password: password

Enable/Disable Revision Planning

Login as instructor6.
Go to an assignment’s edit page. Under the “Review strategy” tab, check the checkbox labeled “Enable Revision Planning?” to enable the revision planning feature.

Enable/Disable the “Revision Planning” link

After the instructor configures the assignment to include the revision planning feature, the “Revision Planning” link will appear on the student's assignment page but will remain disabled and only be enabled during each submission period after round 1. Therefore, to enable the link:
- Login as instructor6.
- Go to an assignment’s edit page. Under the “Due dates” tab, change the round 2 submission date to whenever date in the future.
To disable the link after round 2 submission period:
- Login as instructor6.
- Go to an assignment’s edit page. Under the “Due dates” tab, change the round 2 submission date to whenever date from the past and change the round 2 review date to whenever date in the future.

Create teams for the assignment

Login as instructor6.
Go to an assignment’s edit page. Under the “Other stuff” tab, click “Add participant”.
In the new page, add two students, student8030 and student8031, so one student can create revision plan questions during the submission period while the other can respond to these questions during the review period.
Go back to the assignment’s edit page. Under the “Other stuff” tab, click “Create teams”.
In the new page, group the two added students to separate teams.

Functionalities

Edit a Revision Plan

Login as student8030.
In the assignment page, click on the “Revision Planning” link. which redirects the user to a new page used to create a revision plan. Fill the blanks and click on the “Save review questionnaire” button, and the revision plan should be saved.

Test retrieval of revision plan questions for a specific team

Login as student8030.
In the assignment page, click on the “Revision Planning” link after steps under Edit a Revision Plan have been done. The “Revision Planning” link should redirect the user to the Revision Planning edit page that is populated with previously saved questions.

Check the Revision Plan questions in the questionnaire.

Login as instructor6.
Adjust the time frame to the second round review period.
Log out and log in again as student8031.
In the assignment page, click on the “Others’ work” link, which takes the user to the review page where one requests a new team’s submission to review. Go to the only other team’s review page and check if the questions are properly displayed under the “Revision Planning” subheader.

Check responses to the Revision Plan questions

Login again as student8030.
In the assignment page, click on the “Alternative View” link, and see if student8030 gets responses for both the original rubric questions as well as its revision plan questions.

CSC/ECE 517 Spring 2020 E2016 Revision planning tool