Introduction

About our team

Team members:

Tianji Gao (tgao5@ncsu.edu)
Guoyi Wang (gwang25@ncsu.edu)
Yulin Zhang (yzhan114@ncsu.edu)
Boxuan Zhong (bzhong2@ncsu.edu)

Project mentor: Edward Gehringer (efg@ncsu.edu)

What is Revision Planning?

In the first round of the Expertiza reviews, reviewers are asked to give authors some guidance on how to improve their work. Then in the second round, reviewers rate how well authors have followed their suggestions. Revision planning is a mechanism used to carry the interaction one step further by having authors to supply a revision plan based on the previous round reviews. That is, the authors would derive their plan for code improvement from the previous round reviews and reviewers would later assess how well they did it.

Revision planning is helpful because it makes the author think about what's necessary to improve the work before putting forth the effort to improve it. This leads to a more reflective work process and is likely to produce a better-finished product. When reviewers have an opportunity to give feedback to the author, they too will learn what a good revision plan looks like.

According to the given instructions, a revision plan consists of a description of the plan, followed by any number of questions that would later be appended to the future review questionnaire. The revision plan is per AssignmentTeam-based, which means the authors’ questions would only be used to evaluate their submission and not anyone else. By adding the functionality of revision planning, it helps researchers study the effect of the reviewer’s suggestions on the code improvement.

Previous Implementation

This functionality has previously been done by a team of students from the Fall semester of 2018. Their implementation was merged into the master branch but was reverted due to the following design concerns:

The relationship between `Questionnaire` and `SubmissionRecord` is unclear.
Uses a lot of special-purpose code when existing codes may fulfill the same job.
Revision planning cannot be enabled or disabled for an assignment.
Numeric labelings for the revision plan questions begin from 1 again, instead of continuing after the original rubric questions.
Codebase contains commented codes that are no longer wanted.

Check out the wiki page and the pull request on GitHub if you would like to learn more about the previous implementation of this project.

Please note that unlike the other teams we have reviewed, this project is a complete redo rather than modifications built upon the previous team’s codes because our approach to this problem would be different than theirs. Therefore, we will not mention the previous implementation in the later content.

Problem Statement

For this project, we identified 4 major work items that together fulfill the stated requirements.

Sort out the relationship among classes and introduce the new abstraction of the revision plan to the system in a way that it doesn’t interfere with the majority of codes

We decided to relate each Question object with team_id. A ReviewQuestionnaire will have either questions with no team_id or with a team_id. A question with no team_id indicates that it does not belong to any assignment teams so it is a question set up by the instructor. A question with a team_id, in contrast, indicates that it belongs to a particular team so it is a revision plan question. Both types of questions will be saved under the same questionnaire used for a given round. In this way, we can maximize the usage of existing codes and the only major change should be contained within the Question class.

Modify the existing views and controllers to accommodate the new functionality which includes

Allowing teaching staff to enable/disable revision planning for an assignment.
Allowing team members to create/edit their revision plan during each submission period after the first round.
Allowing both rubric questions and revision plan questions to appear on the same page and be serialized correctly.
Allowing feedback on the revision plan only to be viewed by the team that creates the plan and that team's instructor.

This will involve some minor changes such as appending some method signatures with an optional trailing parameter, adding interactive elements to the views, and slightly adjusting the structure of certain view templates.

In addition, we planned to:

Provide an adequate amount of tests to improve code coverage.
Do necessary refactoring and resolve any CodeClimate issues.

After communicated with our mentor Dr. Gehringer, we have been clarified with the following two problem statements.

Every new question must be linked to the second-round questionnaire.

This means both questions from the team’s revision plan and questions from the review rubric should be displayed together in the frontend. Since we decided to add revision plan questions to the review rubric of the round, we automatically linked every new question to the questionnaire of that round.

Every new question must be linked to the author’s submission (this will probably involve a DB migration)

By saying every new question must be linked to the author’s submission, it means that there should be some relationships between the team and the team’s revision plan questions presented in the database. We addressed this problem by associating them with a team_id field. See Database Design section for more details.

Design

Control Flow Diagram

The below image shows the control flow of the revision planning functionality.

The below image shows the control flow of the revision planning functionality. It involves 3 types of actors, student(reviewee), student(reviewer) and instructor/TA who manages the assignment and review processes. To understand each actor’s responsibility, trace each colored line that arose from each actor in the direction specified by the arrows. The diamond shape represents a decision or precondition, that is, only after the condition meets can the next action proceeds.

Summary of actions

A TA/Instructor can
1. Enable revision planning
2. Impersonate students to perform their responsibility
3. View feedback report of all teams
A student(reviewee) can
1. Make revision during the second round submission period, which includes reading first-round feedback and adding revision plan questions according to that feedback.
2. View feedback report of the team it belongs to
A student(reviewer) can
1. Give feedback on the team’s revised work by answering each question (including the team's revision plan questions) appeared on the review page.
2. View the feedback it wrote to the team.

UI Design

A revision plan should be similar to other review questionnaires. Since functionalities on the review questionnaire have been maturely implemented, we expected to make the least amount of interface changes by utilizing the existing view templates whenever possible. The subsections listed the changes we planned to make.

Enabling revision planning

Implementation of enabling/disabling revision planning for each assignment can be rather straightforward. We looked to add an additional checkbox under the "Review strategy" tab of the assignment’s edit page. This checkbox is labeled as "Enable Revision Planning?" to indicate whether the instructor wants to include this functionality in the newly-created assignment. It is most reasonable to place the checkbox here because it is review related and other similar functionalities like Self Reviews are also implemented in this manner.

Link to the revision planning page

If the instructor decided to include revision planning in this assignment, then the link to “Revision Planning” would appear in the student’s assignment page but would stay disabled until the end of the first round, in which the link becomes clickable. By clicking it, students would be redirected to a whole new page explained in the next subsection.

Revision planning page

The revision plan is just like other questionnaires in that it contains a set of questions for reviewers to answer. The only difference is that the revision plan comes with an additional block of description supplied to help reviewers understand what changes have been made so far. Therefore, it should make use of most existing view templates and controller codes with minimized changes. As the image shown, the only modification made from the instructor’s questionnaire creation page would be to include the description block right before the questions area and leave out (or hide) the place where instructors set the configuration stuff like the range of scores and the questionnaire's visibility. These configurations should use default values defined in the system rather than having students come up with their own.

Review page

The format of the review page remains almost exactly the same. To distinguish between rubric questions set up by the instructor and the revision planning questions created by the team under review, all the revision planning questions are placed after the rubric questions, split by an enlarged “Revision Planning” subheader. Under the “Revision Planning” subheader, the description will be printed to give reviewers directions on what they should look more closely on.

Feedback report

Teaching staff and students have different windows to access the feedback report.

Teaching staff: Manage->Assignments->Edit Assignment->Other stuff->View scores
Students: Assignments->View Assignment->Alternative view

In addition, either instructor and TA can impersonate students to access the feedback report from their views. We would like to consider both cases and illustrate each of them separately.

TA/Instructor

As shown in the figure below, scores for the second round review rubric and the author’s revision planning questions are displayed in two separate tables. We decided to separate them rather than placing them together in the second round table because, without the aid of the subheader (e.g. Code, Writeup, Revision Planning), it is hard for the instructor to say which criteria are defined by the instructors and which are not.

Student

The revision planning section is added to the students’ view as shown in the snapshot below. It displays in the same order as the review page. A “Revision Planning” subheader is also used here to indicate the starting of the revision planning section.

View submission

The instruction suggests that every new question must be linked to the author’s submission. While there is no obvious way to link every new question to the author’s submission (wait for the professor to clarify this part), every revision plan keeps track of its authors by keeping a team_id field inside the RevisionPlanQuestionnaire object. When the instructor checks the author’s submission of a revision plan, in addition to the list of links submitted by the author, the controller method can prepend the list with a link to the team’s revision planning page. The instructor can, therefore, click on the link to view the submitted planning.

File:E2016 view submission.png

Database Design

Here is the diagram of our database design. Our major change is on the Assignment and RevisionPlanQuestionnaire. In the Assignment, we plan to add one new field is_revision_planning_enabled? to indicate whether we would like to enable this feature or not. Additionally, we plan to add a new table: RevisionPlanQuestionnaire with team_id for which team it belongs to and the description that describes the revision plan. In this way, we can achieve the feature without modifying too much of the database.

Code Modifications

Controllers

app/controllers/questionnaires_controller.rb

edit_revision_plan: a new method added to the QuestionnaireController that prepares view template and supplies revision planning questions that belong only to the current team
update_revision_plan: a new method added to the QuestionnaireController that saves revision plan questions under the current round rubric
Require some refactoring to share some existing functionalities with the two new methods we described above

app/controllers/response_controller.rb

Replace all the occurrences of @questionnaire.questions with @questionnaire.questions(@map.reviewee_id) so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding reviewee_id.

app/controllers/grades_controller.rb

Call the retrieve_questions method every time with an extra parameter “@team_id”.

Models

app/models/question.rb

Form association relationship with Team via team_id
- e.g. belongs_to :team, class_name: ‘AssignmentTeam’, foreign_key: ‘team_id’
questions: change the method signature to ‘questions(team_id=nil)’ which uses nil as the parameter team_id’s default value. So users can choose to supply a team_id argument or not. When team_id is supplied, it returns both questions with no team_id as well as questions that have this team_id. In addition, it will add to the return list an unsaved “Revision Planning” QuestionnaireHeader so the list can be displayed nicely on the browser with each section separated.

app/models/assignment_team.rb

Form aggregation relationship with Questions via team_id
- e.g. has_many :revision_plan_questions, class_name: ‘Question’, foreign_key: ‘team_id’

app/models/response.rb

Replace all the occurrences of questionnaire.questions with questionnaire.questions(self.response_map.reviewee_id) so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding reviewee_id.

Views

app/views/questionnaires/edit_revision_plan.html.erb

It is a newly-added view file for students to create and edit their revision plan. It will use some existing codes from the app/views/questionnaires/_questionnaire.html.erb view file to reduce code redundancy.

app/views/questionnaires/_questions.html.erb

Extract codes from the app/views/questionnaires/_questionnaire.html.erb file to make it a standalone partial template that will later be used by the app/views/questionnaires/edit_revision_plan.html.erb view file we are planning to create.

app/views/student_task/view.html.erb

Add a “Revision Planning” link for the students to add their revision plan. The link will lead the students to the “edit revision planning” page. If the revision planning feature is enabled, this link will appear disabled and only become clickable during each submission period after round 1.

app/views/assignments/edit/_review_strategy.html.erb

Add “Enable Revision Planning?” checkbox for each assignment so that the TA/instructor can choose to use this feature or not. The “revision planning” link will disappear in view.html.erb if the checkbox is not checked.

Helpers

app/helpers/grades_helper.rb

retrieve_questions: add an extra parameter team_id whose default value is nil. It then should pass team_id to the Questionnaire model’s questions method and retrieve a proper question set for each team.

Schema

Assignment table: add one column named is_revision_planning_enabled? to indicate whether this feature has been activated.
Question table: add one column named team_id to distinguish whether it is a question from the review questionnaire or the revision plan.

Testing

RSpec Test Plan

Controllers

spec/controllers/questionnaires_controller_spec.rb
- Describe ‘#edit_revision_plan’
  - Context ‘when @questionnaire is not nil’
  - Context ‘when @questionnaire is nil’
  - Context ‘when the type of @questionnaire is ReviewQuestionnaire’
  - Context ‘when the type of @questionnaire is not ReviewQuestionnaire’
- Describe ‘#update_revision_plan’
  - Context ‘have some errors when updating the attributes of questionnaire’
  - Context 'when params[:add_new_questions] is not nil'
  - Context 'when params[:view_advice] is not nil'
  - Context 'successfully updates the questions in a questionnaire'
spec/controllers/grades_controller.rb
- Describe ‘#view’
  - Context ‘When the revision plan is included in one round’
- Describe ‘#view_my_scores’
  - Context ‘When the revision plan is included in one round’

Models

spec/models/question_spec.rb
- Describe ‘#questions’
  - Context ‘when team_id is supplied’
  - Context ‘when team_id is not supplied’

UI Testing Instructions (For Reviewers)

Setup Login information

Visit xxx [expertiza deployment link]
- User name: instructor6/student8030/student8031
- Password: password

Enable/Disable Revision Planning

Login as an instructor.
Go to an assignment’s edit page. Under the “Review strategy” tab, check the checkbox labeled “Enable Revision Planning?” to include the revision planning feature.

Enable/Disable the “Revision Planning” link

After the instructor configures the assignment to include the revision planning feature, the “Revision Planning” link will appear on the student's assignment page but will remain disabled and only be enabled during each submission period after round 1. Therefore, to enable the link:
- Login as an instructor.
- Go to an assignment’s edit page. Under the “Due dates” tab, change the round 2 submission date to whenever date in the future.
To disable the link after round 2 submission period:
- Login as an instructor.
- Go to an assignment’s edit page. Under the “Due dates” tab, change the round 2 submission date to whenever date in the past and change the round 2 review date to whenever date in the future.

Create teams for the assignment

Login as an instructor.
Go to an assignment’s edit page. Under the “Other stuff” tab, click “Add participant”.
In the new page, add two students, student8030 and student8031, so one student can create revision planning questions during the submission period while the other can respond to these questions during the review period.
Go back to the assignment’s edit page. Under the “Other stuff” tab, click “Create teams”.
In the new page, group the two added students to separate teams.

Edit a Revision Plan

Login as student8030.
In the assignment page, click on the “Revision Planning” link. which redirects to a new page used to create a revision plan. Fill the blanks and click on “Save review questionnaire”, and the revision plan should be saved.

Test retrieval of revision plan questions for a specific team

Login as student8030.
In the assignment page, click on the “Revision Planning” link after steps under Edit a Revision Plan have been done. The “Revision Planning” link should redirect to the Revision Planning edit page that is populated with previously saved questions.

Check the Revision Plan questions in the questionnaire.

Login as an instructor.
Adjust the time frame to the second round review period.
Log out and log in again as student8031.
In the assignment page, click on the “Others’ work” link, which takes you to the review page where you request a new team’s submission to review. Go to the only other team’s review page and check if the questions are properly displayed under the “Revision Planning” subheader.

Check responses to the Revision Plan questions

Login again as student8030.
In the assignment page, click on the “Alternative View” link, and see if student8030 gets responses for both the original rubric questions as well as its revision plan questions.

CSC/ECE 517 Spring 2020 E2016 Revision planning tool