Introduction

What is Revision Planning?

In the first round of Expertiza reviews, reviewers are asked to give authors some guidance on how to improve their work. Then in the second round, reviewers rate how well authors have followed their suggestions. Revision planning is a mechanism used to carry the interaction one step further by having authors to supply a revision plan based on the first-round reviews. That is, the authors would describe their plan for code improvement in the second round and second-round reviewers would assess how well they did it.

Revision planning is helpful, because it makes the author think about what's necessary to improve the work before putting forth the effort to improve it. This leads to a more reflective work process and is likely to produce a better finished product. When reviewers have an opportunity to give feedback to the author, they too will learn what a good revision plan looks like.

According to the given instructions, a revision plan consists of a description of the plan, followed by any number of questions that would later be appended to the second-round review questionnaire. The revision plan is per AssignmentTeam-based, which means the authors’ questions would only be used to evaluate their submission and not anyone else. By adding the functionality of revision planning, it helps researchers study the effect of the reviewer’s suggestions on the second-round code improvement.

What needs to be done?

Develop UI for authors to create new questions to add to the second round-rubric. This should be a form that includes the following:
- A description of the revision plan. Eg: We will add feature X to address issues a,b and c. We will modify feature Y and expect it to resolve errors d, c, and e.
- One or more questions for every proposed improvement. Example:
  - How effectively did feature X address / solve issues a, b and c?
  - Did modification of feature Y resolve error d?
The new questionnaire must be part of the second-round questionnaire.
The new questionnaire must be part of the team's submission records.

Problem Statement

For this project, we identified 4 major work items that together fulfill the stated requirements.

Sort out the relationship among classes and introduce the new abstraction of the revision plan to the system in a way that it doesn’t interfere with the majority of codes
Modify the existing views and controllers to accommodate the new functionality which includes
- Allowing teaching staff to enable/disable revision planning for an assignment
- Allowing team members to create/edit their revision plan during the second-round submission period.
- Allowing both rubric questions and revision plan questions to appear on the same page and be serialized correctly
- Allowing feedback on the revision plan only to be viewed by the team that creates the plan and that team's instructor
Provide an adequate amount of tests to improve code coverage
Do necessary refactoring and resolve any CodeClimate issues

Design

Control Flow Diagram

The below image shows the control flow of the revision planning functionality. It involves 3 types of actors, student(reviewee), student(reviewer) and instructor/TA who manages the assignment and review processes. To understand each actor’s responsibility, trace the solid black lines that arose from each actor in the direction specified by the arrows. Two shapes connected by a dashed line represent that the former action triggers the latter event. The diamond shape represents a decision or precondition, that is, only after the condition meets can the next action performs. For example, the reviewee may start making revisions only after 1). TA/instructor has enabled the revision planning for this assignment, and 2). The first draft due date has passed.

UI Design

A revision plan should be similar to other review questionnaires. Since functionalities on the review questionnaire have been well implemented, we expected to make the least amount of interface changes by utilizing the existing views whenever possible. The subsections listed the changes we planned to make.

Enabling revision planning

Implementation of enabling/disabling revision planning for each assignment can be rather straightforward. What we planned to do is adding an extra checkbox under the "Review strategy" tab of the assignment edit page. This checkbox is labeled as "Enable Revision Planning?" to indicate whether the instructor wants to include this functionality in the newly-created assignment. It is most reasonable to place the checkbox here because it is review related and other similar functionalities like Self Reviews are also implemented in this manner.

Link to the revision planning page

If the instructor decided to include revision planning in this assignment, then the link to “Revision Planning” would appear in the student’s assignment page but would stay disabled until the end of the first round, in which the link becomes clickable. By clicking it, students would be redirected to a whole new page explained in the next subsection.

Revision planning page

The revision plan is just like other questionnaires in that it contains a set of questions for reviewers to answer. The only difference is that the revision plan comes with an additional block of description supplied to help reviewers understand what changes have been made so far. Therefore, it should make use of most existing view templates and controller codes with minimized changes. As the image shown, the only modification made from the instructor’s questionnaire creation page would be to include the description block right before the questions area and leave out (or hide) the place where instructors set the configuration stuff like the range of scores and the questionnaire's visibility. These configurations should use default values defined in the system rather than having students come up with their own.

Review page

The format of the review page remains almost exactly the same. To distinguish between rubric questions set up by the instructor and the revision planning questions created by the team under review, all the revision planning questions are placed after the rubric questions, split by an enlarged “Revision Planning” subheader. Under the “Revision Planning” subheader, the description will be printed to give reviewers directions on what they should look more closely on.

Feedback report

Teaching staff and students have different windows to access the feedback report.

Teaching staff: Manage->Assignments->Edit Assignment->Other stuff->View scores
Students: Assignments->View Assignment->Alternative view

In addition, either instructor and TA can impersonate students to access the feedback report from their views. We would like to consider both cases and illustrate each of them separately.

TA/Instructor

As shown in the figure below, scores for the second round review rubric and the author’s revision planning questions are displayed in two separate tables. We decided to separate them rather than placing them together in the second round table because, without the aid of the subheader (e.g. Code, Writeup, Revision Planning), it is hard for the instructor to say which criteria are defined by the instructors and which are not.

Student

The revision planning section is added to the students’ view as shown in the snapshot below. It displays in the same order as the review page. A “Revision Planning” subheader is also used here to indicate the starting of the revision planning section.

View submission

The instruction suggests that every new question must be linked to the author’s submission. While there is no obvious way to link every new question to the author’s submission (wait for the professor to clarify this part), every revision plan keeps track of its authors by keeping a team_id field inside the RevisionPlanQuestionnaire object. When the instructor checks the author’s submission of a revision plan, in addition to the list of links submitted by the author, the controller method can prepend the list with a link to the team’s revision planning page. The instructor can, therefore, click on the link to view the submitted planning.

Database Design

Here is the diagram of our database design. Our major change is on the Assignment and RevisionPlanQuestionnaire. In the Assignment, we plan to add one new field is_revision_planning_enabled? to indicate whether we would like to enable this feature or not. Additionally, we plan to add a new table: RevisionPlanQuestionnaire with team_id for which team it belongs to and the description that describes the revision plan. In this way, we can achieve the feature without modifying too much of the database.

Design Concerns

Not adding team_id attribute to the Question model

If the team_id attribute is used to differentiate different team’s revision planning questions, then all the questions can be dumped into one questionnaire, either the original second rubric or we construct a new ReviewQuestionnaire that holds all the revision planning questions from a revision-planning-enabled assignment. Both solutions would hurt the performance, especially the first one. As more and more outside questions enter the rubric, retrieving the original set of questions becomes harder and harder. Each time it has to traverse through all the irrelevant data before matching the team to its revision plan questions. Our design resolves this issue, making it simpler at least from the coding perspective. (question: will the relational database always have to traverse through each element in the table no matter how the relationship is implemented?)
The question model is shared by all other features, not just revision planning. However, not every questionnaire needs the team_id attribute. This attribute would probably go unused most of the time since the assignment has to enable the revision planning feature to make it useful. Having most Question objects to hold an empty team_id value occupies a lot of memory space, which we can save by only creating an extra RevisionPlanQuestionnaire object.
The description of the revision plan has nowhere to save if we don’t have a concrete RevisionPlanQuestionnaire class. We originally considered storing the description in the AssignmentTeam class since each assignment team has only one revision plan and each revision plan has only one description, which is a one-to-one relationship. However, if we did so, we would encounter the same problem mentioned in bullet point two that a lot of allocated spaces just go unused. Therefore, we came to the agreement that this may not be a good practice.
According to the open-closed principle, software entities should be open for extension but closed for modification.