CSC/ECE 517 Spring 2020 E2016 Revision planning tool

What is Revision Planning?:

In the first round of Expertiza reviews, reviewers are asked to give authors some guidance on how to improve their work. Then in the second round, reviewers rate how well authors have followed their suggestions. Revision planning is a mechanism used to carry the interaction one step further by having authors to supply a revision plan based on the first-round reviews. That is, the authors would describe their plan for code improvement in the second round and second-round reviewers would assess how well they did it.

According to the given instructions, a revision plan consists of a description of the plan, followed by any number of questions that would later be appended to the second-round review questionnaire. The revision plan is per AssignmentTeam-based, which means the authors’ questions would only be used to evaluate their submission and not anyone else. By adding the functionality of revision planning, it helps researchers study the effect of the reviewer’s suggestions on the second-round code improvement.

What needs to be done?

• Develop UI for authors to create new questions to add to the second round-rubric. This should be a form that includes the following:
  • A description of the revision plan. Eg: We will add feature X to address issues a,b and c. We will modify feature Y and expect it to resolve errors d, c and e.
  • One or more questions for every proposed improvement. Example:
    • How effectively did feature X address / solve issues a, b and c?
    • Did modification of feature Y resolve error d?
• The new questionnaire must be linked to the second-round questionnaire.
• The new questionnaire must be part of the team's submission records.

Problem Statement

For this project, we identified 4 major work items that together fulfill the stated requirements.

• Sort out the relationship among classes and introduce the new abstraction of the revision plan to the system in a way that it doesn’t interfere with the majority of codes
• Modify the existing views and controllers to accommodate the new functionality which includes
  • Allowing teaching staff to enable/disable revision planning for an assignment
  • Allowing team members to create/edit their revision plan during the second-round time frame
  • Allowing both rubric questions and revision plan questions to appear on the same page and be serialized correctly
  • Allowing feedback of the revision plan only to be viewed by the team that creates the plan and that team's instructor
• Provide an adequate amount of tests to improve code coverage
• Do necessary refactoring and resolve any CodeClimate issues

Design

Control Flow Diagram

The below image shows the control flow of the revision planning functionality. It involves 3 types of actors, student(reviewee), student(reviewer) and instructor/TA who manages the assignment and review processes. To understand each actor’s responsibility, trace the solid black lines that arose from each actor in the direction specified by the arrows. Two shapes connected by a dashed line represent that the former action triggers the latter event. The diamond shape represents a decision or precondition, that is, only after the condition meets can the next action performs. For example, the reviewee may start making revisions only after 1). TA/instructor has enabled the revision planning for this assignment, and 2). The first draft due date has passed.

UI Design

A revision plan should be similar to other review questionnaires. Since functionalities on the review questionnaire have been well implemented, we expected to make the least amount of interface changes by utilizing the existing views whenever possible. The subsections listed the changes we planned to make.

Enabling revision planning

Implementation of enabling/disabling revision planning for each assignment can be rather straightforward. What we planned to do is adding an extra checkbox under the "Review strategy" tab of the assignment edit page. This checkbox is labeled as "Enable Revision Planning?" to indicate whether the instructor wants to include this functionality in the newly-created assignment. It is most reasonable to place the checkbox here because it is review related and other similar functionalities like Self Reviews are also implemented in this manner.

Link to the revision planning page

If the instructor decided to include revision planning in this assignment, then the link to “Revision Planning” would appear in the student’s assignment page but would stay disabled until the end of the first round, in which the link becomes clickable. By clicking it, students would be redirected to a whole new page explained in the next subsection.

Revision planning page

The revision plan is just like other questionnaires in that it contains a set of questions for reviewers to answer. The only difference is that the revision plan comes with an additional block of description supplied to help reviewers understand what changes have been made so far. Therefore, it should make use of most existing view templates and controller codes with minimized changes. As the image shown, the only modification made from the instructor’s questionnaire creation page would be to include the description block right before the questions area and leave out (or hide) the place where instructors set the configuration stuff like the range of scores and the questionnaire's visibility. These configurations should use default values defined in the system rather than having students come up with their own.

Review page

The format of the review page remains almost exactly the same. To distinguish between rubric questions set up by the instructor and the revision planning questions created by the team under review, all the revision planning questions are placed after the rubric questions, split by an enlarged “Revision Planning” subheader. Under the “Revision Planning” subheader, the description will be printed to give reviewers directions on what they should look more closely on.

Feedback report

Teaching staff and students have different windows to access the feedback report.

• Teaching staff: Manage->Assignments->Edit Assignment->Other stuff->View scores
• Students: Assignments->View Assignment->Alternative view

In addition, either instructor and TA can impersonate students to access the feedback report from their views. We would like to consider both cases and illustrate each of them separately.

TA/Instructor

As shown in the figure below, scores for the second round review rubric and the author’s revision planning questions are displayed in two separate tables. We decided to separate them rather than placing them together in the second round table because without the aid of the subheader (e.g. Code, Writeup, Revision Planning), it is hard for the instructor to say which criterions are defined by the instructors and which are not.

Student

The revision planning section is added to the students’ view as shown in the snapshot below. It displays in the same order as the review page. A “Revision Planning” subheader is also used here to indicate the starting of the revision planning section.