CSC/ECE 517 Spring 2020 E2016 Revision planning tool: Difference between revisions

From Expertiza_Wiki
Jump to navigation Jump to search
(Edit the format)
 
(94 intermediate revisions by 4 users not shown)
Line 1: Line 1:
=Introduction=
=Introduction=
==About our team==
Team members:
*Tianji Gao (tgao5@ncsu.edu)
*Guoyi Wang (gwang25@ncsu.edu)
*Yulin Zhang (yzhan114@ncsu.edu)
*Boxuan Zhong (bzhong2@ncsu.edu)
Project mentor: Edward Gehringer (efg@ncsu.edu)


==What is Revision Planning?==
==What is Revision Planning?==


In the first round of Expertiza reviews, reviewers are asked to give authors some guidance on how to improve their work. Then in the second round, reviewers rate how well authors have followed their suggestions. Revision planning is a mechanism used to carry the interaction one step further by having authors to supply a revision plan based on the first-round reviews. That is, the authors would describe their plan for code improvement in the second round and second-round reviewers would assess how well they did it.
In the first round of the Expertiza reviews, reviewers are asked to give authors some guidance on how to improve their work. Then in the second round, reviewers rate how well authors have followed their suggestions. Revision planning is a mechanism used to carry the interaction one step further by having authors to supply a revision plan based on the previous round reviews. That is, the authors would derive their plan for code improvement from the previous round reviews and reviewers would later assess how well they did it.  


According to the given instructions, a revision plan consists of a description of the plan, followed by any number of questions that would later be appended to the second-round review questionnaire. The revision plan is per AssignmentTeam-based, which means the authors’ questions would only be used to evaluate their submission and not anyone else. By adding the functionality of revision planning, it helps researchers study the effect of the reviewer’s suggestions on the second-round code improvement.
Revision planning is helpful because it makes the author think about what's necessary to improve the work before putting forth the effort to improve it. This leads to a more reflective work process and is likely to produce a better-finished product.  When reviewers have an opportunity to give feedback to the author, they too will learn what a good revision plan looks like.


==What needs to be done?==
According to the given instructions, a revision plan consists of a description of the plan, followed by any number of questions that would later be appended to the future review questionnaire. The revision plan is per AssignmentTeam-based, which means the authors’ questions would only be used to evaluate their submission and not anyone else. By adding the functionality of revision planning, it helps researchers study the effect of the reviewer’s suggestions on the code improvement.
* Develop UI for authors to create new questions to add to the second round-rubric. This should be a form that includes the following:
 
** A description of the revision plan. Eg: We will add feature X to address issues a,b and c. We will modify feature Y and expect it to resolve errors d, c, and e.
==Previous Implementation==
** One or more questions for every proposed improvement. Example:
 
*** How effectively did feature X address / solve issues a, b and c?
This functionality has previously been done by a team of students from the Fall semester of 2018. Their implementation was merged into the master branch but was reverted due to the following design concerns:
*** Did modification of feature Y resolve error d?
*The relationship between `Questionnaire` and `SubmissionRecord` is unclear.
* The new questionnaire must be linked to the second-round questionnaire.
*Uses a lot of '''special-purpose''' code when existing codes may fulfill the same job.
* The new questionnaire must be part of the team's submission records.
*Revision planning cannot be enabled or disabled for an assignment.
*Numeric labelings for the revision plan questions begin from 1 again, instead of continuing after the original rubric questions.
*Codebase contains commented codes that are no longer wanted.
Check out the wiki page and the pull request on GitHub if you would like to learn more about the previous implementation of this project.
*http://wiki.expertiza.ncsu.edu/index.php/E1875_Revision_Planning_Tool
*https://github.com/expertiza/expertiza/pull/1302
Please note that unlike the other teams we have reviewed, this project is a complete redo rather than modifications built upon the previous team’s codes because our approach to this problem would be different than theirs. Therefore, we will not mention the previous implementation in the later content.


==Problem Statement==
==Problem Statement==
For this project, we identified 4 major work items that together fulfill the stated requirements.
For this project, we identified 4 major work items that together fulfill the stated requirements.
*Sort out the relationship among classes and introduce the new abstraction of the revision plan to the system in a way that it doesn’t interfere with the majority of codes
 
*Modify the existing views and controllers to accommodate the new functionality which includes
'''Sort out the relationship among classes and introduce the new abstraction of the revision plan to the system in a way that it doesn’t interfere with the majority of codes'''
**Allowing teaching staff to enable/disable revision planning for an assignment
 
**Allowing team members to create/edit their revision plan during the second-round time frame
We decided to relate each <code>Question</code> object with team_id. A <code>ReviewQuestionnaire</code> may have both questions with no <code>team_id</code> and some questions with a <code>team_id</code>. A question with no team_id indicates that it does not belong to any assignment teams so it is a question set up by the instructor, that will appear on review forms for '''all''' submissions. A question with a team_id, in contrast, indicates that it belongs to a particular team so it is a revision plan question. Both types of questions will be saved under the same questionnaire used for a given round. In this way, we can maximize the usage of existing codes and the only major change should be contained within the <code>Question</code> class.
**Allowing both rubric questions and revision plan questions to appear on the same page and be serialized correctly
 
**Allowing feedback of the revision plan only to be viewed by the team that creates the plan and that team's instructor
'''Modify the existing views and controllers to accommodate the new functionality which includes'''
*Provide an adequate amount of tests to improve code coverage
*Allowing teaching staff to enable/disable revision planning for an assignment.
*Do necessary refactoring and resolve any CodeClimate issues
*Allowing team members to create/edit their revision plan during each submission period after the first round.
*Allowing both rubric questions and revision plan questions to appear on the same page and be serialized correctly.
*Allowing feedback on the revision plan only to be viewed by the team that creates the plan and that team's instructor.
 
This will involve some minor changes such as appending an optional trailing parameter to some method signatures, adding interactive elements to the views, and slightly adjusting the structure of certain view templates.
 
In addition, we planned to:
*Provide an adequate amount of tests to improve code coverage.
*Do necessary refactoring and resolve any CodeClimate issues.
 
After communicated with our mentor Dr. Gehringer, we have been clarified with the following two problem statements.
 
'''Every new question must be linked to the second-round questionnaire.'''
 
This means both questions from the team’s revision plan and questions from the review rubric should be displayed together in the frontend. Since we decided to add revision plan questions to the review rubric of the round, we automatically linked every new question to the questionnaire of that round.
 
'''Every new question must be linked to the author’s submission'''
 
By saying every new question must be linked to the author’s submission, it means that there should be some relationships between the team and the team’s revision plan questions presented in the database. We addressed this problem by associating them with a team_id field. See Database Design section for more details.  This will probably involve a DB migration to add the team_id field to the questions table.


=Design=
=Design=
==Control Flow Diagram==
==Control Flow Diagram==
The below image shows the control flow of the revision planning functionality.
It involves 3 types of actors, '''student(reviewee)''', '''student(reviewer)''' and '''instructor/TA''' who manages the assignment and review processes. To understand each actor’s responsibility, trace the solid black lines that arose from each actor in the direction specified by the arrows. Two shapes connected by a dashed line represent that the former action triggers the latter event. The diamond shape represents a decision or precondition, that is, only after the condition meets can the next action performs.
For example, the reviewee may start making revisions only after 1). TA/instructor has enabled the revision planning for this assignment, and 2). The first draft due date has passed.


[[File:control_flow_diagram_E2016.png]]
The below image shows the control flow of the revision planning functionality.
 
[[File:E2016_control_flow_diagram.png]] [Better to test if you are in a resubmission period, since there are method calls to do that.  If you tested if Round 1 had ended, you'd have to write queries and then do calculations on the data.]
 
The below image shows the control flow of the revision planning functionality. It involves 3 types of actors, student (reviewee), student (reviewer) and instructor/TA who manages the assignment and review processes. To understand each actor’s responsibility, trace each colored line that arose from each actor in the direction specified by the arrows. The diamond shape represents a decision or precondition, that is, only after the condition meets can the next action proceeds.
 
Summary of actions
*A TA/Instructor can
*#Enable revision planning
*#Impersonate students to perform their responsibility
*#View feedback report of all teams
*A student (reviewee) can
*#Make revision during the second round submission period, which includes reading first-round feedback and adding revision plan questions according to that feedback.
*#View feedback report of the team it belongs to
*A student (reviewer) can
*#Give feedback on the team’s revised work by answering each question (including the team's revision plan questions) appeared on the review page.
*#View the feedback it wrote to the team.


==UI Design==
==UI Design==
A revision plan should be similar to other review questionnaires. Since functionalities on the review questionnaire have been well implemented, we expected to make the least amount of interface changes by utilizing the existing views whenever possible. The subsections listed the changes we planned to make.
 
A revision plan should be similar to other review questionnaires. Since functionalities on the review questionnaire have been maturely implemented, we expected to make the least amount of interface changes by utilizing the existing view templates whenever possible. The subsections listed the changes we planned to make.


===Enabling revision planning===
===Enabling revision planning===
Implementation of enabling/disabling revision planning for each assignment can be rather straightforward. What we planned to do is adding an extra checkbox under the "Review strategy" tab of the assignment edit page. This checkbox is labeled as "Enable Revision Planning?" to indicate whether the instructor wants to include this functionality in the newly-created assignment. It is most reasonable to place the checkbox here because it is review related and other similar functionalities like Self Reviews are also implemented in this manner.


[[File:Enabling_revision_planning_3.png]]
Implementation of enabling/disabling revision planning for each assignment can be rather straightforward. We looked to add an additional checkbox under the "Review strategy" tab of the assignment’s edit page. This checkbox is labeled as "Enable Revision Planning?" to indicate whether the instructor wants to include this functionality in the newly-created assignment. It is most reasonable to place the checkbox here because it is review related and other similar functionalities like Self Reviews are also implemented in this manner.
 
[[File:E2016_enabling_revision_planning.png]]


===Link to the revision planning page===
===Link to the revision planning page===
If the instructor decided to include revision planning in this assignment, then the link to “Revision Planning” would appear in the student’s assignment page but would stay disabled until the end of the first round, in which the link becomes clickable. By clicking it, students would be redirected to a whole new page explained in the next subsection.
 
[[File:Link_to_the_revision_planning_page_1.png]]
If the instructor decided to include revision planning in this assignment, then the link to “Revision Planning” would appear on the student’s assignment page but would stay disabled during the first round. After that, It would become clickable during every submission period and greyed again during every review period. By clicking it, students would be redirected to a whole new page explained under the ‘Revision planning page’ subsection. [Revision planning should surely come before "Your scores".]
 
[[File:E2016_link_to_the_revision_planning_page.png]]


===Revision planning page===
===Revision planning page===
The revision plan is just like other questionnaires in that it contains a set of questions for reviewers to answer. The only difference is that the revision plan comes with an additional block of description supplied to help reviewers understand what changes have been made so far. Therefore, it should make use of most existing view templates and controller codes with minimized changes. As the image shown, the only modification made from the instructor’s questionnaire creation page would be to include the description block right before the questions area and leave out (or hide) the place where instructors set the configuration stuff like the range of scores and the questionnaire's visibility. These configurations should use default values defined in the system rather than having students come up with their own.


[[File:Revision_planning_page_2.png]]
The revision plan is just like other questionnaires in that it contains a set of questions for reviewers to answer. The only difference is that the revision plan comes with an additional description to help reviewers understand what changes have been made so far. Therefore, it should make use of most existing view templates and controller codes with minimized changes. As the image is shown, the only modification made from the existing questionnaire creation template would be to include a link that redirects students to the submission page, where the uploading of the revision plan will be handled by the existing implementation. The advantage to upload an external link rather than typing everything to the textbox element is that the description can be well-formatted if it displays outside the form and not causing a distraction effect for reviewers during the review. We also decided to leave out (or hide) the place where instructors set the configuration stuff like the range of scores and the questionnaire's visibility. These configurations should use default values defined in the system rather than having students come up with their own.
 
[[File:E2016_revision_planning_page.png]]


===Review page===
===Review page===
The format of the review page remains almost exactly the same. To distinguish between rubric questions set up by the instructor and the revision planning questions created by the team under review, all the revision planning questions are placed after the rubric questions, split by an enlarged “Revision Planning” subheader. Under the “Revision Planning” subheader, the description will be printed to give reviewers directions on what they should look more closely on.


[[File:Review_page_1.png]]
The format of the review page remains almost exactly the same. To distinguish between rubric questions set up by the instructor and the revision plan questions created by the team under review, all the revision plan questions are placed after the rubric questions, split by an enlarged “Revision planning” subheader. [Having a "Revision planning" header is reasonable, but if the author changes a sequence number to put one of their questions earlier in the rubric, that should be allowed.]
[[File:E2016_review_page.png]]
 
[[File:E2016_review_page_2.png]]


===Feedback report===
===Feedback report===
Teaching staff and students have different windows to access the feedback report.  
Teaching staff and students have different windows to access the feedback report.  
* '''Teaching staff''': Manage->Assignments->Edit Assignment->Other stuff->View scores
*'''Teaching staff''': Manage->Assignments->Edit Assignment->Other stuff->View scores
* '''Students''': Assignments->View Assignment->Alternative view
*'''Students''': Assignments->View Assignment->Alternative View
In addition, either instructor and TA can impersonate students to access the feedback report from their views. We would like to consider both cases and illustrate each of them separately.
In addition, either instructor and TA can impersonate students to access the feedback report from their views. We would like to consider both cases and illustrate each of them separately.


====TA/Instructor====
====TA/Instructor====
As shown in the figure below, scores for the second round review rubric and the author’s revision planning questions are displayed in two separate tables. We decided to separate them rather than placing them together in the second round table because, without the aid of the subheader (e.g. Code, Writeup, Revision Planning), it is hard for the instructor to say which criteria are defined by the instructors and which are not.


[[File:Feedback_report_11.png]]
Scores for the second round review rubric and the author’s revision plan questions will be displayed on the same table and are serialized correctly. See the figure below for an example. Let say the second round rubric has only 5 questions, the remaining questions (6-10) will be revision planning questions written by a particular team.
 
[[File:E2016_feedback_report_instructor.png]]
 
For reviewer: if you reviewed our first draft design, you should notice that we originally chose to place revision plan scores on a distinct table. After our mentor clarified to us, we realized that there can be possibly more than 2 rounds of submission and review periods for a given assignment. Therefore, scores for revision planning questions can vary round by round. Therefore, our previous solution will not work since it confuses the user of which round the revision plan scores refer to.
[It would be nice if there was a way to distinguish instructor questions from questions added by students.  This might be done with a special symbol, or by shading the background of the cell with the question number, etc.]


====Student====
====Student====
The revision planning section is added to the students’ view as shown in the snapshot below. It displays in the same order as the review page. A “Revision Planning” subheader is also used here to indicate the starting of the revision planning section.


[[File:Feedback_report_12.png]]
The revision planning section will be added to the students’ view as shown in the snapshot below. It displays in the same order as how the review page does. A “Revision Planning” subheader is also used here to indicate the starting of the revision planning section.


===View submission===
[[File:E2016_feedback_report_student.png]]


The instruction suggests that every new question must be linked to the author’s submission. While there is no obvious way to link every new question to the author’s submission (wait for the professor to clarify this part), every revision plan keeps track of its authors by keeping a ''team_id'' field inside the ''RevisionPlanQuestionnaire'' object. When the instructor checks the author’s submission of a revision plan, in addition to the list of links submitted by the author, the controller method can prepend the list with a link to the team’s revision planning page. The instructor can, therefore, click on the link to view the submitted planning.
==Database Design==


[[File:View_submission_1.png]]
Here we present the diagram of our database design. As the yellow borders show, we only plan to modify the structure of the Question table and the Assignment table.  


==Database Design==
[[File:E2016_database_design.png]]
Here is the diagram of our database design. Our major change is on the ''Assignment'' and ''RevisionPlanQuestionnaire''. In the ''Assignment'', we plan to add one new field is_revision_planning_enabled? to indicate whether we would like to enable this feature or not. Additionally, we plan to add a new table: ''RevisionPlanQuestionnaire'' with ''team_id'' for which team it belongs to and the description that describes the revision plan. In this way, we can achieve the feature without modifying too much of the database.
 
In the Assignment table, the column <code>is_revision_planning_enabled?</code> will be needed to indicate whether the instructor would like to incorporate the revision planning feature.
 
Additionally, we add <code>team_id</code> to each <code>Question</code> object to distinguish where each question belongs to. A <code>Question</code> object with an empty <code>team_id</code> value will be the question under the original rubric, while the object with a non-empty team_id field will be the question created by the team associated with the team_id. That is, instead of creating a whole new <code>RevisionPlanQuestionnaire</code> class, we decided to dump all the revision plan questions that are created in a given round to the rubric that is used for that round. In this way, we minimize the change to the system to make the original rubric questions and the revision planning questions retrieved together more easily.
 
==Code Modifications==
 
===Controllers===
 
'''controllers/questionnaires_controller.rb'''
*action_allowed?: allow students could edit revision questionnaire.
*edit: when params[:team_id] is exist, "edit_revision_plan" will be redirected.
*update: extract the duplicate code as a new function <code>update_questions</code>.
*add_new_questions: add <code>team_id</code> and round number which is set 2 to question, allow student to add new question for revision plan.
*save_all_questions: The duplicate code is extracted as a new function <code>update_questions</code>. The function <code>adjust_question_seq</code> is used to adjust the sequence of the question in the revision plan based on the origin rubric.
*edit_revision_plan: a new method that prepares view template and supplies revision planning questions that belong to the current team
*update_questions: a new private method that could update questions that are already created. This method is extracted from original methods <code>update</code> and <code>save_all_questions</code>.
*adjust_question_seq: a new private method that could move the questions created by students behind the questions in the original rubric.
*question_params: add team_id
'''controllers/questions_controller.rb'''
*action_allowed?: allow students within a team to remove questions in revision plan.
*destroy: add <code>team_id</code>, so that when student with team_id click "remove" buttom, it will redirect to editing revision plan page.
*is_for_revision_planning?: a new private method that know the question is for revision planning or not.
'''controllers/response_controller.rb'''
*update/ new/ create/ set_content: use <code>set_questions</code> to get the questions from the original rubric and from revision plan proposed by the team with the corresponding <code>reviewee_id</code>.
*set_questions: a new private method that gets a set of questions for each team in a given round through function <code>questions</code> in <code>Questionnaire</code> model.
'''controllers/grades_controller.rb'''
*view_my_scores:add <code>@team_id</code> to retrieve_questions method, so students could see the revision plan question in "Your scores"
*view_team: add <code>@team_id</code> to retrieve_questions method, so students could see the revision plan question in "Alternative View". And a new method <code>sort_questions</code> defined in <code>grades_helper</code> is used to sort questions including questions in revision plans.
*calculate_all_penalties: AssignmentParticipant is used to looking for the participant because the participant_id as a argument in function <code>calculate_penalty</code> is expected a AssignmentParticipant.
 
===Models ===
'''models/due_date.rb'''
*self.get_next_due_date: find all next_due_date and return the first one when all dates in ascending order.
'''models/question.rb'''
*Form association relationship with <code>Team</code> via <code>team_id</code>.
**<code>belongs_to :team, class_name: ‘AssignmentTeam’, foreign_key: ‘team_id’</code>
**<code>validates :team_id, numericality: true, allow_nil: true</code>
'''models/questionnaire.rb'''
*questions: a new method could return questions in the original rubric and the team's revision planning questions if the team_id is supplied. The revision planning questions could be get through <code>questions</code> in <code>RevisionPlanQuestionnaire</code> model.
'''models/response.rb'''
*display_as_html: add team_id and round which could be passed to <code>construct_review_response</code>
*construct_review_respone: add team_id and round which could be passed to <code>questions</code> in <code>Questionnaire</code> model, so the revision plan questions will be included in.
'''models/assignment_team.rb'''
*Form aggregation relationship with <code>Question</code> via <code>team_id</code>
**<code>has_many :revision_plan_questions, class_name: ‘Question’, foreign_key: ‘team_id’</code>
'''models/revision_plan_questionnaire.rb'''
*self.questions: a new model file could return the team's revision planning questions
 
===Views===
'''views/questionnaires/_questioinnaire.html.erb'''
* The code is extracted as a partial template named ''views/questionnaires/_questions.html.erb''.
'''views/questionnaires/edit_revision_plan.html.erb'''
*It is a newly-added view file for students to create and edit their revision plan. It use some existing codes from the ''app/views/questionnaires/_questionnaire.html.erb'' view file to reduce code redundancy.
'''views/questionnaires/_questions.html.erb'''
*Extract codes from the ''app/views/questionnaires/_questionnaire.html.erb'' view file to make it a standalone partial template that will later be loaded by the ''app/views/questionnaires/edit_revision_plan.html.erb'' and ''app/views/questionnaires/_questioinnaire.html.erb''.
'''views/student_task/view.html.erb'''
*Add a “Revision Planning” link for students to edit their revision plan. The link will lead students to the “Edit Revision Plan” page. If the revision planning feature is enabled, this link will appear disabled at first and only become clickable during each submission period after round 1.
'''views/assignments/edit/_review_strategy.html.erb'''
*Add “Enable Revision Planning?” checkbox for each assignment because not every assignment needs this feature. TA/instructor have the option to include this feature more flexibly. The “Revision Planning” link will disappear from the assignment page if the checkbox is not checked.
'''views/response/view.html.erb'''
*Pass the <code>@map.reviewee_id</code> and <code>@response.round</code> to function <code>display_as_html</code>.
'''views/response/response.html.erb'''
*The structure is changed. Because the origin structure couldn't show the sequence of questions correctly which means the first question will always be shown in last position.
'''views/grades/_review.html.erb'''
*Pass the <code>participant.team.id</code> and <code>round</code> to function <code>display_as_html</code>.
 
===Helpers===
 
'''helpers/grades_helper.rb'''
*sort_questions: a new method that could sorts questions by sequence number
*view_heatgrid: use <code>Questionnaire</code> model’s <code>questions</code> method with <code>team_id</code> and <code>round</code> to retrieve a proper question set for each team, and then the method <code>sort_question</code> is used to return a sorted questions.
*retrieve_questions: add an extra parameter <code>team_id</code> whose default value is set to be nil. It then invoke <code>Questionnaire</code> model’s <code>questions</code> method with <code>team_id</code> and <code>round</code>, then retrieve a proper question set for each team.
 
===Schema===
 
'''Assignment table''': add one column named <code>is_revision_planning_enabled?</code> to indicate whether this feature has been activated.
 
'''Question table''': add one column named <code>team_id</code> to distinguish whether the question is from the official review rubric or from a particular team's revision plan.
 
=Testing=
 
==RSpec Test Plan==
 
'''Controllers'''
*spec/controllers/questionnaires_controller_spec.rb:
**describe '#edit_revision_plan': function <code>edit_revision_plan</code> could renders the questionnaires#edit_revision_plan page
 
'''Models'''
*spec/models/question_spec.rb
**Describe ‘#Questions’:function <code>Questions</code> will return the questions with or without team_id in ReviewQuestionnaire.
*spec/models/questionnaire_spec.rb
**Describe ‘#questions’: function<code>questions</code> will return the questions from the original rubric if no team_id and it will return the questions with supplied team_id.
 
'''All tests are passed.''' We delete test of ''#display_as_html'' in ''spec/models/response_spec.rb'' because fixing them would cause more UI errors.
 
==UI Testing Instructions (For Reviewers)==
 
'''Setup'''
 
Login information
*Visit xxx [expertiza deployment link]
    User name: instructor6/student8030/student8031
    Password: password
Enable/Disable Revision Planning
*Login as instructor6.
*Go to an assignment’s edit page. Under the “'''Review strategy'''” tab, check the checkbox labeled “'''Enable Revision Planning?'''” to enable the revision planning feature.
Enable/Disable the “Revision Planning” link
*After the instructor configures the assignment to include the revision planning feature, the “'''Revision Planning'''” link will appear on the student's assignment page but will remain disabled and only be enabled during each submission period after round 1. Therefore, to enable the link:
**Login as instructor6.
**Go to an assignment’s edit page. Under the “'''Due dates'''” tab, change the round 2 submission date to whenever date in the future.
*To disable the link after round 2 submission period:
**Login as instructor6.
**Go to an assignment’s edit page. Under the “'''Due dates'''” tab, change the round 2 submission date to whenever date from the past and change the round 2 review date to whenever date in the future.
Create teams for the assignment
*Login as instructor6.
*Go to an assignment’s edit page. Under the “'''Other stuff'''” tab, click “'''Add participant'''”.
*In the new page, add two students, student8030 and student8031, so one student can create revision plan questions during the submission period while the other can respond to these questions during the review period.
*Go back to the assignment’s edit page. Under the “'''Other stuff'''” tab, click “'''Create teams'''”.
*In the new page, group the two added students to separate teams.


[[File:Database_design_1.png]]
'''Functionalities'''


==Design Concerns==
Edit a Revision Plan
'''Not adding ''team_id'' attribute to the Question model'''
*Login as student8030.
* If the ''team_id'' attribute is used to differentiate different team’s revision planning questions, then all the questions can be dumped into one questionnaire, either the original second rubric or we construct a new ''ReviewQuestionnaire'' that holds all the revision planning questions from a revision-planning-enabled assignment. Both solutions would hurt the performance, especially the first one. As more and more outside questions enter the rubric, retrieving the original set of questions becomes harder and harder. Each time it has to traverse through all the irrelevant data before matching the team to its revision plan questions. Our design resolves this issue, making it simpler at least from the coding perspective. (question: will the relational database always have to traverse through each element in the table no matter how the relationship is implemented?)
*In the assignment page, click on the “'''Revision Planning'''” link. which redirects the user to a new page used to create a revision plan. Fill the blanks and click on the '''Save review questionnaire'''” button, and the revision plan should be saved.
* The question model is shared by all other features, not just revision planning. However, not every questionnaire needs the ''team_id'' attribute. This attribute would probably go unused most of the time since the assignment has to enable the revision planning feature to make it useful. Having most Question objects to hold an empty ''team_id'' value occupies a lot of memory space, which we can save by only creating an extra ''RevisionPlanQuestionnaire'' object.  
Test retrieval of revision plan questions for a specific team
* The description of the revision plan has nowhere to save if we don’t have a concrete ''RevisionPlanQuestionnaire'' class. We originally considered storing the description in the ''AssignmentTeam'' class since each assignment team has only one revision plan and each revision plan has only one description, which is a one-to-one relationship. However, if we did so, we would encounter the same problem mentioned in bullet point two that a lot of allocated spaces just go unused. Therefore, we came to the agreement that this may not be a good practice.
*Login as student8030.
* According to the open-closed principle, software entities should be open for extension but closed for modification.
*In the assignment page, click on the '''Revision Planning'''” link after steps under '''Edit a Revision Plan''' have been done. The “'''Revision Planning'''” link should redirect the user to the Revision Planning edit page that is populated with previously saved questions.
Check the Revision Plan questions in the questionnaire.
*Login as instructor6.
*Adjust the time frame to the second round review period.
*Log out and log in again as student8031.
*In the assignment page, click on the “'''Others’ work'''” link, which takes the user to the review page where one requests a new team’s submission to review. Go to the only other team’s review page and check if the questions are properly displayed under the “Revision Planning” subheader.
Check responses to the Revision Plan questions
*Login again as student8030.
*In the assignment page, click on the “'''Alternative View'''” link, and see if student8030 gets responses for both the original rubric questions as well as its revision plan questions.

Latest revision as of 19:58, 4 May 2020

Introduction

About our team

Team members:

  • Tianji Gao (tgao5@ncsu.edu)
  • Guoyi Wang (gwang25@ncsu.edu)
  • Yulin Zhang (yzhan114@ncsu.edu)
  • Boxuan Zhong (bzhong2@ncsu.edu)

Project mentor: Edward Gehringer (efg@ncsu.edu)

What is Revision Planning?

In the first round of the Expertiza reviews, reviewers are asked to give authors some guidance on how to improve their work. Then in the second round, reviewers rate how well authors have followed their suggestions. Revision planning is a mechanism used to carry the interaction one step further by having authors to supply a revision plan based on the previous round reviews. That is, the authors would derive their plan for code improvement from the previous round reviews and reviewers would later assess how well they did it.

Revision planning is helpful because it makes the author think about what's necessary to improve the work before putting forth the effort to improve it. This leads to a more reflective work process and is likely to produce a better-finished product. When reviewers have an opportunity to give feedback to the author, they too will learn what a good revision plan looks like.

According to the given instructions, a revision plan consists of a description of the plan, followed by any number of questions that would later be appended to the future review questionnaire. The revision plan is per AssignmentTeam-based, which means the authors’ questions would only be used to evaluate their submission and not anyone else. By adding the functionality of revision planning, it helps researchers study the effect of the reviewer’s suggestions on the code improvement.

Previous Implementation

This functionality has previously been done by a team of students from the Fall semester of 2018. Their implementation was merged into the master branch but was reverted due to the following design concerns:

  • The relationship between `Questionnaire` and `SubmissionRecord` is unclear.
  • Uses a lot of special-purpose code when existing codes may fulfill the same job.
  • Revision planning cannot be enabled or disabled for an assignment.
  • Numeric labelings for the revision plan questions begin from 1 again, instead of continuing after the original rubric questions.
  • Codebase contains commented codes that are no longer wanted.

Check out the wiki page and the pull request on GitHub if you would like to learn more about the previous implementation of this project.

Please note that unlike the other teams we have reviewed, this project is a complete redo rather than modifications built upon the previous team’s codes because our approach to this problem would be different than theirs. Therefore, we will not mention the previous implementation in the later content.

Problem Statement

For this project, we identified 4 major work items that together fulfill the stated requirements.

Sort out the relationship among classes and introduce the new abstraction of the revision plan to the system in a way that it doesn’t interfere with the majority of codes

We decided to relate each Question object with team_id. A ReviewQuestionnaire may have both questions with no team_id and some questions with a team_id. A question with no team_id indicates that it does not belong to any assignment teams so it is a question set up by the instructor, that will appear on review forms for all submissions. A question with a team_id, in contrast, indicates that it belongs to a particular team so it is a revision plan question. Both types of questions will be saved under the same questionnaire used for a given round. In this way, we can maximize the usage of existing codes and the only major change should be contained within the Question class.

Modify the existing views and controllers to accommodate the new functionality which includes

  • Allowing teaching staff to enable/disable revision planning for an assignment.
  • Allowing team members to create/edit their revision plan during each submission period after the first round.
  • Allowing both rubric questions and revision plan questions to appear on the same page and be serialized correctly.
  • Allowing feedback on the revision plan only to be viewed by the team that creates the plan and that team's instructor.

This will involve some minor changes such as appending an optional trailing parameter to some method signatures, adding interactive elements to the views, and slightly adjusting the structure of certain view templates.

In addition, we planned to:

  • Provide an adequate amount of tests to improve code coverage.
  • Do necessary refactoring and resolve any CodeClimate issues.

After communicated with our mentor Dr. Gehringer, we have been clarified with the following two problem statements.

Every new question must be linked to the second-round questionnaire.

This means both questions from the team’s revision plan and questions from the review rubric should be displayed together in the frontend. Since we decided to add revision plan questions to the review rubric of the round, we automatically linked every new question to the questionnaire of that round.

Every new question must be linked to the author’s submission

By saying every new question must be linked to the author’s submission, it means that there should be some relationships between the team and the team’s revision plan questions presented in the database. We addressed this problem by associating them with a team_id field. See Database Design section for more details. This will probably involve a DB migration to add the team_id field to the questions table.

Design

Control Flow Diagram

The below image shows the control flow of the revision planning functionality.

[Better to test if you are in a resubmission period, since there are method calls to do that. If you tested if Round 1 had ended, you'd have to write queries and then do calculations on the data.]

The below image shows the control flow of the revision planning functionality. It involves 3 types of actors, student (reviewee), student (reviewer) and instructor/TA who manages the assignment and review processes. To understand each actor’s responsibility, trace each colored line that arose from each actor in the direction specified by the arrows. The diamond shape represents a decision or precondition, that is, only after the condition meets can the next action proceeds.

Summary of actions

  • A TA/Instructor can
    1. Enable revision planning
    2. Impersonate students to perform their responsibility
    3. View feedback report of all teams
  • A student (reviewee) can
    1. Make revision during the second round submission period, which includes reading first-round feedback and adding revision plan questions according to that feedback.
    2. View feedback report of the team it belongs to
  • A student (reviewer) can
    1. Give feedback on the team’s revised work by answering each question (including the team's revision plan questions) appeared on the review page.
    2. View the feedback it wrote to the team.

UI Design

A revision plan should be similar to other review questionnaires. Since functionalities on the review questionnaire have been maturely implemented, we expected to make the least amount of interface changes by utilizing the existing view templates whenever possible. The subsections listed the changes we planned to make.

Enabling revision planning

Implementation of enabling/disabling revision planning for each assignment can be rather straightforward. We looked to add an additional checkbox under the "Review strategy" tab of the assignment’s edit page. This checkbox is labeled as "Enable Revision Planning?" to indicate whether the instructor wants to include this functionality in the newly-created assignment. It is most reasonable to place the checkbox here because it is review related and other similar functionalities like Self Reviews are also implemented in this manner.

Link to the revision planning page

If the instructor decided to include revision planning in this assignment, then the link to “Revision Planning” would appear on the student’s assignment page but would stay disabled during the first round. After that, It would become clickable during every submission period and greyed again during every review period. By clicking it, students would be redirected to a whole new page explained under the ‘Revision planning page’ subsection. [Revision planning should surely come before "Your scores".]

Revision planning page

The revision plan is just like other questionnaires in that it contains a set of questions for reviewers to answer. The only difference is that the revision plan comes with an additional description to help reviewers understand what changes have been made so far. Therefore, it should make use of most existing view templates and controller codes with minimized changes. As the image is shown, the only modification made from the existing questionnaire creation template would be to include a link that redirects students to the submission page, where the uploading of the revision plan will be handled by the existing implementation. The advantage to upload an external link rather than typing everything to the textbox element is that the description can be well-formatted if it displays outside the form and not causing a distraction effect for reviewers during the review. We also decided to leave out (or hide) the place where instructors set the configuration stuff like the range of scores and the questionnaire's visibility. These configurations should use default values defined in the system rather than having students come up with their own.

Review page

The format of the review page remains almost exactly the same. To distinguish between rubric questions set up by the instructor and the revision plan questions created by the team under review, all the revision plan questions are placed after the rubric questions, split by an enlarged “Revision planning” subheader. [Having a "Revision planning" header is reasonable, but if the author changes a sequence number to put one of their questions earlier in the rubric, that should be allowed.]

Feedback report

Teaching staff and students have different windows to access the feedback report.

  • Teaching staff: Manage->Assignments->Edit Assignment->Other stuff->View scores
  • Students: Assignments->View Assignment->Alternative View

In addition, either instructor and TA can impersonate students to access the feedback report from their views. We would like to consider both cases and illustrate each of them separately.

TA/Instructor

Scores for the second round review rubric and the author’s revision plan questions will be displayed on the same table and are serialized correctly. See the figure below for an example. Let say the second round rubric has only 5 questions, the remaining questions (6-10) will be revision planning questions written by a particular team.

For reviewer: if you reviewed our first draft design, you should notice that we originally chose to place revision plan scores on a distinct table. After our mentor clarified to us, we realized that there can be possibly more than 2 rounds of submission and review periods for a given assignment. Therefore, scores for revision planning questions can vary round by round. Therefore, our previous solution will not work since it confuses the user of which round the revision plan scores refer to. [It would be nice if there was a way to distinguish instructor questions from questions added by students. This might be done with a special symbol, or by shading the background of the cell with the question number, etc.]

Student

The revision planning section will be added to the students’ view as shown in the snapshot below. It displays in the same order as how the review page does. A “Revision Planning” subheader is also used here to indicate the starting of the revision planning section.

Database Design

Here we present the diagram of our database design. As the yellow borders show, we only plan to modify the structure of the Question table and the Assignment table.

In the Assignment table, the column is_revision_planning_enabled? will be needed to indicate whether the instructor would like to incorporate the revision planning feature.

Additionally, we add team_id to each Question object to distinguish where each question belongs to. A Question object with an empty team_id value will be the question under the original rubric, while the object with a non-empty team_id field will be the question created by the team associated with the team_id. That is, instead of creating a whole new RevisionPlanQuestionnaire class, we decided to dump all the revision plan questions that are created in a given round to the rubric that is used for that round. In this way, we minimize the change to the system to make the original rubric questions and the revision planning questions retrieved together more easily.

Code Modifications

Controllers

controllers/questionnaires_controller.rb

  • action_allowed?: allow students could edit revision questionnaire.
  • edit: when params[:team_id] is exist, "edit_revision_plan" will be redirected.
  • update: extract the duplicate code as a new function update_questions.
  • add_new_questions: add team_id and round number which is set 2 to question, allow student to add new question for revision plan.
  • save_all_questions: The duplicate code is extracted as a new function update_questions. The function adjust_question_seq is used to adjust the sequence of the question in the revision plan based on the origin rubric.
  • edit_revision_plan: a new method that prepares view template and supplies revision planning questions that belong to the current team
  • update_questions: a new private method that could update questions that are already created. This method is extracted from original methods update and save_all_questions.
  • adjust_question_seq: a new private method that could move the questions created by students behind the questions in the original rubric.
  • question_params: add team_id

controllers/questions_controller.rb

  • action_allowed?: allow students within a team to remove questions in revision plan.
  • destroy: add team_id, so that when student with team_id click "remove" buttom, it will redirect to editing revision plan page.
  • is_for_revision_planning?: a new private method that know the question is for revision planning or not.

controllers/response_controller.rb

  • update/ new/ create/ set_content: use set_questions to get the questions from the original rubric and from revision plan proposed by the team with the corresponding reviewee_id.
  • set_questions: a new private method that gets a set of questions for each team in a given round through function questions in Questionnaire model.

controllers/grades_controller.rb

  • view_my_scores:add @team_id to retrieve_questions method, so students could see the revision plan question in "Your scores"
  • view_team: add @team_id to retrieve_questions method, so students could see the revision plan question in "Alternative View". And a new method sort_questions defined in grades_helper is used to sort questions including questions in revision plans.
  • calculate_all_penalties: AssignmentParticipant is used to looking for the participant because the participant_id as a argument in function calculate_penalty is expected a AssignmentParticipant.

Models

models/due_date.rb

  • self.get_next_due_date: find all next_due_date and return the first one when all dates in ascending order.

models/question.rb

  • Form association relationship with Team via team_id.
    • belongs_to :team, class_name: ‘AssignmentTeam’, foreign_key: ‘team_id’
    • validates :team_id, numericality: true, allow_nil: true

models/questionnaire.rb

  • questions: a new method could return questions in the original rubric and the team's revision planning questions if the team_id is supplied. The revision planning questions could be get through questions in RevisionPlanQuestionnaire model.

models/response.rb

  • display_as_html: add team_id and round which could be passed to construct_review_response
  • construct_review_respone: add team_id and round which could be passed to questions in Questionnaire model, so the revision plan questions will be included in.

models/assignment_team.rb

  • Form aggregation relationship with Question via team_id
    • has_many :revision_plan_questions, class_name: ‘Question’, foreign_key: ‘team_id’

models/revision_plan_questionnaire.rb

  • self.questions: a new model file could return the team's revision planning questions

Views

views/questionnaires/_questioinnaire.html.erb

  • The code is extracted as a partial template named views/questionnaires/_questions.html.erb.

views/questionnaires/edit_revision_plan.html.erb

  • It is a newly-added view file for students to create and edit their revision plan. It use some existing codes from the app/views/questionnaires/_questionnaire.html.erb view file to reduce code redundancy.

views/questionnaires/_questions.html.erb

  • Extract codes from the app/views/questionnaires/_questionnaire.html.erb view file to make it a standalone partial template that will later be loaded by the app/views/questionnaires/edit_revision_plan.html.erb and app/views/questionnaires/_questioinnaire.html.erb.

views/student_task/view.html.erb

  • Add a “Revision Planning” link for students to edit their revision plan. The link will lead students to the “Edit Revision Plan” page. If the revision planning feature is enabled, this link will appear disabled at first and only become clickable during each submission period after round 1.

views/assignments/edit/_review_strategy.html.erb

  • Add “Enable Revision Planning?” checkbox for each assignment because not every assignment needs this feature. TA/instructor have the option to include this feature more flexibly. The “Revision Planning” link will disappear from the assignment page if the checkbox is not checked.

views/response/view.html.erb

  • Pass the @map.reviewee_id and @response.round to function display_as_html.

views/response/response.html.erb

  • The structure is changed. Because the origin structure couldn't show the sequence of questions correctly which means the first question will always be shown in last position.

views/grades/_review.html.erb

  • Pass the participant.team.id and round to function display_as_html.

Helpers

helpers/grades_helper.rb

  • sort_questions: a new method that could sorts questions by sequence number
  • view_heatgrid: use Questionnaire model’s questions method with team_id and round to retrieve a proper question set for each team, and then the method sort_question is used to return a sorted questions.
  • retrieve_questions: add an extra parameter team_id whose default value is set to be nil. It then invoke Questionnaire model’s questions method with team_id and round, then retrieve a proper question set for each team.

Schema

Assignment table: add one column named is_revision_planning_enabled? to indicate whether this feature has been activated.

Question table: add one column named team_id to distinguish whether the question is from the official review rubric or from a particular team's revision plan.

Testing

RSpec Test Plan

Controllers

  • spec/controllers/questionnaires_controller_spec.rb:
    • describe '#edit_revision_plan': function edit_revision_plan could renders the questionnaires#edit_revision_plan page

Models

  • spec/models/question_spec.rb
    • Describe ‘#Questions’:function Questions will return the questions with or without team_id in ReviewQuestionnaire.
  • spec/models/questionnaire_spec.rb
    • Describe ‘#questions’: functionquestions will return the questions from the original rubric if no team_id and it will return the questions with supplied team_id.

All tests are passed. We delete test of #display_as_html in spec/models/response_spec.rb because fixing them would cause more UI errors.

UI Testing Instructions (For Reviewers)

Setup

Login information

  • Visit xxx [expertiza deployment link]
   User name: instructor6/student8030/student8031
   Password: password

Enable/Disable Revision Planning

  • Login as instructor6.
  • Go to an assignment’s edit page. Under the “Review strategy” tab, check the checkbox labeled “Enable Revision Planning?” to enable the revision planning feature.

Enable/Disable the “Revision Planning” link

  • After the instructor configures the assignment to include the revision planning feature, the “Revision Planning” link will appear on the student's assignment page but will remain disabled and only be enabled during each submission period after round 1. Therefore, to enable the link:
    • Login as instructor6.
    • Go to an assignment’s edit page. Under the “Due dates” tab, change the round 2 submission date to whenever date in the future.
  • To disable the link after round 2 submission period:
    • Login as instructor6.
    • Go to an assignment’s edit page. Under the “Due dates” tab, change the round 2 submission date to whenever date from the past and change the round 2 review date to whenever date in the future.

Create teams for the assignment

  • Login as instructor6.
  • Go to an assignment’s edit page. Under the “Other stuff” tab, click “Add participant”.
  • In the new page, add two students, student8030 and student8031, so one student can create revision plan questions during the submission period while the other can respond to these questions during the review period.
  • Go back to the assignment’s edit page. Under the “Other stuff” tab, click “Create teams”.
  • In the new page, group the two added students to separate teams.

Functionalities

Edit a Revision Plan

  • Login as student8030.
  • In the assignment page, click on the “Revision Planning” link. which redirects the user to a new page used to create a revision plan. Fill the blanks and click on the “Save review questionnaire” button, and the revision plan should be saved.

Test retrieval of revision plan questions for a specific team

  • Login as student8030.
  • In the assignment page, click on the “Revision Planning” link after steps under Edit a Revision Plan have been done. The “Revision Planning” link should redirect the user to the Revision Planning edit page that is populated with previously saved questions.

Check the Revision Plan questions in the questionnaire.

  • Login as instructor6.
  • Adjust the time frame to the second round review period.
  • Log out and log in again as student8031.
  • In the assignment page, click on the “Others’ work” link, which takes the user to the review page where one requests a new team’s submission to review. Go to the only other team’s review page and check if the questions are properly displayed under the “Revision Planning” subheader.

Check responses to the Revision Plan questions

  • Login again as student8030.
  • In the assignment page, click on the “Alternative View” link, and see if student8030 gets responses for both the original rubric questions as well as its revision plan questions.