CSC/ECE 517 Spring 2019 E1926 Improve self-review: Difference between revisions

From Expertiza_Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(25 intermediate revisions by 2 users not shown)
Line 11: Line 11:
[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. The Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.
[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. The Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.


===Team and Resources===
* Team Members: Shraddha Bhadauria, Akhil Pabba, Rahul Sethi
* Pull Request: https://github.com/expertiza/expertiza/pull/1462
* Screencast Video: https://drive.google.com/file/d/1wnYxQxETkZM0yU83M80QYGNZkJ7lsIEq/view?usp=sharing
* Code Flow Presentation: https://drive.google.com/file/d/1XE7OK1Vh67WFe0SDAsp3ADJhPA_eu3bA/view?usp=sharing


===Problem Statement and Overview===
===Problem Statement and Overview===
Line 16: Line 23:


The Expertiza provides an option for students to self-review their submissions. In which, each member of the team gets to evaluate their project on the same criteria as the peer review. However, these scores contribute in no way to calculate the overall score for their assignment. So, we focus on improving the self-review system and to extend its functionality. Some of the objectives we try to achieve for this project are as follows –
The Expertiza provides an option for students to self-review their submissions. In which, each member of the team gets to evaluate their project on the same criteria as the peer review. However, these scores contribute in no way to calculate the overall score for their assignment. So, we focus on improving the self-review system and to extend its functionality. Some of the objectives we try to achieve for this project are as follows –
     1) Provide a formula that takes both the peer review score and self-review score into account for calculating the composite score.
      
    2) The composite score should get higher as the self-review score gets closer to the peer review score.
*Provide a formula that takes both the peer review score and self-review score into account for calculating the composite score.
    3) Make sure that the peer review scores are not visible before the self-review submission.
*The composite score should get higher as the self-review score gets closer to the peer review score.
    4) Display the composite score on the “View Scores” page.
*Make sure that the peer review scores are not visible before the self-review submission.
    5) Display the self-review scores in the "View Scores" and heat map of peer-reviews, showing that they are a different type of review.
*Display the composite score on the “View Scores” page.
*Display the self-review scores in the "View Scores" and heat map of peer-reviews, showing that they are a different type of review.


The need for the self-review section is to know how well the students can judge their work and understand how to evaluate their flaws and rectify them. Also, if they can score themselves close to their peers, then it means that they have good self-assessment skills and they are being honest about what they think of their work.  
The need for the self-review section is to know how well the students can judge their work and understand how to evaluate their flaws and rectify them. Also, if they can score themselves close to their peers, then it means that they have good self-assessment skills and they are being honest about what they think of their work.  




===Current Process===
===Code Changes Flow===
 
 
Currently, Expertiza allows students to submit an assignment and provides a link for self-review. The self-reviewing page looks like the picture below-


  INSERT PICTURE


Once the self-reviewing is done, the submitted score gets stored in the database but is not used in calculating the overall score of the assignment for the student. Because of this, students can score themselves higher than what they should get. This helps students in no way. So, to make productive use of this score and to help students learn self-evaluating, we have proposed to make some changes in the review score calculation which is explained in the below section.  
[[File:E1926_code_flow.png]]




===Proposed Process===
===Previous Process===




The new formula will be devised on the basis of which a new composite grade will be derived. After we made the changes needed, the students will be able to see their self-review score, peer review score and the composite score displayed on the “View scores” page as shown below –
Previously, Expertiza allowed students to submit an assignment and provides a link for self-review. Once the self-reviewing was done, the submitted score gets stored in the database but was not used in calculating the overall score of the assignment for the student. Because of this, students can score themselves higher than what they should get. This helps students in no way. So, to make productive use of this score and to help students learn self-evaluating, we have made some changes in the review score calculation which is explained in the section below.


  INSERT PICTURE
===New Process===


As you can see from the picture above, we have a new column for the self-review score along with the peer review score and the calculated composite score.
We derived a new formula that takes into account the self-review score and calculates a composite score, which will be the student's final score. After we made the changes needed, now the students are able to see their peer review score, Final score and the Final derived peer review score displayed on the “View scores” page.
In the same way, we made some changes to the instructor’s page as well because the instructor should be able to see the self-reviewing score from each member of the team, displayed along with the peer review score for that assignment.  
[[File:E1926_scr_1.jpg]]


The figure below shows the changes –
These scores will also be reflected in the "stats" as shown below


  INSERT PICTURE
[[File:E1926_scr_2.jpg]]
  Add some lines describing image.


In the same way, we will make some changes to the instructor’s page as well because the instructor should be able to see the self-reviewing score from each member of the team, displayed along with the peer review score for that assignment.


===Application Flow===
===Application Flow===
Line 61: Line 66:


*In the view scores page, a new column showing the self-review scores will be displayed.
*In the view scores page, a new column showing the self-review scores will be displayed.


===Use case===
===Use case===
Line 75: Line 79:
*Actions
*Actions
   1. Instructor: Create Assignment, Enable Self-Review.
   1. Instructor: Create Assignment, Enable Self-Review.
   2. Student:   Login, Add work and Submit, Submit Self-Review, View Scores.
   2. Student: Login, Add work and Submit, Submit Self-Review, View Scores.
 


===Formula Proposed===
===Formula Proposed===




We have to implement a way to combine self-review and peer-review scores to derive a composite score. We have to come up with a formula to derive the composite score as below.
We have to implement a way to combine self-review and peer-review scores to derive a composite score. We have to come up with a formula to derive the composite score as below with the help of research paper: https://www.ojs.unisa.edu.au/index.php/atna/article/viewFile/343/258.
    
    
   deviated_score_scale = (100 - abs(self_review_score - mean(all_peer_review_scores)))/100;
   SAPA_Factor = sqrt(Self-ratings for individual team member / Average of ratings for the individual by peer team members)
   derived_score =  deviated_score_scale * peer review score;
   derived_score =  SAPA Factor * Average of ratings for the individual by peer team members;
 
The proposed formula first calculate by how much scale the self-review has deviated from the average peer-review score, the participant got.
 
Then the derived score can be calculated by multiplying that scale with the actual peer-review score the participant got.
 


===Test Plan===
===Test Plan===




*Set up expertiza locally on your system. Expertiza repository on Github: https://github.com/expertiza/expertiza
*Now that the set up is done, to test this, you create 6 students and 2 assignments.
*Create yourself a Github account and fork the Expertiza repo. Fork the beta branch.
*One assignment has self-review enabled and the other assignment has self-review disabled.
*Further steps to be followed from the below link. https://docs.google.com/document/d/1Ozw2Bj2u_LyeKUdDqAyQtz1stEIKrxkmRnwmNyUgmzw/edit#
*Then student should submit the assignments and the self-review should be done for the required assignment.
*Once set up is done, figure out the flow of code and files which would be modified to implement the proposed solution.
*Try giving same peer-review scores for both assignments and you should see the final score differs accordingly with the self-review score for one assignment but stays the same for the other.
*We shall Design a way to show them in the regular “View Scores” page and the alternate (heat-map) view.
*You should be able to see the final scores for any student matching to the proposed formula.
*Then implement a way to combine self-review and peer-review scores to derive a composite score
*Implement the functionality in a way that the author(s) should be required to submit their self-evaluation(s) before seeing the results of their peer evaluations.
*Perform testing using “Cucumber” and “rSpecs”
*Perform testing using “Cucumber” and “rSpecs”
*We shall be using “Cucumber” to perform acceptance testing.
*Cucumber is a tool based on Behavior Driven Development framework and is used to write acceptance tests for web application. It lets us automate functional testing in an easily readable and understandable format for Developers, Testers, etc.
*We shall be using “Rspecs” as well along with “Cucumber”.  The approach being followed in rspecs is to test applications is to test behavior rather than only specific methods. RSpec is composed of multiple libraries, which can be used independently with other testing tools like Cucumber.


===Files identified to be modified===
===Files modified===




*app/controllers/grades_controller.rb
*app/controllers/grades_controller.rb
*app/models/assignment_participant.rb
*app/models/author_feedback_questionnaire.rb
*app/models/author_feedback_questionnaire.rb
*app/models/review_questionnaire.rb
*app/models/response_map.rb
*app/models/vm_question_response.rb
*app/models/vm_question_response.rb
*app/models/review_questionnaire.rb
*app/views/grades/_participant.html.erb
*app/models/teammate_review_questionnaire.rb
*app/views/grades/_participant_charts.html.erb
*app/views/grades/_participant_title.html.erb
*app/views/grades/view_team.html.erb
*app/views/reports/_self_review_report.html.erb
*spec/controllers/grades_controller_spec.rb
*spec/controllers/grades_controller_spec.rb
*spec/models/assignment_particpant_spec.rb
*spec/models/assignment_particpant_spec.rb
*app/models/assignment_participant.rb
*app/models/vm_question_response_row.rb


===References===
===References===

Latest revision as of 15:23, 6 May 2019

E1926. Improve self-review, Link peer review & self-review to derive grades

This page provides a description of the Expertiza based Semester project.


About Expertiza

Expertiza is an open source project based on Ruby on Rails framework. The Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.


Team and Resources

Problem Statement and Overview

The Expertiza provides an option for students to self-review their submissions. In which, each member of the team gets to evaluate their project on the same criteria as the peer review. However, these scores contribute in no way to calculate the overall score for their assignment. So, we focus on improving the self-review system and to extend its functionality. Some of the objectives we try to achieve for this project are as follows –

  • Provide a formula that takes both the peer review score and self-review score into account for calculating the composite score.
  • The composite score should get higher as the self-review score gets closer to the peer review score.
  • Make sure that the peer review scores are not visible before the self-review submission.
  • Display the composite score on the “View Scores” page.
  • Display the self-review scores in the "View Scores" and heat map of peer-reviews, showing that they are a different type of review.

The need for the self-review section is to know how well the students can judge their work and understand how to evaluate their flaws and rectify them. Also, if they can score themselves close to their peers, then it means that they have good self-assessment skills and they are being honest about what they think of their work.


Code Changes Flow


Previous Process

Previously, Expertiza allowed students to submit an assignment and provides a link for self-review. Once the self-reviewing was done, the submitted score gets stored in the database but was not used in calculating the overall score of the assignment for the student. Because of this, students can score themselves higher than what they should get. This helps students in no way. So, to make productive use of this score and to help students learn self-evaluating, we have made some changes in the review score calculation which is explained in the section below.

New Process

We derived a new formula that takes into account the self-review score and calculates a composite score, which will be the student's final score. After we made the changes needed, now the students are able to see their peer review score, Final score and the Final derived peer review score displayed on the “View scores” page.

These scores will also be reflected in the "stats" as shown below

In the same way, we will make some changes to the instructor’s page as well because the instructor should be able to see the self-reviewing score from each member of the team, displayed along with the peer review score for that assignment.

Application Flow

  • The above image shows how the instructor will allow users to give a self-review on the work they have done. The instructor has to go to the review strategy tab while assignment creation to enable the self-review option.
  • User then can see the self-review option on the assignment to do tasks to give a self-review.
  • In the view scores page, a new column showing the self-review scores will be displayed.

Use case

  • Actors:
  1. Instructor: This actor is responsible for creating assignments and adding students to the assignment. 
  2. Student:    This actor is responsible for submitting, self-reviewing and viewing the scores.
  • Actions
  1. Instructor: Create Assignment, Enable Self-Review.
  2. Student: Login, Add work and Submit, Submit Self-Review, View Scores.

Formula Proposed

We have to implement a way to combine self-review and peer-review scores to derive a composite score. We have to come up with a formula to derive the composite score as below with the help of research paper: https://www.ojs.unisa.edu.au/index.php/atna/article/viewFile/343/258. 

  SAPA_Factor = sqrt(Self-ratings for individual team member / Average of ratings for the individual by peer team members)
  derived_score =  SAPA Factor * Average of ratings for the individual by peer team members;

Test Plan

  • Now that the set up is done, to test this, you create 6 students and 2 assignments.
  • One assignment has self-review enabled and the other assignment has self-review disabled.
  • Then student should submit the assignments and the self-review should be done for the required assignment.
  • Try giving same peer-review scores for both assignments and you should see the final score differs accordingly with the self-review score for one assignment but stays the same for the other.
  • You should be able to see the final scores for any student matching to the proposed formula.
  • Perform testing using “Cucumber” and “rSpecs”

Files modified

  • app/controllers/grades_controller.rb
  • app/models/assignment_participant.rb
  • app/models/author_feedback_questionnaire.rb
  • app/models/review_questionnaire.rb
  • app/models/response_map.rb
  • app/models/vm_question_response.rb
  • app/views/grades/_participant.html.erb
  • app/views/grades/_participant_charts.html.erb
  • app/views/grades/_participant_title.html.erb
  • app/views/grades/view_team.html.erb
  • app/views/reports/_self_review_report.html.erb
  • spec/controllers/grades_controller_spec.rb
  • spec/models/assignment_particpant_spec.rb

References

Retrieved from "https://wiki.expertiza.ncsu.edu/index.php?title=CSC/ECE_517_Spring_2019_E1926_Improve_self-review&oldid=124879"