CSC/ECE 517 Spring 2019 E1926 Improve self-review: Difference between revisions

From Expertiza_Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(31 intermediate revisions by 2 users not shown)
Line 4: Line 4:


__TOC__
__TOC__


===About Expertiza===
===About Expertiza===


[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.


===Application Flow===
[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. The Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.
 


[[File:E1799_design_1.jpg]]
===Team and Resources===


The instructor has to login and create an assignment by checking allow self-review option. The instructor has to add the topic and assign students to the topic. Now, the student has to login and submit the assignment. The self-review link is visible in the page. The student should give the self-review. After giving the self-review, the student should be able to see the peer review, self-review and the composite score.\
* Team Members: Shraddha Bhadauria, Akhil Pabba, Rahul Sethi
* Pull Request: https://github.com/expertiza/expertiza/pull/1462
* Screencast Video: https://drive.google.com/file/d/1wnYxQxETkZM0yU83M80QYGNZkJ7lsIEq/view?usp=sharing
* Code Flow Presentation: https://drive.google.com/file/d/1XE7OK1Vh67WFe0SDAsp3ADJhPA_eu3bA/view?usp=sharing


===Use case===
===Problem Statement and Overview===
[[File:E1799_design_2.jpg]]


Actors:


    Instructor: This actor is responsible for creating assignments and adding students to the assignment.  
The Expertiza provides an option for students to self-review their submissions. In which, each member of the team gets to evaluate their project on the same criteria as the peer review. However, these scores contribute in no way to calculate the overall score for their assignment. So, we focus on improving the self-review system and to extend its functionality. Some of the objectives we try to achieve for this project are as follows –
     Student: This actor is responsible for submitting, self-reviewing and viewing the scores.  
      
*Provide a formula that takes both the peer review score and self-review score into account for calculating the composite score.
*The composite score should get higher as the self-review score gets closer to the peer review score.
*Make sure that the peer review scores are not visible before the self-review submission.
*Display the composite score on the “View Scores” page.
*Display the self-review scores in the "View Scores" and heat map of peer-reviews, showing that they are a different type of review.


All the other use cases are implemented except “View Scores with self-review column and composite score”
The need for the self-review section is to know how well the students can judge their work and understand how to evaluate their flaws and rectify them. Also, if they can score themselves close to their peers, then it means that they have good self-assessment skills and they are being honest about what they think of their work.


'''Use Case: View score with self-review column and composite score'''


Preconditions:
===Code Changes Flow===


  1.The Instructor creates Assignments and checks the Allow Self Review check box in the Assignment
  2.The Student should submit the assignment and self-review.
  3.The other students should submit the reviews of the work submitted.


Student Sequence:
[[File:E1926_code_flow.png]]


  The student should login.
  The student should browse and upload the assignment.
  The student should submit the assignment.
  The student should submit the self-review.
  The student should choose your scores to view the score.
  The student should be able to see the peer-review score with the self-review scores and composite score.


Post Conditions:
===Previous Process===


  The self-review and composite score will be visible on the page.


===Formula Proposed===
Previously, Expertiza allowed students to submit an assignment and provides a link for self-review. Once the self-reviewing was done, the submitted score gets stored in the database but was not used in calculating the overall score of the assignment for the student. Because of this, students can score themselves higher than what they should get. This helps students in no way. So, to make productive use of this score and to help students learn self-evaluating, we have made some changes in the review score calculation which is explained in the section below.
The composite score is calculated using both self-review score and peer-review score (both are graded over 5 points for each criteria).  
The composite score is calculated as follows:


  composite score = [10 - abs(self-review score - peer review score)] * peer review score / 10
===New Process===


By using this formula for calculating composite score we are discouraging students from either exaggerating or underrating their performance. The composite score model assigns more weight to the peer review score since the peer review score being used is the average of all peer review scores assigned and hence tends to be more appropriate than a single self-review.
We derived a new formula that takes into account the self-review score and calculates a composite score, which will be the student's final score. After we made the changes needed, now the students are able to see their peer review score, Final score and the Final derived peer review score displayed on the “View scores” page.
[[File:E1926_scr_1.jpg]]


Also this composite score formula gives the score on a scale which is identical to the scale of the self and peer review score. Thus this formula is not dependent on the scale of the self-review and peer-review score as it will also return the composite score on the same scale.
These scores will also be reflected in the "stats" as shown below


For example: If someone gets an average peer review score of 3.5/5 and he gives 3.5/5  (self-review score) while reviewing his project, then he will get a composite score of 3.5/5. And instead if the student reviews his project for 2/5 then the student will get a composite score of 2.975/5.
[[File:E1926_scr_2.jpg]]


===Implementation===
In the same way, we will make some changes to the instructor’s page as well because the instructor should be able to see the self-reviewing score from each member of the team, displayed along with the peer review score for that assignment.


As stated above the ability to do self-reviews has already been implemented. This can be seen from the image shown below. The instructor has this ability to enable self-reviews
===Application Flow===
.


[[File:Oodd abhisu.PNG]]


[[File:E1926_Flow_1.jpg]] [[File:E1926_Flow_2.jpg]]


In our project, we were successful in displaying both the self-review score and composite review score on the view scores page. Shown below is an example of the output in the student profile. As we can see the self-review icon has a blue background to indicate that it is different from other peer reviews.
*The above image shows how the instructor will allow users to give a self-review on the work they have done. The instructor has to go to the review strategy tab while assignment creation to enable the self-review option.


*User then can see the self-review option on the assignment to do tasks to give a self-review.


[[File:student view.PNG]]
*In the view scores page, a new column showing the self-review scores will be displayed.


===Use case===




The alternate view was also modified to accommodate a pictorial representation of the self-review scores. The modified alternate view scores page is shown below.
[[File:E1926_use_case.jpg]] [[File:E1926_use_case_2.jpg]]


*Actors:


[[File:alternate view.PNG]]
  1. Instructor: This actor is responsible for creating assignments and adding students to the assignment.
  2. Student:   This actor is responsible for submitting, self-reviewing and viewing the scores.  


*Actions
  1. Instructor: Create Assignment, Enable Self-Review.
  2. Student: Login, Add work and Submit, Submit Self-Review, View Scores.


The same was achieved in the instructor profile too. In the current implementation of expertiza when self-review was not assigned, the instructor would view the scores of the first student in each team (it wouldn't matter much because without self-review all students in a team would have the same scores). But this would be a problem when self-review was enabled so we decided to display all the self-review scores of all students in the team for instructor view. As it can be seen below self-reviews of all students in the team (distinguished by their usernames) are indicated by blue background to differentiate them from peer review scores.
===Formula Proposed===




[[File:Instructor view.PNG]]
We have to implement a way to combine self-review and peer-review scores to derive a composite score. We have to come up with a formula to derive the composite score as below with the help of research paper: https://www.ojs.unisa.edu.au/index.php/atna/article/viewFile/343/258.
 
  SAPA_Factor = sqrt(Self-ratings for individual team member / Average of ratings for the individual by peer team members)
  derived_score =  SAPA Factor * Average of ratings for the individual by peer team members;


===Test Plan===
===Test Plan===
Test cases are as follows:
 
* Peer-reviews cannot be viewed before a self-review is completed.
 
This ensures that the student is not able to see his peer-review score before writing his self-review as the composite score takes into account the difference between the student peer-review and self-review score.
*Now that the set up is done, to test this, you create 6 students and 2 assignments.
* Self-review scores are displayed with peer-review scores.
*One assignment has self-review enabled and the other assignment has self-review disabled.
This enables the student and instructor to view all his scores in one place.
*Then student should submit the assignments and the self-review should be done for the required assignment.
* The composite score must be properly calculated.
*Try giving same peer-review scores for both assignments and you should see the final score differs accordingly with the self-review score for one assignment but stays the same for the other.
The composite score must be calculated so as to give students more points if their self-review score is closer to the peer review score.
*You should be able to see the final scores for any student matching to the proposed formula.
* Composite score must be visible in both view scores page and Alternate view.
*Perform testing using “Cucumber” and “rSpecs”
The scores must be displayed in both the views.
 
===Files modified===
 
 
*app/controllers/grades_controller.rb
*app/models/assignment_participant.rb
*app/models/author_feedback_questionnaire.rb
*app/models/review_questionnaire.rb
*app/models/response_map.rb
*app/models/vm_question_response.rb
*app/views/grades/_participant.html.erb
*app/views/grades/_participant_charts.html.erb
*app/views/grades/_participant_title.html.erb
*app/views/grades/view_team.html.erb
*app/views/reports/_self_review_report.html.erb
*spec/controllers/grades_controller_spec.rb
*spec/models/assignment_particpant_spec.rb
 
===References===
 
 
*https://github.com/expertiza/expertiza
*https://ruby-doc.org
*https://www.softwaretestinghelp.com/cucumber-bdd-tool-selenium-tutorial-30/
*https://www.tutorialspoint.com/cucumber/cucumber_overview.htm
*https://www.toolsqa.com/cucumber-tutorial/
*http://rspec.info
*https://github.com/rspec/rspec-rails
*https://www.tutorialspoint.com/rspec/
*https://blog.teamtreehouse.com/an-introduction-to-rspec
*https://semaphoreci.com/community/tutorials/getting-started-with-rspec

Latest revision as of 15:23, 6 May 2019

E1926. Improve self-review, Link peer review & self-review to derive grades

This page provides a description of the Expertiza based Semester project.


About Expertiza

Expertiza is an open source project based on Ruby on Rails framework. The Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.


Team and Resources

Problem Statement and Overview

The Expertiza provides an option for students to self-review their submissions. In which, each member of the team gets to evaluate their project on the same criteria as the peer review. However, these scores contribute in no way to calculate the overall score for their assignment. So, we focus on improving the self-review system and to extend its functionality. Some of the objectives we try to achieve for this project are as follows –

  • Provide a formula that takes both the peer review score and self-review score into account for calculating the composite score.
  • The composite score should get higher as the self-review score gets closer to the peer review score.
  • Make sure that the peer review scores are not visible before the self-review submission.
  • Display the composite score on the “View Scores” page.
  • Display the self-review scores in the "View Scores" and heat map of peer-reviews, showing that they are a different type of review.

The need for the self-review section is to know how well the students can judge their work and understand how to evaluate their flaws and rectify them. Also, if they can score themselves close to their peers, then it means that they have good self-assessment skills and they are being honest about what they think of their work.


Code Changes Flow


Previous Process

Previously, Expertiza allowed students to submit an assignment and provides a link for self-review. Once the self-reviewing was done, the submitted score gets stored in the database but was not used in calculating the overall score of the assignment for the student. Because of this, students can score themselves higher than what they should get. This helps students in no way. So, to make productive use of this score and to help students learn self-evaluating, we have made some changes in the review score calculation which is explained in the section below.

New Process

We derived a new formula that takes into account the self-review score and calculates a composite score, which will be the student's final score. After we made the changes needed, now the students are able to see their peer review score, Final score and the Final derived peer review score displayed on the “View scores” page.

These scores will also be reflected in the "stats" as shown below

In the same way, we will make some changes to the instructor’s page as well because the instructor should be able to see the self-reviewing score from each member of the team, displayed along with the peer review score for that assignment.

Application Flow

  • The above image shows how the instructor will allow users to give a self-review on the work they have done. The instructor has to go to the review strategy tab while assignment creation to enable the self-review option.
  • User then can see the self-review option on the assignment to do tasks to give a self-review.
  • In the view scores page, a new column showing the self-review scores will be displayed.

Use case

  • Actors:
  1. Instructor: This actor is responsible for creating assignments and adding students to the assignment. 
  2. Student:    This actor is responsible for submitting, self-reviewing and viewing the scores.
  • Actions
  1. Instructor: Create Assignment, Enable Self-Review.
  2. Student: Login, Add work and Submit, Submit Self-Review, View Scores.

Formula Proposed

We have to implement a way to combine self-review and peer-review scores to derive a composite score. We have to come up with a formula to derive the composite score as below with the help of research paper: https://www.ojs.unisa.edu.au/index.php/atna/article/viewFile/343/258. 

  SAPA_Factor = sqrt(Self-ratings for individual team member / Average of ratings for the individual by peer team members)
  derived_score =  SAPA Factor * Average of ratings for the individual by peer team members;

Test Plan

  • Now that the set up is done, to test this, you create 6 students and 2 assignments.
  • One assignment has self-review enabled and the other assignment has self-review disabled.
  • Then student should submit the assignments and the self-review should be done for the required assignment.
  • Try giving same peer-review scores for both assignments and you should see the final score differs accordingly with the self-review score for one assignment but stays the same for the other.
  • You should be able to see the final scores for any student matching to the proposed formula.
  • Perform testing using “Cucumber” and “rSpecs”

Files modified

  • app/controllers/grades_controller.rb
  • app/models/assignment_participant.rb
  • app/models/author_feedback_questionnaire.rb
  • app/models/review_questionnaire.rb
  • app/models/response_map.rb
  • app/models/vm_question_response.rb
  • app/views/grades/_participant.html.erb
  • app/views/grades/_participant_charts.html.erb
  • app/views/grades/_participant_title.html.erb
  • app/views/grades/view_team.html.erb
  • app/views/reports/_self_review_report.html.erb
  • spec/controllers/grades_controller_spec.rb
  • spec/models/assignment_particpant_spec.rb

References

Retrieved from "https://wiki.expertiza.ncsu.edu/index.php?title=CSC/ECE_517_Spring_2019_E1926_Improve_self-review&oldid=124879"