CSC/ECE 517 Fall 2019 - E1992. Add cake type to rubrics design

Introduction:

Expertiza is an open source application running on Ruby on Rails. It is used for management of courses and the assignments for respective courses, by the faculties and the students. Students can form teams in expertiza to work on an assignment in a group. A student team can submit their work through multiple means such as file uploads and embedded links. Expertiza also consists of questionnaires which can be leveraged for many tasks one of them being peer evaluation of submissions. Students can also review their teammates, making use of the teammate review questionnaire, based on their contribution to the projects.

Problem Definition:

Expertiza rubrics are utilized to build questionnaires and these rubrics incorporate several kinds of items, including Criterion (dropdown + comment), Checkbox, MultipleChoice, and Scale. When we use these questionnaires for reviews, for example the teammate review assessment we encounter a few problems. One “problem” with all of these types is that there is nothing to stop a reviewer (say some student) from assigning the maximum score to all the reviewees (student's teammates). This is indeed a problem for teammate assessment, when the faculty asks for what fraction of the work each teammate did.So an alternative is needed, let’s call it a “Cake” item type, that allows a reviewer to divide a “cake” in any way between the reviewees, but does not allow him/her to divvy up more than 100% of the cake.

1. When the reviewer submits a score that would bring the total assigned for this item to > 100%, the system needs to warn.
2. Allow a reviewer to give him/herself a score
3. Proposed design needs to be compatible with existing self reviews code and teammates reviews
4. Should be extensible to other kind of reviews apart from teammate reviews as well

Proposed Solution:

Proposed Solution:

The issue asks us to have a Cake type for the question taking in a participant’s contribution, whenever s/he is reviewing the other teammates. We add in a new Question type ‘Cake’, which will be extended from the Scored Question model [cake < scored question < choice question < question]. We intend to keep the Cake type input in the form of the number of stars, on the lines of the existing code.

Addressing self-reviews for Cake Type:

The questionnaire for reviewing teammates includes the question asking the cake (contribution) factor. The point whether the distribution of the cake should include the reviewer himself, is addressed on the basis of the enabling of ‘Self-Review’. If enabled, we include the self contribution to sum it up to one unit (for instance, five stars.) If not, we exclude the reviewer.

There are two ways to integrate the Cake type to the existing flow:

• Solution A: Self-review at team level as an instance of TeammateReviewResponseMap

In the current system workflow, we found that the teammate reviews are taken as instances of TeammateReviewResponseMap. The questionnaire for reviewing teammates includes the question asking the cake (contribution) factor. Ideally, the cake should include the reviewer himself as well.

Currently, whenever a participant is reviewing his teammates, s/he is displayed a page with his current teammates, and this page excludes the logged in user. We include the logged in user ID as well by removing this check, and let him submit his self-review with the same questionnaire as his other teammates have.

The Teammate Review page would be displayed as:

Steps to reproduce the proposed workflow:

1. Log in to expertiza to view the home page
2. Login as an Instructor, and then impersonate a student or login as a Student
3. Go to Assignments -> Your team
4. You will see a list of your teammates, including self: Choose a member and click on ‘View’
5. You can see the question for asking the contribution as a cake type
6. There will be a text description next to it denoting what part of the cake is taken (what contribution factor of the work is used)

• Solution B: Self-review at team level as an instance of SelfReviewResponseMap

Currently, we have self reviews in place at an assignment level. This means that a team can review their own work, similar to how they review their peers. We want to extend the self review functionality to not only review the project but to also review the student as a team member. We can make use of the existing self review code and extend it to teammate reviews as well.

In the current implementation, a student can perform a self review by going to a particular assignment and clicking on the “Your work” link. If self reviews are enabled for that assignment, then a button will show at the bottom of the page which says “Self review”. Clicking on the button will show a link “Begin”

We want to add another link here, that says “self teammate review”. Such that clicking on the link will render the same questionnaire that is shown for the student to review other teammates.

Steps to reproduce the proposed workflow:

1. Log in to expertiza to view the home page
2. Login as an Instructor, and then impersonate a student or login as a Student
3. Go to Assignments -> Your work
4. Click on self review button at the bottom of the page
5. You will see a new “self teammate review” link here
6. Clicking on the button will render the questionnaire
7. You can see the question for asking the contribution as a cake type
8. There will be a text description next to it denoting what part of the cake is taken (what contribution factor of the work is used)

Final approach: We decide to go with the proposed Solution A (Self-review at team level as an instance of TeammateReviewResponseMap), since it ensures consistency in the flow, while taking teammate reviews and self-reviews.

Use Case Diagram

Files Modified in the Project

1. Edit questionnaire/_questionnaire.html.erb: Add Cake to the list question types
2. Create cake.rb model that inherits from ScoredQuestion model. Add method to calculate currently added weights of the cake to this model.
3. Edit questionnaires_controller.rb, under “add_new_questions” method, adding a cake type and handle the changes.
4. Edit assignments/edit/_review_strategy.html.erb and add ‘Allow teammate self-review’ to the list of options.
5. Edit student_teams/view.html.erb and add reviewer to list of displayed teammates and mark that item as self review in the view.
6. Edit response/response.html.erb and add cake type questions to the displayed response.
7. Edit response/view.html.erb and add cake type questions to the displayed response.

Database Modification

In order to implement the self review feature for teammate reviews we have added a column "is_self_teammate_review_enabled" to the assignments table. Below is the migration for the same:

 class SelfTeammateReview < ActiveRecord::Migration
  def change
	add_column :assignments, :is_self_teammate_review_enabled, :boolean, :default => false
  end
end 

Database Design

Below is the simplified version of the ER diagram which displays the relationships between the entity sets.

Relevant Tables

Test Plan:

UI Testing

• Reviewing teammates (happy case when contribution is < 100%)

1. A student logs on to expertiza and clicks on an assignment
2. Then he clicks on your team link
3. He should be able to see his teammates under team members
4. Click on review link for a fellow team member
5. Fill the responses for the given questions (which include cake type questions as well)
6. Save the review

• Giving > 100% for cake questions

1. A student logs on to expertiza and clicks on an assignment
2. Then he clicks on your team link
3. He should be able to see his teammates under team members
4. While reviewing a teammate, a student gives 60% as contribution to teammate1
5. When he tries to give > 40% for the teammate2. He should see a warning that states that total of the cake is exceeding 100%

• Context: Self reviews can be enabled/ disabled for an assignment as per the choice of the instructor.

1. Self reviews enabled
2. A student logs on to expertiza and clicks on an assignment
3. Then he clicks on your team link
4. He should be able to see his teammates and himself (for self review) under team members
5. Total contribution for a cake type question in a teammate review should include the contribution provided by the student in the self review as well.
6. If the contribution is exceeded, a warning should appear

• Self review disabled

1. The link for a reviewer to review himself as a team member should not show
2. The cake type questions in the teammate review should include only contributions provided by the other team members

• Context: If self review is enabled after the assignment has been created and a few students have already given reviews, then it should give a warning.

1. Self review enabled in the middle of the assignment
2. Add teammate review for teammate1
3. Add teammate review for teammate2 (make sure it adds upto 100%)
4. Enable self review as an instructor
5. As a student, notice that a new review button is visible in the “your team” page.
6. Click on review, try to add contribution for self and make sure a warning appears that says that the cake is already at 100%

• Context: If self review is disabled after the assignment has been created and a few students have already given reviews.

1. Self review disabled in the middle of the assignment
2. Enable self review as an instructor
3. Add teammate review for teammate1
4. Add teammate review for teammate2
5. Add self review (make sure it adds upto 100%)
6. Disable self review as an instructor
7. As a student, notice that the review button is NOT visible in the “your team” page

Open question: what should the expected behavior be? Should we decrease the self review contribution? But this would reduce the total contribution < the desired 100%

Automated testing using Rspec

Both Rspec and Capybara with Rspec tests have been written, with the capybara file - cake_questionnaire.rb added to spec/features folder, and rspec model file - cake_spec.rb added to the spec/models folder.

The capybara test creates a new questionnaire by logging into expertiza as instructor6, and adds the cake type questions to the questionnaire.

Several other test cases such as successful login into expertiza, along with testing the two main scenarios (shown in the snippet above) had been tested. The question_type is a list of values which contain all types of questions, which includes the question type cake. The test cases pass, on creation of a review questionnaire and on creation of a questionnaire with different types of questions, including the cake type questions to the questionnaire rubric.

The RSpec testing has been done on the cake.rb model, and has resulted in a decent code coverage of 82.73% and sample code can be seen in the snippet below:

With most of the code on the model is rendering html code, the correctness of the html code has been tested, to check if rspec also results in code that is expected. To run and verify Rspec results, please do run the following commands from the expertiza folder:

rspec spec/models/cake_spec.rb rspec spec/features/cake_questionnaire.rb

With the right gems installed, the test cases should ideally pass.

Team

Mentor: Carmen Bentley (cnaiken@ncsu.edu)

• Mita Gavade (magavade@ncsu.edu)
• Srujana Rachakonda (srachak@ncsu.edu)
• Ram Chavali (rlchaval@ncsu.edu)
• Abhirav Kariya (akariya@ncsu.edu)