CSC/ECE 517 Spring 2022 - E2234. Calibration submissions should be copied along with calibration assignments

From Expertiza_Wiki
Jump to navigation Jump to search

Introduction

A “calibration assignment” is an assignment where the students are asked to review work that has also been reviewed by a member of the course staff. If the student’s review “resembles” the staff-member’s review, then the student is presumed to be a competent reviewer. Here is a further description of calibration assignments. To set up calibration, the instructor (or TA) adds a few extra participants to the assignment. The instructor (or TA) then impersonates the extra participants, and submits work on behalf of each of the extra participants.

Problem Statement

Having the instructor (or TA) impersonate the extra participants, and submit work on behalf of each of the extra participants is extra trouble. It would be nice if an instructor didn’t have to resubmit the same calibration submissions every semester. Copying the extra participants along with their teams, submissions, and responses when copying an assignment makes things much easier.

Previous Implementation

The previous project did fully implement code that copies over the calibration submissions whenever a calibrated assignment is submitted. It was done by ensuring that when a assignment is copied that 1) the same extra participants as the assignment you are copying are used 2) the URLs and/or files that were submitted to the previous assignment are also copied and 3) the URLs and files are submitted to the assignment as well


Shown below is a diagram of how the current system works (copying the extra participants/reviews):


















The previous implementation needs to be revisited and made more elegant. A description of the needed refactoring and changes is below.

What needs to be done

  • Ensure what gets copied (when and how) is done properly
    • Reviews done by the creator of the assignment are copied, but only reviews that have been started --> how do you recognize a calibration review?: Can check 'calibrate_to' field in response maps where 'reviewed_object_id' equals the assignment ID being copied
    • If an author did any non-calibration reviews in the assignment being copied, those reviews would also be treated as calibration reviews
    • Submitted files are copied too, but if the calibration submissions are done as part of the COPYING of the assignment, they will necessarily use directories 0, 1,
  • Handling the case of students submitting assignments before calibration assignments are done
    • Project assumes that calibrations start at directory 0
    • Not the case if any students submit before the calibration submissions are done
  • Submitter_count needs to be set to the largest directory number that was used for calibration
    • First student to submit will have their submission show up in the same directory with the first calibration submission
  • Several class methods in assignment_form.rb should be moved to different classes as instance methods in those classes
    • Add better comments - eg. a comment for a method that copies an assignment should not reveal details about calibrations. Currently, only 1/4 methods in this class have comments.
    • Currently, for assignment_form.rb we change the name of the copied assignments to “copy of copy” and "copy of copy of copy" etc. We will improve this by making it "Copy of <name> <copy number>" and putting this function in a new method.







  • Clean up participant.rb
    • Look for a more elegant way of (self.create_participant) copying/creating of participants → DRY violation
    • Write the create method in assignment_participant instead of directly in participant.rb
  • Clean up team.rb
    • Create a copy method in team that copies the field used by all teams
    • Rest of the stuff is just used by assignment teams (submitted hyperlinks etc.)
    • Create a copy method and override in assignment_team
  • Reevaluate instance variables. Are these needed, or could they be local variables instead?
  • Look at editing out 9min video due to long waits for responses during copying
  • Verify our work using not only user 'Instructor6' but also Professor Jennifer Kidd's account - 'Instructor 2026'

Design

Files Requiring Modification

  • assignment_form.rb
    • Intead of setting the name of the copied assignments to “copy of copy” and "copy of copy of copy" etc, We will change it to name "Copy of <name> <copy number>" and put this function in a new method.
      This code currently resides in the self.copy method (which is too long and does too much for one method), so we will move it to a new method called copy_name.


    • Implement new method for copying objects needed to recreate calibration reviews
  • participant.rb
    • Write the create method in assignment_participant instead of directly in participant.rb
    • Here is the current create method in participant.rb:


  • response.rb
    • Implement method for copying previous responses
  • response_map.rb
    • Implement method that creates ReviewResponse mapping, based on mapping of copied assignment
  • submission_record.rb
    • Implement method for copying submission records of an assignment
  • team.rb
    • Create a copy method in team.rb that copies all teams of a previous assignment
  • teams_user.rb
    • Create method for creating new teams_users so that new participants and teams are associated the same way they were for the previous assignment
  • assignment_participant.rb
    • Method for copying extra participants should be here, we wouldn't need to copy participants with type 'CourseParticipant'

Final Implementation

We implemented a new class method, 'copy_calibrated_reviews', in the assignment_form model. This method, if the original assignment was calibrated, calls methods in models listed in previous section to copy all objects needed to recreate calibrated reviews. If the original assignment is not calibrated, this method does nothing.

  • We copy all submission records with assignment_id matching the original assignment. See submission record database documentation
  • We copy all participants with parent_id matching the original assignment ID. We did this in the participant model but should have moved it to assignment_participant since we are not interested in course participants. See participant database documentation
  • We copy all teams with a parent_id matching the original assignment ID. See teams database documentation
  • We recreate mappings needed to associate participants with teams and review_response maps.
    • Our teams and participants copy methods return hash maps that are used to lookup previous mappings. For each new team and participant, we can lookup how the original records were mapped
  • We copy all review_responses, using the original review response map as a reference. See response database documentation

Our new 'copy_calibrated_reviews' is called in the existing assignment_form class method 'copy', if the copied assignment is able to be saved (we don't want to create all of these new records pointlessly).

We also added methods for copy naming schemes. This is needed because the assignment model validates uniqueness of both the assignment name and directory_path.

Test Plan

Testing Goals

 The main goal of our testing is to ensure that when a new calibration assignment is added, all of the qualifying previous calibration submissions 
 and reviews are successfully copied over without overwriting any existing submissions.

Automated Unit Tests

Using RSpec, we have to test that the following unit tests pass: 

 1. When the TA adds a calibration assignment, extra participants are added to the assignment.

 2. When extra participants are added to an assignment, calibration submissions are copied over.

 3. When calibration submissions are copied over, the submitter_count value increases by the number of calibration submissions, so that previous submissions are 
    not overwritten.

 4. After calibration assignments are copied over, only calibration reviews from the creator of the assignment are copied over.

 5. After calibration assignments are copied over, only calibration reviews that have been started are copied over.

 6. Calling specific methods in assignment.rb return a NoMethodError to indicate that these methods are no longer class methods and have been moved elsewhere as 
    instance methods.

Conclusion

Our testing was all manual testing. Like the old team, we used the assignment titled "Design exercise" from the expertiza_development database, while logged in as instructor6 to verify our implementation.

The calibration tab, in the assignment edit view, appears identical to the copied assignment.

  • Future Considerations
    • submitter_count, a field in the assignments table is not dealt with in our implementation. Interestingly, the existing 'Design exercise' assignment used for testing had a submitter_count of 0 the database.
    • Directory paths for submissions are still not explicitly considered.
    • Additional rspec tests should be created. An easy test would be to check response_maps for the new assignment after calling the copy method and verifying the same number of records exist.
    • Several database queries are performed in each of our methods. Performance improvements could likely be made by considering another strategy.
    • Our copy naming methods are identical, the only difference being the assignment field used to check for name uniqueness (assignment name vs directory path). We couldn't figure out how to consolidate this to a single method (how can you pass a attribute/field type to a method?)
    • More defensive programming could be done, i.e. when we copy records, we don't check whether saving the new record is successful. If we did check and a copy failed to save, what should be done?

Team Information

Mentor: Dr. Ed Gehringer (efg@ncsu.edu)
Nolan Dowdle (njdowdle@ncsu.edu)
Laura Fox (lmcampb5@ncsu.edu)
Soeun Jo (sjo@ncsu.edu)
Harshil Shah (hshah6@ncsu.edu)

Resources

A link to our pull request can be found here
A link to the Expertiza GitHub can be found here
A link to our repository can be found here
Our implementation video can be found here
A link to the previous Wiki can be found here
The previous implementation video can be found here
A link to the previous GitHub pull request can be found here

Retrieved from "https://wiki.expertiza.ncsu.edu/index.php?title=CSC/ECE_517_Spring_2022_-_E2234._Calibration_submissions_should_be_copied_along_with_calibration_assignments&oldid=145566"