Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Fall 2019 - E1998. Weights in grade calculation

2019-12-11T17:02:15Z

Ssrane2: /* Files modified */

This wiki page has been created to document the changes that will be made under E1998 - Weights in grade calculation as a part of the final project for Fall 2019, CSC/ECE 517.

== '''About Expertiza''' ==

Expertiza is an open source project which can be used by instructors for a multitude of use cases such as creating assignments of various types, creating rubrics, questionnaires, allowing teams to be created (for group assignments) etc. It also allows grading of the assignments created, and provides the ability to let students peer review other students' work.

== '''Project Overview''' ==

'''Background and Problem Statement'''

The questionnaire/rubric creation feature of Expertiza also allows weights to be added to the questions being created, thereby allowing an instructor to create a questionnaire with relative ordering among the questions from a grading standpoint i.e., correctly answering a question with a higher weight will fetch more marks as compared to correctly answering a question with a lower weight.

However, the same functionality is not present for quiz questionnaires. Quiz questionnaires are essentially a way to ensure that students review others' work genuinely. When an assignment has quizzes enabled, each student gets an option to create a quiz for each of his/her reviewers after the initial submission is done and a review round has occurred.

The flow which is followed from quiz creation to score evaluation is as follows:

::::::::::::::[[File:Quiz flow.png]]

While creating this quiz, a student is supposed to ask questions related to his/her work which ideally a reviewer must be able to answer. Once it is created, reviewers are able to answer these quizzes and the corresponding scores are evaluated for each quiz separately. Ideally, each question should have a weight associated with it which a student creating the quiz should be able to specify, which in turn should be used while calculating scores for those taking these quizzes. However, currently, there is no way to specify weights for individual questions while creating quizzes and hence scores are being calculated by giving each question equal weightage.

'''Goal of the Project'''

The goal of this project is to understand the cause of the aforementioned issue and to ensure that the weights associated with questions are taken into consideration when grading all types of questionnaires involving weights. Our main focus will be to implement weights for quiz questionnaires, To achieve this, first we will have to introduce a way for a student creating a quiz to be able to enter weights for individual questions. Once weights for each question are being successfully stored, we can proceed to change how scores for each reviewer(for a particular quiz) are being calculated such that these newly added weights are also being taken into consideration.

== '''Current Implementation''' ==

Expertiza allows multiple questionnaires to be associated with an assignment. For instance, an assignment may have 3 questionnaires - one each for reviewing teammates, one for reviewing others' work, and one for giving feedback to the reviews received. There are various questions that can be created for each type of questionnaire but they can be broadly classified into two types - weighted and non-weighted.

The user-submitted responses to these questions can be accessed via a view called score_view which extracts results from three different tables - questions, questionnaires, and answers. This view has a column named question_weight which is populated by the user entered weights for weighted questions and is set to null for non-weighted questions. This is the view which the code uses for calculating grades for a particular assignment.

Currently, for questionnaires other than quiz questionnaires, an instructor creates a rubric for the questionnaire where he/she can specify weights for each weighted question that exists for that questionnaire. However, the way it is implemented is that the instructor first has to create a question without specifying any weight. The weight for this newly created question is set to 1 by default. After this is done, the instructor has to edit that question to trigger a way to enter a weight for that question. This method for specifying weights is non-intuitive and cumbersome.

For quiz questionnaires on the other hand, there is no way to enter weights for newly added questions at all. The weights are set to 1 by default for all the weighted questions and are not modified henceforth. As a result, scores are calculated for each quiz for a particular reviewer by giving equal weightage for each question. This calls for a complete overhaul of how scores for quiz questionnaires are being calculated to include weights which will need to be accepted from the student creating the quiz.

== '''Changes made by the team''' ==

We followed a two-step approach for this project.

===== Questionnaires except quiz questionnaire =====

* The first step involved handling all questionnaires except the quiz questionnaire, ensuring that all of them store weights associated with any weighted questions they might have. If so, during score calculation, these weights need to be taken into consideration and the appropriate score displayed.

* We found that method ''get_total_score'' was responsible for the score calculation and that it took weights into consideration. A code snippet for the same:

<nowiki>
def self.get_total_score(params)
@response = params[:response].last
if @response
@questions = params[:questions]

weighted_score = 0
sum_of_weights = 0
max_question_score = 0

@questionnaire = Questionnaire.find(@questions[0].questionnaire_id)

questionnaireData = ScoreView.find_by_sql ["SELECT q1_max_question_score ,SUM(question_weight) as sum_of_weights,SUM(question_weight * s_score) as
weighted_score FROM score_views WHERE type in('Criterion', 'Scale') AND q1_id = ? AND s_response_id = ?", @questions[0].questionnaire_id, @response.id]
# zhewei: we should check whether weighted_score is nil,
# which means student did not assign any score before save the peer review.
# If we do not check here, to_f method will convert nil to 0, at that time, we cannot figure out the reason behind 0 point,
# whether is reviewer assign all questions 0 or reviewer did not finish any thing and save directly.
weighted_score = (questionnaireData[0].weighted_score.to_f unless questionnaireData[0].weighted_score.nil?)
sum_of_weights = questionnaireData[0].sum_of_weights.to_f
# Zhewei: we need add questions' weights only their answers are not nil in DB.
all_answers_for_curr_response = Answer.where(response_id: @response.id)
all_answers_for_curr_response.each do |answer|
question = Question.find(answer.question_id)
# if a questions is a scored question (criterion or scale), the weight cannot be null.
# Answer.answer is nil indicates that this scored questions is not filled. Therefore the score of this question is ignored and not counted
# towards the score for this response.
if answer.answer.nil? && question.is_a?(ScoredQuestion)
question_weight = Question.find(answer.question_id).weight
sum_of_weights -= question_weight
end
end
max_question_score = questionnaireData[0].q1_max_question_score.to_f
if sum_of_weights > 0 && max_question_score && !weighted_score.nil?
return (weighted_score / (sum_of_weights * max_question_score)) * 100
else
return -1.0 # indicating no score
end
end
end</nowiki>

* Our next step was to address the issue of how weight entry while creating a rubric was being handled. As mentioned in the previous section, the user had to first create a question and then edit it to specify a weight. What we did was add a text box during the creation of a question itself where the user can enter the appropriate weight. One important point we needed to address was to only allow weight entry for weighted questions(Criterion and Scale). This was taken care by JavaScript and the corresponding changes can be found in ''_question_weight.html.erb''. Finally, we changed the assignment of ''question.weight'' variable in questionnaires_controller.rb to ensure that the user entered weights are being taken into consideration.

Code screenshot:

[[File:Codescr2.png]]

Corresponding UI changes:(We added a Weight field to allow its entry as shown)

[[File:add-weight-new.png]]

* The last stage in this step was to ensure that appropriate changes are made to tests written for questionnaires controller if they fail.

===== Quiz questionnaire =====

* For the quiz questionnaire, the first step was to implement a UI change to allow the student creating a quiz to be able to enter the weight for each question. We've used a number field to input the weights, and ensured that users cannot enter negative weights for a question. In case the user enters a negative weights, an appropriate error message is displayed when the user tries to save the quiz. Further, we've also made changes to the edit view for the quiz which allows students to edit the question weights in addition to other question attributes for the quiz.
A screenshot of this change is provided below:

::::::::::[[File:quiz-weight-field.png]]

The changes in the following screenshot ensure that the user entered weights are being stored corresponding to the questions as compared to the previously hard coded value of 1:

[[File:Codescr2.png]]

* The next course of action was to actually implement a way to use these stored weights while calculating the score for a quiz. This involved making changes in the ''app/models/quiz_response_map.rb'' file and replacing the existing logic to calculate the score.

'''New logic for quiz score calculation:'''
* The "score_views" view is used to calculate the score of the quiz by considering weights set by the quiz creator on each question.
* The score for each answer is stored as 1 if the quiz taker selected the correct option and 0 otherwise.
* The weight of each question is multiplied with the score to compute the score received the by quiz taker for the corresponding answer.
* The total score for all questions is divided by the total quiz score (sums of question weights) and converted into a percentage to get the final score.
* No partial credits are considered for multiple-choice questions, all the options must be correct to gain marks.

Code changes for the above:

[[File:new-logic.PNG]]

* The last stage was to write automated tests for the new functionality we implemented such that it tested as many kinds of quizzes as possible and ensured that the scoring of questions took into consideration the weights being set for individual questions.

The final flow of how quiz questionnaires are handled after our changes can be summarized as follows:

::::::::::[[File:Quiz ScorecalcUML.png]]''

== '''Design Pattern''' ==
We have utilied the Facade Design Pattern to hide the complexity of quiz score calculation and provided a single method quiz_score to calculate and return the score of the quiz considering weights on each question. The method makes the query call which gets the question, the weight, the corresponding answer score obtained by the student. Then it converts the values into a total and calculates the final percent score obtained by the student in the quiz. The method encapsulates the entire logic and returns the total weighted score for the corresponding quiz instance.

== '''Files modified''' ==

The following files were modified to accomplish the aforementioned changes:

*questionnaires_controller.rb
:''add_new_questions''
:The weights were being hard coded earlier, which have now been replaced by the user specified weights via the UI. This method creates new questions when creating questions in questionnaires of types such as review questionnaire, author feedback questionnaire, teammate review questionnaire, etc.

:''save_new_questions''
:This method saves questions when a student creates a quiz questionnaire. There was no way to associate weights with questions in quiz questionnaire earlier, we've made changes to allow adding weights to questions in a quiz questionnaire. The corresponding changes are present in this method.

*_new_question_template.html.erb
:The input field for adding question weights at the time of creation of questions in a quiz has been added to this file. This functionality was not present earlier, all questions in a quiz were associated with a default weight of 1. The weights entered in this input field are used by the save_new_questions method in questionnaires_controller.rb

*_questionnaire.html.erb
:The input field for adding question weights at the time of creation of scored questions for questionnaires of all types except quiz questionnaire has been added in this file. This functionality was not present earlier, as questions had to be created first and then edited to change the default weight (1) associated with it. The weights entered in this input field are used by the add_new_questions method in questionnaires_controller.rb

*_question_weight.html.erb
:This file contains the JavaScript code which handles the conditional display of the field to input weights depending on the type of question the user wants to create in the above view. This code ensures that the field is displayed only if the user is trying to created a question with which weights can be associated - questions of type Criterion and Scale.

*edit.html.erb
:This files was had to be modified to include the above partial which contains the JavaScript for conditional display of the input field for entering question weight.

*quiz_response_map.rb
:''quiz_score''
:This method has been modified to include the logic for calculation of quiz scores. The added logic incorporates the weights which the user can now input corresponding to questions when creating quizzes.

*score_view.rb
:''readonly''
:This method has been modified to allow factories of this class to be created for testing.

The following file was modified to included automated tests for creation and grading of a quiz questionnaire:

*quiz_spec.rb
: This file contains the necessary changes for testing the entire flow, right from creating a quiz to submitting it, and finally scoring it. The test has been created to check all the modifications we've made to the above files. Also, some of the existing tests had to be modified to incorporate the changes in the way the score is calculated using the new logic.

*quiz_factory.rb
: This file contains the factory for score_views which is used by ''quiz_score'' to compute the grade by taking weights into consideration.

== '''Test Plan''' ==

As a part of creation of RSpec tests, we created an example in quiz_spec.rb file which sets up the necessary actions before a student A can create a quiz questionnaire. We then proceed with quiz questionnaire creation and associate weights with questions created. We create all three types of questions that can be created in a quiz questionnaire - True/False questions, Multiple Choice Radio questions and Multiple Choice Checkbox questions. Further, we login as student B who has reviewed student A's work and take the quiz. Finally, we submit the quiz and ensure the that the quiz is scored after taking the weights associated with the questions into consideration.

Please follow the following steps to test our changes and/or reproduce the functionality implemented by us:

<ol>
<li> Find an assignment and find some participants for the assignment using add participant option corresponding to the assignment in the assignment page.</li>
<li> Enable the has quiz option for the corresponding assignment if it is not already enabled.</li>
<li> An input field will appear at the right of the has quiz checkbox, please enter the number of questions in the quiz in that checkbox.</li>
<li> Now, modify the submission deadline of assignment to some date in the near future.</li>
<li> Also, select yes for the stage in which you wish to allow students to take quizzes in the due dates tab.</li>
<li> Login as one of the participants (or impersonate one) say A, submit something in the your work section for the assignment.</li>
<li> This will enable the create quiz link in the your works section, please go ahead and create a quiz.</li>
<li> Now, modify the submission deadline of assignment to some date in the past, so that the assignment is now in review stage.</li>
<li> Now login as another assignment participant say B, review A's work.</li>
<li> Click on the take quizzes link, a radio button with the quiz created by participant A should appear, go ahead and take the quiz.</li>
<li> The quiz will be graded taking into consideration the weights you used for creating the questions in the quiz as participant A.</li>
</ol>

== '''Quick Links''' ==

* Video Walk-through Link: https://drive.google.com/drive/u/1/folders/1GcdSuqH4jcM4Ax-rQ-4FDPYe82NG3cfJ
* Forked Repository Link: https://github.com/theSaurabhMhatre/expertiza
* Pull Request Link: https://github.com/expertiza/expertiza/pull/1633
* Demo Link: http://152.7.99.125:8080/

== '''Team''' ==

Our team:

*Sanket Pai (sgpai2@ncsu.edu)
*Sanveg Rane (ssrane2@ncsu.edu)
*Saurabh Mhatre (smhatre@ncsu.edu)
*Saurabh Shingte (svshingt@ncsu.edu)

Mentor: Carmen Bentley (cnaiken@ncsu.edu)

CSC/ECE 517 Fall 2019 - E1998. Weights in grade calculation

2019-12-02T03:53:20Z

Ssrane2: /* Quiz questionnaire */

This wiki page has been created to document the changes that will be made under E1998 - Weights in grade calculation as a part of the final project for Fall 2019, CSC/ECE 517.

== '''About Expertiza''' ==

Expertiza is an open source project which can be used by instructors for a multitude of use cases such as creating assignments of various types, creating rubrics, questionnaires, allowing teams to be created (for group assignments) etc. It also allows grading of the assignments created, and provides the ability to let students peer review other students' work. Additionally, it also allows an instructor to export data for various objects created into multiple formats for further analysis.

== '''Project Overview''' ==

'''Background and Problem Statement'''

The questionnaire/rubric creation feature of Expertiza also allows weights to be added to the questions being created, thereby allowing an instructor to create a questionnaire with relative ordering among the questions from a grading standpoint i.e., correctly answering a question with a higher weight will fetch more marks as compared to correctly answering a question with a lower weight.

However, the same functionality is not present for quiz questionnaires. Quiz questionnaires are essentially a way to ensure that students review others' work genuinely. When an assignment has quizzes enabled, each student gets an option to create a quiz for each of his/her reviewers after the initial submission is done and a review round has occurred. While creating this quiz, a student is supposed to ask questions related to his/her work which ideally a reviewer must be able to answer. Once it is created, reviewers are able to answer these quizzes and the corresponding scores are evaluated for each quiz separately. Ideally, each question should have a weight associated with it which a student creating the quiz should be able to specify, which in turn should be used while calculating scores for those taking these quizzes. However, currently, there is no way to specify weights for individual questions while creating quizzes and hence scores are being calculated by giving each question equal weightage.

'''Goal of the Project'''

The goal of this project is to understand the cause of the aforementioned issue and to ensure that the weights associated with questions are taken into consideration when grading all types of questionnaires involving weights. Our main focus will be to implement weights for quiz questionnaires, To achieve this, first we will have to introduce a way for a student creating a quiz to be able to enter weights for individual questions. Once weights for each question are being successfully stored, we can proceed to change how scores for each reviewer(for a particular quiz) are being calculated such that these newly added weights are also being taken into consideration.

== '''Current Implementation''' ==

Expertiza allows multiple questionnaires to be associated with an assignment. For instance, an assignment may have 3 questionnaires - one each for reviewing teammates, one for reviewing others' work, and one for giving feedback to the reviews received. There are various questions that can be created for each type of questionnaire but they can be broadly classified into two types - weighted and non-weighted.

The user-submitted responses to these questions can be accessed via a view called score_view which extracts results from three different tables - questions, questionnaires, and answers. This view has a column named question_weight which is populated by the user entered weights for weighted questions and is set to null for non-weighted questions. This is the view which the code uses for calculating grades for a particular assignment.

Currently, for questionnaires other than quiz questionnaires, an instructor creates a rubric for the questionnaire where he/she can specify weights for each weighted question that exists for that questionnaire. However, the way it is implemented is that the instructor first has to create a question without specifying any weight. The weight for this newly created question is set to 1 by default. After this is done, the instructor has to edit that question to trigger a way to enter a weight for that question. This method for specifying weights is non-intuitive and cumbersome.

For quiz questionnaires on the other hand, there is no way to enter weights for newly added questions at all. The weights are set to 1 by default for all the weighted questions and are not modified henceforth. As a result, scores are calculated for each quiz for a particular reviewer by giving equal weightage for each question. This calls for a complete overhaul of how scores for quiz questionnaires are being calculated to include weights which will need to be accepted from the student creating the quiz.

== '''Changes made by the team''' ==

We followed a two-step approach for this project.

===== Questionnaires except quiz questionnaire =====

* The first step involved handling all questionnaires except the quiz questionnaire, ensuring that all of them store weights associated with any weighted questions they might have. If so, during score calculation, these weights need to be taken into consideration and the appropriate score displayed.

* We found that method ''get_total_score'' was responsible for the score calculation and that it took weights into consideration according to this equation:

---def get_total_score screenshot---

* Our next step was to address the issue of how weight entry while creating a rubric was being handled. As mentioned in the previous section, the user had to first create a question and then edit it to specify a weight. What we did was add a text box during the creation of a question itself where the user can enter the appropriate weight. One important point we needed to address was to only allow weight entry for weighted questions(Criterion and Scale). In the file itself, we changed the assignment of ''question.weight'' variable to implement the above change.
Code screenshot:
[[File:Codescr2.png]]
---text box for add question in questionnaire screenshot(weighted and unweighted)---

* The last stage in this step was to ensure that appropriate changes are made to tests written for questionnaires controller if they fail.

===== Quiz questionnaire =====

* For the quiz questionnaire, the first step was to implement a UI change to allow the student creating a quiz to be able to enter the weight for each question.

--UI quiz questionnaire screenshot--

Corresponding code screenshot ensuring entered weights are being stored for the appropriate questions:

[[File:Codescr2.png]]

* The next course of action was to actually implement a way to use these stored weights while calculating the score for a quiz. This involved making changes in the ''app/models/quiz_response_map.rb'' file and replacing the existing logic to calculate the score.

'''New logic for quiz score calculation:'''
::- The "score_views" view is used to calculate the score of quiz by considering weights set by the quiz creator on each question
::- The marks for each answer is stored as 1 if the quiz taker selected correct option and 0 otherwise
::- The weight of each question is multiplied to marks obtained by quiz taker for the corresponding answer
::- The total weighted score is divided by the total quiz score and converted into percentage to get the final score
::- No partical credits are considered for multiple choice questions, all the options must be correct to gain marks

* The last stage was to write automated tests for the new functionality we implemented such that it tested as many kinds of quizzes as possible and ensured that the final score reflected the weights being set for individual weighted questions.

Please find below a high level overview of the approach that we'll be following to tackle this problem:

:::::::::::[[File:high-level-flow.png]]''

== '''Files modified''' ==

The following files were modified to accomplish the aforementioned changes:

*questionnaires_controller.rb
:''add_new_questions''
:The weights were being hard coded earlier, which have now been replaced by the user specified weights via the UI. This method creates new questions when creating questions in questionnaires of types such as review questionnaire, author feedback questionnaire, teammate review questionnaire, etc.

:''save_new_questions''
:This method saves questions when a student creates a quiz questionnaire. There was no way to associate weights with questions in quiz questionnaire earlier, we've made changes to allow adding weights to questions in a quiz questionnaire. The corresponding changes are present in this method.

*_new_question_template.html.erb
:The input field for adding question weights at the time of creation of questions in a quiz has been added to this file. This functionality was not present earlier, all questions in a quiz were associated with a default weight of 1. The weights entered in this input field are used by the save_new_questions method in questionnaires_controller.rb

*_questionnaire.html.erb
:The input field for adding question weights at the time of creation of scored questions for questionnaires of all types except quiz questionnaire has been added in this file. This functionality was not present earlier, as questions had to be created first and then edited to change the default weight (1) associated with it. The weights entered in this input field are used by the add_new_questions method in questionnaires_controller.rb

*_question_weight.html.erb
:This file contains the JavaScript code which handles the conditional display of the field to input weights depending on the type of question the user wants to create in the above view. This code ensures that the field is displayed only if the user is trying to created a question with which weights can be associated - questions of type Criterion and Scale.

*edit.html.erb
:This files was had to be modified to include the above partial which contains the JavaScript for conditional display of the input field for entering question weight.

*quiz_response_map.rb
:''quiz_score''
:This method has been modified to include the logic for calculation of quiz scores. The added logic incorporates the weights which the user can now input corresponding to questions when creating quizzes.

The following file was modified to included automated tests for creation and grading of a quiz questionnaire:

*quiz_spec.rb
: This file contains the necessary changes for testing the entire flow, right from creating a quiz to submitting it, and finally scoring it. The test has been created to check all the modifications we've made to the above files.

== '''Test Plan''' ==

As a part of creation of RSpec tests, we created an example in quiz_spec.rb file which sets up the necessary actions before a student A can create a quiz questionnaire. We then proceed with quiz questionnaire creation and associate weights with questions created. We create all three types of questions that can be created in a quiz questionnaire - True/False questions, Multiple Choice Radio questions and Multiple Choice Checkbox questions. Further, we login as student B who has reviewed student A's work and take the quiz. Finally, we submit the quiz and ensure the that the quiz is scored after taking the weights associated with the questions into consideration.

Please follow the following steps to test our changes and/or reproduce the functionality implemented by us:

<ol>
<li> Find an assignment and find some participants for the assignment using add participant option corresponding to the assignment in the assignment page.</li>
<li> Enable the has quiz option for the corresponding assignment if it is not already enabled.</li>
<li> An input field will appear at the right of the has quiz checkbox, please enter the number of questions in the quiz in that checkbox.</li>
<li> Now, modify the submission deadlines of assignment to some date in the near future.</li>
<li> Login as one of the participants (or impersonate one) say A, submit something in the your work section for the assignment.</li>
<li> This will enable the create quiz link in the your works section, please go ahead and create a quiz.</li>
<li> Now login as another assignment participant say B, review A's work.</li>
<li> Click on the take quizzes link, a radio button with the quiz created by participant A should appear, go ahead and take the quiz.</li>
<li> The quiz will be graded taking into consideration the weights you used for creating the questions in the quiz as participant A.</li>
</ol>

== '''Quick Links''' ==

* Forked Repository Link: https://github.com/theSaurabhMhatre/expertiza
* Pull Request Link: https://github.com/expertiza/expertiza/pull/1633
* Demo Link: To be created

== '''Team''' ==

Our team:

*Sanket Pai (sgpai2@ncsu.edu)
*Sanveg Rane (ssrane2@ncsu.edu)
*Saurabh Mhatre (smhatre@ncsu.edu)
*Saurabh Shingte (svshingt@ncsu.edu)

Mentor: Carmen Bentley (cnaiken@ncsu.edu)

CSC/ECE 517 Fall 2019 - E1998. Weights in grade calculation

2019-11-11T23:31:16Z

Ssrane2: /* Files which require modification (Tentative) */

CSC/ECE 517 Fall 2019 - E1998. Weights in grade calculation

2019-11-11T05:16:50Z

Ssrane2: Created page with "E1998. Weights in grade calculation"

E1998. Weights in grade calculation

CSC/ECE 517 Fall 2019

2019-11-11T05:16:37Z

Ssrane2:

* [[CSC/ECE 517 Fall 2019 - Project E1947. Refactor quiz_questionnaire_controller.rb]]
* [[CSC/ECE 517 Fall 2019 - Project E1965. Review report should link to the usual view for reviews]]
* [[CSC/ECE 517 Fall 2019 - Project E1943. Refactor sign up sheet controller.rb]]
* [[CSC/ECE 517 Fall 2019 - E1972. OSS project J. Skellington: Accessing Assignment Rubrics]]
* [[CSC/ECE 517 Fall 2019 - E1961. Email notification to reviewers and instructors]]
* [[CSC/ECE 517 Fall 2019 - E1971. OSS project Finklestein: Instructors & Institutions]]
* [[CSC/ECE 517 Fall 2019 - E1953. Tagging report for student]]
* [[CSC/ECE 517 Fall 2019 - E1955.Write unit tests for student_task.rb]]
* [[CSC/ECE 517 Fall 2019 - E1954. Auto-generate submission directory names based on assignment names]]
* [[CSC/ECE 517 Fall 2019 - E1958. Two issues related to assignment management]]
* [[CSC/ECE 517 Fall 2019 - E1948. Refactor review_mapping_helper.rb]]
* [[CSC/ECE 517 Fall 2019 - E1959. Intelligent copying of assignments without topics]]
* [[CSC/ECE 517 Fall 2019 - E1968. Fixes for adding members to teams]]
* [[CSC/ECE 517 Fall 2019 - E1969. Fixes for reviews not being available]]
* [[CSC/ECE 517 Fall 2019 - E1951. Remove multiple topics at a time]]
* [[CSC/ECE 517 Fall 2019 - E1957. Time travel Not Allowed..!!! Restrict TAs’ ability to change their own grade + limit file-size upload]]
* [[CSC/ECE 517 Fall 2019 - E1963. Changing assignment participant role]]
* [[CSC/ECE 517 Fall 2019 - E1941. Issues related to topic deadlines]]
* [[CSC/ECE 517 Fall 2019 - E1966. Tabbed_reviews partial file refactor for displaying the alternate view of reviews]]
* [[CSC/ECE 517 Fall 2019 - E1962. Email notification upon account creation]]
* [[CSC/ECE 517 Fall 2019 - E1967. Fix glitches in author feedback]]
* [[CSC/ECE 517 Fall 2019 - E1960. Create new late policy successfully and fixing "Back" link]]
* [[CSC/ECE 517 Fall 2019 - E1939. OSS Project Juniper: Bookmark enhancements]]
* [[CSC/ECE 517 Fall 2019 - M1950. Support Asynchronous Web Assembly Compilation]]
* [[CSC/ECE 517 Fall 2019 - E1938. OSS project Duke Blue: Fix import glitches]]
* [[CSC/ECE 517 Fall 2019 - E1985. Let course staff and students do reviews]]
* [[CSC/ECE 517 Fall 2019 - M1951. Implement missing OffscreenCanvas APIs]]
* [[CSC/ECE 517 Fall 2019 - E1940. Improving email notification]]
* [[CSC/ECE 517 Fall 2019 - E1956. There is no shortcut to get free review points: Review Assignment Bug]]
* [[CSC/ECE 517 Fall 2019 - E1942. Refactor stage deadlines in assignment.rb]]
* [[CSC/ECE 517 Fall 2019 - M1952. Missing DOM features project]]
* [[CSC/ECE 517 Fall 2019 - E1945. Refactor users_controller.rb]]
* [[CSC/ECE 517 Fall 2019 - E1964. Export review scores for projects]]
* [[CSC/ECE 517 Fall 2019 - E1949. Write Unit Tests for Importing assignment participants and import glitches]]
* [[CSC/ECE 517 Fall 2019 - E1944. Refactor review mapping controller]]
* [[CSC/ECE 517 Fall 2019 - E1946. Refactor Questionnaire controller]]
* [[CSC/ECE 517 Fall 2019 - E1982. Regulate changing of rubrics while projects are in progress]]
* [[CSC/ECE 517 Fall 2019 - E1989. Track the time students look at other submissions]]
* [[CSC/ECE 517 Fall 2019 - E1990. Integrate suggestion detection algorithm]]
* [[CSC/ECE 517 Fall 2019 - E1995. Tests for email functionality]]
* [[CSC/ECE 517 Fall 2019 - E1993 Track Time Between Successive Tag Assignments]]
* [[CSC/ECE 517 Fall 2019 - E1979. Completion/Progress view]]
* [[CSC/ECE 517 Fall 2019 - E1995. Weights in grade calculation]]
* [[CSC/ECE 517 Fall 2019 - E1980. Sort instructor reports by name, ID, score, etc.]]
* [[CSC/ECE 517 Fall 2019 - E1984. Improve self-review Link peer review & self-review to derive grades]]
* [[CSC/ECE 517 Fall 2019 - E1994. Mentor management for assignments without topics]]
* [[CSC/ECE 517 Fall 2019 - E1997. Issues related to meta-reviewing]]
* [[CSC/ECE 517 Fall 2019 - E1981. Student-generated questions added to rubric]]
* [[CSC/ECE 517 Fall 2019 - E1996. Enhancements to review grader]]
* [[CSC/ECE 517 Fall 2019 - E1987. Improving search facility in Expertiza]]
* [[CSC/ECE 517 Fall 2019 - E1978. Fix issues related to deadlines and late policies]]
* [[CSC/ECE 517 Fall 2019 - E1991. Improvements to anonymized view]]
* [[CSC/ECE 517 Fall 2019 - E1998. Weights in grade calculation]]

CSC/ECE 517 Fall 2019 - E1941. Issues related to topic deadlines

2019-11-06T22:35:57Z

Ssrane2: /* Functionality */

This wiki page is for the description of changes made under E1941- Issues Related to Topic Deadlines OSS assignment for Fall 2019, CSC/ECE 517.

== '''About Expertiza''' ==

Expertiza is an open source project based on Ruby on Rails framework and the code is available on Github. Expertiza allows the instructor to create new assignments as well as edit new or existing assignments. Instructors can also create a list of topics the students can sign up for and specify deadlines for completion of various tasks. Students can form teams in Expertiza to work on various projects and assignments as well as peer review other students' submissions. Expertiza supports submission across various document types, including the URLs Wiki pages.

== '''Issues in the Current Project''' ==

* There is no deadline that specifies when a team can drop their topics. Due to this, few problems occur: The team might drop a topic later on which is close to the submission date and this results in topic not being assigned to any other team on time such that they can submit their assignment. Also, if one team was wait listed for the same topic which the other team dropped closer to the submission deadline, then the first wait-listed team will be assigned to the dropped topic which is not desirable as they might be working on their assigned topic for long time.

* There are different topics on which students can work during different times. This type of assignment is known as Staggered- deadline assignment in which different topics have different submission and review deadlines. For these assignments too, there is a need for "Drop Topics Deadline". In the current implementation, the drop topic deadline is same for all the topics in a Staggered Deadline assignment which is not desirable.

* The following points were identified in the problem statement, regarding when a team should be dropped automatically:
::- They get a topic they were on the waitlist for
::- The last member leaves the team
::- A drop-topic deadline passes, or a submission passes

Out the three requirements, first two features are already implemented. We have implemented the last one, details are given below.

== '''Current Implementation''' ==

=====Functionality=====

* '''The instructor cannot add a Drop Topic Deadline for topics'''

::The instructor has the functionality to specify Review and Submission deadlines, while editing a topic. This page however, doesn’t have any provision on the form to let the instructor add Drop Topic Deadlines. According to the current implementation, s/he can add in a date for Drop Topic Deadline as well.

* ''' The wait-list for a topic doesn’t get cleared on reaching the Drop Topic Deadline'''

::Whenever the Drop Topic Deadline is crossed, we expect:

::- The waitlist queue for the topic is cleared

::- The teams are not permitted to drop the assigned topics anymore.

::Previously, we had no job scheduled for dropping the waitlists. This segment involves a queue set-up for job scheduling (Sidekiq), which is not fully functional in the current implementation. To fix the issue, we have written the code for dropping teams from the waitlist which will work as expected once the queue starts working.

* ''' Calculation of Drop Topic Deadline'''

::Issues could occur if user accidently sets a drop topic deadline which is after the submission date of the topic. This case can be handled by checking the submission deadline of the topic against the date user sets. The flow works as follows:

::- Case: 'no deadline date is set on topic' - the topic submission deadline is used to clear the teams waitlisted for the topic (but the drop topic deadline is not set)
::- Case: 'date prior to the topic submission deadline is set as drop topic deadline' - the date set by user is set and used for clearing waitlisted teams
::- Case: 'date after the topic submission deadline is set as drop topic deadline' - the date set as submission date is logically correct date in this case and gets set as the drop topic deadline as well and is used to clear the teams wailisted

=====Drawbacks and Solutions=====

* '''Problem 1''': No provision for taking user input for Drop Topic Deadline and persisting it in the database. Although the current schema supports a Topic/Assignment having drop Topic Deadlines, there is not form input field for the same.

* '''Solution''': The implementation has been changed in such a way that the user can input the date, and the same is persisted under the due_dates table.

:Below is the code snippet from ''app/models/topic_due_date.rb'', dealing with the above change:

<pre>
# adds a new deadline if not present,
# updates the date if already present
def self.upsert_drop_deadline_job(assignment_id, topic, drop_topic_input)
deadline_type_id = DeadlineHelper::DEADLINE_TYPE_DROP_TOPIC
drop_topic_date = TopicDueDate.where(parent_id: topic.id, deadline_type_id: deadline_type_id).first rescue nil

calc_drop_topic_date = get_drop_topic_deadline_date(assignment_id, topic.id, drop_topic_input)

# if drop topic deadline is not in db, make an entry
if drop_topic_date.nil?
# if user sets a date before creating the first entry
due_at = (drop_topic_input.nil? || drop_topic_input.blank?) ? nil : calc_drop_topic_date

# add delayed job to drop waitlisted teams after deadline passes
delayed_job_id = modify_delayed_job(topic.id, calc_drop_topic_date, nil, false)

TopicDueDate.create(due_at: due_at,
parent_id: topic.id,
deadline_type_id: deadline_type_id,
type: DeadlineHelper::TOPIC_DEADLINE_TYPE,
delayed_job_id: delayed_job_id)
else
update_delayed_job = true

if !drop_topic_date.due_at.nil? && (drop_topic_input.nil? || drop_topic_input.blank?)
# if drop topic deadline is deleted
due_at = nil
elsif drop_topic_date.due_at.nil? && !(drop_topic_input.nil? || drop_topic_input.blank?)
# if drop topic deadline is entered first time
due_at = calc_drop_topic_date
elsif !drop_topic_date.due_at.nil? &&
drop_topic_date.due_at.to_datetime.strftime(DeadlineHelper::DATE_FORMATTER_DROP_DEADLINE) != calc_drop_topic_date.strftime(DeadlineHelper::DATE_FORMATTER_DROP_DEADLINE)
# if drop topic deadline is updated
due_at = calc_drop_topic_date
else
# if updated date is same as existing, don't update the delayed job
update_delayed_job = false
end

if update_delayed_job
delayed_job_id = modify_delayed_job(topic.id, calc_drop_topic_date, drop_topic_date.delayed_job_id, true)
drop_topic_date.update_attributes(due_at: due_at, delayed_job_id: delayed_job_id)
end
end
end

# check the user input date and return the accurate deadline for dropping topics
private
def self.get_drop_topic_deadline_date(assignment_id, topic_id, drop_topic_input)
expected_drop_deadline_date = DueDate.get_deadline_to_drop_topic(assignment_id, topic_id)

if drop_topic_input.nil? || drop_topic_input.blank?
return expected_drop_deadline_date
end

drop_topic_date = DateTime.parse(drop_topic_input)

# if user sets drop topic deadline ahead of submission date, use submission date
return (drop_topic_date.utc > expected_drop_deadline_date.utc) ? expected_drop_deadline_date : drop_topic_date
end
</pre>

[[File:Screen Shot 2019-10-31 at 2.34.32 PM.png]]

* '''Problem 2''': Waitlisted topics not getting dropped after the Drop Topic Deadline. For executing the timed jobs, Expertiza uses Sidekiq, and creates jobs for every Topic. Referred to as delayed_jobs, it currently has some defined for tasks like dropping a team when the last member leaves, etc.

* '''Solution''': We need a job to be executed on the day of the specified Drop Deadline, which would drop the waitlist. To resolve this, we added another job which gets created on every new Topic creation, to be executed at the Drop Topic Deadline. In case there is an update on a previously defined deadline by the instructor, we fetch the existing job, using the delayed_job_id, and modify the same as per the new Deadline.

:Edge cases addressed:

::1. In case the Topic Drop Deadline is not defined, we set the default based on the following dates, whichever is earlier/populated:
:::a. Topic Submission Deadline
:::b. Assignment Drop Topic Deadline
:::c. Assignment Submission Deadline
::2. In case of an updation on Drop Deadline, we check if the new entry is different from the existing date, and reschedule the job only if it is, thereby avoiding any redundant updates.
::3. Whenever the user-entered Topic Drop Deadline is earlier than the Submission Deadline, we set the date as per (1).

:Below is the code snippet from ''app/models/topic_due_date.rb'', dealing with the above change:

<pre>
# This method either adds a new job to the queue or deletes
# an existing job and replaces it with a new one
def self.modify_delayed_job(topic_id, drop_topic_date, delayed_job_id, job_present)
if job_present
remove_job_from_queue(delayed_job_id)
end

mins_left = calculate_mins_left(drop_topic_date)
return add_job_to_queue(mins_left, topic_id, "drop_topic", drop_topic_date)
end

def self.calculate_mins_left(drop_topic_date)
curr_time = DateTime.now
time_in_min = ((curr_time - drop_topic_date) * 24 * 60).to_i
end

def self.add_job_to_queue(min_left, topic_id, deadline_type, due_at)
delayed_job_id = MailWorker.perform_in(min_left * 60, topic_id, deadline_type, due_at)
return delayed_job_id
end

def self.remove_job_from_queue(job_id)
queue = Sidekiq::ScheduledSet.new
queue.each do |job|
current_job_id = job.args.first
job.delete if job_id == current_job_id
end
end
</pre>

== '''Files modified in Current Project''' ==

1. app/helpers/sign_up_sheet_helper.rb

2. app/models/due_date.rb

3. app/models/signed_up_team.rb

4. app/models/waitlist.rb

5. app/controllers/sign_up_sheet_controller.rb

6. app/helpers/deadline_helper.rb

7. app/mailers/mail_worker.rb

8. app/models/topic_due_date.rb

9. spec/controllers/sign_up_sheet_controller_spec.rb

== '''Test Plan''' ==

*'''RSpec:'''

::It was discussed that project does not require any Unit Test cases to be implemented as the changes are mostly visible in views.
::Of all the code changes, we have one controller modification that had a rspec test case. We modified the same as per the changes we made to the code.
::Further, all the additional methods created are private methods and so, we've not created any rspec test cases for those.

*'''UI Testing:'''
::'''[Problem 1]'''
::::1. Login to Expertiza using username: instructor6 & password: password.
::::2. Select an existing course/Create a course and an assignment under that course. Check 'Staggered deadline assignment?' for the assignment.
::::3. Create topics under the assignment.
::::4. Go to Assignment -> Topics. Click 'Show start/due date' at the bottom of the screen.
::::5. Enter 'Drop Topic Deadline' and hit 'Save start/due date'.
::::6. Reopen the assignment topic to cross verify if the deadline has been saved.

::Caveat: While creating an assignment the page would error out, but the assignment does get created, which is not related to any of the changes we have done. In case unable to do so, use 'MadeUp Problem' with Creation Date of 'Aug 30, 2017 - 9:03 AM'

::'''[Problem 2]'''
::::When a deadline is assigned for a topic, we create a job (identified with delayed_job_id) to be executed on the deadline date. The module for job and queue set-up on expertiza is not fully functional, and hence this cannot be tested directly. However, we have written the code for job scheduling, here's a snapshot of the due_dates table with the delayed_job_id populated:

::::[[File:DB Snapshot.png]]

== '''References''' ==

1. Expertiza on GITHUB: https://github.com/expertiza/expertiza

2. GitHub Project Repository Fork: https://github.com/sanveg-rane-13/expertiza

3. Demo link: https://drive.google.com/open?id=1jUwbeeiI2KaIH3l0C7FD0QHgBL6PVwF1

4. Pull Request: https://github.com/expertiza/expertiza/pull/1560

5. Expertiza project documentation wiki: http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1941._Issues_related_to_topic_deadlines