Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T09:52:06Z

Jcui23: /* Future Tasks */

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

==== aes_encrypt & aes_decrypt ====

These two methods are counterparts of each other. Instead of testing them separately, we test both in the '''same''' rspect context. We generate a random string mixed with numbers and then encrypt it with the method <code>aes_encrypt</code> and receive the tuple <code>[cipher, key, iv]</code>. Then the test invokes the <code>aes_decrypt</code> method with the tuples to retrieve the plain text. Finally, the test checks whether the decrypted text is the same as the original random data. The test covers both the <code>aes_encrypt</code> and <code>aes_decrpyt</code> methods in the reputation web service controller.

<pre>
context 'test aes_decrypt' do
it 'return the correct plain text' do
data = (0...8).map { (65 + rand(26)).chr }.join
cipher, key, iv = ReputationWebServiceController.new.aes_encrypt(data)
plain = ReputationWebServiceController.new.aes_decrypt(cipher, key, iv)
expect(plain).to eq(data)
end
end
</pre>

== Results ==
=== Testing Results ===

All of current examples passed the test. There are '''5''' examples in the <code>reputation_web_service_controller_spec.rb</code> file and there is no failure.

[[File:E2168-test-passed.png|600px]]

There are total '''10''' method in the <code>reputation_web_service_controller.rb</code> controller:

# action_allowed?
# db_query
# db_query_with_quiz_score
# json_generator
# client
# send_post_request
# rsa_public_key1
# rsa_private_key2
# aes_encrypt
# aes_decrypt

Our test covers '''7''' of them:

# action_allowed?
# db_query
# json_generator
# client
# send_post_request
# aes_encrypt
# aes_decrypt

The <code>rsa_public_key1</code> and <code>rsa_public_key2</code> methods requires the public key file
'''''public1.pem''''' and the private key file '''''private1.pem'''''. However, these two files are missing and therefore we could not test the related two methods.

=== Coverage ===
The test coverage boosts from '''0%''' to '''50.31%'''.

[[File:Test-coverage.png|1080px]]

There is no testing implemented for the <code>reputation_web_service_controller.rb</code> prior to our work and therefore the previous testing coverage is '''0%'''.

From the coverage report generated by the <code>Simplecov</code> gem, there are '''80''' lines covered by our test. As for the residual '''72''' lines of code, they are related to the public key file and deprecated functions for gathering data for a paper [https://doi.org/10.1109/FIE.2015.7344292] published in 2015.

The code <code>case</code> under the first <code>if</code> statement in the <code>send_post_request</code> method of the <code>reputation_web_service_controller.rb</code> should be removed. The team in Fall 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2064._Refactor_reputation_web_service_controller.rb#Issues_to_be_fixed] also mentioned the issues.

If the redundant method <code>send_post-request</code> in the controller <code>reputation_web_service_controller.rb</code> is commented out, the test coverage can achieve '''52.63%'''.

[[File:Coverage-more.png|1080px]]

=== Relevant Links ===
Github Repo : [https://github.com/HenryChen34/expertiza/tree/beta]

Pull request: [https://github.com/expertiza/expertiza/pull/2128]

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Current test cases are implemented only based on round 1 reputation scoring even though the assignment_1 and assignment_2 are subjective to be 2 rounds of review assignment. As the result of no accessibility to reputation web service, creation of round 2 object is meaningless because of the absence of round 1 reputation score. Therefore, future test cases need to stub the behavior of fulfilling assignment_questionnaire_1_2 (2nd round questionnaire) and assignment_questionnaire_2_2 (2nd round questionnaire) respectively assuming reputation web service available that time.
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T09:51:53Z

Jcui23: /* Future Tasks */

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T09:50:53Z

Jcui23: /* send_post_request */

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

==== aes_encrypt & aes_decrypt ====

These two methods are counterparts of each other. Instead of testing them separately, we test both in the '''same''' rspect context. We generate a random string mixed with numbers and then encrypt it with the method <code>aes_encrypt</code> and receive the tuple <code>[cipher, key, iv]</code>. Then the test invokes the <code>aes_decrypt</code> method with the tuples to retrieve the plain text. Finally, the test checks whether the decrypted text is the same as the original random data. The test covers both the <code>aes_encrypt</code> and <code>aes_decrpyt</code> methods in the reputation web service controller.

<pre>
context 'test aes_decrypt' do
it 'return the correct plain text' do
data = (0...8).map { (65 + rand(26)).chr }.join
cipher, key, iv = ReputationWebServiceController.new.aes_encrypt(data)
plain = ReputationWebServiceController.new.aes_decrypt(cipher, key, iv)
expect(plain).to eq(data)
end
end
</pre>

== Results ==
=== Testing Results ===

All of current examples passed the test. There are '''5''' examples in the <code>reputation_web_service_controller_spec.rb</code> file and there is no failure.

[[File:E2168-test-passed.png|600px]]

There are total '''10''' method in the <code>reputation_web_service_controller.rb</code> controller:

# action_allowed?
# db_query
# db_query_with_quiz_score
# json_generator
# client
# send_post_request
# rsa_public_key1
# rsa_private_key2
# aes_encrypt
# aes_decrypt

Our test covers '''7''' of them:

# action_allowed?
# db_query
# json_generator
# client
# send_post_request
# aes_encrypt
# aes_decrypt

The <code>rsa_public_key1</code> and <code>rsa_public_key2</code> methods requires the public key file
'''''public1.pem''''' and the private key file '''''private1.pem'''''. However, these two files are missing and therefore we could not test the related two methods.

=== Coverage ===
The test coverage boosts from '''0%''' to '''50.31%'''.

[[File:Test-coverage.png|1080px]]

There is no testing implemented for the <code>reputation_web_service_controller.rb</code> prior to our work and therefore the previous testing coverage is '''0%'''.

From the coverage report generated by the <code>Simplecov</code> gem, there are '''80''' lines covered by our test. As for the residual '''72''' lines of code, they are related to the public key file and deprecated functions for gathering data for a paper [https://doi.org/10.1109/FIE.2015.7344292] published in 2015.

The code <code>case</code> under the first <code>if</code> statement in the <code>send_post_request</code> method of the <code>reputation_web_service_controller.rb</code> should be removed. The team in Fall 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2064._Refactor_reputation_web_service_controller.rb#Issues_to_be_fixed] also mentioned the issues.

If the redundant method <code>send_post-request</code> in the controller <code>reputation_web_service_controller.rb</code> is commented out, the test coverage can achieve '''52.63%'''.

[[File:Coverage-more.png|1080px]]

=== Relevant Links ===
Github Repo : [https://github.com/HenryChen34/expertiza/tree/beta]

Pull request: [https://github.com/expertiza/expertiza/pull/2128]

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Current test cases are implemented only based on round 1 reputation scoring even though the assignment_1 and assignment_2 are subjective to be 2 rounds of review assignment. As the result of no accessibility to reputation web service, creation of round 2 object is meaningless because of the absence of round 1 reputation score. Therefore, future test cases need to stub the behavior of fulfilling assignment_questionnaire_1_2 (2nd round questionnaire) and assignment_questionnaire_2_2 (2nd round questionnaire) respectively assuming reputation web service available that time.
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T09:40:32Z

Jcui23: /* Testing Results */

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

All of current examples passed the test. There are '''5''' examples in the <code>reputation_web_service_controller_spec.rb</code> file and there is no failure.

[[File:E2168-test-passed.png|600px]]

There are total '''10''' method in the <code>reputation_web_service_controller.rb</code> controller:

# action_allowed?
# db_query
# db_query_with_quiz_score
# json_generator
# client
# send_post_request
# rsa_public_key1
# rsa_private_key2
# aes_encrypt
# aes_decrypt

Our test covers '''7''' of them:

# action_allowed?
# db_query
# json_generator
# client
# send_post_request
# aes_encrypt
# aes_decrypt

The <code>rsa_public_key1</code> and <code>rsa_public_key2</code> methods requires the public key file
'''''public1.pem''''' and the private key file '''''private1.pem'''''. However, these two files are missing and therefore we could not test the related two methods.

=== Coverage ===
The test coverage boosts from '''0%''' to '''50.31%'''.

[[File:Test-coverage.png|1080px]]

There is no testing implemented for the <code>reputation_web_service_controller.rb</code> prior to our work and therefore the previous testing coverage is '''0%'''.

From the coverage report generated by the <code>Simplecov</code> gem, there are '''80''' lines covered by our test. As for the residual '''72''' lines of code, they are related to the public key file and deprecated functions for gathering data for a paper [https://doi.org/10.1109/FIE.2015.7344292] published in 2015.

The code <code>case</code> under the first <code>if</code> statement in the <code>send_post_request</code> method of the <code>reputation_web_service_controller.rb</code> should be removed. The team in Fall 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2064._Refactor_reputation_web_service_controller.rb#Issues_to_be_fixed] also mentioned the issues.

If the redundant method <code>send_post-request</code> in the controller <code>reputation_web_service_controller.rb</code> is commented out, the test coverage can achieve '''52.63%'''.

[[File:Coverage-more.png|1080px]]

=== Relevant Links ===
Github Repo : [https://github.com/HenryChen34/expertiza/tree/beta]

Pull request: [https://github.com/expertiza/expertiza/pull/2128]

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Current test cases are implemented only based on round 1 reputation scoring even though the assignment_1 and assignment_2 are subjective to be 2 rounds of review assignment. As the result of no accessibility to reputation web service, creation of round 2 object is meaningless because of the absence of round 1 reputation score. Therefore, future test cases need to stub the behavior of fulfilling assignment_questionnaire_1_2 (2nd round questionnaire) and assignment_questionnaire_2_2 (2nd round questionnaire) respectively assuming reputation web service available that time.
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

File:E2168-test-passed.png

2021-11-30T09:37:29Z

Jcui23: Rspec running result for the Fall 2021 - E2168. Testing - Reputations

Rspec running result for the Fall 2021 - E2168. Testing - Reputations

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T09:34:44Z

Jcui23: /* Coverage */

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

All of our tests are passed.

There are total '''10''' method in the <code>reputation_web_service_controller.rb</code> controller:

# action_allowed?
# db_query
# db_query_with_quiz_score
# json_generator
# client
# send_post_request
# rsa_public_key1
# rsa_private_key2
# aes_encrypt
# aes_decrypt

Our test covers '''7''' of them:

# action_allowed?
# db_query
# json_generator
# client
# send_post_request
# aes_encrypt
# aes_decrypt

The <code>rsa_public_key1</code> and <code>rsa_public_key2</code> methods requires the public key file
'''''public1.pem''''' and the private key file '''''private1.pem'''''. However, these two files are missing and therefore we could not test the related two methods.

=== Coverage ===
The test coverage boosts from '''0%''' to '''50.31%'''.

[[File:Test-coverage.png|1080px]]

There is no testing implemented for the <code>reputation_web_service_controller.rb</code> prior to our work and therefore the previous testing coverage is '''0%'''.

From the coverage report generated by the <code>Simplecov</code> gem, there are '''80''' lines covered by our test. As for the residual '''72''' lines of code, they are related to the public key file and deprecated functions for gathering data for a paper [https://doi.org/10.1109/FIE.2015.7344292] published in 2015.

The code <code>case</code> under the first <code>if</code> statement in the <code>send_post_request</code> method of the <code>reputation_web_service_controller.rb</code> should be removed. The team in Fall 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2064._Refactor_reputation_web_service_controller.rb#Issues_to_be_fixed] also mentioned the issues.

If the redundant method <code>send_post-request</code> in the controller <code>reputation_web_service_controller.rb</code> is commented out, the test coverage can achieve '''52.63%'''.

[[File:Coverage-more.png|1080px]]

=== Relevant Links ===
Github Repo : [https://github.com/HenryChen34/expertiza/tree/beta]

Pull request: [https://github.com/expertiza/expertiza/pull/2128]

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Current test cases are implemented only based on round 1 reputation scoring even though the assignment_1 and assignment_2 are subjective to be 2 rounds of review assignment. As the result of no accessibility to reputation web service, creation of round 2 object is meaningless because of the absence of round 1 reputation score. Therefore, future test cases need to stub the behavior of fulfilling assignment_questionnaire_1_2 (2nd round questionnaire) and assignment_questionnaire_2_2 (2nd round questionnaire) respectively assuming reputation web service available that time.
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

File:Test-coverage.png

2021-11-30T09:30:42Z

Jcui23: Test coverage without commenting redundant code for CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

Test coverage without commenting redundant code for CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

File:Coverage-more.png

2021-11-30T09:27:25Z

Jcui23: Highest Test coverage report for CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations.

Highest Test coverage report for CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations.

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T09:25:52Z

Jcui23: /* Coverage */

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

All of our tests are passed.

There are total '''10''' method in the <code>reputation_web_service_controller.rb</code> controller:

# action_allowed?
# db_query
# db_query_with_quiz_score
# json_generator
# client
# send_post_request
# rsa_public_key1
# rsa_private_key2
# aes_encrypt
# aes_decrypt

Our test covers '''7''' of them:

# action_allowed?
# db_query
# json_generator
# client
# send_post_request
# aes_encrypt
# aes_decrypt

The <code>rsa_public_key1</code> and <code>rsa_public_key2</code> methods requires the public key file
'''''public1.pem''''' and the private key file '''''private1.pem'''''. However, these two files are missing and therefore we could not test the related two methods.

=== Coverage ===
The test coverage boosts from '''0%''' to '''52.63%'''.
[[File:Example.jpg]]

There is no testing implemented for the <code>reputation_web_service_controller.rb</code> prior to our work and therefore the previous testing coverage is '''0%'''.

From the coverage report generated by the <code>Simplecov</code> gem, there are '''127''' lines covered by our test. As for the residual '''87''' lines of code, they are related to the public key file and deprecated functions for gathering data for a paper [https://doi.org/10.1109/FIE.2015.7344292] published in 2015.

The method <code>send_post_request</code> inside the <code>reputation_web_service_controller.rb</code> should be removed. The team in Fall 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2064._Refactor_reputation_web_service_controller.rb#Issues_to_be_fixed] also mentioned the issues.

If the redundant method <code>send_post-request</code> in the controller <code>reputation_web_service_controller.rb</code> is commented out, the test coverage can achieve '''87.22%'''.

=== Relevant Links ===
Github Repo : [https://github.com/HenryChen34/expertiza/tree/beta]

Pull request: [https://github.com/expertiza/expertiza/pull/2128]

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Current test cases are implemented only based on round 1 reputation scoring even though the assignment_1 and assignment_2 are subjective to be 2 rounds of review assignment. As the result of no accessibility to reputation web service, creation of round 2 object is meaningless because of the absence of round 1 reputation score. Therefore, future test cases need to stub the behavior of fulfilling assignment_questionnaire_1_2 (2nd round questionnaire) and assignment_questionnaire_2_2 (2nd round questionnaire) respectively assuming reputation web service available that time.
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T09:24:16Z

Jcui23: /* Testing Results */

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

All of our tests are passed.

There are total '''10''' method in the <code>reputation_web_service_controller.rb</code> controller:

# action_allowed?
# db_query
# db_query_with_quiz_score
# json_generator
# client
# send_post_request
# rsa_public_key1
# rsa_private_key2
# aes_encrypt
# aes_decrypt

Our test covers '''7''' of them:

# action_allowed?
# db_query
# json_generator
# client
# send_post_request
# aes_encrypt
# aes_decrypt

The <code>rsa_public_key1</code> and <code>rsa_public_key2</code> methods requires the public key file
'''''public1.pem''''' and the private key file '''''private1.pem'''''. However, these two files are missing and therefore we could not test the related two methods.

=== Coverage ===
The test coverage boosts from '''0%''' to '''52.77%'''.

There is no testing implemented for the <code>reputation_web_service_controller.rb</code> prior to our work and therefore the previous testing coverage is '''0%'''.

From the coverage report generated by the <code>Simplecov</code> gem, there are '''127''' lines covered by our test. As for the residual '''87''' lines of code, they are related to the public key file and deprecated functions for gathering data for a paper [https://doi.org/10.1109/FIE.2015.7344292] published in 2015.

The method <code>send_post_request</code> inside the <code>reputation_web_service_controller.rb</code> should be removed. The team in Fall 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2064._Refactor_reputation_web_service_controller.rb#Issues_to_be_fixed] also mentioned the issues.

If the redundant method <code>send_post-request</code> in the controller <code>reputation_web_service_controller.rb</code> is commented out, the test coverage can achieve '''87.22%'''.

=== Relevant Links ===
Github Repo : [https://github.com/HenryChen34/expertiza/tree/beta]

Pull request: [https://github.com/expertiza/expertiza/pull/2128]

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Current test cases are implemented only based on round 1 reputation scoring even though the assignment_1 and assignment_2 are subjective to be 2 rounds of review assignment. As the result of no accessibility to reputation web service, creation of round 2 object is meaningless because of the absence of round 1 reputation score. Therefore, future test cases need to stub the behavior of fulfilling assignment_questionnaire_1_2 (2nd round questionnaire) and assignment_questionnaire_2_2 (2nd round questionnaire) respectively assuming reputation web service available that time.
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T09:15:38Z

Jcui23: /* Testing Results */

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

All of our tests are passed.

There are total '''10''' method in the <code>reputation_web_service_controller.rb</code> controller:

# action_allowed?
# db_query
# db_query_with_quiz_score
# json_generator
# client
# send_post_request
# rsa_public_key1
# rsa_private_key2
# aes_encrypt
# aes_decrypt

Our test covers '''7''' of them:

# action_allowed?
# db_query
# json_generator
# client
# send_post_request
# aes_encrypt
# aes_decrypt

The <code>rsa_public_key1</code> and <code>rsa_public_key2</code> methods requires the public key file
'''''public1.pem''''' and the private key file '''''private1.pem'''''. However, they are missing. We have consulted the author Yang Song about the missing file but got no response.

=== Coverage ===
The test coverage boosts from '''0%''' to '''52.77%'''.

There is no testing implemented for the <code>reputation_web_service_controller.rb</code> prior to our work and therefore the previous testing coverage is '''0%'''.

From the coverage report generated by the <code>Simplecov</code> gem, there are '''127''' lines covered by our test. As for the residual '''87''' lines of code, they are related to the public key file and deprecated functions for gathering data for a paper [https://doi.org/10.1109/FIE.2015.7344292] published in 2015.

The method <code>send_post_request</code> inside the <code>reputation_web_service_controller.rb</code> should be removed. The team in Fall 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2064._Refactor_reputation_web_service_controller.rb#Issues_to_be_fixed] also mentioned the issues.

If the redundant method <code>send_post-request</code> in the controller <code>reputation_web_service_controller.rb</code> is commented out, the test coverage can achieve '''87.22%'''.

=== Relevant Links ===
Github Repo : [https://github.com/HenryChen34/expertiza/tree/beta]

Pull request: [https://github.com/expertiza/expertiza/pull/2128]

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Current test cases are implemented only based on round 1 reputation scoring even though the assignment_1 and assignment_2 are subjective to be 2 rounds of review assignment. As the result of no accessibility to reputation web service, creation of round 2 object is meaningless because of the absence of round 1 reputation score. Therefore, future test cases need to stub the behavior of fulfilling assignment_questionnaire_1_2 (2nd round questionnaire) and assignment_questionnaire_2_2 (2nd round questionnaire) respectively assuming reputation web service available that time.
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T09:15:24Z

Jcui23: /* Testing Results */

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

All of our tests are passed.

There are total '''10''' method in the <code>reputation_web_service_controller.rb</code> controller:

# action_allowed?
# db_query
# db_query_with_quiz_score
# json_generator
# client
# send_post_request
# rsa_public_key1
# rsa_private_key2
# aes_encrypt
# aes_decrypt

Our test covers '''7'' of them:

# action_allowed?
# db_query
# json_generator
# client
# send_post_request
# aes_encrypt
# aes_decrypt

The <code>rsa_public_key1</code> and <code>rsa_public_key2</code> methods requires the public key file
'''''public1.pem''''' and the private key file '''''private1.pem'''''. However, they are missing. We have consulted the author Yang Song about the missing file but got no response.

=== Coverage ===
The test coverage boosts from '''0%''' to '''52.77%'''.

There is no testing implemented for the <code>reputation_web_service_controller.rb</code> prior to our work and therefore the previous testing coverage is '''0%'''.

From the coverage report generated by the <code>Simplecov</code> gem, there are '''127''' lines covered by our test. As for the residual '''87''' lines of code, they are related to the public key file and deprecated functions for gathering data for a paper [https://doi.org/10.1109/FIE.2015.7344292] published in 2015.

The method <code>send_post_request</code> inside the <code>reputation_web_service_controller.rb</code> should be removed. The team in Fall 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2064._Refactor_reputation_web_service_controller.rb#Issues_to_be_fixed] also mentioned the issues.

If the redundant method <code>send_post-request</code> in the controller <code>reputation_web_service_controller.rb</code> is commented out, the test coverage can achieve '''87.22%'''.

=== Relevant Links ===
Github Repo : [https://github.com/HenryChen34/expertiza/tree/beta]

Pull request: [https://github.com/expertiza/expertiza/pull/2128]

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Current test cases are implemented only based on round 1 reputation scoring even though the assignment_1 and assignment_2 are subjective to be 2 rounds of review assignment. As the result of no accessibility to reputation web service, creation of round 2 object is meaningless because of the absence of round 1 reputation score. Therefore, future test cases need to stub the behavior of fulfilling assignment_questionnaire_1_2 (2nd round questionnaire) and assignment_questionnaire_2_2 (2nd round questionnaire) respectively assuming reputation web service available that time.
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T06:04:50Z

Jcui23: /* Testing Results */

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

All of our tests are passed.

There are total '''10''' method in the <code>reputation_web_service_controller.rb</code> controller:

# action_allowed?
# db_query
# db_query_with_quiz_score
# json_generator
# client
# send_post_request
# rsa_public_key1
# rsa_private_key2
# aes_encrypt
# aes_decrypt

Our test covers '''3''' of them:

# db_query
# json_generator
# send_post_request

The <code>rsa_public_key1</code> and <code>rsa_public_key2</code> methods requires the public key file
'''''public1.pem''''' and the private key file '''''private1.pem'''''. However, they are missing. We have consulted the author Yang Song about the missing file but got no response. We cannot proceed on testing the other two related methods, <code>aes_encrypt</code> and <code>aes_decrypt</code>. The '''reputation''' algorithm generates reputation scores for students in the peer-review assignmentsm, however, the method <code>db_query_with_quiz_score</code> is used when the assignment type is <code>'quiz scores'</code>

=== Coverage ===
The test coverage boosts from '''0%''' to '''52.77%'''.

There is no testing implemented for the <code>reputation_web_service_controller.rb</code> prior to our work and therefore the previous testing coverage is '''0%'''.

From the coverage report generated by the <code>Simplecov</code> gem, there are '''127''' lines covered by our test. As for the residual '''87''' lines of code, they are related to the public key file and deprecated functions for gathering data for a paper [https://doi.org/10.1109/FIE.2015.7344292] published in 2015.

The method <code>send_post_request</code> inside the <code>reputation_web_service_controller.rb</code> should be removed. The team in Fall 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2064._Refactor_reputation_web_service_controller.rb#Issues_to_be_fixed] also mentioned the issues.

If the redundant method <code>send_post-request</code> in the controller <code>reputation_web_service_controller.rb</code> is commented out, the test coverage can achieve '''87.22%'''.

=== Relevant Links ===
Github Repo : [https://github.com/HenryChen34/expertiza/tree/beta]

Pull request: [https://github.com/expertiza/expertiza/pull/2128]

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Current test cases are implemented only based on round 1 reputation scoring even though the assignment_1 and assignment_2 are subjective to be 2 rounds of review assignment. As the result of no accessibility to reputation web service, creation of round 2 object is meaningless because of the absence of round 1 reputation score. Therefore, future test cases need to stub the behavior of fulfilling assignment_questionnaire_1_2 (2nd round questionnaire) and assignment_questionnaire_2_2 (2nd round questionnaire) respectively assuming reputation web service available that time.
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T05:29:00Z

Jcui23: /* Coverage */

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

=== Coverage ===
The test coverage boosts from '''0%''' to '''52.77%'''.

There is no testing implemented for the <code>reputation_web_service_controller.rb</code> prior to our work and therefore the previous testing coverage is '''0%'''.

From the coverage report generated by the <code>Simplecov</code> gem, there are '''127''' lines covered by our test. As for the residual '''87''' lines of code, they are related to the public key file and deprecated functions for gathering data for a paper [https://doi.org/10.1109/FIE.2015.7344292] published in 2015.

The method <code>send_post_request</code> inside the <code>reputation_web_service_controller.rb</code> should be removed. The team in Fall 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2064._Refactor_reputation_web_service_controller.rb#Issues_to_be_fixed] also mentioned the issues.

If the redundant method <code>send_post-request</code> in the controller <code>reputation_web_service_controller.rb</code> is commented out, the test coverage can achieve '''87.22%'''.

=== Relevant Links ===
Github Repo : [https://github.com/HenryChen34/expertiza/tree/beta]

Pull request: [https://github.com/expertiza/expertiza/pull/2128]

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Current test cases are implemented only based on round 1 reputation scoring even though the assignment_1 and assignment_2 are subjective to be 2 rounds of review assignment. As the result of no accessibility to reputation web service, creation of round 2 object is meaningless because of the absence of round 1 reputation score. Therefore, future test cases need to stub the behavior of fulfilling assignment_questionnaire_1_2 (2nd round questionnaire) and assignment_questionnaire_2_2 (2nd round questionnaire) respectively assuming reputation web service available that time.
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T04:57:19Z

Jcui23: /* Coverage */

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

=== Coverage ===
The test coverage boosts from '''0%''' to '''52.77%'''.

There is no testing implemented for the <code>reputation_web_service_controller.rb</code> prior to our work and therefore the previous testing coverage is '''0'''.

From the coverage report generated by the <code>Simplecov</code> gem, there are '''127''' lines covered by our test. As for the residual '''87''' lines of code, they are related to the public key file and deprecated functions for the

=== Relevant Links ===
Github Repo : [https://github.com/HenryChen34/expertiza/tree/beta]

Pull request: [https://github.com/expertiza/expertiza/pull/2128]

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Current test cases are implemented only based on round 1 reputation scoring even though the assignment_1 and assignment_2 are subjective to be 2 rounds of review assignment. As the result of no accessibility to reputation web service, creation of round 2 object is meaningless because of the absence of round 1 reputation score. Therefore, future test cases need to stub the behavior of fulfilling assignment_questionnaire_1_2 (2nd round questionnaire) and assignment_questionnaire_2_2 (2nd round questionnaire) respectively assuming reputation web service available that time.
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T04:41:59Z

Jcui23: /* Relevant Links */

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

=== Coverage ===

=== Relevant Links ===
Github Repo : [https://github.com/HenryChen34/expertiza/tree/beta]

Pull request: [https://github.com/expertiza/expertiza/pull/2128]

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Round 1 and round 2?
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T04:41:25Z

Jcui23: /* Relevant Links */

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

=== Coverage ===

=== Relevant Links ===
Github Repo : [https://github.com/HenryChen34/expertiza/tree/beta]
Pull request: [https://github.com/expertiza/expertiza/pull/2128]

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Round 1 and round 2?
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T04:28:57Z

Jcui23: /* Results */

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

=== Coverage ===

=== Relevant Links ===

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Round 1 and round 2?
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-04T03:08:14Z

Jcui23: /* Testing Objects */

== Project Overview ==

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score expectation

=== Files Involved ===
*reputation_web_service_controller_spec
*hamer_spec
*<del>lauw_spec (?)</del>

== Test Plan ==

=== Testing Objects ===
Following objects will be created and confined for the purpose of testing.

*[[assignments]] 
*[[submission_records]] 
*[[questionnaires]] 
*[[assignment_questionnaires]] 
*[[participants]] 
*[[teams]] 
*[[teams_users]] 
*[[users]] 
*[[responses]] 
*[[response_maps]]

The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

=== Relevant Methods ===
==== hamer_spec ====
*calculate_weighted_scores_and_reputation
This is the main method of Hamer algorithm. Reviewers inaccuracy will be calculated and updated from past average, then weight of the review will be adjusted based on the inaccuracy. We plan to create two contexts to test it as below: 1. When the review is the same from average, reviewer’s inaccuracy should be minimized and the review’s weight should be maximized; 2. When the review is furthest from average, reviewer’s inaccuracy should be maximized and the review’s weight should be minimized.
*converged?
This method ensures the final result of calculated reviews' weights is converged, as the algorithm requires. A simple fixture test can be done by giving the method a set of input data and compare it to expected result.
==== ReputationController_spec ====
*db_query
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect. 1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

*db_query_with_quiz_score
This is the special db query, call this method will return quiz scores with given assignment id. We will test this method with same logic as the first one.

*json_generator
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

*client
client method will fill many instance variables with corresponding class variables, we need to test this method with send_post_request.

*send_post_request
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result
in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the
update value in database.

== Results ==
=== Collaborators ===
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021

2021-11-04T02:50:01Z

Jcui23: /* Final Projects */

== OSS Projects ==

* [[CSC/ECE 517 Fall 2021 - E2117. Refactor questionaires_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2128. Refactor student_quizzes_controller.rb & late_policies_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2129. Refactor auth_controller.rb & password_retrieval_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2132. Add tests cases for review mapping helper.rb]]
* [[CSC/ECE 517 Fall 2021 - E2134. Write unit tests for admin_controller.rb and institution_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment]]
* [[CSC/ECE 517 Fall 2021 - E2142. Improve e-mail notifications]]
* [[CSC/ECE 517 Fall 2021 - E2133. Write tests for popup_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2120. Refactor reputation_web_service_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2139. Remove multiple topics at a time]]
* [[CSC/ECE 517 Fall 2021 - E2131. Improve assessment360_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2121. Refactor suggestion_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2122. Refactor impersonate_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2123. Refactor sign_up_sheet_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2126. Refactor account_request_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2124. Refactor review_mapping_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2125. Refactor review_mapping_helper.rb]]
* [[CSC/ECE 517 Fall 2021 - E2127. Refactor teams_controller]]
* [[CSC/ECE 517 Fall 2021 - E2130. Refactor submitted_content_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2140. Create new late policy successfully and fix Bank link]]
* [[CSC/ECE 517 Fall 2021 - E2141. OSS project Finklestein: Instructors & Institutions]]
* [[CSC/ECE 517 Fall 2021 - E2144. Refactor delayed mailer and scheduled task]]
* [[CSC/ECE 517 Fall 2021 - E2147. Role-based reviewing]]
* [[CSC/ECE 517 Fall 2021 - E2145. OSS Project Beige]]
* [[CSC/ECE 517 Fall 2021 - E2146. Introduce a Student View for instructors]]
* [[CSC/ECE 517 Fall 2021 - Refactor Evaluation of SQL Queries]]
* [[CSC/ECE 517 Fall 2021 - E2135. Email notification to reviewers and instructors]]
* [[CSC/ECE 517 Fall 2021 - E2149. Finish Github metrics integration - Reputations]]

== Final Projects ==
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2166._Testing_-_Scoring_%26_Grades#Description_about_project CSC/ECE 517 Fall 2021 - E2166. Testing - Scoring_and_Grades]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2150._Integrate_suggestion_detection_algorithm#Description_about_project CSC/ECE 517 Fall 2021 - E2150. Integrate suggestion detection algorithm]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2151._Allow_reviewers_to_bid_on_what_to_review CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2152._Revision_planning_tool#Description_about_project CSC/ECE 517 Fall 2021 - E2152. Revision_planning_tool]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2170._Testing_-_Response_Maps#Description_about_project CSC/ECE 517 Fall 2021 - E2170. Testing - Response Maps]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2153._Improving_search_facility_in_Expertiza#Description_about_project CSC/ECE 517 Fall 2021 - E2153. Improving search facility in Expertiza]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2162._Further_refactoring_and_improvement_of_review_mapping_helper CSC/ECE 517 Fall 2021 - E2162. Further refactoring and improvement of review mapping helper]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2163._Refactor_waitlist_functionality CSC/ECE 517 Fall 2021 - E2163. Refactor waitlist functionality]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2165._Fix_teammate-review_view CSC/ECE 517 Fall 2021 - E2165. Fix teammate review view ]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2155._Calibration_submissions_should_be_copied_along_with_calibration_assignments CSC/ECE 517 Fall 2021 - E2155. Calibration submissions should be copied along with calibration assignments]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2148._Completion/Progress_View CSC/ECE 517 Fall 2021 - E2148. Completion/Progress view]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2158._Grading_audit_trail CSC/ECE 517 Fall 2021 - E2158. Grading audit trail ]
*[https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2159._Expertiza_internationalization CSC/ECE 517 Fall 2021 - E2159. Expertiza internationalization]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2160._Implementing_and_testing_import_export_controllers#Description_about_project CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import and export controllers]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2164._Heatgrid_fixes_and_improvements#Description_about_project CSC/ECE 517 Fall 2021 - E2164. Heatgrid fixes and improvements]
* [[CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations]]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2126._Testing_-_Team_Related_Files#Description_about_project CSC/ECE 517 Fall 2021 - E2126. Testing - Team Related Files]

CSC/ECE 517 Fall 2021

2021-11-04T02:49:07Z

Jcui23: /* OSS Projects */

== OSS Projects ==

* [[CSC/ECE 517 Fall 2021 - E2117. Refactor questionaires_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2128. Refactor student_quizzes_controller.rb & late_policies_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2129. Refactor auth_controller.rb & password_retrieval_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2132. Add tests cases for review mapping helper.rb]]
* [[CSC/ECE 517 Fall 2021 - E2134. Write unit tests for admin_controller.rb and institution_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment]]
* [[CSC/ECE 517 Fall 2021 - E2142. Improve e-mail notifications]]
* [[CSC/ECE 517 Fall 2021 - E2133. Write tests for popup_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2120. Refactor reputation_web_service_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2139. Remove multiple topics at a time]]
* [[CSC/ECE 517 Fall 2021 - E2131. Improve assessment360_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2121. Refactor suggestion_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2122. Refactor impersonate_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2123. Refactor sign_up_sheet_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2126. Refactor account_request_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2124. Refactor review_mapping_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2125. Refactor review_mapping_helper.rb]]
* [[CSC/ECE 517 Fall 2021 - E2127. Refactor teams_controller]]
* [[CSC/ECE 517 Fall 2021 - E2130. Refactor submitted_content_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2140. Create new late policy successfully and fix Bank link]]
* [[CSC/ECE 517 Fall 2021 - E2141. OSS project Finklestein: Instructors & Institutions]]
* [[CSC/ECE 517 Fall 2021 - E2144. Refactor delayed mailer and scheduled task]]
* [[CSC/ECE 517 Fall 2021 - E2147. Role-based reviewing]]
* [[CSC/ECE 517 Fall 2021 - E2145. OSS Project Beige]]
* [[CSC/ECE 517 Fall 2021 - E2146. Introduce a Student View for instructors]]
* [[CSC/ECE 517 Fall 2021 - Refactor Evaluation of SQL Queries]]
* [[CSC/ECE 517 Fall 2021 - E2135. Email notification to reviewers and instructors]]
* [[CSC/ECE 517 Fall 2021 - E2149. Finish Github metrics integration - Reputations]]

== Final Projects ==
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2166._Testing_-_Scoring_%26_Grades#Description_about_project CSC/ECE 517 Fall 2021 - E2166. Testing - Scoring_and_Grades]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2150._Integrate_suggestion_detection_algorithm#Description_about_project CSC/ECE 517 Fall 2021 - E2150. Integrate suggestion detection algorithm]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2151._Allow_reviewers_to_bid_on_what_to_review CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2152._Revision_planning_tool#Description_about_project CSC/ECE 517 Fall 2021 - E2152. Revision_planning_tool]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2170._Testing_-_Response_Maps#Description_about_project CSC/ECE 517 Fall 2021 - E2170. Testing - Response Maps]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2153._Improving_search_facility_in_Expertiza#Description_about_project CSC/ECE 517 Fall 2021 - E2153. Improving search facility in Expertiza]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2162._Further_refactoring_and_improvement_of_review_mapping_helper CSC/ECE 517 Fall 2021 - E2162. Further refactoring and improvement of review mapping helper]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2163._Refactor_waitlist_functionality CSC/ECE 517 Fall 2021 - E2163. Refactor waitlist functionality]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2165._Fix_teammate-review_view CSC/ECE 517 Fall 2021 - E2165. Fix teammate review view ]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2155._Calibration_submissions_should_be_copied_along_with_calibration_assignments CSC/ECE 517 Fall 2021 - E2155. Calibration submissions should be copied along with calibration assignments]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2148._Completion/Progress_View CSC/ECE 517 Fall 2021 - E2148. Completion/Progress view]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2158._Grading_audit_trail CSC/ECE 517 Fall 2021 - E2158. Grading audit trail ]
*[https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2159._Expertiza_internationalization CSC/ECE 517 Fall 2021 - E2159. Expertiza internationalization]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2160._Implementing_and_testing_import_export_controllers#Description_about_project CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import and export controllers]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2164._Heatgrid_fixes_and_improvements#Description_about_project CSC/ECE 517 Fall 2021 - E2164. Heatgrid fixes and improvements]
* [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2021_-_E2126._Testing_-_Team_Related_Files#Description_about_project CSC/ECE 517 Fall 2021 - E2126. Testing - Team Related Files]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-10-27T03:41:19Z

Jcui23:

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-10-27T03:22:47Z

Jcui23: /* E2168. Testing - Reputations */

Initial Create of the page.

More update will be added.

CSC/ECE 517 Fall 2021

2021-10-27T03:21:23Z

Jcui23: /* OSS Projects */

CSC/ECE 517 Fall 2021 - E2135. Email notification to reviewers and instructors

2021-10-27T03:09:36Z

Jcui23: /* Create the page in the list of OSS Projects of CSC/ECE 517 Fall 2021*/

This page provides a description of the Expertiza based OSS project.
__TOC__

==About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an open-source project developed using the Ruby on Rails framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. The application allows students to submit and peer-review learning objects (articles, code, websites, etc)[1]. Expertiza supports submission across various document types, including the URLs and wiki pages.

==Problem Statement==

<li> When an assignment or review approaches its deadline on Expertiza, students initially should receive deadline reminder emails at a specific time before the deadline that the instructor has preconfigured. Lack of this functionality sometimes results in students missing their assignment submission deadlines and thus losing marks. Students should receive this type of deadline reminder email. So this amendment to the project involves adding an asynchronous deadline reminder mailer to the application.

==Modified Files==
* app/models/due_date.rb
* test/models/due_date.rb
* db/migrate/20210319212323_create_delayed_jobs.rb

==Pre-config==

We use delayed jobs library to deal with the delayed email.
We need run following command in order to install '''delayed_job''' binary executable.

rails generate delayed_job

Then we need to run

RAILS_ENV=development bin/delayed_job start

This will start a backend server to deal with the delayed jobs.
If we want to stop, just run following.

RAILS_ENV=development bin/delayed_job stop

==Implementation approach==

'''1) Reminder email sent when assignment or review is approaching deadline:'''
In the '''due_date.rb''' file, whenever a new due date is created or an existing due date is updated, the '''start_reminder''' method will be fired which will eventually be added to the delayed_job queue. This job will be executed at a preconfigured time before the deadline, where it will fire the method '''reminder''' which will be added to the delayed job queue by the handle_asynchronously method of gem '''delayed_job_active_record'''. Inside the reminder method, we will fetch three attributes - '''''assignment_id''''', '''''deadline_type''''', '''''due_at'''''. These three attributes will be used to decide the deadline type ( submission or review or teammate review ), fetch the participant email for that assignment, fetch the deadline threshold and at the end send the email reminder at a specified threshold time before the deadline which will contain all the details such as assignment names link to assignment and assignment type ( submission or review or teammate review).

'''2) Implement code:'''

def create_mailer_object
Mailer.new
end
def create_mailworker_object
MailWorker.new(self.parent_id, self.deadline_type, self.due_at)
end

# main function to start email reminder
def start_reminder
puts when_to_run_reminder
if self.changed?
@extra_param = self.parent_id.to_s + "," + self.deadline_type_id.to_s
# first deleted existed delayed jobs with same parent_id(which is assignment id actually)
Delayed::Job.where(extra_param: @extra_param).each do |job|
job.delete
end
# add a delayed job to the delayed job queue, the job will run at what when_to_run_reminder return
run_at_time = when_to_run_reminder
if run_at_time >= 0.seconds.from_now
self.delay(run_at: run_at_time, :extra_param => @extra_param).reminder
end
end
end
####
def reminder
deadline_text = self.deadline_type if %w[submission review].include? self.deadline_type
deadline_text = "Team Review" if self.deadline_type == 'metareview'
mail_worker = create_mailworker_object
email_reminder(mail_worker.find_participant_emails, deadline_text) unless mail_worker.find_participant_emails.empty?
end
####
def email_reminder(emails, deadline_type)
assignment = Assignment.find(self.parent_id)
subject = "Message regarding #{deadline_type} for assignment #{assignment.name}"
body = "This is a reminder to complete #{deadline_type} for assignment #{assignment.name}. \
Deadline is #{self.due_at}.If you have already done the #{deadline_type}, Please ignore this mail."
emails.each do |mail|
Rails.logger.info mail
end
Mailer.delayed_message(bcc: emails, subject: subject, body: body).deliver_now
end

# after duedate - threshold hours, then we can send the reminder email
def when_to_run_reminder
hours_before_deadline = self.threshold.hours
result = (self.due_at.in_time_zone - hours_before_deadline).to_dateti

==Automated Testing using RSPEC==

We have used Rspec for testing the delayed_job functionalities. Using the test-driven development (TDD) approach, we have added a Rspec test which checks whether the mail is enqueued upon the firing of the reminder method.

We have used Rspec for testing the delayed_job functionalities. Using the test-driven development (TDD) approach, we have added a Rspec test which checks whether the call for reminder method is enqueued upon the firing of the '''start_reminder''' method, with the proper scheduled execution time.

'''NOTE''': All the reminder mails except the ones for the reviewer are sent to '''''expertiza.development@gmail.com''''' ,as this is already set in the development environment.

==Steps to verify Functionality==

*Test Email
Email: expertiza_test123@outlook.com
Password: password98@

*Test Expertiza account
name/login: Email_Test_ID1
password: password98@

1. Change the existing assignment's due date
First login in as instructor6, and go to Manage → Assignments

[[File:manage_page.png|700px]]

Then click change edit in the right side menu under **Actions**.
Go to Due dates and change the **reminder** and **Due & Time**, and click save.

[[File:edit_assignment.png|700px]]

Then you should receive email at expertiza_test123@outlook.com in the appropriate time. For this example,16 hours before 2021/10/25 04:08 (US Eastern time zone), the system shall send a reminder email to expertiza_test123@outlook.com.

2. Create a new assignment with valid due date
Create a new Assignment

[[File:create_assignment.png|700px]]

Assign a valid due date and reminder hours

[[File:new_date.png|700px]]

Then add the previous user login to it as a participant

[[File:add_participate.png|700px]]

Because the participants was added later, so we need to change the due date once more in order to trigger the email notification jobs.

==Additional Links==

*Git forked repository link: https://github.com/CuiJinku/expertiza/tree/beta
*VCL deployment: http://152.7.176.117:8080/
*Pull request link: https://github.com/expertiza/expertiza/pull/2099

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[https://github.com/CuiJinku/expertiza GitHub Project Repository Fork]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://relishapp.com/rspec Rspec Documentation]

==Team==
[mailto:dli35@ncsu.edu Dong Li] 
[mailto:ldu2@ncsu.edu Liwen Du] 
[mailto:jcui23@ncsu.edu Jinku Cui]

CSC/ECE 517 Fall 2021

2021-10-27T03:08:29Z

Jcui23: /* OSS Projects */

CSC/ECE 517 Fall 2021 - E2135 Email notification to reviewers and instructors

2021-10-27T02:48:22Z

Jcui23: /* Pre-config */

CSC/ECE 517 Fall 2021 - E2135 Email notification to reviewers and instructors

2021-10-27T02:47:50Z

Jcui23: /* Additional Links */

This page provides a description of the Expertiza based OSS project.
__TOC__

==About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an open-source project developed using the Ruby on Rails framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. The application allows students to submit and peer-review learning objects (articles, code, websites, etc)[1]. Expertiza supports submission across various document types, including the URLs and wiki pages.

==Problem Statement==

<li> When an assignment or review approaches its deadline on Expertiza, students initially should receive deadline reminder emails at a specific time before the deadline that the instructor has preconfigured. Lack of this functionality sometimes results in students missing their assignment submission deadlines and thus losing marks. Students should receive this type of deadline reminder email. So this amendment to the project involves adding an asynchronous deadline reminder mailer to the application.

==Modified Files==
* app/models/due_date.rb
* test/models/due_date.rb
* db/migrate/20210319212323_create_delayed_jobs.rb

==Pre-config==

We use delayed jobs library to deal with the delayed email.
We need run following command in order to install delayed_job binary executable.

rails generate delayed_job

Then we need to run

RAILS_ENV=development bin/delayed_job start

This will start a backend server to deal with the delayed jobs.
If we want to stop, just run following.

RAILS_ENV=development bin/delayed_job stop

==Implementation approach==

'''1) Reminder email sent when assignment or review is approaching deadline:'''
In the '''due_date.rb''' file, whenever a new due date is created or an existing due date is updated, the '''start_reminder''' method will be fired which will eventually be added to the delayed_job queue. This job will be executed at a preconfigured time before the deadline, where it will fire the method '''reminder''' which will be added to the delayed job queue by the handle_asynchronously method of gem '''delayed_job_active_record'''. Inside the reminder method, we will fetch three attributes - '''''assignment_id''''', '''''deadline_type''''', '''''due_at'''''. These three attributes will be used to decide the deadline type ( submission or review or teammate review ), fetch the participant email for that assignment, fetch the deadline threshold and at the end send the email reminder at a specified threshold time before the deadline which will contain all the details such as assignment names link to assignment and assignment type ( submission or review or teammate review).

'''2) Implement code:'''

def create_mailer_object
Mailer.new
end
def create_mailworker_object
MailWorker.new(self.parent_id, self.deadline_type, self.due_at)
end

# main function to start email reminder
def start_reminder
puts when_to_run_reminder
if self.changed?
@extra_param = self.parent_id.to_s + "," + self.deadline_type_id.to_s
# first deleted existed delayed jobs with same parent_id(which is assignment id actually)
Delayed::Job.where(extra_param: @extra_param).each do |job|
job.delete
end
# add a delayed job to the delayed job queue, the job will run at what when_to_run_reminder return
run_at_time = when_to_run_reminder
if run_at_time >= 0.seconds.from_now
self.delay(run_at: run_at_time, :extra_param => @extra_param).reminder
end
end
end
####
def reminder
deadline_text = self.deadline_type if %w[submission review].include? self.deadline_type
deadline_text = "Team Review" if self.deadline_type == 'metareview'
mail_worker = create_mailworker_object
email_reminder(mail_worker.find_participant_emails, deadline_text) unless mail_worker.find_participant_emails.empty?
end
####
def email_reminder(emails, deadline_type)
assignment = Assignment.find(self.parent_id)
subject = "Message regarding #{deadline_type} for assignment #{assignment.name}"
body = "This is a reminder to complete #{deadline_type} for assignment #{assignment.name}. \
Deadline is #{self.due_at}.If you have already done the #{deadline_type}, Please ignore this mail."
emails.each do |mail|
Rails.logger.info mail
end
Mailer.delayed_message(bcc: emails, subject: subject, body: body).deliver_now
end

# after duedate - threshold hours, then we can send the reminder email
def when_to_run_reminder
hours_before_deadline = self.threshold.hours
result = (self.due_at.in_time_zone - hours_before_deadline).to_dateti

==Automated Testing using RSPEC==

We have used Rspec for testing the delayed_job functionalities. Using the test-driven development (TDD) approach, we have added a Rspec test which checks whether the mail is enqueued upon the firing of the reminder method.

We have used Rspec for testing the delayed_job functionalities. Using the test-driven development (TDD) approach, we have added a Rspec test which checks whether the call for reminder method is enqueued upon the firing of the '''start_reminder''' method, with the proper scheduled execution time.

'''NOTE''': All the reminder mails except the ones for the reviewer are sent to '''''expertiza.development@gmail.com''''' ,as this is already set in the development environment.

==Steps to verify Functionality==

*Test Email
Email: expertiza_test123@outlook.com
Password: password98@

*Test Expertiza account
name/login: Email_Test_ID1
password: password98@

1. Change the existing assignment's due date
First login in as instructor6, and go to Manage → Assignments

[[File:manage_page.png|700px]]

Then click change edit in the right side menu under **Actions**.
Go to Due dates and change the **reminder** and **Due & Time**, and click save.

[[File:edit_assignment.png|700px]]

Then you should receive email at expertiza_test123@outlook.com in the appropriate time. For this example,16 hours before 2021/10/25 04:08 (US Eastern time zone), the system shall send a reminder email to expertiza_test123@outlook.com.

2. Create a new assignment with valid due date
Create a new Assignment

[[File:create_assignment.png|700px]]

Assign a valid due date and reminder hours

[[File:new_date.png|700px]]

Then add the previous user login to it as a participant

[[File:add_participate.png|700px]]

Because the participants was added later, so we need to change the due date once more in order to trigger the email notification jobs.

==Additional Links==

*Git forked repository link: https://github.com/CuiJinku/expertiza/tree/beta
*VCL deployment: http://152.7.176.117:8080/
*Pull request link: https://github.com/expertiza/expertiza/pull/2099

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[https://github.com/CuiJinku/expertiza GitHub Project Repository Fork]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://relishapp.com/rspec Rspec Documentation]

==Team==
[mailto:dli35@ncsu.edu Dong Li] 
[mailto:ldu2@ncsu.edu Liwen Du] 
[mailto:jcui23@ncsu.edu Jinku Cui]

CSC/ECE 517 Fall 2021 - E2135 Email notification to reviewers and instructors

2021-10-27T02:46:51Z

Jcui23: /* Automated Testing using RSPEC */

This page provides a description of the Expertiza based OSS project.
__TOC__

==About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an open-source project developed using the Ruby on Rails framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. The application allows students to submit and peer-review learning objects (articles, code, websites, etc)[1]. Expertiza supports submission across various document types, including the URLs and wiki pages.

==Problem Statement==

<li> When an assignment or review approaches its deadline on Expertiza, students initially should receive deadline reminder emails at a specific time before the deadline that the instructor has preconfigured. Lack of this functionality sometimes results in students missing their assignment submission deadlines and thus losing marks. Students should receive this type of deadline reminder email. So this amendment to the project involves adding an asynchronous deadline reminder mailer to the application.

==Modified Files==
* app/models/due_date.rb
* test/models/due_date.rb
* db/migrate/20210319212323_create_delayed_jobs.rb

==Pre-config==

We use delayed jobs library to deal with the delayed email.
We need run following command in order to install delayed_job binary executable.

rails generate delayed_job

Then we need to run

RAILS_ENV=development bin/delayed_job start

This will start a backend server to deal with the delayed jobs.
If we want to stop, just run following.

RAILS_ENV=development bin/delayed_job stop

==Implementation approach==

'''1) Reminder email sent when assignment or review is approaching deadline:'''
In the '''due_date.rb''' file, whenever a new due date is created or an existing due date is updated, the '''start_reminder''' method will be fired which will eventually be added to the delayed_job queue. This job will be executed at a preconfigured time before the deadline, where it will fire the method '''reminder''' which will be added to the delayed job queue by the handle_asynchronously method of gem '''delayed_job_active_record'''. Inside the reminder method, we will fetch three attributes - '''''assignment_id''''', '''''deadline_type''''', '''''due_at'''''. These three attributes will be used to decide the deadline type ( submission or review or teammate review ), fetch the participant email for that assignment, fetch the deadline threshold and at the end send the email reminder at a specified threshold time before the deadline which will contain all the details such as assignment names link to assignment and assignment type ( submission or review or teammate review).

'''2) Implement code:'''

def create_mailer_object
Mailer.new
end
def create_mailworker_object
MailWorker.new(self.parent_id, self.deadline_type, self.due_at)
end

# main function to start email reminder
def start_reminder
puts when_to_run_reminder
if self.changed?
@extra_param = self.parent_id.to_s + "," + self.deadline_type_id.to_s
# first deleted existed delayed jobs with same parent_id(which is assignment id actually)
Delayed::Job.where(extra_param: @extra_param).each do |job|
job.delete
end
# add a delayed job to the delayed job queue, the job will run at what when_to_run_reminder return
run_at_time = when_to_run_reminder
if run_at_time >= 0.seconds.from_now
self.delay(run_at: run_at_time, :extra_param => @extra_param).reminder
end
end
end
####
def reminder
deadline_text = self.deadline_type if %w[submission review].include? self.deadline_type
deadline_text = "Team Review" if self.deadline_type == 'metareview'
mail_worker = create_mailworker_object
email_reminder(mail_worker.find_participant_emails, deadline_text) unless mail_worker.find_participant_emails.empty?
end
####
def email_reminder(emails, deadline_type)
assignment = Assignment.find(self.parent_id)
subject = "Message regarding #{deadline_type} for assignment #{assignment.name}"
body = "This is a reminder to complete #{deadline_type} for assignment #{assignment.name}. \
Deadline is #{self.due_at}.If you have already done the #{deadline_type}, Please ignore this mail."
emails.each do |mail|
Rails.logger.info mail
end
Mailer.delayed_message(bcc: emails, subject: subject, body: body).deliver_now
end

# after duedate - threshold hours, then we can send the reminder email
def when_to_run_reminder
hours_before_deadline = self.threshold.hours
result = (self.due_at.in_time_zone - hours_before_deadline).to_dateti

==Automated Testing using RSPEC==

We have used Rspec for testing the delayed_job functionalities. Using the test-driven development (TDD) approach, we have added a Rspec test which checks whether the mail is enqueued upon the firing of the reminder method.

We have used Rspec for testing the delayed_job functionalities. Using the test-driven development (TDD) approach, we have added a Rspec test which checks whether the call for reminder method is enqueued upon the firing of the '''start_reminder''' method, with the proper scheduled execution time.

'''NOTE''': All the reminder mails except the ones for the reviewer are sent to '''''expertiza.development@gmail.com''''' ,as this is already set in the development environment.

==Steps to verify Functionality==

*Test Email
Email: expertiza_test123@outlook.com
Password: password98@

*Test Expertiza account
name/login: Email_Test_ID1
password: password98@

1. Change the existing assignment's due date
First login in as instructor6, and go to Manage → Assignments

[[File:manage_page.png|700px]]

Then click change edit in the right side menu under **Actions**.
Go to Due dates and change the **reminder** and **Due & Time**, and click save.

[[File:edit_assignment.png|700px]]

Then you should receive email at expertiza_test123@outlook.com in the appropriate time. For this example,16 hours before 2021/10/25 04:08 (US Eastern time zone), the system shall send a reminder email to expertiza_test123@outlook.com.

2. Create a new assignment with valid due date
Create a new Assignment

[[File:create_assignment.png|700px]]

Assign a valid due date and reminder hours

[[File:new_date.png|700px]]

Then add the previous user login to it as a participant

[[File:add_participate.png|700px]]

Because the participants was added later, so we need to change the due date once more in order to trigger the email notification jobs.

==Additional Links==

*Git pull link: https://github.com/CuiJinku/expertiza/tree/beta
*VCL deployment: http://152.7.176.117:8080/

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[https://github.com/CuiJinku/expertiza GitHub Project Repository Fork]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://relishapp.com/rspec Rspec Documentation]

==Team==
[mailto:dli35@ncsu.edu Dong Li] 
[mailto:ldu2@ncsu.edu Liwen Du] 
[mailto:jcui23@ncsu.edu Jinku Cui]

CSC/ECE 517 Fall 2021 - E2135 Email notification to reviewers and instructors

2021-10-27T02:46:20Z

Jcui23: /* Automated Testing using RSPEC */

This page provides a description of the Expertiza based OSS project.
__TOC__

==About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an open-source project developed using the Ruby on Rails framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. The application allows students to submit and peer-review learning objects (articles, code, websites, etc)[1]. Expertiza supports submission across various document types, including the URLs and wiki pages.

==Problem Statement==

<li> When an assignment or review approaches its deadline on Expertiza, students initially should receive deadline reminder emails at a specific time before the deadline that the instructor has preconfigured. Lack of this functionality sometimes results in students missing their assignment submission deadlines and thus losing marks. Students should receive this type of deadline reminder email. So this amendment to the project involves adding an asynchronous deadline reminder mailer to the application.

==Modified Files==
* app/models/due_date.rb
* test/models/due_date.rb
* db/migrate/20210319212323_create_delayed_jobs.rb

==Pre-config==

We use delayed jobs library to deal with the delayed email.
We need run following command in order to install delayed_job binary executable.

rails generate delayed_job

Then we need to run

RAILS_ENV=development bin/delayed_job start

This will start a backend server to deal with the delayed jobs.
If we want to stop, just run following.

RAILS_ENV=development bin/delayed_job stop

==Implementation approach==

'''1) Reminder email sent when assignment or review is approaching deadline:'''
In the '''due_date.rb''' file, whenever a new due date is created or an existing due date is updated, the '''start_reminder''' method will be fired which will eventually be added to the delayed_job queue. This job will be executed at a preconfigured time before the deadline, where it will fire the method '''reminder''' which will be added to the delayed job queue by the handle_asynchronously method of gem '''delayed_job_active_record'''. Inside the reminder method, we will fetch three attributes - '''''assignment_id''''', '''''deadline_type''''', '''''due_at'''''. These three attributes will be used to decide the deadline type ( submission or review or teammate review ), fetch the participant email for that assignment, fetch the deadline threshold and at the end send the email reminder at a specified threshold time before the deadline which will contain all the details such as assignment names link to assignment and assignment type ( submission or review or teammate review).

'''2) Implement code:'''

def create_mailer_object
Mailer.new
end
def create_mailworker_object
MailWorker.new(self.parent_id, self.deadline_type, self.due_at)
end

# main function to start email reminder
def start_reminder
puts when_to_run_reminder
if self.changed?
@extra_param = self.parent_id.to_s + "," + self.deadline_type_id.to_s
# first deleted existed delayed jobs with same parent_id(which is assignment id actually)
Delayed::Job.where(extra_param: @extra_param).each do |job|
job.delete
end
# add a delayed job to the delayed job queue, the job will run at what when_to_run_reminder return
run_at_time = when_to_run_reminder
if run_at_time >= 0.seconds.from_now
self.delay(run_at: run_at_time, :extra_param => @extra_param).reminder
end
end
end
####
def reminder
deadline_text = self.deadline_type if %w[submission review].include? self.deadline_type
deadline_text = "Team Review" if self.deadline_type == 'metareview'
mail_worker = create_mailworker_object
email_reminder(mail_worker.find_participant_emails, deadline_text) unless mail_worker.find_participant_emails.empty?
end
####
def email_reminder(emails, deadline_type)
assignment = Assignment.find(self.parent_id)
subject = "Message regarding #{deadline_type} for assignment #{assignment.name}"
body = "This is a reminder to complete #{deadline_type} for assignment #{assignment.name}. \
Deadline is #{self.due_at}.If you have already done the #{deadline_type}, Please ignore this mail."
emails.each do |mail|
Rails.logger.info mail
end
Mailer.delayed_message(bcc: emails, subject: subject, body: body).deliver_now
end

# after duedate - threshold hours, then we can send the reminder email
def when_to_run_reminder
hours_before_deadline = self.threshold.hours
result = (self.due_at.in_time_zone - hours_before_deadline).to_dateti

==Automated Testing using RSPEC==

We have used Rspec for testing the delayed_job functionalities. Using the test-driven development (TDD) approach, we have added a Rspec test which checks whether the mail is enqueued upon the firing of the reminder method.

We have used Rspec for testing the delayed_job functionalities. Using the test-driven development (TDD) approach, we have added a Rspec test which checks whether the call for reminder method is enqueued upon the firing of the '''start_reminder''' method, with the proper scheduled execution time.

NOTE: All the reminder mails except the ones for the reviewer are sent to expertiza.development@gmail.com ,as this is already set in the development environment.

==Steps to verify Functionality==

*Test Email
Email: expertiza_test123@outlook.com
Password: password98@

*Test Expertiza account
name/login: Email_Test_ID1
password: password98@

1. Change the existing assignment's due date
First login in as instructor6, and go to Manage → Assignments

[[File:manage_page.png|700px]]

Then click change edit in the right side menu under **Actions**.
Go to Due dates and change the **reminder** and **Due & Time**, and click save.

[[File:edit_assignment.png|700px]]

Then you should receive email at expertiza_test123@outlook.com in the appropriate time. For this example,16 hours before 2021/10/25 04:08 (US Eastern time zone), the system shall send a reminder email to expertiza_test123@outlook.com.

2. Create a new assignment with valid due date
Create a new Assignment

[[File:create_assignment.png|700px]]

Assign a valid due date and reminder hours

[[File:new_date.png|700px]]

Then add the previous user login to it as a participant

[[File:add_participate.png|700px]]

Because the participants was added later, so we need to change the due date once more in order to trigger the email notification jobs.

==Additional Links==

*Git pull link: https://github.com/CuiJinku/expertiza/tree/beta
*VCL deployment: http://152.7.176.117:8080/

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[https://github.com/CuiJinku/expertiza GitHub Project Repository Fork]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://relishapp.com/rspec Rspec Documentation]

==Team==
[mailto:dli35@ncsu.edu Dong Li] 
[mailto:ldu2@ncsu.edu Liwen Du] 
[mailto:jcui23@ncsu.edu Jinku Cui]

CSC/ECE 517 Fall 2021 - E2135 Email notification to reviewers and instructors

2021-10-27T02:45:42Z

Jcui23: /* Implementation approach */

This page provides a description of the Expertiza based OSS project.
__TOC__

==About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an open-source project developed using the Ruby on Rails framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. The application allows students to submit and peer-review learning objects (articles, code, websites, etc)[1]. Expertiza supports submission across various document types, including the URLs and wiki pages.

==Problem Statement==

<li> When an assignment or review approaches its deadline on Expertiza, students initially should receive deadline reminder emails at a specific time before the deadline that the instructor has preconfigured. Lack of this functionality sometimes results in students missing their assignment submission deadlines and thus losing marks. Students should receive this type of deadline reminder email. So this amendment to the project involves adding an asynchronous deadline reminder mailer to the application.

==Modified Files==
* app/models/due_date.rb
* test/models/due_date.rb
* db/migrate/20210319212323_create_delayed_jobs.rb

==Pre-config==

We use delayed jobs library to deal with the delayed email.
We need run following command in order to install delayed_job binary executable.

rails generate delayed_job

Then we need to run

RAILS_ENV=development bin/delayed_job start

This will start a backend server to deal with the delayed jobs.
If we want to stop, just run following.

RAILS_ENV=development bin/delayed_job stop

==Implementation approach==

'''1) Reminder email sent when assignment or review is approaching deadline:'''
In the '''due_date.rb''' file, whenever a new due date is created or an existing due date is updated, the '''start_reminder''' method will be fired which will eventually be added to the delayed_job queue. This job will be executed at a preconfigured time before the deadline, where it will fire the method '''reminder''' which will be added to the delayed job queue by the handle_asynchronously method of gem '''delayed_job_active_record'''. Inside the reminder method, we will fetch three attributes - '''''assignment_id''''', '''''deadline_type''''', '''''due_at'''''. These three attributes will be used to decide the deadline type ( submission or review or teammate review ), fetch the participant email for that assignment, fetch the deadline threshold and at the end send the email reminder at a specified threshold time before the deadline which will contain all the details such as assignment names link to assignment and assignment type ( submission or review or teammate review).

'''2) Implement code:'''

def create_mailer_object
Mailer.new
end
def create_mailworker_object
MailWorker.new(self.parent_id, self.deadline_type, self.due_at)
end

# main function to start email reminder
def start_reminder
puts when_to_run_reminder
if self.changed?
@extra_param = self.parent_id.to_s + "," + self.deadline_type_id.to_s
# first deleted existed delayed jobs with same parent_id(which is assignment id actually)
Delayed::Job.where(extra_param: @extra_param).each do |job|
job.delete
end
# add a delayed job to the delayed job queue, the job will run at what when_to_run_reminder return
run_at_time = when_to_run_reminder
if run_at_time >= 0.seconds.from_now
self.delay(run_at: run_at_time, :extra_param => @extra_param).reminder
end
end
end
####
def reminder
deadline_text = self.deadline_type if %w[submission review].include? self.deadline_type
deadline_text = "Team Review" if self.deadline_type == 'metareview'
mail_worker = create_mailworker_object
email_reminder(mail_worker.find_participant_emails, deadline_text) unless mail_worker.find_participant_emails.empty?
end
####
def email_reminder(emails, deadline_type)
assignment = Assignment.find(self.parent_id)
subject = "Message regarding #{deadline_type} for assignment #{assignment.name}"
body = "This is a reminder to complete #{deadline_type} for assignment #{assignment.name}. \
Deadline is #{self.due_at}.If you have already done the #{deadline_type}, Please ignore this mail."
emails.each do |mail|
Rails.logger.info mail
end
Mailer.delayed_message(bcc: emails, subject: subject, body: body).deliver_now
end

# after duedate - threshold hours, then we can send the reminder email
def when_to_run_reminder
hours_before_deadline = self.threshold.hours
result = (self.due_at.in_time_zone - hours_before_deadline).to_dateti

==Automated Testing using RSPEC==

We have used Rspec for testing the delayed_job functionalities. Using the test-driven development (TDD) approach, we have added a Rspec test which checks whether the mail is enqueued upon the firing of the reminder method.

We have used Rspec for testing the delayed_job functionalities. Using the test-driven development (TDD) approach, we have added a Rspec test which checks whether the call for reminder method is enqueued upon the firing of the start_reminder method, with the proper scheduled execution time.

NOTE: All the reminder mails except the ones for the reviewer are sent to expertiza.development@gmail.com ,as this is already set in the development environment.

==Steps to verify Functionality==

*Test Email
Email: expertiza_test123@outlook.com
Password: password98@

*Test Expertiza account
name/login: Email_Test_ID1
password: password98@

1. Change the existing assignment's due date
First login in as instructor6, and go to Manage → Assignments

[[File:manage_page.png|700px]]

Then click change edit in the right side menu under **Actions**.
Go to Due dates and change the **reminder** and **Due & Time**, and click save.

[[File:edit_assignment.png|700px]]

Then you should receive email at expertiza_test123@outlook.com in the appropriate time. For this example,16 hours before 2021/10/25 04:08 (US Eastern time zone), the system shall send a reminder email to expertiza_test123@outlook.com.

2. Create a new assignment with valid due date
Create a new Assignment

[[File:create_assignment.png|700px]]

Assign a valid due date and reminder hours

[[File:new_date.png|700px]]

Then add the previous user login to it as a participant

[[File:add_participate.png|700px]]

Because the participants was added later, so we need to change the due date once more in order to trigger the email notification jobs.

==Additional Links==

*Git pull link: https://github.com/CuiJinku/expertiza/tree/beta
*VCL deployment: http://152.7.176.117:8080/

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[https://github.com/CuiJinku/expertiza GitHub Project Repository Fork]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://relishapp.com/rspec Rspec Documentation]

==Team==
[mailto:dli35@ncsu.edu Dong Li] 
[mailto:ldu2@ncsu.edu Liwen Du] 
[mailto:jcui23@ncsu.edu Jinku Cui]

CSC/ECE 517 Fall 2021 - E2135 Email notification to reviewers and instructors

2021-10-27T02:20:39Z

Jcui23: /* Steps to verify Functionality */

This page provides a description of the Expertiza based OSS project.
__TOC__

==About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an open-source project developed using the Ruby on Rails framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. The application allows students to submit and peer-review learning objects (articles, code, websites, etc)[1]. Expertiza supports submission across various document types, including the URLs and wiki pages.

==Problem Statement==

<li> When an assignment or review approaches its deadline on Expertiza, students initially should receive deadline reminder emails at a specific time before the deadline that the instructor has preconfigured. Lack of this functionality sometimes results in students missing their assignment submission deadlines and thus losing marks. Students should receive this type of deadline reminder email. So this amendment to the project involves adding an asynchronous deadline reminder mailer to the application.

==Modified Files==
* app/models/due_date.rb
* test/models/due_date.rb
* db/migrate/20210319212323_create_delayed_jobs.rb

==Pre-config==

We use delayed jobs library to deal with the delayed email.
We need run following command in order to install delayed_job binary executable.

rails generate delayed_job

Then we need to run

RAILS_ENV=development bin/delayed_job start

This will start a backend server to deal with the delayed jobs.
If we want to stop, just run following.

RAILS_ENV=development bin/delayed_job stop

==Implementation approach==

'''1) Reminder email sent when assignment or review is approaching deadline:'''
In the due_date.rb file, whenever a new due date is created or an existing due date is updated, the 'start_reminder' method will be fired which will eventually be added to the delayed_job queue. This job will be executed at a preconfigured time before deadline, where it will fire the method 'reminder' which will be added to the delayed job queue by the handle_asynchronously method of gem 'delayed_job_active_record'. Inside the reminder method, we will fetch three attributes - assignment_id, deadline_type, due_at. These three attributes will be used to decide the deadline type ( submission or review or teammate review ), fetch the participant email for that assignment, fetch the deadline threshold and at the end send the email reminder at a specified threshold time before the deadline which will contain all the details such as assignment names link to assignment and assignment type ( submission or review or teammate review).

'''2) Implement code:'''

def create_mailer_object
Mailer.new
end
def create_mailworker_object
MailWorker.new(self.parent_id, self.deadline_type, self.due_at)
end

# main function to start email reminder
def start_reminder
puts when_to_run_reminder
if self.changed?
@extra_param = self.parent_id.to_s + "," + self.deadline_type_id.to_s
# first deleted existed delayed jobs with same parent_id(which is assignment id actually)
Delayed::Job.where(extra_param: @extra_param).each do |job|
job.delete
end
# add a delayed job to the delayed job queue, the job will run at what when_to_run_reminder return
run_at_time = when_to_run_reminder
if run_at_time >= 0.seconds.from_now
self.delay(run_at: run_at_time, :extra_param => @extra_param).reminder
end
end
end
####
def reminder
deadline_text = self.deadline_type if %w[submission review].include? self.deadline_type
deadline_text = "Team Review" if self.deadline_type == 'metareview'
mail_worker = create_mailworker_object
email_reminder(mail_worker.find_participant_emails, deadline_text) unless mail_worker.find_participant_emails.empty?
end
####
def email_reminder(emails, deadline_type)
assignment = Assignment.find(self.parent_id)
subject = "Message regarding #{deadline_type} for assignment #{assignment.name}"
body = "This is a reminder to complete #{deadline_type} for assignment #{assignment.name}. \
Deadline is #{self.due_at}.If you have already done the #{deadline_type}, Please ignore this mail."
emails.each do |mail|
Rails.logger.info mail
end
Mailer.delayed_message(bcc: emails, subject: subject, body: body).deliver_now
end

# after duedate - threshold hours, then we can send the reminder email
def when_to_run_reminder
hours_before_deadline = self.threshold.hours
result = (self.due_at.in_time_zone - hours_before_deadline).to_dateti

==Automated Testing using RSPEC==

We have used Rspec for testing the delayed_job functionalities. Using the test-driven development (TDD) approach, we have added a Rspec test which checks whether the mail is enqueued upon the firing of the reminder method.

We have used Rspec for testing the delayed_job functionalities. Using the test-driven development (TDD) approach, we have added a Rspec test which checks whether the call for reminder method is enqueued upon the firing of the start_reminder method, with the proper scheduled execution time.

NOTE: All the reminder mails except the ones for the reviewer are sent to expertiza.development@gmail.com ,as this is already set in the development environment.

==Steps to verify Functionality==

*Test Email
Email: expertiza_test123@outlook.com
Password: password98@

*Test Expertiza account
name/login: Email_Test_ID1
password: password98@

1. Change the existing assignment's due date
First login in as instructor6, and go to Manage → Assignments

[[File:manage_page.png|700px]]

Then click change edit in the right side menu under **Actions**.
Go to Due dates and change the **reminder** and **Due & Time**, and click save.

[[File:edit_assignment.png|700px]]

Then you should receive email at expertiza_test123@outlook.com in the appropriate time. For this example,16 hours before 2021/10/25 04:08 (US Eastern time zone), the system shall send a reminder email to expertiza_test123@outlook.com.

2. Create a new assignment with valid due date
Create a new Assignment

[[File:create_assignment.png|700px]]

Assign a valid due date and reminder hours

[[File:new_date.png|700px]]

Then add the previous user login to it as a participant

[[File:add_participate.png|700px]]

Because the participants was added later, so we need to change the due date once more in order to trigger the email notification jobs.

==Additional Links==

*Git pull link: https://github.com/CuiJinku/expertiza/tree/beta
*VCL deployment: http://152.7.176.117:8080/

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[https://github.com/CuiJinku/expertiza GitHub Project Repository Fork]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://relishapp.com/rspec Rspec Documentation]

==Team==
[mailto:dli35@ncsu.edu Dong Li] 
[mailto:ldu2@ncsu.edu Liwen Du] 
[mailto:jcui23@ncsu.edu Jinku Cui]

CSC/ECE 517 Fall 2021 - E2135 Email notification to reviewers and instructors

2021-10-27T02:04:31Z

Jcui23: /* Correct links to be new VCL*/

This page provides a description of the Expertiza based OSS project.
__TOC__

==About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an open-source project developed using the Ruby on Rails framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. The application allows students to submit and peer-review learning objects (articles, code, websites, etc)[1]. Expertiza supports submission across various document types, including the URLs and wiki pages.

==Problem Statement==

<li> When an assignment or review approaches its deadline on Expertiza, students initially should receive deadline reminder emails at a specific time before the deadline that the instructor has preconfigured. Lack of this functionality sometimes results in students missing their assignment submission deadlines and thus losing marks. Students should receive this type of deadline reminder email. So this amendment to the project involves adding an asynchronous deadline reminder mailer to the application.

==Modified Files==
* app/models/due_date.rb
* test/models/due_date.rb
* db/migrate/20210319212323_create_delayed_jobs.rb

==Pre-config==

We use delayed jobs library to deal with the delayed email.
We need run following command in order to install delayed_job binary executable.

rails generate delayed_job

Then we need to run

RAILS_ENV=development bin/delayed_job start

This will start a backend server to deal with the delayed jobs.
If we want to stop, just run following.

RAILS_ENV=development bin/delayed_job stop

==Implementation approach==

'''1) Reminder email sent when assignment or review is approaching deadline:'''
In the due_date.rb file, whenever a new due date is created or an existing due date is updated, the 'start_reminder' method will be fired which will eventually be added to the delayed_job queue. This job will be executed at a preconfigured time before deadline, where it will fire the method 'reminder' which will be added to the delayed job queue by the handle_asynchronously method of gem 'delayed_job_active_record'. Inside the reminder method, we will fetch three attributes - assignment_id, deadline_type, due_at. These three attributes will be used to decide the deadline type ( submission or review or teammate review ), fetch the participant email for that assignment, fetch the deadline threshold and at the end send the email reminder at a specified threshold time before the deadline which will contain all the details such as assignment names link to assignment and assignment type ( submission or review or teammate review).

'''2) Implement code:'''

def create_mailer_object
Mailer.new
end
def create_mailworker_object
MailWorker.new(self.parent_id, self.deadline_type, self.due_at)
end

# main function to start email reminder
def start_reminder
puts when_to_run_reminder
if self.changed?
@extra_param = self.parent_id.to_s + "," + self.deadline_type_id.to_s
# first deleted existed delayed jobs with same parent_id(which is assignment id actually)
Delayed::Job.where(extra_param: @extra_param).each do |job|
job.delete
end
# add a delayed job to the delayed job queue, the job will run at what when_to_run_reminder return
run_at_time = when_to_run_reminder
if run_at_time >= 0.seconds.from_now
self.delay(run_at: run_at_time, :extra_param => @extra_param).reminder
end
end
end
####
def reminder
deadline_text = self.deadline_type if %w[submission review].include? self.deadline_type
deadline_text = "Team Review" if self.deadline_type == 'metareview'
mail_worker = create_mailworker_object
email_reminder(mail_worker.find_participant_emails, deadline_text) unless mail_worker.find_participant_emails.empty?
end
####
def email_reminder(emails, deadline_type)
assignment = Assignment.find(self.parent_id)
subject = "Message regarding #{deadline_type} for assignment #{assignment.name}"
body = "This is a reminder to complete #{deadline_type} for assignment #{assignment.name}. \
Deadline is #{self.due_at}.If you have already done the #{deadline_type}, Please ignore this mail."
emails.each do |mail|
Rails.logger.info mail
end
Mailer.delayed_message(bcc: emails, subject: subject, body: body).deliver_now
end

# after duedate - threshold hours, then we can send the reminder email
def when_to_run_reminder
hours_before_deadline = self.threshold.hours
result = (self.due_at.in_time_zone - hours_before_deadline).to_dateti

==Automated Testing using RSPEC==

We have used Rspec for testing the delayed_job functionalities. Using the test-driven development (TDD) approach, we have added a Rspec test which checks whether the mail is enqueued upon the firing of the reminder method.

We have used Rspec for testing the delayed_job functionalities. Using the test-driven development (TDD) approach, we have added a Rspec test which checks whether the call for reminder method is enqueued upon the firing of the start_reminder method, with the proper scheduled execution time.

NOTE: All the reminder mails except the ones for the reviewer are sent to expertiza.development@gmail.com ,as this is already set in the development environment.

==Steps to verify Functionality==

*Test Email
Email: [expertiza_test123@outlook.com](mailto:expertiza_test123@outlook.com)
Password: password98@

*Test Expertiza account
name/login: Email_Test_ID1
password: password98@

1. Change the existing assignment's due date
First login in as instructor6, and go to Manage → Assignments

[[File:manage_page.png|700px]]

Then click change edit in the right side menu under **Actions**.
Go to Due dates and change the **reminder** and **Due & Time**, and click save.

[[File:edit_assignment.png|700px]]

Then you should receive email at expertiza_test123@outlook.com in the appropriate time. For this example,16 hours before 2021/10/25 04:08 (US Eastern time zone), the system shall send a reminder email to expertiza_test123@outlook.com.

2. Create a new assignment with valid due date
Create a new Assignment

[[File:create_assignment.png|700px]]

Assign a valid due date and reminder hours

[[File:new_date.png|700px]]

Then add the previous user login to it as a participant

[[File:add_participate.png|700px]]

Because the participants was added later, so we need to change the due date once more in order to trigger the email notification jobs.

==Additional Links==

*Git pull link: https://github.com/CuiJinku/expertiza/tree/beta
*VCL deployment: http://152.7.176.117:8080/

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[https://github.com/CuiJinku/expertiza GitHub Project Repository Fork]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://relishapp.com/rspec Rspec Documentation]

==Team==
[mailto:dli35@ncsu.edu Dong Li] 
[mailto:ldu2@ncsu.edu Liwen Du] 
[mailto:jcui23@ncsu.edu Jinku Cui]

CSC/ECE 517 Fall 2021 - E2135 Email notification to reviewers and instructors

2021-10-27T02:03:27Z

Jcui23: /* delete the TODO comment */

This page provides a description of the Expertiza based OSS project.
__TOC__

==About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an open-source project developed using the Ruby on Rails framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. The application allows students to submit and peer-review learning objects (articles, code, websites, etc)[1]. Expertiza supports submission across various document types, including the URLs and wiki pages.

==Problem Statement==

<li> When an assignment or review approaches its deadline on Expertiza, students initially should receive deadline reminder emails at a specific time before the deadline that the instructor has preconfigured. Lack of this functionality sometimes results in students missing their assignment submission deadlines and thus losing marks. Students should receive this type of deadline reminder email. So this amendment to the project involves adding an asynchronous deadline reminder mailer to the application.

==Modified Files==
* app/models/due_date.rb
* test/models/due_date.rb
* db/migrate/20210319212323_create_delayed_jobs.rb

==Pre-config==

We use delayed jobs library to deal with the delayed email.
We need run following command in order to install delayed_job binary executable.

rails generate delayed_job

Then we need to run

RAILS_ENV=development bin/delayed_job start

This will start a backend server to deal with the delayed jobs.
If we want to stop, just run following.

RAILS_ENV=development bin/delayed_job stop

==Implementation approach==

'''1) Reminder email sent when assignment or review is approaching deadline:'''
In the due_date.rb file, whenever a new due date is created or an existing due date is updated, the 'start_reminder' method will be fired which will eventually be added to the delayed_job queue. This job will be executed at a preconfigured time before deadline, where it will fire the method 'reminder' which will be added to the delayed job queue by the handle_asynchronously method of gem 'delayed_job_active_record'. Inside the reminder method, we will fetch three attributes - assignment_id, deadline_type, due_at. These three attributes will be used to decide the deadline type ( submission or review or teammate review ), fetch the participant email for that assignment, fetch the deadline threshold and at the end send the email reminder at a specified threshold time before the deadline which will contain all the details such as assignment names link to assignment and assignment type ( submission or review or teammate review).

'''2) Implement code:'''

def create_mailer_object
Mailer.new
end
def create_mailworker_object
MailWorker.new(self.parent_id, self.deadline_type, self.due_at)
end

# main function to start email reminder
def start_reminder
puts when_to_run_reminder
if self.changed?
@extra_param = self.parent_id.to_s + "," + self.deadline_type_id.to_s
# first deleted existed delayed jobs with same parent_id(which is assignment id actually)
Delayed::Job.where(extra_param: @extra_param).each do |job|
job.delete
end
# add a delayed job to the delayed job queue, the job will run at what when_to_run_reminder return
run_at_time = when_to_run_reminder
if run_at_time >= 0.seconds.from_now
self.delay(run_at: run_at_time, :extra_param => @extra_param).reminder
end
end
end
####
def reminder
deadline_text = self.deadline_type if %w[submission review].include? self.deadline_type
deadline_text = "Team Review" if self.deadline_type == 'metareview'
mail_worker = create_mailworker_object
email_reminder(mail_worker.find_participant_emails, deadline_text) unless mail_worker.find_participant_emails.empty?
end
####
def email_reminder(emails, deadline_type)
assignment = Assignment.find(self.parent_id)
subject = "Message regarding #{deadline_type} for assignment #{assignment.name}"
body = "This is a reminder to complete #{deadline_type} for assignment #{assignment.name}. \
Deadline is #{self.due_at}.If you have already done the #{deadline_type}, Please ignore this mail."
emails.each do |mail|
Rails.logger.info mail
end
Mailer.delayed_message(bcc: emails, subject: subject, body: body).deliver_now
end

# after duedate - threshold hours, then we can send the reminder email
def when_to_run_reminder
hours_before_deadline = self.threshold.hours
result = (self.due_at.in_time_zone - hours_before_deadline).to_dateti

==Automated Testing using RSPEC==

We have used Rspec for testing the delayed_job functionalities. Using the test-driven development (TDD) approach, we have added a Rspec test which checks whether the mail is enqueued upon the firing of the reminder method.

We have used Rspec for testing the delayed_job functionalities. Using the test-driven development (TDD) approach, we have added a Rspec test which checks whether the call for reminder method is enqueued upon the firing of the start_reminder method, with the proper scheduled execution time.

NOTE: All the reminder mails except the ones for the reviewer are sent to expertiza.development@gmail.com ,as this is already set in the development environment.

==Steps to verify Functionality==

*Test Email
Email: [expertiza_test123@outlook.com](mailto:expertiza_test123@outlook.com)
Password: password98@

*Test Expertiza account
name/login: Email_Test_ID1
password: password98@

1. Change the existing assignment's due date
First login in as instructor6, and go to Manage → Assignments

[[File:manage_page.png|700px]]

Then click change edit in the right side menu under **Actions**.
Go to Due dates and change the **reminder** and **Due & Time**, and click save.

[[File:edit_assignment.png|700px]]

Then you should receive email at expertiza_test123@outlook.com in the appropriate time. For this example,16 hours before 2021/10/25 04:08 (US Eastern time zone), the system shall send a reminder email to expertiza_test123@outlook.com.

2. Create a new assignment with valid due date
Create a new Assignment

[[File:create_assignment.png|700px]]

Assign a valid due date and reminder hours

[[File:new_date.png|700px]]

Then add the previous user login to it as a participant

[[File:add_participate.png|700px]]

Because the participants was added later, so we need to change the due date once more in order to trigger the email notification jobs.

==Additional Links==

*Git pull link: https://github.com/CuiJinku/expertiza/tree/beta
*VCL deployment: http://152.7.98.122:8080/

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[https://github.com/CuiJinku/expertiza GitHub Project Repository Fork]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://relishapp.com/rspec Rspec Documentation]

==Team==
[mailto:dli35@ncsu.edu Dong Li] 
[mailto:ldu2@ncsu.edu Liwen Du] 
[mailto:jcui23@ncsu.edu Jinku Cui]