Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-12-01T04:05:08Z

Hchen34: format

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

==== aes_encrypt & aes_decrypt ====

These two methods are counterparts of each other. Instead of testing them separately, we test both in the '''same''' rspect context. We generate a random string mixed with numbers and then encrypt it with the method <code>aes_encrypt</code> and receive the tuple <code>[cipher, key, iv]</code>. Then the test invokes the <code>aes_decrypt</code> method with the tuples to retrieve the plain text. Finally, the test checks whether the decrypted text is the same as the original random data. The test covers both the <code>aes_encrypt</code> and <code>aes_decrpyt</code> methods in the reputation web service controller.

<pre>
context 'test aes_decrypt' do
it 'return the correct plain text' do
data = (0...8).map { (65 + rand(26)).chr }.join
cipher, key, iv = ReputationWebServiceController.new.aes_encrypt(data)
plain = ReputationWebServiceController.new.aes_decrypt(cipher, key, iv)
expect(plain).to eq(data)
end
end
</pre>

== Results ==
=== Testing Results ===

All of current examples passed the test. There are '''5''' examples in the <code>reputation_web_service_controller_spec.rb</code> file and there is no failure.

[[File:E2168-test-passed.png|600px]]

There are total '''10''' method in the <code>reputation_web_service_controller.rb</code> controller:

# action_allowed?
# db_query
# db_query_with_quiz_score
# json_generator
# client
# send_post_request
# rsa_public_key1
# rsa_private_key2
# aes_encrypt
# aes_decrypt

Our test covers '''7''' of them:

# action_allowed?
# db_query
# json_generator
# client
# send_post_request
# aes_encrypt
# aes_decrypt

The <code>rsa_public_key1</code> and <code>rsa_public_key2</code> methods requires the public key file
'''''public1.pem''''' and the private key file '''''private1.pem'''''. However, these two files are missing and therefore we could not test the related two methods.

=== Coverage ===
The test coverage boosts from '''0%''' to '''50.31%'''.

[[File:Test-coverage.png|1080px]]

There is no testing implemented for the <code>reputation_web_service_controller.rb</code> prior to our work and therefore the previous testing coverage is '''0%'''.

From the coverage report generated by the <code>Simplecov</code> gem, there are '''80''' lines covered by our test. As for the residual '''72''' lines of code, they are related to the public key file and deprecated functions for gathering data for a paper [https://doi.org/10.1109/FIE.2015.7344292] published in 2015.

The code <code>case</code> under the first <code>if</code> statement in the <code>send_post_request</code> method of the <code>reputation_web_service_controller.rb</code> should be removed. The team in Fall 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2064._Refactor_reputation_web_service_controller.rb#Issues_to_be_fixed] also mentioned the issues.

If the redundant method <code>send_post-request</code> in the controller <code>reputation_web_service_controller.rb</code> is commented out, the test coverage can achieve '''52.63%'''.

[[File:Coverage-more.png|1080px]]

=== Relevant Links ===
#[https://github.com/HenryChen34/expertiza/tree/beta Github Repo]
#[https://github.com/expertiza/expertiza/pull/2128 Pull Request]
#[https://youtu.be/sL0xQTol61w Demo Video]

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed.
# Implement reputation score correctness test for both Lauw's and Hamer's Algorithm assuming reputation web service available in the future.
# Current test cases are implemented only based on round 1 reputation scoring even though the assignment_1 and assignment_2 are subjective to be 2 rounds of review assignment. As the result of no accessibility to reputation web service, creation of round 2 object is meaningless because of the absence of round 1 reputation score. Therefore, future test cases need to stub the behavior of fulfilling assignment_questionnaire_1_2 (2nd round questionnaire) and assignment_questionnaire_2_2 (2nd round questionnaire) respectively assuming reputation web service available that time.
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-12-01T04:03:28Z

Hchen34:

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

==== aes_encrypt & aes_decrypt ====

These two methods are counterparts of each other. Instead of testing them separately, we test both in the '''same''' rspect context. We generate a random string mixed with numbers and then encrypt it with the method <code>aes_encrypt</code> and receive the tuple <code>[cipher, key, iv]</code>. Then the test invokes the <code>aes_decrypt</code> method with the tuples to retrieve the plain text. Finally, the test checks whether the decrypted text is the same as the original random data. The test covers both the <code>aes_encrypt</code> and <code>aes_decrpyt</code> methods in the reputation web service controller.

<pre>
context 'test aes_decrypt' do
it 'return the correct plain text' do
data = (0...8).map { (65 + rand(26)).chr }.join
cipher, key, iv = ReputationWebServiceController.new.aes_encrypt(data)
plain = ReputationWebServiceController.new.aes_decrypt(cipher, key, iv)
expect(plain).to eq(data)
end
end
</pre>

== Results ==
=== Testing Results ===

All of current examples passed the test. There are '''5''' examples in the <code>reputation_web_service_controller_spec.rb</code> file and there is no failure.

[[File:E2168-test-passed.png|600px]]

There are total '''10''' method in the <code>reputation_web_service_controller.rb</code> controller:

# action_allowed?
# db_query
# db_query_with_quiz_score
# json_generator
# client
# send_post_request
# rsa_public_key1
# rsa_private_key2
# aes_encrypt
# aes_decrypt

Our test covers '''7''' of them:

# action_allowed?
# db_query
# json_generator
# client
# send_post_request
# aes_encrypt
# aes_decrypt

The <code>rsa_public_key1</code> and <code>rsa_public_key2</code> methods requires the public key file
'''''public1.pem''''' and the private key file '''''private1.pem'''''. However, these two files are missing and therefore we could not test the related two methods.

=== Coverage ===
The test coverage boosts from '''0%''' to '''50.31%'''.

[[File:Test-coverage.png|1080px]]

There is no testing implemented for the <code>reputation_web_service_controller.rb</code> prior to our work and therefore the previous testing coverage is '''0%'''.

From the coverage report generated by the <code>Simplecov</code> gem, there are '''80''' lines covered by our test. As for the residual '''72''' lines of code, they are related to the public key file and deprecated functions for gathering data for a paper [https://doi.org/10.1109/FIE.2015.7344292] published in 2015.

The code <code>case</code> under the first <code>if</code> statement in the <code>send_post_request</code> method of the <code>reputation_web_service_controller.rb</code> should be removed. The team in Fall 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2064._Refactor_reputation_web_service_controller.rb#Issues_to_be_fixed] also mentioned the issues.

If the redundant method <code>send_post-request</code> in the controller <code>reputation_web_service_controller.rb</code> is commented out, the test coverage can achieve '''52.63%'''.

[[File:Coverage-more.png|1080px]]

=== Relevant Links ===
Github Repo : [https://github.com/HenryChen34/expertiza/tree/beta]

Pull request: [https://github.com/expertiza/expertiza/pull/2128]

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed.
# Implement reputation score correctness test for both Lauw's and Hamer's Algorithm assuming reputation web service available in the future.
# Current test cases are implemented only based on round 1 reputation scoring even though the assignment_1 and assignment_2 are subjective to be 2 rounds of review assignment. As the result of no accessibility to reputation web service, creation of round 2 object is meaningless because of the absence of round 1 reputation score. Therefore, future test cases need to stub the behavior of fulfilling assignment_questionnaire_1_2 (2nd round questionnaire) and assignment_questionnaire_2_2 (2nd round questionnaire) respectively assuming reputation web service available that time.
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]
#[https://youtu.be/sL0xQTol61w Demo Video]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-12-01T04:02:53Z

Hchen34: add demo video link

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

==== aes_encrypt & aes_decrypt ====

These two methods are counterparts of each other. Instead of testing them separately, we test both in the '''same''' rspect context. We generate a random string mixed with numbers and then encrypt it with the method <code>aes_encrypt</code> and receive the tuple <code>[cipher, key, iv]</code>. Then the test invokes the <code>aes_decrypt</code> method with the tuples to retrieve the plain text. Finally, the test checks whether the decrypted text is the same as the original random data. The test covers both the <code>aes_encrypt</code> and <code>aes_decrpyt</code> methods in the reputation web service controller.

<pre>
context 'test aes_decrypt' do
it 'return the correct plain text' do
data = (0...8).map { (65 + rand(26)).chr }.join
cipher, key, iv = ReputationWebServiceController.new.aes_encrypt(data)
plain = ReputationWebServiceController.new.aes_decrypt(cipher, key, iv)
expect(plain).to eq(data)
end
end
</pre>

== Results ==
=== Testing Results ===

All of current examples passed the test. There are '''5''' examples in the <code>reputation_web_service_controller_spec.rb</code> file and there is no failure.

[[File:E2168-test-passed.png|600px]]

There are total '''10''' method in the <code>reputation_web_service_controller.rb</code> controller:

# action_allowed?
# db_query
# db_query_with_quiz_score
# json_generator
# client
# send_post_request
# rsa_public_key1
# rsa_private_key2
# aes_encrypt
# aes_decrypt

Our test covers '''7''' of them:

# action_allowed?
# db_query
# json_generator
# client
# send_post_request
# aes_encrypt
# aes_decrypt

The <code>rsa_public_key1</code> and <code>rsa_public_key2</code> methods requires the public key file
'''''public1.pem''''' and the private key file '''''private1.pem'''''. However, these two files are missing and therefore we could not test the related two methods.

=== Coverage ===
The test coverage boosts from '''0%''' to '''50.31%'''.

[[File:Test-coverage.png|1080px]]

There is no testing implemented for the <code>reputation_web_service_controller.rb</code> prior to our work and therefore the previous testing coverage is '''0%'''.

From the coverage report generated by the <code>Simplecov</code> gem, there are '''80''' lines covered by our test. As for the residual '''72''' lines of code, they are related to the public key file and deprecated functions for gathering data for a paper [https://doi.org/10.1109/FIE.2015.7344292] published in 2015.

The code <code>case</code> under the first <code>if</code> statement in the <code>send_post_request</code> method of the <code>reputation_web_service_controller.rb</code> should be removed. The team in Fall 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2064._Refactor_reputation_web_service_controller.rb#Issues_to_be_fixed] also mentioned the issues.

If the redundant method <code>send_post-request</code> in the controller <code>reputation_web_service_controller.rb</code> is commented out, the test coverage can achieve '''52.63%'''.

[[File:Coverage-more.png|1080px]]

=== Relevant Links ===
Github Repo : [https://github.com/HenryChen34/expertiza/tree/beta]

Pull request: [https://github.com/expertiza/expertiza/pull/2128]

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed.
# Implement reputation score correctness test for both Lauw's and Hamer's Algorithm assuming reputation web service available in the future.
# Current test cases are implemented only based on round 1 reputation scoring even though the assignment_1 and assignment_2 are subjective to be 2 rounds of review assignment. As the result of no accessibility to reputation web service, creation of round 2 object is meaningless because of the absence of round 1 reputation score. Therefore, future test cases need to stub the behavior of fulfilling assignment_questionnaire_1_2 (2nd round questionnaire) and assignment_questionnaire_2_2 (2nd round questionnaire) respectively assuming reputation web service available that time.
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]
#[https://youtu.be/sL0xQTol61w]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T03:38:40Z

Hchen34: add comment for db_query test

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
# create 5 answers for 5 related questions

create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect to see a data array return generated by the score given.
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

=== Coverage ===

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Round 1 and round 2?
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T03:32:43Z

Hchen34:

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
# assignment_questionnaire__<j> means assignment #I'd #j th round of review.
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

=== Coverage ===

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Round 1 and round 2?
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T03:29:51Z

Hchen34:

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

=== Coverage ===

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Round 1 and round 2?
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T03:29:04Z

Hchen34:

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_i_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means response of reviewer #j's to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means response of reviewer #j's to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
# create 5 answers for 5 related questions
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# choose hammer algorithm without expert grade(intructor's given grade)
params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# if it functions correctly, it will update the reviewer's reputation score according to the selected reputation algorithm.

# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

=== Coverage ===

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Round 1 and round 2?
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T03:28:02Z

Hchen34:

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_j_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means reviewer #j's review to reviewee #i.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means reviewer #j's review to reviewee #i.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

=== Coverage ===

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Round 1 and round 2?
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T03:24:30Z

Hchen34:

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_j_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means reviewer #1's review to reviewee #j.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means reviewer #1's review to reviewee #j.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

=== Coverage ===

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Round 1 and round 2?
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-30T03:23:49Z

Hchen34: Some comments for code.

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to set up assignments and reviews that would produce specific reputation scores, and test that the correct reputations are in fact being produced.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score using the Hamer's and Lauw's algorithm.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
# question_j_j means question #j in questionnaire #i.
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
# response_map__<j> means reviewer #1's review to reviewee #j.
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
# response__<j> means reviewer #1's review to reviewee #j.
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
# reivewer_1's review for reviewee_1: [1, 2, 3, 4, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

=== Coverage ===

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# Current setup is based on assignment_1, which uses lauw's algorithm to calculate reputation score. Future testing team need to setup similar object set for assignment_2, which uses hamer's algorithm.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. (Need to replace by the comment Professor mentioned in the meeting today)
# Round 1 and round 2?
# The db_query violates the DRY principle as it repetitively calculates sum of the assignment. Such sum calculation should be handled in the assignment.rb.

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-29T21:51:26Z

Hchen34:

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to write tests to verify Hamer's and Lauw’s algorithm from the reputation system.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score expectation on different reputation range of Hamer's and Lauw's algorithm with or without instructor score impact.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

#result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect(result).to eq([[2, 1, 100.0], [3, 1, 60.0], [4, 1, 20.0]])
result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have given the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

=== Coverage ===

=== Future Tasks ===
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# There is a lot of unused/commented code, which should be removed.
# Figure out what the code is doing and write appropriate comments for it.
# In the case of db_query, the name should say what it queries for. Also, this method not only queries, but calculates sums. Since each method should do only one thing, the code for calculating sums should be in another method. And there should be comments in the code!
#json_generator should be generate_json. There needs to be a method comment saying what the parameters are.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. send_post_request is 91 lines long, far too long.
#There is a password for a private key in the code (and the code is open-sourced!) It should be in the db instead.
#Fix spelling of “dimention”
#client is a bad method name; why is stuff being copied from class variables to instance variables?

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-29T21:49:02Z

Hchen34:

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to write tests to verify Hamer's and Lauw’s algorithm from the reputation system.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score expectation on different reputation range of Hamer's and Lauw's algorithm with or without instructor score impact.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.
 
The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = <target_assignment>.id ; 
reviewer_id = <target_reviewer>.id ; 
reviewee_id = <target_reviewee>.id ; 
Note: The response map is setup to determine the relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

#result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect(result).to eq([[2, 1, 100.0], [3, 1, 60.0], [4, 1, 20.0]])
result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have gave the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

=== Coverage ===

=== Future Tasks ===
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# There is a lot of unused/commented code, which should be removed.
# Figure out what the code is doing and write appropriate comments for it.
# In the case of db_query, the name should say what it queries for. Also, this method not only queries, but calculates sums. Since each method should do only one thing, the code for calculating sums should be in another method. And there should be comments in the code!
#json_generator should be generate_json. There needs to be a method comment saying what the parameters are.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. send_post_request is 91 lines long, far too long.
#There is a password for a private key in the code (and the code is open-sourced!) It should be in the db instead.
#Fix spelling of “dimention”
#client is a bad method name; why is stuff being copied from class variables to instance variables?

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-29T21:42:26Z

Hchen34: Explain why send_post_request was not tested

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to write tests to verify Hamer's and Lauw’s algorithm from the reputation system.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score expectation on different reputation range of Hamer's and Lauw's algorithm with or without instructor score impact.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
instructor_id = from_fixture; 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2, id: 4)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = assignment_id; 
reviewer_id = participants; 
reviewee_id = AssignmentTeam; 
Note: we will setup response map to determine relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

#result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect(result).to eq([[2, 1, 100.0], [3, 1, 60.0], [4, 1, 20.0]])
result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the update value in database.

Note that this method is not functioning due to the missing public & private key file for RSA encryption. The algorithm involving "expert grade" was also not implemented.
Thus, this method could not be properly tested. However we have gave the template to create tests for future use.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

=== Coverage ===

=== Future Tasks ===
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# There is a lot of unused/commented code, which should be removed.
# Figure out what the code is doing and write appropriate comments for it.
# In the case of db_query, the name should say what it queries for. Also, this method not only queries, but calculates sums. Since each method should do only one thing, the code for calculating sums should be in another method. And there should be comments in the code!
#json_generator should be generate_json. There needs to be a method comment saying what the parameters are.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. send_post_request is 91 lines long, far too long.
#There is a password for a private key in the code (and the code is open-sourced!) It should be in the db instead.
#Fix spelling of “dimention”
#client is a bad method name; why is stuff being copied from class variables to instance variables?

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-29T21:37:28Z

Hchen34: Added testing results and coverage category for future use

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to write tests to verify Hamer's and Lauw’s algorithm from the reputation system.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score expectation on different reputation range of Hamer's and Lauw's algorithm with or without instructor score impact.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
instructor_id = from_fixture; 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2, id: 4)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = assignment_id; 
reviewer_id = participants; 
reviewee_id = AssignmentTeam; 
Note: we will setup response map to determine relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

#result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect(result).to eq([[2, 1, 100.0], [3, 1, 60.0], [4, 1, 20.0]])
result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result
in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the
update value in database.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==
=== Testing Results ===

=== Coverage ===

=== Future Tasks ===
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# There is a lot of unused/commented code, which should be removed.
# Figure out what the code is doing and write appropriate comments for it.
# In the case of db_query, the name should say what it queries for. Also, this method not only queries, but calculates sums. Since each method should do only one thing, the code for calculating sums should be in another method. And there should be comments in the code!
#json_generator should be generate_json. There needs to be a method comment saying what the parameters are.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. send_post_request is 91 lines long, far too long.
#There is a password for a private key in the code (and the code is open-sourced!) It should be in the db instead.
#Fix spelling of “dimention”
#client is a bad method name; why is stuff being copied from class variables to instance variables?

== Collaborators ==
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-29T21:35:23Z

Hchen34:

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to write tests to verify Hamer's and Lauw’s algorithm from the reputation system.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score expectation on different reputation range of Hamer's and Lauw's algorithm with or without instructor score impact.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
instructor_id = from_fixture; 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2, id: 4)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = assignment_id; 
reviewer_id = participants; 
reviewee_id = AssignmentTeam; 
Note: we will setup response map to determine relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

#result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect(result).to eq([[2, 1, 100.0], [3, 1, 60.0], [4, 1, 20.0]])
result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result
in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the
update value in database.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==

=== Future Tasks ===
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# There is a lot of unused/commented code, which should be removed.
# Figure out what the code is doing and write appropriate comments for it.
# In the case of db_query, the name should say what it queries for. Also, this method not only queries, but calculates sums. Since each method should do only one thing, the code for calculating sums should be in another method. And there should be comments in the code!
#json_generator should be generate_json. There needs to be a method comment saying what the parameters are.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. send_post_request is 91 lines long, far too long.
#There is a password for a private key in the code (and the code is open-sourced!) It should be in the db instead.
#Fix spelling of “dimention”
#client is a bad method name; why is stuff being copied from class variables to instance variables?

=== Collaborators ===
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

=== References ===
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-29T21:33:08Z

Hchen34:

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to write tests to verify Hamer's and Lauw’s algorithm from the reputation system.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score expectation on different reputation range of Hamer's and Lauw's algorithm with or without instructor score impact.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
instructor_id = from_fixture; 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2, id: 4)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = assignment_id; 
reviewer_id = participants; 
reviewee_id = AssignmentTeam; 
Note: we will setup response map to determine relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

#result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect(result).to eq([[2, 1, 100.0], [3, 1, 60.0], [4, 1, 20.0]])
result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result
in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the
update value in database.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==

=== Future Tasks ===
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# There is a lot of unused/commented code, which should be removed.
# Figure out what the code is doing and write appropriate comments for it.
# In the case of db_query, the name should say what it queries for. Also, this method not only queries, but calculates sums. Since each method should do only one thing, the code for calculating sums should be in another method. And there should be comments in the code!
#json_generator should be generate_json. There needs to be a method comment saying what the parameters are.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. send_post_request is 91 lines long, far too long.
#There is a password for a private key in the code (and the code is open-sourced!) It should be in the db instead.
#Fix spelling of “dimention”
#client is a bad method name; why is stuff being copied from class variables to instance variables?

=== Collaborators ===
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

=== References ===
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-29T21:32:25Z

Hchen34:

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to write tests to verify Hamer's and Lauw’s algorithm from the reputation system.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score expectation on different reputation range of Hamer's and Lauw's algorithm with or without instructor score impact.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.

==== Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
instructor_id = from_fixture; 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2, id: 4)
</pre>

==== Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = assignment_id; 
reviewer_id = participants; 
reviewee_id = AssignmentTeam; 
Note: we will setup response map to determine relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

#result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect(result).to eq([[2, 1, 100.0], [3, 1, 60.0], [4, 1, 20.0]])
result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result
in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the
update value in database.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# There is a lot of unused/commented code, which should be removed.
# Figure out what the code is doing and write appropriate comments for it.
# In the case of db_query, the name should say what it queries for. Also, this method not only queries, but calculates sums. Since each method should do only one thing, the code for calculating sums should be in another method. And there should be comments in the code!
#json_generator should be generate_json. There needs to be a method comment saying what the parameters are.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. send_post_request is 91 lines long, far too long.
#There is a password for a private key in the code (and the code is open-sourced!) It should be in the db instead.
#Fix spelling of “dimention”
#client is a bad method name; why is stuff being copied from class variables to instance variables?

=== Collaborators ===
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-29T21:30:09Z

Hchen34:

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to write tests to verify Hamer's and Lauw’s algorithm from the reputation system.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score expectation on different reputation range of Hamer's and Lauw's algorithm with or without instructor score impact.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.

==== 1. Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== 2. Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
instructor_id = from_fixture; 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2, id: 4)
</pre>

==== 3. Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== 4. Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== 5. Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = assignment_id; 
reviewer_id = participants; 
reviewee_id = AssignmentTeam; 
Note: we will setup response map to determine relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

=== Relevant Methods ===
==== db_query ====
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect.
1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
expect(result).to eq([[2, 1, 60.0]])
end
end
</pre>

==== json_generator ====
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

#result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect(result).to eq([[2, 1, 100.0], [3, 1, 60.0], [4, 1, 20.0]])
result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end
end
</pre>

==== send_post_request ====
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result
in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the
update value in database.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end
</pre>

== Results ==

== Future Tasks ==
As our team could not obtain the public/private key pair to access the reputation web service, we were only able get to the step prior to sending the JSON to the web service of reputation algorithms. Therefore, future steps are required to test on reputation system.

# There is a lot of unused/commented code, which should be removed.
# Figure out what the code is doing and write appropriate comments for it.
# In the case of db_query, the name should say what it queries for. Also, this method not only queries, but calculates sums. Since each method should do only one thing, the code for calculating sums should be in another method. And there should be comments in the code!
#json_generator should be generate_json. There needs to be a method comment saying what the parameters are.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. send_post_request is 91 lines long, far too long.
#There is a password for a private key in the code (and the code is open-sourced!) It should be in the db instead.
#Fix spelling of “dimention”
#client is a bad method name; why is stuff being copied from class variables to instance variables?

=== Collaborators ===
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-29T21:23:18Z

Hchen34:

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to write tests to verify Hamer's and Lauw’s algorithm from the reputation system.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score expectation on different reputation range of Hamer's and Lauw's algorithm with or without instructor score impact.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Setup Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.

==== 1. Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== 2. Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
instructor_id = from_fixture; 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2, id: 4)
</pre>

==== 3. Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== 4. Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== 5. Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = assignment_id; 
reviewer_id = participants; 
reviewee_id = AssignmentTeam; 
Note: we will setup response map to determine relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

=== Relevant Methods ===
==== ReputationController_spec ====
*db_query
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect. 1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
expect(result).to eq([[2, 1, 60.0]])
end
end

</pre>

*json_generator
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

#result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect(result).to eq([[2, 1, 100.0], [3, 1, 60.0], [4, 1, 20.0]])
result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end

end
</pre>

*send_post_request
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result
in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the
update value in database.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end

</pre>

== Results ==

== Future Tasks ==
As our team was not able to obtain the public/private key pair to access the reputation web service, we
reputation_web_service_controller.rb is a fairly complex file. It contains a lot of methods that are long and hard to understand (for e.g. send_post_request). These methods need to be broken down into simpler and more specific methods that are easier to read/understand. Also, the few instances of code duplication that exist should be removed.

# There is a lot of unused/commented code, which should be removed.
# Figure out what the code is doing and write appropriate comments for it.
# In the case of db_query, the name should say what it queries for. Also, this method not only queries, but calculates sums. Since each method should do only one thing, the code for calculating sums should be in another method. And there should be comments in the code!
#json_generator should be generate_json. There needs to be a method comment saying what the parameters are.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. send_post_request is 91 lines long, far too long.
#There is a password for a private key in the code (and the code is open-sourced!) It should be in the db instead.
#Fix spelling of “dimention”
#client is a bad method name; why is stuff being copied from class variables to instance variables?

=== Collaborators ===
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-29T21:21:14Z

Hchen34: updates testing objects

== Project Overview ==

=== Introduction ===

<q>Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.</q> The reputation system is meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.
During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.
Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to write tests to verify Hamer's and Lauw’s algorithm from the reputation system.

=== System Design ===
The below is referenced from project E1625, which would give us the overall description of the reputation system. 
[[File:Diagram E1625.png|1000px]] 
There are two algorithms intended for use in calculation of the reputation values for participants.

There is a [https://expertiza.ncsu.edu/reputation_web_service/client web-service] (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the ''participants'' table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the [https://docs.google.com/viewer?url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FYang_Song36%2Fpublication%2F289528736_Pluggable_Reputation_Systems_for_Peer_Review_a_Web-Service_Approach%2Flinks%2F568ec8d008ae78cc05160aed.pdf%3FinViewer%3D0%26pdfJsDownload%3D0%26origin%3Dpublication_detail paper] on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”
:;Reputation range of Hamer’s algorithm is :
:red value < 0.5
:yellow value is >= 0.5 and <= 1
:orange value is > 1 and <= 1.5
:light green value is > 1.5 and <= 2
:green value is > 2

 
The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].
:;Reputation range of Lauw’s algorithm is
:red value is < 0.2
:yellow value is >= 0.2 and <= 0.4
:orange value is > 0.4 and <= 0.6
:light green value is > 0.6 and <= 0.8
:green value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score expectation on different reputation range of Hamer's and Lauw's algorithm with or without instructor score impact.

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Testing Objects ===
In order to implement testing on reputation, it is crucial to create sample reviews so that we could possibly obtain reputation score. During the kickoff meeting, our team defined four necessary steps to follow for the purpose of testing. Also, appropriate objects could be created and confined as discussed below.

==== 1. Setup Assignment ====
* Objects involved 
[[assignments]] 

* Essential Parameters to be configured 
submitter_count = 0; 
num_reviews = 5; 
num_reviewers = 5; 
num_reviews_allowed = 5; 
rounds_of_reviews = 2; 
reputation_algorithm = lauw/hamer; 
Note: Two assignment objects were created. Assignment_1 used lauw's algorithm, whereas assignment_2 used hamer's alogorithm. 

* Code Implemented 
<pre>
@assignment_1 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'lauw', id: 1)
@assignment_2 = create(:assignment, created_at: DateTime.now.in_time_zone - 13.day, submitter_count: 0, num_reviews: 3, num_reviewers: 5, num_reviews_allowed: 5, rounds_of_reviews: 2, reputation_algorithm: 'hamer', id: 2)
</pre>

==== 2. Setup Questionnaires(Rubrics) ====
* Objects involved 
[[questionnaires]] 
[[assignment_questionnaires]] 

* Essential Parameters to be configured 
instructor_id = from_fixture; 
min_question_score = 0; 
max_question_score = 5; 
type = ReviewQuestionnaire; 
Note: We will define the assignment with ReviewQuestionnaire type rubric. 

* Code Implemented 
<pre>
@questionnaire_1 = create(:questionnaire, min_question_score: 0, max_question_score: 5, type: 'ReviewQuestionnaire', id: 1)
@assignment_questionnaire_1_1 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_1_2 = create(:assignment_questionnaire, assignment_id: @assignment_1.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2)
@assignment_questionnaire_2_1 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 1)
@assignment_questionnaire_2_2 = create(:assignment_questionnaire, assignment_id: @assignment_2.id, questionnaire_id: @questionnaire_1.id, used_in_round: 2, id: 4)
</pre>

==== 3. Setup Questions under Questionnaires ====
* Objects involved 
[[questions]] 

* Code Implemented 
<pre>
@question_1_1 = create(:question, questionnaire_id: @questionnaire_1.id, id: 1)
@question_1_2 = create(:question, questionnaire_id: @questionnaire_1.id, id: 2)
@question_1_3 = create(:question, questionnaire_id: @questionnaire_1.id, id: 3)
@question_1_4 = create(:question, questionnaire_id: @questionnaire_1.id, id: 4)
@question_1_5 = create(:question, questionnaire_id: @questionnaire_1.id, id: 5)
</pre>

==== 4. Setup Reviewers and Reviewees ====
* Objects involved 
[[participants]] 
[[teams]] 

* Code Implemented 
Reviewers (Participant):
<pre>
@reviewer_1 = create(:participant, can_review: 1)
@reviewer_2 = create(:participant, can_review: 1)
@reviewer_3 = create(:participant, can_review: 1)
</pre>

Reviewees (Teams):
<pre>
@reviewee_1 = create(:assignment_team, assignment: @assignment)
@reviewee_2 = create(:assignment_team, assignment: @assignment)
@reviewee_3 = create(:assignment_team, assignment: @assignment)
</pre>

==== 5. Setup Responses ====
* Objects involved 
[[response_maps]] 
[[responses]] 

* Essential Parameters to be configured 
reviewed_object_id = assignment_id; 
reviewer_id = participants; 
reviewee_id = AssignmentTeam; 
Note: we will setup response map to determine relationship between reviewer and reviewee of an assignment. 

* Code Implemented 
Response_maps:
<pre>
@response_map_1_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_1.id)
@response_map_1_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_1.id)
@response_map_1_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_1.id)

@response_map_2_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_2.id)
@response_map_2_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_2.id)
@response_map_2_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_2.id)

@response_map_3_1 = create(:review_response_map, reviewer_id: @reviewer_1.id, reviewee_id: @reviewee_3.id)
@response_map_3_2 = create(:review_response_map, reviewer_id: @reviewer_2.id, reviewee_id: @reviewee_3.id)
@response_map_3_3 = create(:review_response_map, reviewer_id: @reviewer_3.id, reviewee_id: @reviewee_3.id)
</pre>

Responses:
<pre>
@response_1_1 = create(:response, is_submitted: true, map_id: @response_map_1_1.id)
@response_1_2 = create(:response, is_submitted: true, map_id: @response_map_1_2.id)
@response_1_3 = create(:response, is_submitted: true, map_id: @response_map_1_3.id)

@response_2_1 = create(:response, is_submitted: true, map_id: @response_map_2_1.id)
@response_2_2 = create(:response, is_submitted: true, map_id: @response_map_2_2.id)
@response_2_3 = create(:response, is_submitted: true, map_id: @response_map_2_3.id)

@response_3_1 = create(:response, is_submitted: true, map_id: @response_map_3_1.id)
@response_3_2 = create(:response, is_submitted: true, map_id: @response_map_3_2.id)
@response_3_3 = create(:response, is_submitted: true, map_id: @response_map_3_3.id)
</pre>

The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

=== Relevant Methods ===
==== ReputationController_spec ====
*db_query
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect. 1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

<pre>
context 'test db_query' do
it 'return average score' do
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 2)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 4)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)
result = ReputationWebServiceController.new.db_query(1, 1, false)
expect(result).to eq([[2, 1, 60.0]])
end
end

</pre>

*json_generator
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

<pre>
context 'test json_generator' do
it 'test 3 reviewer for one reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_2's review for reviewee_1: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_2.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_2.id, answer: 3)

# reivewer_3's review for reviewee_1: [1, 1, 1, 1, 1]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_3.id, answer: 1)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_3.id, answer: 1)

#result = ReputationWebServiceController.new.db_query(1, 1, false)
#expect(result).to eq([[2, 1, 100.0], [3, 1, 60.0], [4, 1, 20.0]])
result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq({"submission1"=>{"stu2"=>100.0, "stu3"=>60.0, "stu4"=>20.0}})
#repeat for different answers
end

it 'test same reviewer for different reviewee' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

# reivewer_1's review for reviewee_2: [3, 3, 3, 3, 3]
create(:answer, question_id: @question_1_1.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_2.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_3.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_4.id, response_id: @response_2_1.id, answer: 3)
create(:answer, question_id: @question_1_5.id, response_id: @response_2_1.id, answer: 3)

result = ReputationWebServiceController.new.json_generator(1, 0, 1)
expect(result).to eq("submission1"=>{"stu2"=>100.0}, "submission2"=>{"stu2"=>60.0})
#repeat for different answers
end

end
</pre>

*send_post_request
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result
in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the
update value in database.

<pre>
context 'test send_post_request' do
it 'failed because of no public key file' do
# reivewer_1's review for reviewee_1: [5, 5, 5, 5, 5]
create(:answer, question_id: @question_1_1.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_2.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_3.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_4.id, response_id: @response_1_1.id, answer: 5)
create(:answer, question_id: @question_1_5.id, response_id: @response_1_1.id, answer: 5)

params = {assignment_id: 1, round_num: 1, algorithm: 'hammer', checkbox: {expert_grade: "empty"}}
session = {user: build(:instructor, id: 1)}

expect(true).to eq(true)

# comment out because send_post_request method request public key file while this file is missing
# so at this time send_post_request is not functioning normally
# get :send_post_request, params, session
# expect(response).to redirect_to '/reputation_web_service/client'
end
end

</pre>

== Results ==

== Future Tasks ==
As our team was not able to obtain the public/private key pair to access the reputation web service, we
reputation_web_service_controller.rb is a fairly complex file. It contains a lot of methods that are long and hard to understand (for e.g. send_post_request). These methods need to be broken down into simpler and more specific methods that are easier to read/understand. Also, the few instances of code duplication that exist should be removed.

# There is a lot of unused/commented code, which should be removed.
# Figure out what the code is doing and write appropriate comments for it.
# In the case of db_query, the name should say what it queries for. Also, this method not only queries, but calculates sums. Since each method should do only one thing, the code for calculating sums should be in another method. And there should be comments in the code!
#json_generator should be generate_json. There needs to be a method comment saying what the parameters are.
# In send_post_request, there are references to specific assignments, such as 724, 735, and 756. They were put in to gather data for a paper published in 2015. They are no longer relevant and should be removed. send_post_request is 91 lines long, far too long.
#There is a password for a private key in the code (and the code is open-sourced!) It should be in the db instead.
#Fix spelling of “dimention”
#client is a bad method name; why is stuff being copied from class variables to instance variables?

=== Collaborators ===
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-09T02:22:40Z

Hchen34:

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-05T02:50:27Z

Hchen34:

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-05T02:50:04Z

Hchen34:

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-05T02:48:48Z

Hchen34:

== Project Overview ==

=== Introduction ===

Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system.
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to write tests to verify the reputation system.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score expectation

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Testing Objects ===
Following objects will be created and confined for the purpose of testing.

*[[assignments]] 
*[[submission_records]] 
*[[questionnaires]] 
*[[assignment_questionnaires]] 
*[[participants]] 
*[[teams]] 
*[[teams_users]] 
*[[users]] 
*[[responses]] 
*[[response_maps]]

The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

=== Relevant Methods ===
==== ReputationController_spec ====
*db_query
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect. 1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

*db_query_with_quiz_score
This is the special db query, call this method will return quiz scores with given assignment id. We will test this method with same logic as the first one.

*json_generator
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

*client
client method will fill many instance variables with corresponding class variables, we need to test this method with send_post_request.

*send_post_request
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result
in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the
update value in database.

== Results ==
=== Collaborators ===
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-05T02:48:30Z

Hchen34: added introduction to reputation system

== Project Overview ==

=== Introduction ===

Online peer-review systems are now in common use in higher education. They free the instructor and course staff from having to provide personally all the feedback that students receive on their work. However, if we want to assure that all students receive competent feedback, or even use peer-assigned grades, we need a way to judge which peer reviewers are most credible. The solution is to use a reputation system. 
Currently the reputation system is implemented in Expertiza through web-service, but there's no test written for it. Thus our goal is to write tests to verify the reputation system.

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score expectation

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Testing Objects ===
Following objects will be created and confined for the purpose of testing.

*[[assignments]] 
*[[submission_records]] 
*[[questionnaires]] 
*[[assignment_questionnaires]] 
*[[participants]] 
*[[teams]] 
*[[teams_users]] 
*[[users]] 
*[[responses]] 
*[[response_maps]]

The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

=== Relevant Methods ===
==== ReputationController_spec ====
*db_query
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect. 1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

*db_query_with_quiz_score
This is the special db query, call this method will return quiz scores with given assignment id. We will test this method with same logic as the first one.

*json_generator
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

*client
client method will fill many instance variables with corresponding class variables, we need to test this method with send_post_request.

*send_post_request
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result
in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the
update value in database.

== Results ==
=== Collaborators ===
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-05T02:37:32Z

Hchen34: remove hamer.rb related methods

== Project Overview ==

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score expectation

=== Files Involved ===
*reputation_web_service_controller_spec

== Test Plan ==

=== Testing Objects ===
Following objects will be created and confined for the purpose of testing.

*[[assignments]] 
*[[submission_records]] 
*[[questionnaires]] 
*[[assignment_questionnaires]] 
*[[participants]] 
*[[teams]] 
*[[teams_users]] 
*[[users]] 
*[[responses]] 
*[[response_maps]]

The manifestation of each object will contribute to the success of the following test on reputations. Some fields in each object can be empty or have default values. Some attributes are not relevant to the test. When implementing the test, the test scripts need to generate or set fixed values for corresponding fields.

=== Relevant Methods ===
==== ReputationController_spec ====
*db_query
This is the normal db query method, call this method will return peer review grades with given assignment id.
We will test this method in two aspect. 1. Test whether or not the grade return is right based on the specified algorithm.
2. We need to test the correctness of the query.

*db_query_with_quiz_score
This is the special db query, call this method will return quiz scores with given assignment id. We will test this method with same logic as the first one.

*json_generator
This method will generate the hash format of the review, we will test this method by calling to and convert the result to json format the print to test its correctness.

*client
client method will fill many instance variables with corresponding class variables, we need to test this method with send_post_request.

*send_post_request
This method send a post request to peerlogic.csc.ncsu.edu/reputation/calculations/reputation_algorithms to calculate get the reputation result and use show the result
in corresponding UI and update given reviewer's reputation. We will test this method based on the algorithm in the paper, first to test the result reputation value, second to test the
update value in database.

== Results ==
=== Collaborators ===
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-03T23:13:01Z

Hchen34:

== Project Overview ==

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score expectation

=== Files Involved ===
*reputation_web_service_controller_spec
*hamer_spec
*<del>lauw_spec (?)</del>

== Test Plan ==

=== Testing Objects ===
Following objects will be created and confined for the purpose of testing.

*[[assignments]] 
*[[submission_records]] 
*[[questionnaires]] 
*[[assignment_questionnaires]] 
*[[participants]] 
*[[teams]] 
*[[teams_users]] 
*[[users]] 
*[[responses]] 
*[[response_maps]]

=== Relevant Methods ===
==== hamer_spec ====
*calculate_weighted_scores_and_reputation
This is the main method of Hamer algorithm. Reviewers inaccuracy will be calculated and updated from past average, then weight of the review will be adjusted based on the inaccuracy. We plan to create two contexts to test it as below: 1. When the review is the same from average, reviewer’s inaccuracy should be minimized and the review’s weight should be maximized; 2. When the review is furthest from average, reviewer’s inaccuracy should be maximized and the review’s weight should be minimized.
*converged?
This method ensures the final result of calculated reviews' weights is converged, as the algorithm requires. A simple fixture test can be done by giving the method a set of input data and compare it to expected result.
==== ReputationController_spec ====
*db_query
*db_query_with_quiz_score
*json_generator
*client
*send_post_request

== Results ==
=== Collaborators ===
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-03T23:08:16Z

Hchen34:

== Project Overview ==

=== Objectives ===

Our objectives for this project are the following:
* Double and stub an assignment, a few submissions to the assignment, under different review rubrics
* Manually calculate reputation scores based on paper "Pluggable reputation systems for peer review: A web-service approach"
* Validate correct reputation scores based on different review rubrics generated by reputation management VS manual computation of reputation score expectation

=== Files Involved ===
*reputation_web_service_controller_spec
*hamer_spec
*<del>lauw_spec (?)</del>

== Test Plan ==

=== Testing Objects ===
Following objects will be created and confined for the purpose of testing.

*[[assignments]] 
*[[submission_records]] 
*[[questionnaires]] 
*[[assignment_questionnaires]] 
*[[participants]] 
*[[teams]] 
*[[teams_users]] 
*[[users]] 
*[[responses]] 
*[[response_maps]]

=== Relevant Methods ===
==== hamer_spec ====
*calculate_weighted_scores_and_reputation
This is the main method of Hamer algorithm. Reviewers inaccuracy will be calculated and updated from past average, then weight of the review will be adjusted based on the inaccuracy. We plan to create two contexts to test it as below: 1. When the review is the same from average, reviewer’s inaccuracy should be minimized and the review’s weight should be maximized; 2. When the review is furthest from average, reviewer’s inaccuracy should be maximized and the review’s weight should be minimized.
*<del>converged?</del>
<del>This method ensures the final result of calculated reviews' weights is converged, as the algorithm requires. No testing is needed on this method.</del>
==== ReputationController_spec ====
*db_query
*db_query_with_quiz_score
*json_generator
*client
*send_post_request

== Results ==
=== Collaborators ===
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-03T22:46:46Z

Hchen34: Added Test Objects section

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-02T02:38:13Z

Hchen34: added methods (subject to changes)

== Project Overview ==

=== Objectives ===

Our objectives for this project are the following:
*
*

=== Files Involved ===
*ReputationController_spec

== Test Plan ==

=== Relevant Methods ===
ReputationController_spec
*db_query
*db_query_with_quiz_score
*json_generator
*client
*send_post_request

== ReputationController_spec ==

=== db_query ===

=== db_query_with_quiz_score ===

=== json_generator ===

=== client ===

=== send_post_request ===

== Results ==

=== Collaborators ===
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-02T02:29:54Z

Hchen34:

== Project Overview ==

=== Objectives ===

Our objectives for this project are the following:
*
*

=== Files Involved ===
*ReputationController_spec

== Test Plan ==
=== Relevant Methods ===
ReputationController_spec
*
*

== ReputationController_spec ==

== Results ==
=== Collaborators ===
Jinku Cui (jcui23)

Henry Chen (hchen34)

Dong Li (dli35)

Zijun Lu (zlu5)

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2168. Testing - Reputations

2021-11-02T02:29:11Z

Hchen34: Created sections

== Project Overview ==

=== Objectives ===

Our objectives for this project are the following:
*
*

=== Files Involved ===
*ReputationController_spec

== Test Plan ==
=== Relevant Methods ===
ReputationController_spec
*
*

== ReputationController_spec ==

== Results ==
=== Collaborators ===
Jinku Cui (jcui23)
Henry Chen (hchen34)
Dong Li (dli35)
Zijun Lu (zlu5)

==References==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[https://doi.org/10.1109/FIE.2015.7344292 Pluggable reputation systems for peer review: A web-service approach]

CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment

2021-10-25T16:41:09Z

Hchen34:

This wiki page describes the changes made under E2138, in order to auto-generate submission directory names based on assignment names for Fall 2021, CSC/ECE 517.

__TOC__

= Overview =

== About Expertiza ==

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

== Issues for this project ==

When instructors give an assignment a name, submission directories should be auto-generated based on the assignment name, and also be unique. A number of issues existed when a previous team worked on this assignment (E2054), including
* Issue #1: The directory name should be auto-generated from the assignment name.
* Issue #2: It should be done by changing spaces in the names to underscores. E.g., the directory for Program 1 is by default "Program_1".
* Issue #3: A check should be added to prevent two assignments in the same course from having the same name.
* Issue #4: Verify or add if not present - a check to stop two assignments from sharing the same directory.
* Issue #5: On changing the name of an assignment while creating it, the code shouldn't throw a NoMethodError.

= Project implementation =

== Submitted work and demonstration of project ==

* [http://152.7.98.88:8080/ Link to deployed code on NCSU VCL]
* [https://github.com/SN-18/expertiza Link to our Github repository]

== Files involved ==
*app/views/assignments/edit/_general.html.erb - added function that auto generate directory name, and substitute its spacebar with underscore.
*app/models/assignment.rb - added presence and uniqueness validation to assignment.
*app/controllers/assignments_controller.rb - modify create method to check for existing assignment name & directory.
*spec/controllers/assignments_controller_spec.rb - edited & added tests for the project.
*spec/models/due_date_spec.rb - fixed issues where tests can fail due to creating assignments with same names & directories.
*spec/models/student_task_spec.rb - fixed issues where tests can fail due to creating assignments with same names & directories.

== Changes made to code ==

=== views ===
The following code was added to app/views/assignments/edit/_general.html.erb. The purpose is to generate submission directory name from the assignment name and substitute the spacebar with underscore.

<pre>
<script>
...
$(function() {
$("#assignment_form_assignment_name").change(function() {
filename = $( "#assignment_form_assignment_name" ).val().replace(/ /g,"_").replace(/[/\\?%*:|"<>/$&!#%^@]/g, '');;
$('#assignment_form_assignment_directory_path').val(filename);
});
});

function autogenerate_submission(){
assignment_form.assignment.directory_path = assignment_form.assignment.name;
}
</script>
</pre>

=== models ===
The following code was added to app/models/assignment.rb's Assignment class, to add validation for assignment name and directory.
<pre>
validates :directory_path, presence: true # E2138 Validation for unique submission directory
validates :directory_path, uniqueness: {scope: :course_id}
</pre>

=== controllers ===
The following code was modified in app/controllers/assignments_controller.rb's create method, adding checks for assignment name and directory when creating assignment.

'''Before'''
<pre>
def create
@assignment_form = AssignmentForm.new(assignment_form_params)
if params[:button]
if @assignment_form.save
@assignment_form.create_assignment_node
exist_assignment = Assignment.find_by(id: @assignment_form.assignment.id)
assignment_form_params[:assignment][:id] = exist_assignment.id.to_s
if assignment_form_params[:assignment][:directory_path].blank?
assignment_form_params[:assignment][:directory_path] = "assignment_#{assignment_form_params[:assignment][:id]}"
end
ques_array = assignment_form_params[:assignment_questionnaire]
due_array = assignment_form_params[:due_date]
ques_array.each do |cur_questionnaire|
cur_questionnaire[:assignment_id] = exist_assignment.id.to_s
end
due_array.each do |cur_due|
cur_due[:parent_id] = exist_assignment.id.to_s
end
assignment_form_params[:assignment_questionnaire] = ques_array
assignment_form_params[:due_date] = due_array
@assignment_form.update(assignment_form_params, current_user)
aid = Assignment.find_by(id: @assignment_form.assignment.id).id
ExpertizaLogger.info "Assignment created: #{@assignment_form.as_json}"
redirect_to edit_assignment_path aid
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
return
else
flash.now[:error] = "Failed to create assignment"
render 'new'
end
else
render 'new'
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
end
end
</pre>

'''After'''
<pre>
def create
@assignment_form = AssignmentForm.new(assignment_form_params)
if params[:button]
# E2138 issue #3
find_existing_assignment = Assignment.find_by(name: @assignment_form.assignment.name, course_id: @assignment_form.assignment.course_id)
dir_path = assignment_form_params[:assignment][:directory_path]
find_existing_directory = Assignment.find_by(directory_path: dir_path, course_id: @assignment_form.assignment.course_id)
if !find_existing_assignment and !find_existing_directory and @assignment_form.save #No existing names/directories
@assignment_form.create_assignment_node
current_assignment = Assignment.find_by(name: @assignment_form.assignment.name, course_id: @assignment_form.assignment.course_id)
assignment_form_params[:assignment][:id] = current_assignment.id.to_s
ques_array = assignment_form_params[:assignment_questionnaire]
due_array = assignment_form_params[:due_date]
ques_array.each do |cur_questionnaire|
cur_questionnaire[:assignment_id] = current_assignment.id.to_s
end
due_array.each do |cur_due|
cur_due[:parent_id] = current_assignment.id.to_s
end
assignment_form_params[:assignment_questionnaire] = ques_array
assignment_form_params[:due_date] = due_array
@assignment_form.update(assignment_form_params, current_user)
aid = Assignment.find_by(name: @assignment_form.assignment.name, course_id: @assignment_form.assignment.course_id).id
ExpertizaLogger.info "Assignment created: #{@assignment_form.as_json}"
redirect_to edit_assignment_path aid
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
return
else
flash[:error] = "Failed to create assignment."
if find_existing_assignment
flash[:error] << " " + @assignment_form.assignment.name + " already exists as an assignment name"
end
if find_existing_directory
flash[:error] << " " + dir_path + " already exists as a submission directory name"
end
redirect_to "/assignments/new?private=1"
end
else
render 'new'
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
end
end
</pre>

= Test Plan =

== Manual UI Testing ==
The following steps must be performed to test the project UI: 
'''Step 1:''' Log in as an Instructor, with Username - instructor6, Password - password. 
[[File:expertiza_login.PNG]] 
'''Step 2:''' Create a new assignment. Assignment is named as "Test Assignment", under course CSC/ECE 517 Fall 2020.
The directory name gets auto generated with name replacing space by underscore in assignment name. 
[[File:auto_generated.PNG]] 
'''Step 3:''' Assignment gets successfully created. Assignment is named as "Test Assignment" and gets saved in the directory "Test_assignment", under course CSC/ECE 517 Fall 2020 
[[File:save_success.PNG]] 
'''Step 4:''' Create and save another assignment with same name. Following error is observed. 
[[File:error.PNG]] 
== RSpec Testing ==
The following RSpec test is added in the assignments_contoller_spec.rb file
<pre>
# Assignment Names Cannot be duplicated
context 'when assignment_form name already exists and is not saved properly' do
it 'redirects to assignment#new page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(course_id:1, name:'test assignment').and_return(assignment)
post :create, @params
expect(flash[:error]).to eq('Failed to create assignment. test assignment already exists as an assignment name')
expect(response).to redirect_to('/assignments/new?private=1')
end
end
</pre>
 
The following RSpec tests are corrected in the assignments_contoller_spec.rb file

'''Before'''
<pre>
context 'when assignment_form is saved successfully' do
it 'redirects to assignment#edit page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(id: 1).and_return(assignment)
allow_any_instance_of(AssignmentsController).to receive(:undo_link)
.with('Assignment "test assignment" has been created successfully. ').and_return(true)
post :create, @params
expect(response).to redirect_to('/assignments/1/edit')
end
end
</pre>
'''After'''
<pre>
context 'when assignment_form is saved successfully' do
it 'redirects to assignment#edit page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(course_id:1, name:'test assignment').and_return(assignment)
allow_any_instance_of(AssignmentsController).to receive(:undo_link)
.with('Assignment "test assignment" has been created successfully. ').and_return(true)
post :create, @params
expect(response).to redirect_to('/assignments/1/edit')
end
end
</pre>
= Team Information =
Mentor: Nicholas Himes (nnhimes)

Henry Chen (hchen34)

Saurabh Nanda (snanda2)

Snehapriyaa Mathiyalaghan (smathiy)

CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment

2021-10-25T16:39:56Z

Hchen34: added RSpec files in files involved

This wiki page describes the changes made under E2138, in order to auto-generate submission directory names based on assignment names for Fall 2021, CSC/ECE 517.

__TOC__

= Overview =

== About Expertiza ==

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

== Issues for this project ==

When instructors give an assignment a name, submission directories should be auto-generated based on the assignment name, and also be unique. A number of issues existed when a previous team worked on this assignment (E2054), including
* Issue #1: The directory name should be auto-generated from the assignment name.
* Issue #2: It should be done by changing spaces in the names to underscores. E.g., the directory for Program 1 is by default "Program_1".
* Issue #3: A check should be added to prevent two assignments in the same course from having the same name.
* Issue #4: Verify or add if not present - a check to stop two assignments from sharing the same directory.
* Issue #5: On changing the name of an assignment while creating it, the code shouldn't throw a NoMethodError.

= Project implementation =

== Submitted work and demonstration of project ==

* [http://152.7.98.88:8080/ Link to deployed code on NCSU VCL]
* [https://github.com/SN-18/expertiza Link to our Github repository]

== Files involved ==
*app/views/assignments/edit/_general.html.erb - added function that auto generate directory name, and substitute its spacebar with underscore.
*app/models/assignment.rb - added presence and uniqueness validation to assignment.
*app/controllers/assignments_controller.rb - modify create method to check for existing assignment name & directory.
*spec/controllers/assignments_controller_spec.rb - edited & added tests for the project.
*spec/models/due_date_spec.rb - fixed issues where tests can fail due to creating assignments with same names & directories.
*spec/models/student_task_spec.rb - fixed issues where tests can fail due to creating assignments with same names & directories.

== Changes made to code ==

=== views ===
The following code was added to app/views/assignments/edit/_general.html.erb. The purpose is to generate submission directory name from the assignment name and substitute the spacebar with underscore.

<pre>
<script>
...
$(function() {
$("#assignment_form_assignment_name").change(function() {
filename = $( "#assignment_form_assignment_name" ).val().replace(/ /g,"_").replace(/[/\\?%*:|"<>/$&!#%^@]/g, '');;
$('#assignment_form_assignment_directory_path').val(filename);
});
});

function autogenerate_submission(){
assignment_form.assignment.directory_path = assignment_form.assignment.name;
}
</script>
</pre>

=== models ===
The following code was added to app/models/assignment.rb's Assignment class, to add validation for assignment name and directory.
<pre>
validates :directory_path, presence: true # E2138 Validation for unique submission directory
validates :directory_path, uniqueness: {scope: :course_id}
</pre>

=== controllers ===
The following code was modified in app/controllers/assignments_controller.rb's create method, adding checks for assignment name and directory when creating assignment.

'''Before'''
<pre>
def create
@assignment_form = AssignmentForm.new(assignment_form_params)
if params[:button]
if @assignment_form.save
@assignment_form.create_assignment_node
exist_assignment = Assignment.find_by(id: @assignment_form.assignment.id)
assignment_form_params[:assignment][:id] = exist_assignment.id.to_s
if assignment_form_params[:assignment][:directory_path].blank?
assignment_form_params[:assignment][:directory_path] = "assignment_#{assignment_form_params[:assignment][:id]}"
end
ques_array = assignment_form_params[:assignment_questionnaire]
due_array = assignment_form_params[:due_date]
ques_array.each do |cur_questionnaire|
cur_questionnaire[:assignment_id] = exist_assignment.id.to_s
end
due_array.each do |cur_due|
cur_due[:parent_id] = exist_assignment.id.to_s
end
assignment_form_params[:assignment_questionnaire] = ques_array
assignment_form_params[:due_date] = due_array
@assignment_form.update(assignment_form_params, current_user)
aid = Assignment.find_by(id: @assignment_form.assignment.id).id
ExpertizaLogger.info "Assignment created: #{@assignment_form.as_json}"
redirect_to edit_assignment_path aid
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
return
else
flash.now[:error] = "Failed to create assignment"
render 'new'
end
else
render 'new'
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
end
end
</pre>

'''After'''
<pre>
def create
@assignment_form = AssignmentForm.new(assignment_form_params)
if params[:button]
# E2138 issue #3
find_existing_assignment = Assignment.find_by(name: @assignment_form.assignment.name, course_id: @assignment_form.assignment.course_id)
dir_path = assignment_form_params[:assignment][:directory_path]
find_existing_directory = Assignment.find_by(directory_path: dir_path, course_id: @assignment_form.assignment.course_id)
if !find_existing_assignment and !find_existing_directory and @assignment_form.save #No existing names/directories
@assignment_form.create_assignment_node
current_assignment = Assignment.find_by(name: @assignment_form.assignment.name, course_id: @assignment_form.assignment.course_id)
assignment_form_params[:assignment][:id] = current_assignment.id.to_s
ques_array = assignment_form_params[:assignment_questionnaire]
due_array = assignment_form_params[:due_date]
ques_array.each do |cur_questionnaire|
cur_questionnaire[:assignment_id] = current_assignment.id.to_s
end
due_array.each do |cur_due|
cur_due[:parent_id] = current_assignment.id.to_s
end
assignment_form_params[:assignment_questionnaire] = ques_array
assignment_form_params[:due_date] = due_array
@assignment_form.update(assignment_form_params, current_user)
aid = Assignment.find_by(name: @assignment_form.assignment.name, course_id: @assignment_form.assignment.course_id).id
ExpertizaLogger.info "Assignment created: #{@assignment_form.as_json}"
redirect_to edit_assignment_path aid
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
return
else
flash[:error] = "Failed to create assignment."
if find_existing_assignment
flash[:error] << " " + @assignment_form.assignment.name + " already exists as an assignment name"
end
if find_existing_directory
flash[:error] << " " + dir_path + " already exists as a submission directory name"
end
redirect_to "/assignments/new?private=1"
end
else
render 'new'
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
end
end
</pre>

= Test Plan =

== Manual UI Testing ==
The following steps must be performed to test the project UI: 
'''Step 1:''' Log in as an Instructor, with Username - instructor6, Password - password. 
[[File:expertiza_login.PNG]] 
'''Step 2:''' Create a new assignment. Assignment is named as "Test Assignment", under course CSC/ECE 517 Fall 2020.
The directory name gets auto generated with name replacing space by underscore in assignment name. 
[[File:auto_generated.PNG]] 
'''Step 3:''' Assignment gets successfully created. Assignment is named as "Test Assignment" and gets saved in the directory "Test_assignment", under course CSC/ECE 517 Fall 2020 
[[File:save_success.PNG]] 
'''Step 4:''' Create and save another assignment with same name. Following error is observed. 
[[File:error.PNG]] 
== RSpec Testing ==
The following RSpec test is added in the assignments_contoller_spec.rb file
<pre>
# Assignment Names Cannot be duplicated
context 'when assignment_form name already exists and is not saved properly' do
it 'redirects to assignment#new page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(course_id:1, name:'test assignment').and_return(assignment)
post :create, @params
expect(flash[:error]).to eq('Failed to create assignment. test assignment already exists as an assignment name')
expect(response).to redirect_to('/assignments/new?private=1')
end
end
</pre>
 
The following RSpec tests are corrected in the assignments_contoller_spec.rb file
'''Before'''
<pre>
context 'when assignment_form is saved successfully' do
it 'redirects to assignment#edit page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(id: 1).and_return(assignment)
allow_any_instance_of(AssignmentsController).to receive(:undo_link)
.with('Assignment "test assignment" has been created successfully. ').and_return(true)
post :create, @params
expect(response).to redirect_to('/assignments/1/edit')
end
end
</pre>
'''After'''
<pre>
context 'when assignment_form is saved successfully' do
it 'redirects to assignment#edit page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(course_id:1, name:'test assignment').and_return(assignment)
allow_any_instance_of(AssignmentsController).to receive(:undo_link)
.with('Assignment "test assignment" has been created successfully. ').and_return(true)
post :create, @params
expect(response).to redirect_to('/assignments/1/edit')
end
end
</pre>
= Team Information =
Mentor: Nicholas Himes (nnhimes)

Henry Chen (hchen34)

Saurabh Nanda (snanda2)

Snehapriyaa Mathiyalaghan (smathiy)

CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment

2021-10-20T21:50:31Z

Hchen34: Undo revision 139523 by Hdu5 (talk)

This wiki page describes the changes made under E2138, in order to auto-generate submission directory names based on assignment names for Fall 2021, CSC/ECE 517.

__TOC__

= Overview =

== About Expertiza ==

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

== Issues for this project ==

When instructors give an assignment a name, submission directories should be auto-generated based on the assignment name, and also be unique. A number of issues existed when a previous team worked on this assignment (E2054), including
* Issue #1: The directory name should be auto-generated from the assignment name.
* Issue #2: It should be done by changing spaces in the names to underscores. E.g., the directory for Program 1 is by default "Program_1".
* Issue #3: A check should be added to prevent two assignments in the same course from having the same name.
* Issue #4: Verify or add if not present - a check to stop two assignments from sharing the same directory.
* Issue #5: On changing the name of an assignment while creating it, the code shouldn't throw a NoMethodError.

= Project implementation =

== Submitted work and demonstration of project ==

* [http://152.7.98.88:8080/ Link to deployed code on NCSU VCL]
* [https://github.com/SN-18/expertiza Link to our Github repository]

== Files involved ==
*app/views/assignments/edit/_general.html.erb - added function that auto generate directory name, and substitute its spacebar with underscore.
*app/models/assignment.rb - added presence and uniqueness validation to assignment.
*app/controllers/assignments_controller.rb - modify create method to check for existing assignment name & directory.

== Changes made to code ==

=== views ===
The following code was added to app/views/assignments/edit/_general.html.erb. The purpose is to generate submission directory name from the assignment name and substitute the spacebar with underscore.

<pre>
<script>
...
$(function() {
$("#assignment_form_assignment_name").change(function() {
filename = $( "#assignment_form_assignment_name" ).val().replace(/ /g,"_").replace(/[/\\?%*:|"<>/$&!#%^@]/g, '');;
$('#assignment_form_assignment_directory_path').val(filename);
});
});

function autogenerate_submission(){
assignment_form.assignment.directory_path = assignment_form.assignment.name;
}
</script>
</pre>

=== models ===
The following code was added to app/models/assignment.rb's Assignment class, to add validation for assignment name and directory.
<pre>
validates :directory_path, presence: true # E2138 Validation for unique submission directory
validates :directory_path, uniqueness: {scope: :course_id}
</pre>

=== controllers ===
The following code was modified in app/controllers/assignments_controller.rb's create method, adding checks for assignment name and directory when creating assignment.

'''Before'''
<pre>
def create
@assignment_form = AssignmentForm.new(assignment_form_params)
if params[:button]
if @assignment_form.save
@assignment_form.create_assignment_node
exist_assignment = Assignment.find_by(id: @assignment_form.assignment.id)
assignment_form_params[:assignment][:id] = exist_assignment.id.to_s
if assignment_form_params[:assignment][:directory_path].blank?
assignment_form_params[:assignment][:directory_path] = "assignment_#{assignment_form_params[:assignment][:id]}"
end
ques_array = assignment_form_params[:assignment_questionnaire]
due_array = assignment_form_params[:due_date]
ques_array.each do |cur_questionnaire|
cur_questionnaire[:assignment_id] = exist_assignment.id.to_s
end
due_array.each do |cur_due|
cur_due[:parent_id] = exist_assignment.id.to_s
end
assignment_form_params[:assignment_questionnaire] = ques_array
assignment_form_params[:due_date] = due_array
@assignment_form.update(assignment_form_params, current_user)
aid = Assignment.find_by(id: @assignment_form.assignment.id).id
ExpertizaLogger.info "Assignment created: #{@assignment_form.as_json}"
redirect_to edit_assignment_path aid
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
return
else
flash.now[:error] = "Failed to create assignment"
render 'new'
end
else
render 'new'
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
end
end
</pre>

'''After'''
<pre>
def create
@assignment_form = AssignmentForm.new(assignment_form_params)
if params[:button]
# E2138 issue #3
find_existing_assignment = Assignment.find_by(name: @assignment_form.assignment.name, course_id: @assignment_form.assignment.course_id)
dir_path = assignment_form_params[:assignment][:directory_path]
find_existing_directory = Assignment.find_by(directory_path: dir_path, course_id: @assignment_form.assignment.course_id)
if !find_existing_assignment and !find_existing_directory and @assignment_form.save #No existing names/directories
@assignment_form.create_assignment_node
current_assignment = Assignment.find_by(name: @assignment_form.assignment.name, course_id: @assignment_form.assignment.course_id)
assignment_form_params[:assignment][:id] = current_assignment.id.to_s
ques_array = assignment_form_params[:assignment_questionnaire]
due_array = assignment_form_params[:due_date]
ques_array.each do |cur_questionnaire|
cur_questionnaire[:assignment_id] = current_assignment.id.to_s
end
due_array.each do |cur_due|
cur_due[:parent_id] = current_assignment.id.to_s
end
assignment_form_params[:assignment_questionnaire] = ques_array
assignment_form_params[:due_date] = due_array
@assignment_form.update(assignment_form_params, current_user)
aid = Assignment.find_by(name: @assignment_form.assignment.name, course_id: @assignment_form.assignment.course_id).id
ExpertizaLogger.info "Assignment created: #{@assignment_form.as_json}"
redirect_to edit_assignment_path aid
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
return
else
flash[:error] = "Failed to create assignment."
if find_existing_assignment
flash[:error] << " " + @assignment_form.assignment.name + " already exists as an assignment name"
end
if find_existing_directory
flash[:error] << " " + dir_path + " already exists as a submission directory name"
end
redirect_to "/assignments/new?private=1"
end
else
render 'new'
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
end
end
</pre>

= Test Plan =

== Manual UI Testing ==
The following steps must be performed to test the project UI: 
'''Step 1:''' Log in as an Instructor, with Username - instructor6, Password - password 
[[File:expertiza_login.PNG]] 
'''Step 2:''' Create a new assignment. Assignment is named as "Test Assignment", under course CSC/ECE 517 Fall 2020.
The directory name gets auto generated with replacing space by underscore in assignment name. 
[[File:auto_generated.PNG]] 
'''Step 3:''' Assignment gets successfully created. Assignment is named as "Test Assignment" and gets saved in the directory "Test_assignment", under course CSC/ECE 517 Fall 2020 
[[File:save_success.PNG]] 
'''Step 4:''' Try saving another assignment with same name. Following error is observed. br> 
[[File:error.PNG]] 
== RSpec Testing ==
The following RSpec tests are corrected in the assignments_contoller_spec.rb file
'''Before'''
<pre>
context 'when assignment_form is saved successfully' do
it 'redirects to assignment#edit page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(course_id:1, name:'test assignment').and_return(assignment)
allow_any_instance_of(AssignmentsController).to receive(:undo_link)
.with('Assignment "test assignment" has been created successfully. ').and_return(true)
post :create, @params
expect(response).to redirect_to('/assignments/1/edit')
end
end
</pre>
'''After'''
<pre>
context 'when assignment_form is saved successfully' do
it 'redirects to assignment#edit page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(course_id:1, name:'test assignment').and_return(assignment)
allow_any_instance_of(AssignmentsController).to receive(:undo_link)
.with('Assignment "test assignment" has been created successfully. ').and_return(true)
post :create, @params
expect(response).to redirect_to('/assignments/1/edit')
end
end
</pre>
= Team Information =
Mentor: Nicholas Himes (nnhimes)

Henry Chen (hchen34)

Saurabh Nanda (snanda2)

Snehapriyaa Mathiyalaghan (smathiy)

CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment

2021-10-20T21:50:16Z

Hchen34: Undo revision 139528 by Hdu5 (talk)

This wiki page describes the changes made under E2138, in order to auto-generate submission directory names based on assignment names for Fall 2021, CSC/ECE 517.

__TOC__

= Overview =

== About Expertiza ==

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

== Issues for this project ==

When instructors give an assignment a name, submission directories should be auto-generated based on the assignment name, and also be unique. A number of issues existed when a previous team worked on this assignment (E2054), including
* Issue #1: The directory name should be auto-generated from the assignment name.
* Issue #2: It should be done by changing spaces in the names to underscores. E.g., the directory for Program 1 is by default "Program_1".
* Issue #3: A check should be added to prevent two assignments in the same course from having the same name.
* Issue #4: Verify or add if not present - a check to stop two assignments from sharing the same directory.
* Issue #5: On changing the name of an assignment while creating it, the code shouldn't throw a NoMethodError.

= Project implementation =

== Submitted work and demonstration of project ==

* [http://152.7.98.88:8080/ Link to deployed code on NCSU VCL]
* [https://github.com/SN-18/expertiza Link to our Github repository]

== Files involved ==
*app/controllers/impersonate_controller.rb - refactored the codes related to the issues mentioned above.
*spec/controllers/impersonate_controller_spec.rb - implemented and added some related tests.

== Changes made to code ==

=== views ===
The following code was added to app/views/assignments/edit/_general.html.erb. The purpose is to generate submission directory name from the assignment name and substitute the spacebar with underscore.

<pre>
<script>
...
$(function() {
$("#assignment_form_assignment_name").change(function() {
filename = $( "#assignment_form_assignment_name" ).val().replace(/ /g,"_").replace(/[/\\?%*:|"<>/$&!#%^@]/g, '');;
$('#assignment_form_assignment_directory_path').val(filename);
});
});

function autogenerate_submission(){
assignment_form.assignment.directory_path = assignment_form.assignment.name;
}
</script>
</pre>

=== models ===
The following code was added to app/models/assignment.rb's Assignment class, to add validation for assignment name and directory.
<pre>
validates :directory_path, presence: true # E2138 Validation for unique submission directory
validates :directory_path, uniqueness: {scope: :course_id}
</pre>

=== controllers ===
The following code was modified in app/controllers/assignments_controller.rb's create method, adding checks for assignment name and directory when creating assignment.

'''Before'''
<pre>
def create
@assignment_form = AssignmentForm.new(assignment_form_params)
if params[:button]
if @assignment_form.save
@assignment_form.create_assignment_node
exist_assignment = Assignment.find_by(id: @assignment_form.assignment.id)
assignment_form_params[:assignment][:id] = exist_assignment.id.to_s
if assignment_form_params[:assignment][:directory_path].blank?
assignment_form_params[:assignment][:directory_path] = "assignment_#{assignment_form_params[:assignment][:id]}"
end
ques_array = assignment_form_params[:assignment_questionnaire]
due_array = assignment_form_params[:due_date]
ques_array.each do |cur_questionnaire|
cur_questionnaire[:assignment_id] = exist_assignment.id.to_s
end
due_array.each do |cur_due|
cur_due[:parent_id] = exist_assignment.id.to_s
end
assignment_form_params[:assignment_questionnaire] = ques_array
assignment_form_params[:due_date] = due_array
@assignment_form.update(assignment_form_params, current_user)
aid = Assignment.find_by(id: @assignment_form.assignment.id).id
ExpertizaLogger.info "Assignment created: #{@assignment_form.as_json}"
redirect_to edit_assignment_path aid
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
return
else
flash.now[:error] = "Failed to create assignment"
render 'new'
end
else
render 'new'
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
end
end
</pre>

'''After'''
<pre>
def create
@assignment_form = AssignmentForm.new(assignment_form_params)
if params[:button]
# E2138 issue #3
find_existing_assignment = Assignment.find_by(name: @assignment_form.assignment.name, course_id: @assignment_form.assignment.course_id)
dir_path = assignment_form_params[:assignment][:directory_path]
find_existing_directory = Assignment.find_by(directory_path: dir_path, course_id: @assignment_form.assignment.course_id)
if !find_existing_assignment and !find_existing_directory and @assignment_form.save #No existing names/directories
@assignment_form.create_assignment_node
current_assignment = Assignment.find_by(name: @assignment_form.assignment.name, course_id: @assignment_form.assignment.course_id)
assignment_form_params[:assignment][:id] = current_assignment.id.to_s
ques_array = assignment_form_params[:assignment_questionnaire]
due_array = assignment_form_params[:due_date]
ques_array.each do |cur_questionnaire|
cur_questionnaire[:assignment_id] = current_assignment.id.to_s
end
due_array.each do |cur_due|
cur_due[:parent_id] = current_assignment.id.to_s
end
assignment_form_params[:assignment_questionnaire] = ques_array
assignment_form_params[:due_date] = due_array
@assignment_form.update(assignment_form_params, current_user)
aid = Assignment.find_by(name: @assignment_form.assignment.name, course_id: @assignment_form.assignment.course_id).id
ExpertizaLogger.info "Assignment created: #{@assignment_form.as_json}"
redirect_to edit_assignment_path aid
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
return
else
flash[:error] = "Failed to create assignment."
if find_existing_assignment
flash[:error] << " " + @assignment_form.assignment.name + " already exists as an assignment name"
end
if find_existing_directory
flash[:error] << " " + dir_path + " already exists as a submission directory name"
end
redirect_to "/assignments/new?private=1"
end
else
render 'new'
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
end
end
</pre>

= Test Plan =

== Manual UI Testing ==
The following steps must be performed to test the project UI: 
'''Step 1:''' Log in as an Instructor, with Username - instructor6, Password - password 
[[File:expertiza_login.PNG]] 
'''Step 2:''' Create a new assignment. Assignment is named as "Test Assignment", under course CSC/ECE 517 Fall 2020.
The directory name gets auto generated with replacing space by underscore in assignment name. 
[[File:auto_generated.PNG]] 
'''Step 3:''' Assignment gets successfully created. Assignment is named as "Test Assignment" and gets saved in the directory "Test_assignment", under course CSC/ECE 517 Fall 2020 
[[File:save_success.PNG]] 
'''Step 4:''' Try saving another assignment with same name. Following error is observed. br> 
[[File:error.PNG]] 
== RSpec Testing ==
The following RSpec tests are corrected in the assignments_contoller_spec.rb file
'''Before'''
<pre>
context 'when assignment_form is saved successfully' do
it 'redirects to assignment#edit page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(course_id:1, name:'test assignment').and_return(assignment)
allow_any_instance_of(AssignmentsController).to receive(:undo_link)
.with('Assignment "test assignment" has been created successfully. ').and_return(true)
post :create, @params
expect(response).to redirect_to('/assignments/1/edit')
end
end
</pre>
'''After'''
<pre>
context 'when assignment_form is saved successfully' do
it 'redirects to assignment#edit page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(course_id:1, name:'test assignment').and_return(assignment)
allow_any_instance_of(AssignmentsController).to receive(:undo_link)
.with('Assignment "test assignment" has been created successfully. ').and_return(true)
post :create, @params
expect(response).to redirect_to('/assignments/1/edit')
end
end
</pre>
= Team Information =
Mentor: Nicholas Himes (nnhimes)

Henry Chen (hchen34)

Saurabh Nanda (snanda2)

Snehapriyaa Mathiyalaghan (smathiy)

CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment

2021-10-20T21:50:05Z

Hchen34: Undo revision 139529 by Hdu5 (talk)

This wiki page describes the changes made under E2138, in order to auto-generate submission directory names based on assignment names for Fall 2021, CSC/ECE 517.

__TOC__

= Overview =

== About Expertiza ==

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

== Issues for this project ==

When instructors give an assignment a name, submission directories should be auto-generated based on the assignment name, and also be unique. A number of issues existed when a previous team worked on this assignment (E2054), including
* Issue #1: The directory name should be auto-generated from the assignment name.
* Issue #2: It should be done by changing spaces in the names to underscores. E.g., the directory for Program 1 is by default "Program_1".
* Issue #3: A check should be added to prevent two assignments in the same course from having the same name.
* Issue #4: Verify or add if not present - a check to stop two assignments from sharing the same directory.
* Issue #5: On changing the name of an assignment while creating it, the code shouldn't throw a NoMethodError.

= Project implementation =

== Submitted work and demonstration of project ==

* [http://152.7.98.88:8080/ Link to deployed code on NCSU VCL]
* [https://github.com/SN-18/expertiza Link to our Github repository]

== Files involved ==
*app/controllers/impersonate_controller.rb - refactored the codes related to the issues mentioned above.
*spec/controllers/impersonate_controller_spec.rb - implemented and added some related tests.

== Changes made to code ==

The following code was modified in app/controllers/impersonate_controller.rb, in order to refactor to:
* make the code more readable.
* properly delete unused variable .

'''Before'''
<pre>
def create
@assignment_form = AssignmentForm.new(assignment_form_params)
if params[:button]
if @assignment_form.save
@assignment_form.create_assignment_node
exist_assignment = Assignment.find_by(id: @assignment_form.assignment.id)
assignment_form_params[:assignment][:id] = exist_assignment.id.to_s
if assignment_form_params[:assignment][:directory_path].blank?
assignment_form_params[:assignment][:directory_path] = "assignment_#{assignment_form_params[:assignment][:id]}"
end
ques_array = assignment_form_params[:assignment_questionnaire]
due_array = assignment_form_params[:due_date]
ques_array.each do |cur_questionnaire|
cur_questionnaire[:assignment_id] = exist_assignment.id.to_s
end
due_array.each do |cur_due|
cur_due[:parent_id] = exist_assignment.id.to_s
end
assignment_form_params[:assignment_questionnaire] = ques_array
assignment_form_params[:due_date] = due_array
@assignment_form.update(assignment_form_params, current_user)
aid = Assignment.find_by(id: @assignment_form.assignment.id).id
ExpertizaLogger.info "Assignment created: #{@assignment_form.as_json}"
redirect_to edit_assignment_path aid
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
return
else
flash.now[:error] = "Failed to create assignment"
render 'new'
end
else
render 'new'
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
end
end
</pre>

'''After'''
<pre>
def create
@assignment_form = AssignmentForm.new(assignment_form_params)
if params[:button]
# E2138 issue #3
find_existing_assignment = Assignment.find_by(name: @assignment_form.assignment.name, course_id: @assignment_form.assignment.course_id)
dir_path = assignment_form_params[:assignment][:directory_path]
find_existing_directory = Assignment.find_by(directory_path: dir_path, course_id: @assignment_form.assignment.course_id)
if !find_existing_assignment and !find_existing_directory and @assignment_form.save #No existing names/directories
@assignment_form.create_assignment_node
current_assignment = Assignment.find_by(name: @assignment_form.assignment.name, course_id: @assignment_form.assignment.course_id)
assignment_form_params[:assignment][:id] = current_assignment.id.to_s
ques_array = assignment_form_params[:assignment_questionnaire]
due_array = assignment_form_params[:due_date]
ques_array.each do |cur_questionnaire|
cur_questionnaire[:assignment_id] = current_assignment.id.to_s
end
due_array.each do |cur_due|
cur_due[:parent_id] = current_assignment.id.to_s
end
assignment_form_params[:assignment_questionnaire] = ques_array
assignment_form_params[:due_date] = due_array
@assignment_form.update(assignment_form_params, current_user)
aid = Assignment.find_by(name: @assignment_form.assignment.name, course_id: @assignment_form.assignment.course_id).id
ExpertizaLogger.info "Assignment created: #{@assignment_form.as_json}"
redirect_to edit_assignment_path aid
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
return
else
flash[:error] = "Failed to create assignment."
if find_existing_assignment
flash[:error] << " " + @assignment_form.assignment.name + " already exists as an assignment name"
end
if find_existing_directory
flash[:error] << " " + dir_path + " already exists as a submission directory name"
end
redirect_to "/assignments/new?private=1"
end
else
render 'new'
undo_link("Assignment \"#{@assignment_form.assignment.name}\" has been created successfully. ")
end
end
</pre>

= Test Plan =

== Manual UI Testing ==
The following steps must be performed to test the project UI: 
'''Step 1:''' Log in as an Instructor, with Username - instructor6, Password - password 
[[File:expertiza_login.PNG]] 
'''Step 2:''' Create a new assignment. Assignment is named as "Test Assignment", under course CSC/ECE 517 Fall 2020.
The directory name gets auto generated with replacing space by underscore in assignment name. 
[[File:auto_generated.PNG]] 
'''Step 3:''' Assignment gets successfully created. Assignment is named as "Test Assignment" and gets saved in the directory "Test_assignment", under course CSC/ECE 517 Fall 2020 
[[File:save_success.PNG]] 
'''Step 4:''' Try saving another assignment with same name. Following error is observed. br> 
[[File:error.PNG]] 
== RSpec Testing ==
The following RSpec tests are corrected in the assignments_contoller_spec.rb file
'''Before'''
<pre>
context 'when assignment_form is saved successfully' do
it 'redirects to assignment#edit page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(course_id:1, name:'test assignment').and_return(assignment)
allow_any_instance_of(AssignmentsController).to receive(:undo_link)
.with('Assignment "test assignment" has been created successfully. ').and_return(true)
post :create, @params
expect(response).to redirect_to('/assignments/1/edit')
end
end
</pre>
'''After'''
<pre>
context 'when assignment_form is saved successfully' do
it 'redirects to assignment#edit page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(course_id:1, name:'test assignment').and_return(assignment)
allow_any_instance_of(AssignmentsController).to receive(:undo_link)
.with('Assignment "test assignment" has been created successfully. ').and_return(true)
post :create, @params
expect(response).to redirect_to('/assignments/1/edit')
end
end
</pre>
= Team Information =
Mentor: Nicholas Himes (nnhimes)

Henry Chen (hchen34)

Saurabh Nanda (snanda2)

Snehapriyaa Mathiyalaghan (smathiy)

CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment

2021-10-20T21:49:51Z

Hchen34: Undo revision 139530 by Hdu5 (talk)

This wiki page describes the changes made under E2138, in order to auto-generate submission directory names based on assignment names for Fall 2021, CSC/ECE 517.

__TOC__

= Overview =

== About Expertiza ==

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

== Issues for this project ==

When instructors give an assignment a name, submission directories should be auto-generated based on the assignment name, and also be unique. A number of issues existed when a previous team worked on this assignment (E2054), including
* Issue #1: The directory name should be auto-generated from the assignment name.
* Issue #2: It should be done by changing spaces in the names to underscores. E.g., the directory for Program 1 is by default "Program_1".
* Issue #3: A check should be added to prevent two assignments in the same course from having the same name.
* Issue #4: Verify or add if not present - a check to stop two assignments from sharing the same directory.
* Issue #5: On changing the name of an assignment while creating it, the code shouldn't throw a NoMethodError.

= Project implementation =

== Submitted work and demonstration of project ==

* [http://152.7.98.88:8080/ Link to deployed code on NCSU VCL]
* [https://github.com/SN-18/expertiza Link to our Github repository]

== Files involved ==
*app/controllers/impersonate_controller.rb - refactored the codes related to the issues mentioned above.
*spec/controllers/impersonate_controller_spec.rb - implemented and added some related tests.

== Changes made to code ==

The following code was modified in app/controllers/impersonate_controller.rb, in order to refactor to:
* make the code more readable.
* properly delete unused variable .

'''Before'''
<pre>
def overwrite_session
# If not impersonatable, then original user's session remains
if params[:impersonate].nil?
# E1991 : check whether instructor is currently in anonymized view
user = User.anonymized_view?(session[:ip]) ? User.real_user_from_anonymized_name(params[:user][:name]) : user = User.find_by(name: params[:user][:name])
session[:super_user] = session[:user] if session[:super_user].nil?
AuthController.clear_user_info(session, nil)
session[:original_user] = @original_user
session[:impersonate] = true
session[:user] = user
else
# If some user is to be impersonated, their session details are overwritten onto the current to impersonate
if !params[:impersonate][:name].empty?
# E1991 : check whether instructor is currently in anonymized view
user = User.anonymized_view?(session[:ip]) ? User.real_user_from_anonymized_name(params[:impersonate][:name]) : user = user = User.find_by(name: params[:impersonate][:name])
AuthController.clear_user_info(session, nil)
session[:user] = user
session[:impersonate] = true
session[:original_user] = @original_user
else
# E1991 : check whether instructor is currently in anonymized view
user = User.anonymized_view?(session[:ip]) ? User.real_user_from_anonymized_name(params[:user][:name]) : user = User.find_by(name: params[:user][:name])
AuthController.clear_user_info(session, nil)
session[:user] = session[:super_user]
user = session[:user]
session[:super_user] = nil
end
end
end
</pre>

'''After'''
<pre>
def overwrite_session
# If not impersonatable, then original user's session remains
if params[:impersonate].nil?
# E1991 : check whether instructor is currently in anonymized view
if User.anonymized_view?(session[:ip])
user = User.real_user_from_anonymized_name(params[:user][:name])
else
user = User.find_by(name: params[:user][:name])
end
session[:super_user] = session[:user] if session[:super_user].nil?
AuthController.clear_user_info(session, nil)
session[:original_user] = @original_user
session[:impersonate] = true
session[:user] = user
# If some user is to be impersonated, their session details are overwritten onto the current to impersonate
elsif !params[:impersonate][:name].empty?
# E1991 : check whether instructor is currently in anonymized view
if User.anonymized_view?(session[:ip])
user = User.real_user_from_anonymized_name(params[:impersonate][:name])
else
user = User.find_by(name: params[:impersonate][:name])
end
AuthController.clear_user_info(session, nil)
session[:user] = user
session[:impersonate] = true
session[:original_user] = @original_user
else
# E1991 : check whether instructor is currently in anonymized view
AuthController.clear_user_info(session, nil)
session[:user] = session[:super_user]
session[:super_user] = nil
end
end
</pre>

= Test Plan =

== Manual UI Testing ==
The following steps must be performed to test the project UI: 
'''Step 1:''' Log in as an Instructor, with Username - instructor6, Password - password 
[[File:expertiza_login.PNG]] 
'''Step 2:''' Create a new assignment. Assignment is named as "Test Assignment", under course CSC/ECE 517 Fall 2020.
The directory name gets auto generated with replacing space by underscore in assignment name. 
[[File:auto_generated.PNG]] 
'''Step 3:''' Assignment gets successfully created. Assignment is named as "Test Assignment" and gets saved in the directory "Test_assignment", under course CSC/ECE 517 Fall 2020 
[[File:save_success.PNG]] 
'''Step 4:''' Try saving another assignment with same name. Following error is observed. br> 
[[File:error.PNG]] 
== RSpec Testing ==
The following RSpec tests are corrected in the assignments_contoller_spec.rb file
'''Before'''
<pre>
context 'when assignment_form is saved successfully' do
it 'redirects to assignment#edit page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(course_id:1, name:'test assignment').and_return(assignment)
allow_any_instance_of(AssignmentsController).to receive(:undo_link)
.with('Assignment "test assignment" has been created successfully. ').and_return(true)
post :create, @params
expect(response).to redirect_to('/assignments/1/edit')
end
end
</pre>
'''After'''
<pre>
context 'when assignment_form is saved successfully' do
it 'redirects to assignment#edit page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(course_id:1, name:'test assignment').and_return(assignment)
allow_any_instance_of(AssignmentsController).to receive(:undo_link)
.with('Assignment "test assignment" has been created successfully. ').and_return(true)
post :create, @params
expect(response).to redirect_to('/assignments/1/edit')
end
end
</pre>
= Team Information =
Mentor: Nicholas Himes (nnhimes)

Henry Chen (hchen34)

Saurabh Nanda (snanda2)

Snehapriyaa Mathiyalaghan (smathiy)

CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment

2021-10-20T21:49:21Z

Hchen34: Undo revision 139531 by Hdu5 (talk)

This wiki page describes the changes made under E2138, in order to auto-generate submission directory names based on assignment names for Fall 2021, CSC/ECE 517.

__TOC__

= Overview =

== About Expertiza ==

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

== Issues for this project ==

When instructors give an assignment a name, submission directories should be auto-generated based on the assignment name, and also be unique. A number of issues existed when a previous team worked on this assignment (E2054), including
* Issue #1: The directory name should be auto-generated from the assignment name.
* Issue #2: It should be done by changing spaces in the names to underscores. E.g., the directory for Program 1 is by default "Program_1".
* Issue #3: A check should be added to prevent two assignments in the same course from having the same name.
* Issue #4: Verify or add if not present - a check to stop two assignments from sharing the same directory.
* Issue #5: On changing the name of an assignment while creating it, the code shouldn't throw a NoMethodError.

= Project implementation =

== Submitted work and demonstration of project ==

* [http://152.7.98.88:8080/ Link to deployed code on NCSU VCL]
* [https://github.com/SN-18/expertiza Link to our Github repository]

== Files involved ==
*app/controllers/impersonate_controller.rb - refactored the codes related to the issues mentioned above.
*spec/controllers/impersonate_controller_spec.rb - implemented and added some related tests.

== Changes made to code ==

* The following code was modified in app/controllers/impersonate_controller.rb, in order to refactor to:
** make the code more readable.
** properly delete unused variable.

* In function overwrite_session:
'''Before'''
<pre>
def overwrite_session
# If not impersonatable, then original user's session remains
if params[:impersonate].nil?
# E1991 : check whether instructor is currently in anonymized view
user = User.anonymized_view?(session[:ip]) ? User.real_user_from_anonymized_name(params[:user][:name]) : user = User.find_by(name: params[:user][:name])
session[:super_user] = session[:user] if session[:super_user].nil?
AuthController.clear_user_info(session, nil)
session[:original_user] = @original_user
session[:impersonate] = true
session[:user] = user
else
# If some user is to be impersonated, their session details are overwritten onto the current to impersonate
if !params[:impersonate][:name].empty?
# E1991 : check whether instructor is currently in anonymized view
user = User.anonymized_view?(session[:ip]) ? User.real_user_from_anonymized_name(params[:impersonate][:name]) : user = user = User.find_by(name: params[:impersonate][:name])
AuthController.clear_user_info(session, nil)
session[:user] = user
session[:impersonate] = true
session[:original_user] = @original_user
else
# E1991 : check whether instructor is currently in anonymized view
user = User.anonymized_view?(session[:ip]) ? User.real_user_from_anonymized_name(params[:user][:name]) : user = User.find_by(name: params[:user][:name])
AuthController.clear_user_info(session, nil)
session[:user] = session[:super_user]
user = session[:user]
session[:super_user] = nil
end
end
end
</pre>

'''After'''
<pre>
def overwrite_session
# If not impersonatable, then original user's session remains
if params[:impersonate].nil?
# E1991 : check whether instructor is currently in anonymized view
if User.anonymized_view?(session[:ip])
user = User.real_user_from_anonymized_name(params[:user][:name])
else
user = User.find_by(name: params[:user][:name])
end
session[:super_user] = session[:user] if session[:super_user].nil?
AuthController.clear_user_info(session, nil)
session[:original_user] = @original_user
session[:impersonate] = true
session[:user] = user
# If some user is to be impersonated, their session details are overwritten onto the current to impersonate
elsif !params[:impersonate][:name].empty?
# E1991 : check whether instructor is currently in anonymized view
if User.anonymized_view?(session[:ip])
user = User.real_user_from_anonymized_name(params[:impersonate][:name])
else
user = User.find_by(name: params[:impersonate][:name])
end
AuthController.clear_user_info(session, nil)
session[:user] = user
session[:impersonate] = true
session[:original_user] = @original_user
else
# E1991 : check whether instructor is currently in anonymized view
AuthController.clear_user_info(session, nil)
session[:user] = session[:super_user]
session[:super_user] = nil
end
end
</pre>

= Test Plan =

== Manual UI Testing ==
The following steps must be performed to test the project UI: 
'''Step 1:''' Log in as an Instructor, with Username - instructor6, Password - password 
[[File:expertiza_login.PNG]] 
'''Step 2:''' Create a new assignment. Assignment is named as "Test Assignment", under course CSC/ECE 517 Fall 2020.
The directory name gets auto generated with replacing space by underscore in assignment name. 
[[File:auto_generated.PNG]] 
'''Step 3:''' Assignment gets successfully created. Assignment is named as "Test Assignment" and gets saved in the directory "Test_assignment", under course CSC/ECE 517 Fall 2020 
[[File:save_success.PNG]] 
'''Step 4:''' Try saving another assignment with same name. Following error is observed. br> 
[[File:error.PNG]] 
== RSpec Testing ==
The following RSpec tests are corrected in the assignments_contoller_spec.rb file
'''Before'''
<pre>
context 'when assignment_form is saved successfully' do
it 'redirects to assignment#edit page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(course_id:1, name:'test assignment').and_return(assignment)
allow_any_instance_of(AssignmentsController).to receive(:undo_link)
.with('Assignment "test assignment" has been created successfully. ').and_return(true)
post :create, @params
expect(response).to redirect_to('/assignments/1/edit')
end
end
</pre>
'''After'''
<pre>
context 'when assignment_form is saved successfully' do
it 'redirects to assignment#edit page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(course_id:1, name:'test assignment').and_return(assignment)
allow_any_instance_of(AssignmentsController).to receive(:undo_link)
.with('Assignment "test assignment" has been created successfully. ').and_return(true)
post :create, @params
expect(response).to redirect_to('/assignments/1/edit')
end
end
</pre>
= Team Information =
Mentor: Nicholas Himes (nnhimes)

Henry Chen (hchen34)

Saurabh Nanda (snanda2)

Snehapriyaa Mathiyalaghan (smathiy)

CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment

2021-10-20T21:49:02Z

Hchen34: Undo revision 139532 by Hdu5 (talk)

This wiki page describes the changes made under E2138, in order to auto-generate submission directory names based on assignment names for Fall 2021, CSC/ECE 517.

__TOC__

= Overview =

== About Expertiza ==

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

== Issues for this project ==

When instructors give an assignment a name, submission directories should be auto-generated based on the assignment name, and also be unique. A number of issues existed when a previous team worked on this assignment (E2054), including
* Issue #1: The directory name should be auto-generated from the assignment name.
* Issue #2: It should be done by changing spaces in the names to underscores. E.g., the directory for Program 1 is by default "Program_1".
* Issue #3: A check should be added to prevent two assignments in the same course from having the same name.
* Issue #4: Verify or add if not present - a check to stop two assignments from sharing the same directory.
* Issue #5: On changing the name of an assignment while creating it, the code shouldn't throw a NoMethodError.

= Project implementation =

== Submitted work and demonstration of project ==

* [http://152.7.98.88:8080/ Link to deployed code on NCSU VCL]
* [https://github.com/SN-18/expertiza Link to our Github repository]

== Files involved ==
*app/controllers/impersonate_controller.rb - refactored the codes related to the issues mentioned above.
*spec/controllers/impersonate_controller_spec.rb - implemented and added some related tests.

== Changes made to code ==

* The following code was modified in app/controllers/impersonate_controller.rb, in order to refactor to:
** make the code more readable.
** properly delete unused variable.

=== In function overwrite_session ===
'''Before'''
<pre>
def overwrite_session
# If not impersonatable, then original user's session remains
if params[:impersonate].nil?
# E1991 : check whether instructor is currently in anonymized view
user = User.anonymized_view?(session[:ip]) ? User.real_user_from_anonymized_name(params[:user][:name]) : user = User.find_by(name: params[:user][:name])
session[:super_user] = session[:user] if session[:super_user].nil?
AuthController.clear_user_info(session, nil)
session[:original_user] = @original_user
session[:impersonate] = true
session[:user] = user
else
# If some user is to be impersonated, their session details are overwritten onto the current to impersonate
if !params[:impersonate][:name].empty?
# E1991 : check whether instructor is currently in anonymized view
user = User.anonymized_view?(session[:ip]) ? User.real_user_from_anonymized_name(params[:impersonate][:name]) : user = user = User.find_by(name: params[:impersonate][:name])
AuthController.clear_user_info(session, nil)
session[:user] = user
session[:impersonate] = true
session[:original_user] = @original_user
else
# E1991 : check whether instructor is currently in anonymized view
user = User.anonymized_view?(session[:ip]) ? User.real_user_from_anonymized_name(params[:user][:name]) : user = User.find_by(name: params[:user][:name])
AuthController.clear_user_info(session, nil)
session[:user] = session[:super_user]
user = session[:user]
session[:super_user] = nil
end
end
end
</pre>

'''After'''
<pre>
def overwrite_session
# If not impersonatable, then original user's session remains
if params[:impersonate].nil?
# E1991 : check whether instructor is currently in anonymized view
if User.anonymized_view?(session[:ip])
user = User.real_user_from_anonymized_name(params[:user][:name])
else
user = User.find_by(name: params[:user][:name])
end
session[:super_user] = session[:user] if session[:super_user].nil?
AuthController.clear_user_info(session, nil)
session[:original_user] = @original_user
session[:impersonate] = true
session[:user] = user
# If some user is to be impersonated, their session details are overwritten onto the current to impersonate
elsif !params[:impersonate][:name].empty?
# E1991 : check whether instructor is currently in anonymized view
if User.anonymized_view?(session[:ip])
user = User.real_user_from_anonymized_name(params[:impersonate][:name])
else
user = User.find_by(name: params[:impersonate][:name])
end
AuthController.clear_user_info(session, nil)
session[:user] = user
session[:impersonate] = true
session[:original_user] = @original_user
else
# E1991 : check whether instructor is currently in anonymized view
AuthController.clear_user_info(session, nil)
session[:user] = session[:super_user]
session[:super_user] = nil
end
end
</pre>

= Test Plan =

== Manual UI Testing ==
The following steps must be performed to test the project UI: 
'''Step 1:''' Log in as an Instructor, with Username - instructor6, Password - password 
[[File:expertiza_login.PNG]] 
'''Step 2:''' Create a new assignment. Assignment is named as "Test Assignment", under course CSC/ECE 517 Fall 2020.
The directory name gets auto generated with replacing space by underscore in assignment name. 
[[File:auto_generated.PNG]] 
'''Step 3:''' Assignment gets successfully created. Assignment is named as "Test Assignment" and gets saved in the directory "Test_assignment", under course CSC/ECE 517 Fall 2020 
[[File:save_success.PNG]] 
'''Step 4:''' Try saving another assignment with same name. Following error is observed. br> 
[[File:error.PNG]] 
== RSpec Testing ==
The following RSpec tests are corrected in the assignments_contoller_spec.rb file
'''Before'''
<pre>
context 'when assignment_form is saved successfully' do
it 'redirects to assignment#edit page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(course_id:1, name:'test assignment').and_return(assignment)
allow_any_instance_of(AssignmentsController).to receive(:undo_link)
.with('Assignment "test assignment" has been created successfully. ').and_return(true)
post :create, @params
expect(response).to redirect_to('/assignments/1/edit')
end
end
</pre>
'''After'''
<pre>
context 'when assignment_form is saved successfully' do
it 'redirects to assignment#edit page' do
allow(assignment_form).to receive(:assignment).and_return(assignment)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(Assignment).to receive(:find_by).with(any_args).and_return(false)
allow(assignment_form).to receive(:save).and_return(true)
allow(assignment_form).to receive(:create_assignment_node).and_return(double('node'))
allow(assignment_form).to receive(:update).with(any_args).and_return(true)
allow(assignment).to receive(:id).and_return(1)
allow(Assignment).to receive(:find_by).with(course_id:1, name:'test assignment').and_return(assignment)
allow_any_instance_of(AssignmentsController).to receive(:undo_link)
.with('Assignment "test assignment" has been created successfully. ').and_return(true)
post :create, @params
expect(response).to redirect_to('/assignments/1/edit')
end
end
</pre>
= Team Information =
Mentor: Nicholas Himes (nnhimes)

Henry Chen (hchen34)

Saurabh Nanda (snanda2)

Snehapriyaa Mathiyalaghan (smathiy)

CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment

2021-10-20T00:36:06Z

Hchen34: minor formatting changes

CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment

2021-10-20T00:34:28Z

Hchen34: edited Project implementation section

CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment

2021-10-18T20:04:19Z

Hchen34: update VCL address

This wiki page describes the changes made under E2138, in order to auto-generate submission directory names based on assignment names for Fall 2021, CSC/ECE 517.

__TOC__

==About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

==Issues for this project==
When instructors give an assignment a name, submission directories should be auto-generated based on the assignment name, and also be unique. A number of issues existed when a previous team worked on this assignment (E2054), including
* Issue #1: The directory name should be auto-generated from the assignment name.
* Issue #2: It should be done by changing spaces in the names to underscores. E.g., the directory for Program 1 is by default "Program_1".
* Issue #3: A check should be added to prevent two assignments in the same course from having the same name.
* Issue #4: Verify or add if not present - a check to stop two assignments from sharing the same directory.
* Issue #5: On changing the name of an assignment while creating it, the code shouldn't throw a NoMethodError.

= Project implementation =

== Submitted work and demonstration of project ==
* [http://152.7.98.88:8080/ Link to deployed code on NCSU VCL]
* [https://github.com/SN-18/expertiza Link to our Github repository]

== Files involved ==

== Changes made to code ==

= Test Plan =

== Manual UI Testing ==

== RSpec Testing ==

= Team Information =

Henry Chen (hchen34)

Saurabh Nanda (snanda2)

Snehapriyaa Mathiyalaghan (smathiy)

Mentor: Nicholas Himes (nnhimes)

CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment

2021-10-17T20:47:35Z

Hchen34:

This wiki page describes the changes made under E2138, in order to auto-generate submission directory names based on assignment names for Fall 2021, CSC/ECE 517.

__TOC__

==About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

==Issues for this project==
When instructors give an assignment a name, submission directories should be auto-generated based on the assignment name, and also be unique. A number of issues existed when a previous team worked on this assignment (E2054), including
* Issue #1: The directory name should be auto-generated from the assignment name.
* Issue #2: It should be done by changing spaces in the names to underscores. E.g., the directory for Program 1 is by default "Program_1".
* Issue #3: A check should be added to prevent two assignments in the same course from having the same name.
* Issue #4: Verify or add if not present - a check to stop two assignments from sharing the same directory.
* Issue #5: On changing the name of an assignment while creating it, the code shouldn't throw a NoMethodError.

= Project implementation =

== Submitted work and demonstration of project ==
* [http://152.7.99.133:8080/ Link to deployed code on NCSU VCL]
* [https://github.com/SN-18/expertiza Link to our Github repository]

== Files involved ==

== Changes made to code ==

= Test Plan =

== Manual UI Testing ==

== RSpec Testing ==

= Team Information =

Henry Chen (hchen34)

Saurabh Nanda (snanda2)

Snehapriyaa Mathiyalaghan (smathiy)

Mentor: Nicholas Himes (nnhimes)

CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment

2021-10-17T20:46:06Z

Hchen34: initial edit

This wiki page describes the changes made under E2138, in order to auto-generate submission directory names based on assignment names for Fall 2021, CSC/ECE 517.

__TOC__

==About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

==Issues for this project==
When instructors give an assignment a name, submission directories should be auto-generated based on the assignment name, and also be unique. A number of issues existed when a previous team worked on this assignment (E2054), including
* Issue #1: The directory name should be auto-generated from the assignment name.
* Issue #2: It should be done by changing spaces in the names to underscores. E.g., the directory for Program 1 is by default "Program_1".
* Issue #3: A check should be added to prevent two assignments in the same course from having the same name.
* Issue #4: Verify or add if not present - a check to stop two assignments from sharing the same directory.
* Issue #5: On changing the name of an assignment while creating it, the code shouldn't throw a NoMethodError.

= Project implementation =

== Submitted work and demonstration of project ==
* [http://152.7.99.133:8080/ Link to deployed code on NCSU VCL]
* [https://github.com/SN-18/expertiza Link to our Github repository]

== Files involved ==

== Changes made to code ==

= Test Plan =

== Manual UI Testing ==

== RSpec Testing ==

= Team Information =

1) Henry Chen (hchen34)
2) Saurabh Nanda (snanda2)
3) Snehapriyaa Mathiyalaghan (smathiy)

Mentor: Nicholas Himes (nnhimes)

CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment

2021-10-17T20:27:36Z

Hchen34: Created page with "Test"

Test

CSC/ECE 517 Fall 2021

2021-10-17T20:26:37Z

Hchen34: add E2138

== OSS Projects ==
* [[CSC/ECE 517 Fall 2021 - {$num}. {$desc.title}]]
* [[CSC/ECE 517 Fall 2021 - E2128. Refactor student_quizzes_controller.rb & late_policies_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2132. Add tests cases for review mapping helper.rb]]
* [[CSC/ECE 517 Fall 2021 - E2134. Write unit tests for admin_controller.rb and institution_controller.rb]]
* [[CSC/ECE 517 Fall 2021 - E2138. Auto-generate submission directory names based on assignment]]
* [[CSC/ECE 517 Fall 2021 - E2142. Improve e-mail notifications]]