CSC/ECE 517 Spring 2018 E1814 Write unit tests for collusion cycle.rb

Introduction

Expertiza Background

Expertiza is an opensource web application maintained by students and faculty members of North Carolina State University. Through it students can submit and peer-review learning objects (articles, code, web sites, etc). More information on http://wiki.expertiza.ncsu.edu/index.php/Expertiza_documentation.

Project Description

Collusion_cycle.rb is a part of the Expertiza project used to calculate potential collusion cycles between peer reviews. Our purpose is to write corresponding unit tests (collusion_cycle_spec.rb) for it.

Project Details

Model introduction

Collusion_Cycle is a model of Expertiza which is used to detect if there is a cycle between reviewers and record the scores. And at this time it can detect the cycle only up to 4 people.

Model purpose

The file collusion_cycle.rb in app/models is used to calculate potential collusion cycles between peer reviews which means two assignment participants are accidentally assigned to review each other's assignments directly or through a so-called collusion cycle. For example, if participant A was reviewed by participant B, participant B was reviewed by participant C, participant C was reviewed by participant A and all the reviews were indeed finished, then a three nodes cycle exists. Cases including two nodes, three nodes and four nodes have been covered in collusion_cycle.rb.

Functions introduction

1. n_node_cycles(assignment_participant)

As we talked above, this model can only detect the cycle up to 4 people. So there are three functions corresponding to two people, three people and four people separately.

The basic algorithm of these detection functions is first to find a closed loop among reviewers. Take three people as an example, a closed loop means A is a reviewer of B, B is a reviewer of C and C is a reviewer of A. After finding the loop among reviewers, the functions will test the validation of the loop. Validation test is necessary because there may be a situation that A was assigned to be the reviewer of B but didn't do the review.That will create an invalid review response which will break the loop. Finally after the functions find a validate closed loop, they will record all the reviewers and there scores in an array named collusion_cycles.

2. cycle_similarity_score(cycle)

The function below is used to calculate the similarity which is the average difference between each pair of scores.

For example, if we input a cycle with data like [[participant1, 100], [participant2, 95], [participant3, 85]] (PS: 100 is the score participant1 receives.), the similarity score will be (|100-95|+|100-85|+|95-85|)/3 = 10.0 .

def cycle_similarity_score(cycle)
	similarity_score = 0.0
	count = 0.0
	for pivot in 0...cycle.size - 1 do
		pivot_score = cycle[pivot][1]
		for other in pivot + 1...cycle.size do
			similarity_score += (pivot_score - cycle[other][1]).abs
			count += 1.0
      		end
    	end
    	similarity_score /= count unless count == 0.0
    	similarity_score
end

3. cycle_deviation_score(cycle)

The function below is used to calculate the deviation of one cycle which is the average difference of all participants between the standard review score and the review score from this particular cycle.

For example, if we input a cycle with data like [[participant1, 95], [participant2, 90], [participant3, 91]] (PS: 95 is the score participant1 receives.) and the standard scores for each are 90, 80 and 91, the deviation score will be (|95-90|+|90-80|+|91-91|)/3 = 5.0 .

def cycle_deviation_score(cycle)
  	deviation_score = 0.0
  	count = 0.0
  	for member in 0...cycle.size do
    		participant = AssignmentParticipant.find(cycle[member][0].id)
    		total_score = participant.review_score
    		deviation_score += (total_score - cycle[member][1]).abs
    		count += 1.0
  	end
  	deviation_score /= count unless count == 0.0
  	deviation_score
end

Test Description

Cycle Detection

For all three functions of collusion detection, the tests can be divided into three parts: test the loop closure, test the review validation and test the final output result.

1. two-node

As shown in the figure above, Two_node_cycle is very simple. If two people both create validate review response to each other, a two_node_cycle is generated. The line response ab represents b's review of a's work. So for the unit test of this function, we first test no collusion situation that participant1 is not a reviewer of participant2. Then we test if response12 or response21 is invalid. Finally we give it a totally validate two_node_cycle and test its output.

2. three node

For three_node_cycle, we first test no collusion situation. And for no collusion test we don't have to test all the disconnected situation, such as participant2 is not a reviewer of participant1. We only need to test the last connection of the loop--participant1 is not a reviewer of participant3. Then we test validation of all the responses. Set these responses to be nil separately and see what will happen. In the end we input a validate three_node cycle to see if the output result is what it supposed to be.

3. four node

Four_node_cycle is similar to three_node_cycle. The test for no collusion sets participant1 not to be a reviewer of participant4. Then set the responses to be nil separately to test the validation. Finally create a correct loop and test the output result.

Scores

1. cycle_similarity_score(cycle)

The code below is the unit test for the cycle_similarity_score(cycle) function in CollusionCycle class. The function is very simple, as it doesn't call any other functions. We only need to input the appropriate data and check whether the output is correct. So for each cases, 2 nodes or 3 nodes, I set different cycle as input and check the result of the function.

context "#cycle_similarity_score" do
    	it "similarity score of 2 node test" do
      		cycle = [[participant1, 100], [participant2, 90]]
      		expect(colcyc.cycle_similarity_score(cycle)).to eq (10.0)
    	end

    	it "similarity score of 3 node test" do
      		cycle = [[participant1, 100], [participant2, 95], [participant3, 85]]
      		expect(colcyc.cycle_similarity_score(cycle)).to eq (10)
    	end

	it "similarity score of 4 node test" do
		cycle = [[participant1, 100], [participant2, 95], [participant3, 95], [participant4, 90]]
      		expect(colcyc.cycle_similarity_score(cycle)).to eq (5)
	end
end

2. cycle_deviation_score(cycle)

The code below is the unit test for the cycle_deviation_score(cycle) function in CollusionCycle class, It calls review_score method from AssignmentParticipant class. This method will return a standard score of the participant. But in our project, how the review_score method works and whether the result of the review_score method is correct are not parts of our duty (PS: I think it should be tested in unit tests of AssignmentParticipant class). So I set the results of review_score method for each participant. And also set different cycles as input and check the result of the function.

context "#cycle_deviation_score" do
	before(:each) do
		allow(participant1).to receive(:id).and_return(1)
		allow(participant2).to receive(:id).and_return(2)
		allow(participant3).to receive(:id).and_return(3)
		allow(participant4).to receive(:id).and_return(4)     
		allow(AssignmentParticipant).to receive(:find).with(1).and_return(participant1)
		allow(AssignmentParticipant).to receive(:find).with(2).and_return(participant2)
		allow(AssignmentParticipant).to receive(:find).with(3).and_return(participant3)
		allow(AssignmentParticipant).to receive(:find).with(4).and_return(participant4)
		allow(participant1).to receive(:review_score).and_return(90) 			
		allow(participant2).to receive(:review_score).and_return(80)
		allow(participant3).to receive(:review_score).and_return(91)
		allow(participant4).to receive(:review_score).and_return(95)
	end

    	it "deviation score for 2 nodes test" do
      		cycle = [[participant1, 95], [participant2, 90]]
      		expect(colcyc.cycle_deviation_score(cycle)).to eq (7.5)
    	end

    	it "deviation score for 3 nodes test" do
      		cycle = [[participant1, 95], [participant2, 90], [participant3, 91]]
      		expect(colcyc.cycle_deviation_score(cycle)).to eq (5.0)
    	end

    	it "deviation score for 4 nodes test" do
      		cycle = [[participant1, 95], [participant2, 90], [participant3, 91], [participant4, 90]]
      		expect(colcyc.cycle_deviation_score(cycle)).to eq (5.0)
    	end
  end

Coverage Result

Reference

1. rspec *2

2. expertiza *3