CSC/ECE 517 Spring 2018 E1814 Write unit tests for collusion cycle.rb

From Expertiza_Wiki
Jump to navigation Jump to search

Introduction

Expertiza Background

Expertiza is an opensource web application maintained by students and faculty members of North Carolina State University. Through it students can submit and peer-review learning objects (articles, code, web sites, etc). More information on http://wiki.expertiza.ncsu.edu/index.php/Expertiza_documentation.

Project Description

Collusion_cycle.rb is a part of the Expertiza project used to calculate potential collusion cycles between peer reviews. Our purpose is to write corresponding unit tests (collusion_cycle_spec.rb) for it.

Project Details

Model introduction

Collusion_Cycle is a model of Expertiza which is used to detect if there is a cycle among reviewers and to record the scores. At this time, it only has functions to detect the cycle up to 4 people.

Model purpose

The file collusion_cycle.rb in app/models is used to calculate potential collusion cycles between peer reviews which means two assignment participants are accidentally assigned to review each other's assignments directly or through a so-called collusion cycle. For example, if participant A was reviewed by participant B, participant B was reviewed by participant C, participant C was reviewed by participant A and all the reviews were indeed finished, then a three nodes cycle exists. Cases including two nodes, three nodes and four nodes have been covered in collusion_cycle.rb.

Functions introduction

1. n_node_cycles(assignment_participant)

As we discussed above, this model can only detect the cycle up to 4 people. So there are three functions corresponding to two people, three people and four people separately.

The basic algorithm of these detection functions is first to find a closed loop among reviewers. Take three people as an example, a closed loop means A is a reviewer of B, B is a reviewer of C and C is a reviewer of A. After finding the loop among reviewers, the functions will test the validation of the loop. Validation test is necessary because there may be a situation that A was assigned to be the reviewer of B but didn't do the review.That will create an invalid review response which will break the loop. Finally after the functions find a valid closed loop, the function will record all the reviewers and there scores in an array named collusion_cycles.

2. cycle_similarity_score(cycle)

The function below is used to calculate the similarity which is the average difference between each pair of scores.

For example, if we input a cycle with data like [[participant1, 100], [participant2, 95], [participant3, 85]] (PS: 100 is the score participant1 receives.), the similarity score will be (|100-95|+|100-85|+|95-85|)/3 = 10.0 . 

def cycle_similarity_score(cycle)
	similarity_score = 0.0
	count = 0.0
	for pivot in 0...cycle.size - 1 do
		pivot_score = cycle[pivot][1]
		for other in pivot + 1...cycle.size do
			similarity_score += (pivot_score - cycle[other][1]).abs
			count += 1.0
      		end
    	end
    	similarity_score /= count unless count == 0.0
    	similarity_score
end

3. cycle_deviation_score(cycle)

The function below is used to calculate the deviation of one cycle which is the average difference of all participants between the standard review score and the review score from this particular cycle.

For example, if we input a cycle with data like [[participant1, 95], [participant2, 90], [participant3, 91]] (PS: 95 is the score participant1 receives.) and the standard scores for each are 90, 80 and 91, the deviation score will be (|95-90|+|90-80|+|91-91|)/3 = 5.0 . 

def cycle_deviation_score(cycle)
  	deviation_score = 0.0
  	count = 0.0
  	for member in 0...cycle.size do
    		participant = AssignmentParticipant.find(cycle[member][0].id)
    		total_score = participant.review_score
    		deviation_score += (total_score - cycle[member][1]).abs
    		count += 1.0
  	end
  	deviation_score /= count unless count == 0.0
  	deviation_score
end

Test Plan

Cycle Detection

For all three functions of collusion detection, the tests can be divided into three parts: test the loop closure, test the review validation and test the final output result.

1. two_node_cycle

As shown in the figure above, Two_node_cycle is very simple. If two people both create validate review response to each other, a two_node_cycle is generated. The line response ab represents b's review of a's work. So for the unit test of this function, we first test no collusion situation that participant1 is not a reviewer of participant2. Then we test if response12 or response21 is invalid. Finally we offer the function a totally valid two_node_cycle and test its output. 

  context "two_node_cycle" do
    before(:each) do
      allow(participant1).to receive(:reviewers).and_return([participant2])
      allow(response12).to receive(:total_score).and_return(100)
      allow(response21).to receive(:total_score).and_return(90)
    end

    it "no collusion" do
      allow(participant2).to receive(:reviewers).and_return([])
      expect(colcyc.two_node_cycles(participant1)).to eq ([])
    end

    it "Cycle collude but review12 is nil" do
      allow(participant2).to receive(:reviewers).and_return([participant1])
      allow(participant1).to receive(:reviews_by_reviewer).with(participant2).and_return(nil)
      expect(colcyc.two_node_cycles(participant1)).to eq ([])
    end

    it "Cycle collude but review21 is nil" do
      allow(participant2).to receive(:reviewers).and_return([participant1])
      allow(participant1).to receive(:reviews_by_reviewer).with(participant2).and_return(response12)
      allow(participant2).to receive(:reviews_by_reviewer).with(participant1).and_return(nil)
      expect(colcyc.two_node_cycles(participant1)).to eq ([])
    end

    it "Cycle collude and all reviews are not nil" do
      allow(participant2).to receive(:reviewers).and_return([participant1])
      allow(participant1).to receive(:reviews_by_reviewer).with(participant2).and_return(response12)
      allow(participant2).to receive(:reviews_by_reviewer).with(participant1).and_return(response21)
      expect(colcyc.two_node_cycles(participant1)).to eq ([[[participant1, 100],[participant2, 90]]])
    end
  end

Above is the code for two_node_cycle function test. It's easy to see the test is divided into 3 situations--no collusion, collude but one of the review responses is nil and collude and all responses are valid. These three situations are the basic format of two_node_cycles test, and the following functions are tested by the same way. The only differences are some details, such as the number of responses or how to set the no collusion situation. Theoretically speaking, this format is suitable for all n_node_cycles functions built in the same way as these three functions.

2. three_node_cycle

For three_node_cycle, we first test no collusion situation. And for no collusion test we don't have to test all the disconnected situation, such as participant2 is not a reviewer of participant1. We only need to test the last connection of the loop--participant1 is not a reviewer of participant3. Then we test validation of all the responses. Set these responses to be nil separately and see what will happen. In the end we input a valid three_node cycle to see if the output result is what it supposed to be.

3. four_node_cycle

Four_node_cycle is similar to three_node_cycle. The test for no collusion sets participant1 not to be a reviewer of participant4. Then set the responses to be nil separately to test the validation. Finally create a valid four_node_cycle and test the output result.

Scores

1. cycle_similarity_score(cycle)

The code below is the unit test for the cycle_similarity_score(cycle) function in CollusionCycle class. The function is very simple, as it doesn't call any other functions. We only need to input the appropriate data and check whether the output is correct. So for each cases, 2 nodes or 3 nodes, I set different cycle as input and check the result of the function. 

context "#cycle_similarity_score" do
    	it "similarity score of 2 node test" do
      		cycle = [[participant1, 100], [participant2, 90]]
      		expect(colcyc.cycle_similarity_score(cycle)).to eq (10.0)
    	end

    	it "similarity score of 3 node test" do
      		cycle = [[participant1, 100], [participant2, 95], [participant3, 85]]
      		expect(colcyc.cycle_similarity_score(cycle)).to eq (10)
    	end

	it "similarity score of 4 node test" do
		cycle = [[participant1, 100], [participant2, 95], [participant3, 95], [participant4, 90]]
      		expect(colcyc.cycle_similarity_score(cycle)).to eq (5)
	end
end

2. cycle_deviation_score(cycle)

The code below is the unit test for the cycle_deviation_score(cycle) function in CollusionCycle class, It calls review_score method from AssignmentParticipant class. This method will return a standard score of the participant. But in our project, how the review_score method works and whether the result of the review_score method is correct are not parts of our duty (PS: I think it should be tested in unit tests of AssignmentParticipant class). So I set the results of review_score method for each participant. And also set different cycles as input and check the result of the function. 

context "#cycle_deviation_score" do
	before(:each) do
		allow(participant1).to receive(:id).and_return(1)
		allow(participant2).to receive(:id).and_return(2)
		allow(participant3).to receive(:id).and_return(3)
		allow(participant4).to receive(:id).and_return(4)     
		allow(AssignmentParticipant).to receive(:find).with(1).and_return(participant1)
		allow(AssignmentParticipant).to receive(:find).with(2).and_return(participant2)
		allow(AssignmentParticipant).to receive(:find).with(3).and_return(participant3)
		allow(AssignmentParticipant).to receive(:find).with(4).and_return(participant4)
		allow(participant1).to receive(:review_score).and_return(90) 			
		allow(participant2).to receive(:review_score).and_return(80)
		allow(participant3).to receive(:review_score).and_return(91)
		allow(participant4).to receive(:review_score).and_return(95)
	end

    	it "deviation score for 2 nodes test" do
      		cycle = [[participant1, 95], [participant2, 90]]
      		expect(colcyc.cycle_deviation_score(cycle)).to eq (7.5)
    	end

    	it "deviation score for 3 nodes test" do
      		cycle = [[participant1, 95], [participant2, 90], [participant3, 91]]
      		expect(colcyc.cycle_deviation_score(cycle)).to eq (5.0)
    	end

    	it "deviation score for 4 nodes test" do
      		cycle = [[participant1, 95], [participant2, 90], [participant3, 91], [participant4, 90]]
      		expect(colcyc.cycle_deviation_score(cycle)).to eq (5.0)
    	end
  end

Coverage Result

The following figure is the coverage result of our test. Obviously, we achieve a perfect coverage result.

Future Work

  • This model can only test cycle containing up to 4 nodes, maybe functions for cycle with more nodes can be added.
  • Apart from adding more functions, combination of all these cycle functions into a new function with parameter n, which represents the number of nodes, will be a better choice. So that detecting cycle with different nodes can be realized by calling the same function with different parameter rather than adding a new function.

Reference

Retrieved from "https://wiki.expertiza.ncsu.edu/index.php?title=CSC/ECE_517_Spring_2018_E1814_Write_unit_tests_for_collusion_cycle.rb&oldid=115484"