CSC/ECE 517 Spring 2024 - E2440 Testing for questionnaire helper, review bids helper: Difference between revisions

From Expertiza_Wiki
Jump to navigation Jump to search
Line 115: Line 115:
== Objective 1: Develop code testing scenarios for questionnaire_helper ==
== Objective 1: Develop code testing scenarios for questionnaire_helper ==


== Objective 2: Reimplement the algorithm if discrepancies arise in the reputation web server's Hamer values. ==
== Objective 2: Develop code testing scenarios for review_bids_helper ==
 
As established before, the values returned by reputation server do not match the expected values. Hence, we concluded that the PeerLogic Webservice is implemented incorrectly. In this phase, we implemented the algorithm in Ruby as a function in a controller file : /app/controllers/reputation_web_service_controller.rb
 
=== Changes made in implementation ===
* coded this algorithm in Ruby in a controller.
* Included a way for the algorithm to handle nil values.
 
=== Code Snippet ===
<pre>
 
# Method: calculate_reputation_score
# This method calculates the reputation scores for each reviewer based on the provided input data.
# It first parses the input JSON string to extract the submissions and their corresponding scores.
# Then, it calculates the average weighted grades per reviewer and the delta R values.
# Next, it calculates the weight prime values based on the delta R values.
# Finally, it calculates the reputation weights for each reviewer using the weight prime values.
#
# Params
#  - input_json: a JSON string representing the input data with submission scores
#
# Returns
#  An array of reputation scores, one score per reviewer, indicating their reputation in the system.
def calculate_reputation_score(reviews)
  # Parse the input JSON string
  # reviews = JSON.parse(input_json)
 
  # Initialize arrays to store intermediate values
  grades = []
  delta_r = []
  weight_prime = []
  weight = []
 
  # Calculate Average Weighted Grades per Reviewer
  reviews.each do |reviewer_marks|
    # Skip nil values when calculating the sum
    reviewer_marks_without_nil = reviewer_marks.compact
    assignment_grade_average = reviewer_marks_without_nil.sum.to_f / reviewer_marks_without_nil.length
    grades << assignment_grade_average
  end
 
  # Calculate delta R
  reviews.each do |reviewer_marks|
    reviewer_delta_r = 0
    # Skip nil values when calculating the sum
    reviewer_marks_without_nil = reviewer_marks.compact
    reviewer_marks_without_nil.each_with_index do |grade, student_index|
      reviewer_delta_r += (grade - grades[student_index]) ** 2
    end
    delta_r << reviewer_delta_r / reviewer_marks_without_nil.length
  end
 
  # Calculate weight prime
  average_delta_r = delta_r.sum / delta_r.length.to_f
 
  delta_r.each do |reviewer_delta_r|
    weight_prime << average_delta_r / reviewer_delta_r
  end
 
  # Calculate reputation weight
  weight_prime.each do |reviewer_weight_prime|
    if reviewer_weight_prime <= 2
      weight << reviewer_weight_prime.round(2)
    else
      weight << (2 + Math.log(reviewer_weight_prime - 1)).round(2)
    end
  end
 
  # Return the reputation weights
  weight
  end
end
</pre>


== Objective 3: Validate the accuracy of the newly implemented Hamer algorithm ==
== Objective 3: Validate the accuracy of the newly implemented Hamer algorithm ==

Revision as of 16:55, 8 April 2024

This page describes the changes made for the Spring 2024 E2440. Testing for questionnaire_helper, review_bids_helper

Project Overview

Problem Statement

Our project involves writing test cases for the `questionnaire_helper` and `review_bids_helper` files in Expertiza, an open-source assignment/project management portal built on the Ruby on Rails framework. This platform facilitates collaborative learning and feedback among both instructors and students. Instructors can create and customize assignments, define topics for student sign-ups, and manage teams for various projects. Students, on the other hand, can sign up for topics, form teams, and participate in peer reviews to enhance each other's learning experiences. Our goal is to develop comprehensive test plans and increase code coverage for these helper files to ensure their reliability and effectiveness in the Expertiza platform.

Current Code Coverage

Objectives

  • Develop test plans/scenarios for questionnaire_helper.rb
  • Develop test plans/scenarios for review_bids_helper.rb
  • Improve code coverage for questionnaire_helper.rb
  • Improve code coverage for review_bids_helper.rb

Files Involved

  • Questionnaire_helper.rb: /app/helpers/questionnaire_helper.rb
  • Review_bids_helper.rb: /app/helpers/review_bids_helper.rb

Mentor

  • Muhammet Mustafa Olmez (molmez@ncsu.edu)

Team Members

  • Neha Vijay Patil (npatil2@ncsu.edu)
  • Prachit Mhalgi (psmhalgi@ncsu.edu)
  • Sahil Santosh Sawant (ssawant2@ncsu.edu)

Class and Method Overview

QuestionnaireHelper

The QuestionnaireHelper module contains several methods to assist with managing questionnaires in expertiza. QuestionnaireHelper provides methods for adjusting advice size, updating questionnaire questions, and creating questionnaire instances based on types. It also defines constants to facilitate these functionalities. These methods are likely used within the application to handle questionnaire-related tasks efficiently. Let's break down the class and its methods:

Constants

CSV_QUESTION, CSV_TYPE, CSV_PARAM, CSV_WEIGHT 

  - These constants define indices for specific columns in a CSV file.

QUESTIONNAIRE_MAP 

  - This constant is a hash that maps questionnaire types to their respective questionnaire classes.
  - It's used by the `questionnaire_factory` method to determine the appropriate class to instantiate.

Methods

1. adjust_advice_size(questionnaire, question) 

  - This method adjusts the size of advice associated with a given question in a questionnaire.
  - Parameters:
    - `questionnaire`: The questionnaire object.
    - `question`: The question object whose advice size needs adjustment.
  - Functionality:
    - Checks if the question is a `ScoredQuestion`.
    - Deletes any existing advice for the question outside the score range.
    - Iterates over the score range and ensures each score has an associated advice.
    - Deletes any duplicate advice records.

2. update_questionnaire_questions 

  - This method updates attributes of questionnaire questions based on form data, without modifying unchanged attributes.
  - Functionality:
    - Checks for presence of `params[:question]`.
    - Iterates through each question and its attributes in the parameters.
    - Compares each attribute's current value with the new value from the parameters and updates if changed.
    - Saves the question.

3. questionnaire_factory(type) 

  - This method acts as a factory to create an appropriate questionnaire object based on the type provided.
  - Parameters:
    - `type`: The type of questionnaire.
  - Functionality:
    - Retrieves the questionnaire class from `QUESTIONNAIRE_MAP` based on the provided type.
    - If the type is not found in the map, it sets an error flash message.
    - Otherwise, it initializes a new instance of the corresponding questionnaire class and returns it.

ReviewBidsHelper

This Ruby module, `ReviewBidsHelper` serves as a helper module for views related to reviewing bids in expertiza. `ReviewBidsHelper` provides helper methods for rendering topic rows and determining the background color for topics based on their bid status and the number of participants. These methods are likely used in the views associated with reviewing bids in the application. Let's break down the class and its methods:

Methods

1. get_intelligent_topic_row_review_bids(topic, selected_topics, num_participants) 

  - This method seems to be responsible for generating HTML markup for a row in a table displaying topics for review bids.
  - Parameters:
    - `topic`: Represents a specific topic being reviewed.
    - `selected_topics`: An array of topics that have been selected.
    - `num_participants`: The number of participants involved in the review process.
  - Functionality:
    - Iterates through the `selected_topics`.
    - Depending on the conditions (whether the topic is selected and not waitlisted, or selected and waitlisted), it generates a specific row HTML.
    - Returns the generated row HTML as safe HTML.

2. get_topic_bg_color_review_bids(topic, num_participants) 

  - This method calculates the background color for a topic based on the number of participants and the number of bids for that topic.
  - Parameters:
    - `topic`: Represents the topic for which the background color is being determined.
    - `num_participants`: The total number of participants.
  - Functionality:
    - Calculates the number of bids for the given `topic`.
    - Determines the proportion of bids compared to the total number of participants and adjusts the color accordingly.
    - Returns a string representing the RGB value of the background color.

Hamer value calculation




Objective 1: Develop code testing scenarios for questionnaire_helper

Objective 2: Develop code testing scenarios for review_bids_helper

Objective 3: Validate the accuracy of the newly implemented Hamer algorithm

We test the newly implemented Hamer algorithm function with our scenarios and verify if they match the expected values.

Test Code Snippet

describe ReputationWebServiceController do
    it "should calculate correct Hamer calculation" do
      weights = ReputationWebServiceController.new.calculate_reputation_score(reviews)
      keys = ["maxtoall", "mintoall", "mediantoall", "incomplete_review", "sametoall", "passing1", "passing2", "passing3"]
      rounded_weights = weights.map { |w| w.round(1) }
      result_hash = keys.zip(rounded_weights).to_h
      expect(result_hash).to eq(JSON.parse(EXPECTED)["Hamer"])
    end
end

Results

Conclusion

The observed results indicate a tendency towards lower values, primarily due to our decision to include nil values and treat them as zeros in our analysis. This treatment has led to a skew in the scores, favoring lower values and potentially impacting the accuracy of our findings. To address this issue and improve the robustness of our analysis, it is advisable to explore alternative approaches such as using median or random values instead of treating nil values as zeros. However, we must also carefully consider how to handle incomplete reviews that contain nil values in our input dataset, as this can significantly influence the overall integrity and reliability of our results and conclusions.

Conclusion

In this project, we aimed to test the accuracy of the Hamer algorithm used for assessing the credibility of reviewers in a peer assessment system. We began by developing code testing scenarios to validate the Hamer algorithm and ensure the accuracy of its output values. These scenarios covered various review scenarios, including cases where reviewers provided extreme scores.

It was established that the original reputation web server was implemented incorrectly.

As a result, we proceeded to reimplement the Hamer algorithm in Ruby, incorporating adjustments to handle nil values appropriately. Subsequently, we validated the accuracy of the newly implemented algorithm using the same testing scenarios. While the results initially showed a skew towards lower values due to our treatment of nil values, we acknowledge the need for further refinement to handle these cases more effectively.

In conclusion, this project highlights the importance of rigorous testing and implementation adjustments in ensuring the reliability of algorithms used in peer assessment systems. Moving forward, we recommend further refinements and validations to enhance the accuracy and robustness of the Hamer algorithm.

Links

Link to Expertiza repository: here

Link to the forked repository: here

Link to pull request: here

Link to Github Project page: here

Link to Testing Video: here

References

1. Expertiza on GitHub (https://github.com/expertiza/expertiza)
2. The live Expertiza website (http://expertiza.ncsu.edu/)
3. Pluggable reputation systems for peer review: A web-service approach (https://doi.org/10.1109/FIE.2015.7344292)

Retrieved from "https://wiki.expertiza.ncsu.edu/index.php?title=CSC/ECE_517_Spring_2024_-_E2440_Testing_for_questionnaire_helper,_review_bids_helper&oldid=154991"