Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-30T02:05:19Z

Aluthra: /* review_bid.rb */

==Topic Overview==
===Statement of Problem===
Reviewing is an important stage of learning and improving, both as a reviewer and a reviewee, and so allowing reviewers more freedom and choice in what they review would result in better reviews and more engaged reviewers. However, currently the only way to choose a topic to review is a first-come-first-served basis, penalizing students who do not sign up quickly by not allowing them to review the topics they would be most interesting in engaging with.

A better alternative would be to implement a bidding system, similar to that used for topics, which would allow students to establish preferences for which topics they would most like to review. A good bidding algorithm implementing with this idea would ensure that all students (who care enough to bid) receive a topic to review that they actively want.

A prior project group created a prototype for this function, allowing students to bid on what topics they wanted to review. While the functionality was as desired, the feature was not able to be merged due to having too many DRY violations. Thus, the goal of this topic was to bring this prior implementation into the current version of the expertiza beta branch, and remove these DRY violations, so that it may be merged.

As development continued, it became clear this prototype also had several bugs and issues that made it unable to work under several conditions. Thus, a secondary goal was to fix any issues which existed such that the feature would generally work with the functionality the prior implementation demonstrated.

===Prior Work===
The prior work which is the basis for this topic is linked here [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review is linked here]. This work was done in Fall of 2020, and as previously mentioned, achieved much of the desired functionality for this task. This prior work will be referenced several times within this documentation, as it has a more comprehensive explanation of the underlying design principles behind the views, controller, model, algorithm, and RSpec tests implemented. These details will only be briefly summarized here.

As previously noted, this prior implementation was rejected due to DRY violations, and so correcting these violations--and explaining how those corrections were done--will be the major focus of this project documentation.

==Explanation of Feature==

===Existing Bidding Functionality for Topics===
Currently in expertiza, students may bid on which topics they would like to work on. An image showcasing this functionality is given below:

[[File:expected_review_UI.PNG]]

Students may choose a number of topics to bid on, in order of priority, and once the deadline is reached, all students are given a topic.

Behind the scenes, this works by utilizing a modified version of the Gale-Shapely Algorithm. This algorithm seeks to find a 'stable' matching of each student to each topic, such that in the end, all students have a topic they wanted (at least to some degree), and that the preferences of students is respected as well as possible. To help break situations where there are ties in preferences, each topic assigns a Priority to each student. This Priority is determined by A. whether the student is part of a group bidding for this topic (groups getting higher priority), and B. the time at which the bid for the topic was made (earlier bidders are given Priority). Once this has been determined and the deadline for bidding has ended, the following steps occur:
# Each student is matched with each topic they most prefer.
# If the number of students assigned to a topic is above the maximum allowed for that topic, the students with the highest Priority are assigned the topic
# All maximally filled topics are removed from the pool of available topics
# All unmatched students begin this cycle again, now choosing the most preferred topic remaining
# Repeat until all students are assigned a topic
# If any matches are 'unstable' (two students would have their preferences better met if they were swapped), perform any swaps until matches are as stable as possible.

This algorithm has a time complexity of O(n^2). For more details on this algorithm and its implementation, refer to the previous year's work, where this algorithm was implemented.

===Desired Functionality for Reviews===

The desired feature for reviews should have an almost identical structure to this, with reviews instead of topics. Students should be shown a list of the topics that were worked on, and may bid on which of those topics they would like to review the most. The Priority system could be handled in the same way, with the only caveat being that a student should not be able to review their own work.

In terms of what this feature looks like, it can be enabled as an option when editing/creating a new assignment, under the "Topics" section, using interface drop down show below.

[[File:Bidding_interface.png|center|800px]]

Once this option is enabled, during the review phase of an assignment, students will be able to access a new view listing all the topics of all the submitted assignments (except for their own). They will then be able to bid on as many topics as they like, assigning them a preference, in a manner similar to the interface for topic bidding. The table that allows for this drag-and-drop priorities is shown below.

[[File:Bidding_options.png|center|900px]]

Whenever the instructor decides they would like to close the window for review bidding, they may return to the options for the given assignment and navigate to "Topics", where they will see a button marked "Run Review Algorithm". This will make a call to the external webservice to determine which students are assigned what reviews, and will then assign those reviews to all students. Once this button is pressed, it locks out the ability for students to perform any more bidding, and cannot be pressed again. The button allowing this option is shown below.

[[File:Bidding_run_algorithm.png|center|900px]]

After this point, students may navigate to the "Others Work" option under their assignment and see all of their assigned reviews. They will see up to the maximum number of allowed reviews, but will be informed of how many they are required to do (for example, a student may see 4 reviews, but will be told they are only required to complete 2).

===Use Case Diagram===
The following Use Case Diagram demonstrates the primary functionality explained in the previous section for different types of users.

[[File:Use_Case_Diagram_Rev1_E2151.jpg|center]]

==Plan and List of Work Done==

===General Design Goals===

* Move all prior work for this topic into a new version of the expertiza beta branch, without loss in functionality.
* Ensure all prior/expected functionality is working correctly, and if not, make any repairs or changes necessary.
* DRY out prior implementation's controller and model methods which have functionality that already exists within the expertiza system.
* Remove repetitive views and partials that exist by merging views for review bids into the set of views for sign up sheets, from which it uses partials.
* Ensure all present test cases still work properly after making the above changes.

===Specific Tasks Completed===

* Transferred over all code files relevant to the review bidding feature
* Re-deployed the external website which handles the review bidding algorithm
* Added that website's url to the webservices url's within expertiza
* Enhanced the stability of the webservice for the bidding algorithm by testing with edge cases and optimize the endpoint by restoring the original access endpoint reference
* Analyzed the necessity of the phase finding related logic code section
* Optimized phase finding related logic by removing unused code segment
* Removed an instance variable within the ''review_bids_controller'' which would have made any practical deployment fail, removing the unnecessary functionality it was created for
* Updated method calls to ''assignment.find_current_stage'' to their new form ''assignment.current_stage''
* Removed the review_bid model method ''self.reviewer_self_topic'' and replaced its usage with the already-existing ''SignedUpTeam.topic_id'' method
* Removed DRY-violating code used to calculate the current assignment phase which was never utilized by the controller or views
* Moved several view files from the ''review_bids'' view directory into the ''sign_up_sheet'' view directory. Partials which were duplicates in the two directories (''_all_actions.html.erb'' and ''_topic_names.html.erb'' were no longer necessary and deleted
* Refactored the naming of the variable that represents the id of the reviewer object from 'reviewer' to 'reviewer_id' for clarity
* Removed the unnecessary 'create' function within the review_bids_controller
* Removed the sparse testing files under the 'test' directory in keeping with typical expertiza structure
* Removed a test case from the unit tests which relied upon the removed 'create' method of the review_bids_controller

==Initial Implementation==

A considerable amount of time and effort was necessary to review, understand, and re-implement the previous implementation of the Review Bidding feature. While the feature was only created a year ago, several underlying functions and methods have since been refactored, and so needed to be changed in the implementation. Additionally, certain parts of how the implementation was designed did not seem to be properly implemented, and so had to be revised.

===Code Transfer===
First, all files that were added in the original pull request were copied and moved to the new version of the expertiza beta branch. This includes the following files:

*'''controllers/review_bids_controller.rb''': the primary controller that handles calls to review bids functionality from views, managing them itself or directing them to the appropriate model
*'''helpers/review_bids_helper.rb''': contains helper methods for rendering the rows of topics available to choose a review for
*'''models/review_bid.rb''': contains the methods that each individual review bid knows and manages, including assigning a review to a student, getting info on the topic it applies to, etc.
*'''views/review_bids/_all_actions.html.erb''': a view partial used to render the actions available to a student on the review bidding view. Nearly identical to the _all_actions file for topic bidding.
*'''views/review_bids/_table_header.html.erb''': a view partial used to render the head of the table for review bids. Though structurally similar to that of topics reviews, contains significantly less information, as less needs to be known by a student for particular topics to review.
*'''views/review_bids/_table_line.html.erb''': a view partial used to render individual rows of the review bids table. Like the table header, is structurally similar to that of topic bidding, but functions differently.
*'''views/review_bids/show.html.erb''': the view which displays all the available topics to bid on reviewing. Uses the aforementioned partials to render this bidding table.
*'''views/review_bids/others_work.html.erb''': displays the available review work which has the user has bid for.
*'''db/migrate/20201107203204_create_review_bids.rb''': the migration which creates the review bid objects within the expertiza database.
*'''spec/controllers/review_bids_controller_spec.rb''': the unit tests for the review bids controller
*'''spec/models/review_bid_spec.rb''': the unit tests for the review bids model
*'''test/controllers/review_bids_controller_test.rb''': a more elaborate feature test for the review bids functionality
*'''test/fixtures/review_bids.yml''': fixtures that are utilized for the review_bids_controller_test
*'''test/models/review_bid_test.rb''': A more elaborate feature test of the review bid model.

In addition to adding the previous files, the following files were also added to existing pages as modifications:

*'''controllers/student_review_controller.rb''': the student review controller's "list" method now redirects to the review_bid_controller's "index" method if the assignment in question has a review_choosing_algorithm set to "Bidding" (the option an instructor sets when they wish to allow students to bid on reviews).
*'''student_task/view.html.erb''': Instead of simply linking directly to the student_review controller's "list" method, the view instead checks if the assignment this view pertains to has their review_choosing_algorithm set to Bidding. If it does, and the assignment allows students to choose the topic to review (meaning the algorithm to assign bids has not yet been run), they are redirected to the 'show' method of the review_bids_controller. Otherwise, they are redirected to the 'index' method (showing them the info on their currently available topics to review).
*'''config/webservices.yml''': added a new url for the website that is called to run the review bidding algorithm
*'''config/routes.rb''': routes added for review_bids.
*'''views/assignments/edit/_topics.html.erb''': Added a new option box which allows an instructor to choose to allow bidding to occur using the review bidding algorithm, rather than the default 'Simple Choose.' Once this option is selected, it also offers a button that, when clicked, runs the review bidding algorithm on all currently bid upon reviews.

===Website Reimplementation===
In the previous implementation, the algorithm which handles the assigning of reviews was hosted on an external website. This was done for the sake of improved performance. Thus, to get the feature in working condition, we pulled the code necessary to run the algorithm and hosted it on our own website (linked at the end of this documentation). We also made a note in the pull request that the place where this website is referenced (in the "assign_reviews_algorithm" method of the review_bids_controller) should be replaced by a proper external web tool, as is done for similar web tools in other parts of expertiza.

===Modifications to Methods===
Certain methods utilized in the original version of the webpages were changed or renamed in later versions, and so, needed to be modified. The primary example of this is the method that assignments use to determine what stage of assignment completion they are in. The previous year's implementation called this method as "assignment.find_current_stage(topic_id)," but this failed when trying to execute. After comparing how this page worked to how normal review pages work, we determined the new call to this method is "assignment.current_stage(topic_id)." All calls to this method were replaced in all files.

Another change to methods was the removal of a method from the review_bids_controller named 'create'. This method primarily served as a redirect for an aspect of the feature that was unnecessary and so removed (see the following section), that being the button to show another review. With this feature removed, this method was no longer used, and so it was removed from the controller.

===Errors Corrected===
A notable period of time was also spent on determining the purpose of a particular local area that caused issues when trying to view the bids, that being a variable named "@number_of_reviews_to_show". This variable apparently kept track of how many reviews a student needed to be shown, so that they could request another. The issue was that this was created as a local variable within the review_bids_controller, and therefore would never work outside of a manual testing environment, as the variable would need to be different from every student and every assignment. After some investigation, we realized that all this variable did was limit the number of already-assigned reviews the student was able to see (as the review bidding algorithm already assigned all possible reviews). Thus, rather than try to recreate this functionality, we simply removed it.

This means that, when the review bidding algorithm runs, a student is able to see all of the reviews assigned to them, up to the maximum number of reviews for the assignment, and therefore have no reason to ask for any more. It may be the case that this feature could be improved by adding a possibility for students to request any reviews not assigned to any other student, or that still have available slots, using the "normal" review assignment scheme. For the time being, we did not consider this part of the baseline necessary functionality, and so did not implement it.

==Final Implementation==

===DRY Principle and Code Duplication Refactoring===
Per comments and feedbacks on the previous implementation [https://github.com/expertiza/expertiza/pull/1822 E2085], there were 3 sections of the code that potentially were DRY violations. For each of those violations, we have had made appropriate and provided the justification on the results.

====review_bids_controller.rb====
For the refactoring of review bids controller.eb, the original issue is the DRY principle violation on the internal logic of finding the phase of assignments.
The original implementation to be refactored and optimized are shown as follow.
<pre>
# Finding the current phase that we are in
due_dates = AssignmentDueDate.where(parent_id: @assignment.id)
@very_last_due_date = AssignmentDueDate.where(parent_id: @assignment.id).order("due_at DESC").limit(1)
next_due_date = @very_last_due_date[0]
for due_date in due_dates
if due_date.due_at > Time.now
next_due_date = due_date if due_date.due_at < next_due_date.due_at
end
end
@review_phase = next_due_date.deadline_type_id
</pre>
Through some discovery process, we have found some related logics that is already implemented from the system which enable the refactoring and reuse of the method. Some code snippets are demonstrated as follow.
<pre>
def self.current_due_date(due_dates)
#Get the current due date from list of due dates
due_dates.each do |due_date|
if due_date.due_at > Time.now
current_due_date = due_date
return current_due_date
end
end
#in case current due date not found
return nil
end
</pre>
<pre>
def self.teammate_review_allowed(student)
# time when teammate review is allowed
due_date = current_due_date(student.assignment.due_dates)
student.assignment.find_current_stage == 'Finished' ||
due_date &&
(due_date.teammate_review_allowed_id == 3 ||
due_date.teammate_review_allowed_id == 2) # late(2) or yes(3)
end
</pre>
As we can observed above, both methods of <code>current_due_date</code> and <code>teammate_review_allowed</code> provides the capability of getting the <code>review_phase</code> without the repetition. Additionally, we can see that those existing implementation still have some difference in implementation and purpose to be fully reusable by the logic that find current phase of a topic because the methods provided are helper methods for broader abstraction.

Furthermore, the section that we attempted to refactoring out have a similar logic that is used to get the current stage as shown below.
<pre>
def find_current_stage(topic_id = nil)
next_due_date = DueDate.get_next_due_date(self.id, topic_id)
return 'Finished' if next_due_date.nil?
next_due_date
end
</pre>
However, it is worth pointing out that the method is not fully substitutable, particularly the internal functionality that finds the <code>@very_last_due_date</code>. A similar logic does exist that is meant to use to acquire current due date and assist with finding current stage.
<pre>
def current_stage_name(topic_id = nil)
if self.staggered_deadline?
return (topic_id.nil? ? 'Unknown' : current_stage(topic_id))
end

due_date = find_current_stage(topic_id)
unless due_date == 'Finished' || due_date.nil? || due_date.deadline_name.nil?
return due_date.deadline_name
end
current_stage(topic_id)
end
</pre>
With all the findings, we have come to conclusion that the portion of the code to be dry is in similar, but required full restructure, form of existing methods. The internal logic flow itself is not fully matching the reusable methods consists across the controllers. Because of the clarity of limited reusability of existing methods, we have come to conclusion that the portions to be refactored are not duplication of any existing methods. Additionally, after a more detailed discovery, it has come to realization that the code portion of getting the current phase of the next due date was, in fact, meant to find the last due date. After further analysis of the purpose of the code, the portion has been deleted from <code>review_bids_controller.rb</code> because there was no usage observed with the result of the execution.

====review_bid.rb====
Per the previous iteration's feedback, the review_bid.rb model file presented some DRY violations. These violations are highlighted in the method below:
<pre>
def self.reviewer_self_topic(reviewer_id)
participant = AssignmentParticipant.find_by(id: reviewer_id)
team_id = participant.team.try(:id)
self_topic = SignedUpTeam.where(team_id: team_id).pluck(:topic_id).first

return self_topic
end
</pre>
This method returns the topic corresponding to the reviewer that is passed into the method. Line 1 gets the participant from the reviewer_id parameter. Line 2 extracts the team that the participant is in. Line 3 will get the topic which is assigned to the participant's team. This topic is returned. Our goal while refactoring was to find a similar method that can replace this method.

While searching through the project for similar methods which perform the same action, we came across this method in the signed_up_team.rb model:
<pre>
def self.topic_id(assignment_id, user_id)
# team_id variable represents the team_id for this user in this assignment
team_id = TeamsUser.team_id(assignment_id, user_id)
topic_id_by_team_id(team_id) if team_id
end
</pre>
This method gets the team_id from the user_id and isolates the topic from said team_id. This method performs a similar function to reviewer_self_topic. We decided to replace all uses of reviewer_self_topic with topic_id. By reusing an existing method, we are DRYing out this code. By removing reviewer_self_topic from the code base and using an existing method, less code is being reused.
 
The previous uses of reviewer_self_topic such as:
<pre>
def self.reviewer_bidding_data(reviewer,assignment_id)
self_topic = self.reviewer_self_topic(reviewer)
bidding_data = {'tid'=> [], 'otid' => self_topic, 'priority' => [], 'time'=>[]}
bids = ReviewBid.where(participant_id: reviewer)
</pre>
were instead rewritten to adapt the existing topic_id method:
<pre>
def self.reviewer_bidding_data(reviewer_id,assignment_id)
reviewer_user_id = AssignmentParticipant.find(reviewer_id).user_id
self_topic = SignedUpTeam.topic_id(assignment_id, reviewer_user_id)
bidding_data = {'tid'=> [], 'otid' => self_topic, 'priority' => [], 'time'=>[]}
bids = ReviewBid.where(participant_id: reviewer_id)
</pre>
The use of existing methods has lead to less DRY violations in the review_bid.rb file.

====Refactoring Views (_all_actions.html.erb and _topic_names.html.erb)====
The last element of the prior implementation that caused DRY violations had to do with its implementation of views. Two primary views are necessary for the review bids functionality: one which shows a table where students can bid on which topic they wish to review (rendered by a file called 'show.html.erb'), and a list of assigned reviews which a student must select from (rendered by a file called 'others_work.html.erb').

In order to render the elements in 'show.html.erb' in particular, a number of partials must be utilized:
*a partial for the table's header (_table_header.html.erb)
*a partial for rendering each of the table's rows, describing a topic to bid on (_table_line.html.erb)
*a partial to contain information related to what actions are available for a student to do for a review, like edit or view (_all_actions.html.erb)
*a partial for finding and rendering the name of each topic one must review (_topic_names.html.erb)

Both of these views and all four of these partials were contained within the "review_bids" directory within the views directory. The DRY violation incurred by these views has to do with these partials: in particular, '_topic_names.html.erb' and '_all_actions.html.erb' are near identical code copies to those within the sign_up_sheet views, used for rendering and displaying information for topic bidding and normal reviews.

In order to DRY out this portion of the code, we decided that having a separate set of views for the results related to review bids were unnecessary. Within expertiza's structure, most of the views related to students choosing which topics or reviews to work on are all contained within the sign_up_sheet views. The review_vids views were copying the partials within the sign_up_sheet views because they were using the same fundamental components. Thus, we thought the best solution would be to move the two primary views for review_bids ('show.html.erb' and 'others_work.html.erb') into the sign_up_sheet views, and leave behind all redundant partials. In doing so, the redundancy of '_all_actions.html.erb' and '_topic_names.html.erb' were eliminated.

It could be argued that this implementation could be further DRY-ed by leaving behind the other two partials, '_table_header.html.erb' and '_table_line.html.erb', as they are structurally similar to those used for topic bidding. However, this is not actually permissible, as the header and lines for topics when bidding on them for reviews are significantly different than when bidding on them for assignments. More information needs to be available when bidding on topics for an assignment, such as the number of slots and number on the waitlist, but such an idea is not applicable for bids on reviews. While it would be possible to have a single file which produces one view or another depending upon if it's bidding for topics or reviews, we believed this violated the Expert Pattern, by attempting to create a single view for two distinct purposes. Instead, we thought it better to simply have two separate partials, each which precisely gave the desired functionality. Thus, the other two partials were brought over, being renamed '_review_bids_table_header.html.erb' and '_review_bids_table_line.html.erb'.

==Tests==

Throughout the refactoring and merging of [https://github.com/expertiza/expertiza/pull/1822 E2085] to [https://github.com/expertiza/expertiza/pull/2150 E2151], there were some updates that has been made to the test suits to maintain corrosponding testing automation. Since our implementations and changes are focusing on DRY principal optimization, most of the test are based on the previous test that interact with the system and observe the behavior of the user views.

===Video Demonstration===
Visit [https://www.youtube.com/watch?v=wxXALQNWFcc Feature Demo] to see a video demonstrating the functionality of the review bidding feature that is working properly after our DRY principle optimization.

===RSpec Testing===
One of the primary design goals when doing this refactoring was to ensure that all test cases still passed after all changes were made.

There was a single change that needed to be made to the RSpec files for the review bidding algorithm, which was the removal of a test relying on the 'create' method of the review_bids_controller. As previously mentioned, this function was removed because it had become obsolete. Thus, the test case for this function (which simply checked if it redirected to the right page) was no longer necessary, and thus was deleted.

Otherwise, the test cases for both the review_bids_controller and the review_bids model were manually run and found to have no errors, as shown below.

[[File:Bidding_test_results.png|800px]]

For details on what these specific tests were, please refer to the previous work referenced before.

==Conclusions and Future Work==
While the current state of the review bidding code is operational, and is certainly more DRY and stable than the previous incarnation of this work, there are still several issues with the current state of this feature that make it difficult to recommend an immediate merge into expertiza.

===Enhancement of Algorithm Service Stability===

Across the process of refining the refactored front end page view along with more robust edge cases testing towards the algorithm service, we have discovered an underlying issue that could impact the availability of the review bidding algorithm service. Through cause isolation, we have observed a high correlation to flask application crash with the edge case scenario input where the algorithm service is trying to match a student who was not signed up with any topic with other students.

Because the scope of our project was focused on refactoring, we did not have the time or prior warning to need to address these edge cases. However, we will provide the error log for the reference of any future teams who may work on the stability improvement of the algorithm service.

<pre>
"POST /match_topics HTTP/1.1" 500 -
{'tid': [5133, 5134, 5135, None], 'users': {'45514': {'otid': 5135, 'tid': [5133, 5134], 'time': ['2021-11-30T17:57:20.000-05:00', '2021-11-30T17:57:24.000-05:00'], 'priority': [2, 1]}, '45513': {'otid': 5134, 'tid': [5133, 5135], 'time': ['2021-11-30T17:57:41.000-05:00', '2021-11-30T17:57:43.000-05:00'], 'priority': [2, 1]}, '45512': {'otid': 5133, 'tid': [5134, 5135], 'time': ['2021-11-30T17:57:58.000-05:00', '2021-11-30T17:58:00.000-05:00'], 'priority': [2, 1]}, '45515': {'otid': None, 'tid': [], 'time': [0], 'priority': [3, 1, 4, 2]}}, 'max_accepted_proposals': 3}
[2021-11-28 18:10:40,576] ERROR in app: Exception on /match_topics [POST]
Traceback (most recent call last):
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/usr/.local/lib/python3.5/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/usr/IntelligentAssignment/app/app.py", line 19, in topic_matching
return jsonify(assigned_topics_model.get_student_topic_matches())
File "/home/usr/IntelligentAssignment/app/topics_matcher.py", line 57, in get_student_topic_matches
student.accept_proposal()
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in accept_proposal
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in <lambda>
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
ValueError: None is not in list
</pre>

As we can see from the additional information provided above, the error is due to an unhandled edge case of attempting to access missing value. Our further observation also concluded that such missing value is due to the fact that one or more to-be-matched students did not have any topic signed up at the time of algorithm execution. To address the issue, a general enhancement can be implemented on the algorithm service side with Flask to increase the crash recovery stability. Further refinement of interaction flow with user in the front end can also reduce the possibility of algorithm service error.

Additional to the unhandled cases from the algorithm service, we also noticed an inconsistency of the system due to the performance limitation of the Heroku deployment.

[[File:Out_of_scope_External-Issue_E2151.gif|frameless|x1000px]]

As we can see from the above recorded screen capture animation, with the exact same input, the output and algorithm service result are unstable. Because of the performance limitation of the Heroku, sometime the Flask app that deployed were overloaded. Furthermore, we have also attempted to deploy on VCL, however, we notice the limitation of VCL deployment that will become offline once the VCL reservation expired. After all, because of the focus of our tasks is on the DRY principal optimization, the external algorithm service can be improved in the future. To access and deploy the algorithm on local environment or another environment, setup guide and resource are available on [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment] repository.

===Lack of Comprehensive Testing===

A notable contradiction that was noticed when working with this project was that the previously-mentioned failed edge cases existed, yet all tests for this feature passed. This is illustrative of the fact that this feature requires better and more exhaustive testing. With the scope limitation of our tasks that focusing on the DRY principal enhancement of the Expertiza internally, the external algorithm service module was not within our focus. However, with our experience of deployment with the algorithm service, we can offer some suggestions for the future team who will address those edge cases of the algorithm service.

When dealing with an issue with so many factors as how students bid on reviews, an extensive suite of edge cases should be considered. Any future project group working on this function should consider the following edge cases to cover:
* Does a student who does not bid on any reviews still get reviews?
* Does a student who did not submit anything for their assignment still have a review to bid on?
* What if there are less assignments to have reviews for than the minimum number required for all students to have the minimum number of reviews?

===Conclusion===
While we believe the current state of this feature is greatly improved, with many of its DRY violations removed and several noteworthy bugs corrected, we believe the feature of allowing students to bid on what to review still needs further development before it is deployed. While the basic functionality is working under ideal situations, the issues mentioned above should be addressed before this feature is put in the hands of actual instructors, as if this feature fails, it could ruin the entire reviewing process for an assignment.

Still, we think there is notable promise to this feature, and it would be a very engaging and interesting feature if deployed successfully, so we believe develop on it should continue until it is ready for deployment.

==Github and Related Link==
Github repository corresponding to this task is publicly available at [https://github.com/WeiRui-Wang/expertiza E2151 GitHub].

Github pull request and file changes can be viewed at [https://github.com/expertiza/expertiza/pull/2150/files E2151 Pull Request].

Github of source code of externally hosted bidding algorithm service deployment manual and files can be access at [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment].

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: Geoff Garrido, Ayush Luthra, and WeiRui Wang. Our project mentor was Luis Delossantos (lgdeloss@ncsu.edu)

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-30T01:59:51Z

Aluthra: /* review_bid.rb */

==Topic Overview==
===Statement of Problem===
Reviewing is an important stage of learning and improving, both as a reviewer and a reviewee, and so allowing reviewers more freedom and choice in what they review would result in better reviews and more engaged reviewers. However, currently the only way to choose a topic to review is a first-come-first-served basis, penalizing students who do not sign up quickly by not allowing them to review the topics they would be most interesting in engaging with.

A better alternative would be to implement a bidding system, similar to that used for topics, which would allow students to establish preferences for which topics they would most like to review. A good bidding algorithm implementing with this idea would ensure that all students (who care enough to bid) receive a topic to review that they actively want.

A prior project group created a prototype for this function, allowing students to bid on what topics they wanted to review. While the functionality was as desired, the feature was not able to be merged due to having too many DRY violations. Thus, the goal of this topic was to bring this prior implementation into the current version of the expertiza beta branch, and remove these DRY violations, so that it may be merged.

As development continued, it became clear this prototype also had several bugs and issues that made it unable to work under several conditions. Thus, a secondary goal was to fix any issues which existed such that the feature would generally work with the functionality the prior implementation demonstrated.

===Prior Work===
The prior work which is the basis for this topic is linked here [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review is linked here]. This work was done in Fall of 2020, and as previously mentioned, achieved much of the desired functionality for this task. This prior work will be referenced several times within this documentation, as it has a more comprehensive explanation of the underlying design principles behind the views, controller, model, algorithm, and RSpec tests implemented. These details will only be briefly summarized here.

As previously noted, this prior implementation was rejected due to DRY violations, and so correcting these violations--and explaining how those corrections were done--will be the major focus of this project documentation.

==Explanation of Feature==

===Existing Bidding Functionality for Topics===
Currently in expertiza, students may bid on which topics they would like to work on. An image showcasing this functionality is given below:

[[File:expected_review_UI.PNG]]

Students may choose a number of topics to bid on, in order of priority, and once the deadline is reached, all students are given a topic.

Behind the scenes, this works by utilizing a modified version of the Gale-Shapely Algorithm. This algorithm seeks to find a 'stable' matching of each student to each topic, such that in the end, all students have a topic they wanted (at least to some degree), and that the preferences of students is respected as well as possible. To help break situations where there are ties in preferences, each topic assigns a Priority to each student. This Priority is determined by A. whether the student is part of a group bidding for this topic (groups getting higher priority), and B. the time at which the bid for the topic was made (earlier bidders are given Priority). Once this has been determined and the deadline for bidding has ended, the following steps occur:
# Each student is matched with each topic they most prefer.
# If the number of students assigned to a topic is above the maximum allowed for that topic, the students with the highest Priority are assigned the topic
# All maximally filled topics are removed from the pool of available topics
# All unmatched students begin this cycle again, now choosing the most preferred topic remaining
# Repeat until all students are assigned a topic
# If any matches are 'unstable' (two students would have their preferences better met if they were swapped), perform any swaps until matches are as stable as possible.

This algorithm has a time complexity of O(n^2). For more details on this algorithm and its implementation, refer to the previous year's work, where this algorithm was implemented.

===Desired Functionality for Reviews===

The desired feature for reviews should have an almost identical structure to this, with reviews instead of topics. Students should be shown a list of the topics that were worked on, and may bid on which of those topics they would like to review the most. The Priority system could be handled in the same way, with the only caveat being that a student should not be able to review their own work.

In terms of what this feature looks like, it can be enabled as an option when editing/creating a new assignment, under the "Topics" section, using interface drop down show below.

[[File:Bidding_interface.png|center|800px]]

Once this option is enabled, during the review phase of an assignment, students will be able to access a new view listing all the topics of all the submitted assignments (except for their own). They will then be able to bid on as many topics as they like, assigning them a preference, in a manner similar to the interface for topic bidding. The table that allows for this drag-and-drop priorities is shown below.

[[File:Bidding_options.png|center|900px]]

Whenever the instructor decides they would like to close the window for review bidding, they may return to the options for the given assignment and navigate to "Topics", where they will see a button marked "Run Review Algorithm". This will make a call to the external webservice to determine which students are assigned what reviews, and will then assign those reviews to all students. Once this button is pressed, it locks out the ability for students to perform any more bidding, and cannot be pressed again. The button allowing this option is shown below.

[[File:Bidding_run_algorithm.png|center|900px]]

After this point, students may navigate to the "Others Work" option under their assignment and see all of their assigned reviews. They will see up to the maximum number of allowed reviews, but will be informed of how many they are required to do (for example, a student may see 4 reviews, but will be told they are only required to complete 2).

===Use Case Diagram===
The following Use Case Diagram demonstrates the primary functionality explained in the previous section for different types of users.

[[File:Use_Case_Diagram_Rev1_E2151.jpg|center]]

==Plan and List of Work Done==

===General Design Goals===

* Move all prior work for this topic into a new version of the expertiza beta branch, without loss in functionality.
* Ensure all prior/expected functionality is working correctly, and if not, make any repairs or changes necessary.
* DRY out prior implementation's controller and model methods which have functionality that already exists within the expertiza system.
* Remove repetitive views and partials that exist by merging views for review bids into the set of views for sign up sheets, from which it uses partials.
* Ensure all present test cases still work properly after making the above changes.

===Specific Tasks Completed===

* Transferred over all code files relevant to the review bidding feature
* Re-deployed the external website which handles the review bidding algorithm
* Added that website's url to the webservices url's within expertiza
* Enhanced the stability of the webservice for the bidding algorithm by testing with edge cases and optimize the endpoint by restoring the original access endpoint reference
* Analyzed the necessity of the phase finding related logic code section
* Optimized phase finding related logic by removing unused code segment
* Removed an instance variable within the ''review_bids_controller'' which would have made any practical deployment fail, removing the unnecessary functionality it was created for
* Updated method calls to ''assignment.find_current_stage'' to their new form ''assignment.current_stage''
* Removed the review_bid model method ''self.reviewer_self_topic'' and replaced its usage with the already-existing ''SignedUpTeam.topic_id'' method
* Removed DRY-violating code used to calculate the current assignment phase which was never utilized by the controller or views
* Moved several view files from the ''review_bids'' view directory into the ''sign_up_sheet'' view directory. Partials which were duplicates in the two directories (''_all_actions.html.erb'' and ''_topic_names.html.erb'' were no longer necessary and deleted
* Refactored the naming of the variable that represents the id of the reviewer object from 'reviewer' to 'reviewer_id' for clarity
* Removed the unnecessary 'create' function within the review_bids_controller
* Removed the sparse testing files under the 'test' directory in keeping with typical expertiza structure
* Removed a test case from the unit tests which relied upon the removed 'create' method of the review_bids_controller

==Initial Implementation==

A considerable amount of time and effort was necessary to review, understand, and re-implement the previous implementation of the Review Bidding feature. While the feature was only created a year ago, several underlying functions and methods have since been refactored, and so needed to be changed in the implementation. Additionally, certain parts of how the implementation was designed did not seem to be properly implemented, and so had to be revised.

===Code Transfer===
First, all files that were added in the original pull request were copied and moved to the new version of the expertiza beta branch. This includes the following files:

*'''controllers/review_bids_controller.rb''': the primary controller that handles calls to review bids functionality from views, managing them itself or directing them to the appropriate model
*'''helpers/review_bids_helper.rb''': contains helper methods for rendering the rows of topics available to choose a review for
*'''models/review_bid.rb''': contains the methods that each individual review bid knows and manages, including assigning a review to a student, getting info on the topic it applies to, etc.
*'''views/review_bids/_all_actions.html.erb''': a view partial used to render the actions available to a student on the review bidding view. Nearly identical to the _all_actions file for topic bidding.
*'''views/review_bids/_table_header.html.erb''': a view partial used to render the head of the table for review bids. Though structurally similar to that of topics reviews, contains significantly less information, as less needs to be known by a student for particular topics to review.
*'''views/review_bids/_table_line.html.erb''': a view partial used to render individual rows of the review bids table. Like the table header, is structurally similar to that of topic bidding, but functions differently.
*'''views/review_bids/show.html.erb''': the view which displays all the available topics to bid on reviewing. Uses the aforementioned partials to render this bidding table.
*'''views/review_bids/others_work.html.erb''': displays the available review work which has the user has bid for.
*'''db/migrate/20201107203204_create_review_bids.rb''': the migration which creates the review bid objects within the expertiza database.
*'''spec/controllers/review_bids_controller_spec.rb''': the unit tests for the review bids controller
*'''spec/models/review_bid_spec.rb''': the unit tests for the review bids model
*'''test/controllers/review_bids_controller_test.rb''': a more elaborate feature test for the review bids functionality
*'''test/fixtures/review_bids.yml''': fixtures that are utilized for the review_bids_controller_test
*'''test/models/review_bid_test.rb''': A more elaborate feature test of the review bid model.

In addition to adding the previous files, the following files were also added to existing pages as modifications:

*'''controllers/student_review_controller.rb''': the student review controller's "list" method now redirects to the review_bid_controller's "index" method if the assignment in question has a review_choosing_algorithm set to "Bidding" (the option an instructor sets when they wish to allow students to bid on reviews).
*'''student_task/view.html.erb''': Instead of simply linking directly to the student_review controller's "list" method, the view instead checks if the assignment this view pertains to has their review_choosing_algorithm set to Bidding. If it does, and the assignment allows students to choose the topic to review (meaning the algorithm to assign bids has not yet been run), they are redirected to the 'show' method of the review_bids_controller. Otherwise, they are redirected to the 'index' method (showing them the info on their currently available topics to review).
*'''config/webservices.yml''': added a new url for the website that is called to run the review bidding algorithm
*'''config/routes.rb''': routes added for review_bids.
*'''views/assignments/edit/_topics.html.erb''': Added a new option box which allows an instructor to choose to allow bidding to occur using the review bidding algorithm, rather than the default 'Simple Choose.' Once this option is selected, it also offers a button that, when clicked, runs the review bidding algorithm on all currently bid upon reviews.

===Website Reimplementation===
In the previous implementation, the algorithm which handles the assigning of reviews was hosted on an external website. This was done for the sake of improved performance. Thus, to get the feature in working condition, we pulled the code necessary to run the algorithm and hosted it on our own website (linked at the end of this documentation). We also made a note in the pull request that the place where this website is referenced (in the "assign_reviews_algorithm" method of the review_bids_controller) should be replaced by a proper external web tool, as is done for similar web tools in other parts of expertiza.

===Modifications to Methods===
Certain methods utilized in the original version of the webpages were changed or renamed in later versions, and so, needed to be modified. The primary example of this is the method that assignments use to determine what stage of assignment completion they are in. The previous year's implementation called this method as "assignment.find_current_stage(topic_id)," but this failed when trying to execute. After comparing how this page worked to how normal review pages work, we determined the new call to this method is "assignment.current_stage(topic_id)." All calls to this method were replaced in all files.

Another change to methods was the removal of a method from the review_bids_controller named 'create'. This method primarily served as a redirect for an aspect of the feature that was unnecessary and so removed (see the following section), that being the button to show another review. With this feature removed, this method was no longer used, and so it was removed from the controller.

===Errors Corrected===
A notable period of time was also spent on determining the purpose of a particular local area that caused issues when trying to view the bids, that being a variable named "@number_of_reviews_to_show". This variable apparently kept track of how many reviews a student needed to be shown, so that they could request another. The issue was that this was created as a local variable within the review_bids_controller, and therefore would never work outside of a manual testing environment, as the variable would need to be different from every student and every assignment. After some investigation, we realized that all this variable did was limit the number of already-assigned reviews the student was able to see (as the review bidding algorithm already assigned all possible reviews). Thus, rather than try to recreate this functionality, we simply removed it.

This means that, when the review bidding algorithm runs, a student is able to see all of the reviews assigned to them, up to the maximum number of reviews for the assignment, and therefore have no reason to ask for any more. It may be the case that this feature could be improved by adding a possibility for students to request any reviews not assigned to any other student, or that still have available slots, using the "normal" review assignment scheme. For the time being, we did not consider this part of the baseline necessary functionality, and so did not implement it.

==Final Implementation==

===DRY Principle and Code Duplication Refactoring===
Per comments and feedbacks on the previous implementation [https://github.com/expertiza/expertiza/pull/1822 E2085], there were 3 sections of the code that potentially were DRY violations. For each of those violations, we have had made appropriate and provided the justification on the results.

====review_bids_controller.rb====
For the refactoring of review bids controller.eb, the original issue is the DRY principle violation on the internal logic of finding the phase of assignments.
The original implementation to be refactored and optimized are shown as follow.
<pre>
# Finding the current phase that we are in
due_dates = AssignmentDueDate.where(parent_id: @assignment.id)
@very_last_due_date = AssignmentDueDate.where(parent_id: @assignment.id).order("due_at DESC").limit(1)
next_due_date = @very_last_due_date[0]
for due_date in due_dates
if due_date.due_at > Time.now
next_due_date = due_date if due_date.due_at < next_due_date.due_at
end
end
@review_phase = next_due_date.deadline_type_id
</pre>
Through some discovery process, we have found some related logics that is already implemented from the system which enable the refactoring and reuse of the method. Some code snippets are demonstrated as follow.
<pre>
def self.current_due_date(due_dates)
#Get the current due date from list of due dates
due_dates.each do |due_date|
if due_date.due_at > Time.now
current_due_date = due_date
return current_due_date
end
end
#in case current due date not found
return nil
end
</pre>
<pre>
def self.teammate_review_allowed(student)
# time when teammate review is allowed
due_date = current_due_date(student.assignment.due_dates)
student.assignment.find_current_stage == 'Finished' ||
due_date &&
(due_date.teammate_review_allowed_id == 3 ||
due_date.teammate_review_allowed_id == 2) # late(2) or yes(3)
end
</pre>
As we can observed above, both methods of <code>current_due_date</code> and <code>teammate_review_allowed</code> provides the capability of getting the <code>review_phase</code> without the repetition. Additionally, we can see that those existing implementation still have some difference in implementation and purpose to be fully reusable by the logic that find current phase of a topic because the methods provided are helper methods for broader abstraction.

Furthermore, the section that we attempted to refactoring out have a similar logic that is used to get the current stage as shown below.
<pre>
def find_current_stage(topic_id = nil)
next_due_date = DueDate.get_next_due_date(self.id, topic_id)
return 'Finished' if next_due_date.nil?
next_due_date
end
</pre>
However, it is worth pointing out that the method is not fully substitutable, particularly the internal functionality that finds the <code>@very_last_due_date</code>. A similar logic does exist that is meant to use to acquire current due date and assist with finding current stage.
<pre>
def current_stage_name(topic_id = nil)
if self.staggered_deadline?
return (topic_id.nil? ? 'Unknown' : current_stage(topic_id))
end

due_date = find_current_stage(topic_id)
unless due_date == 'Finished' || due_date.nil? || due_date.deadline_name.nil?
return due_date.deadline_name
end
current_stage(topic_id)
end
</pre>
With all the findings, we have come to conclusion that the portion of the code to be dry is in similar, but required full restructure, form of existing methods. The internal logic flow itself is not fully matching the reusable methods consists across the controllers. Because of the clarity of limited reusability of existing methods, we have come to conclusion that the portions to be refactored are not duplication of any existing methods. Additionally, after a more detailed discovery, it has come to realization that the code portion of getting the current phase of the next due date was, in fact, meant to find the last due date. After further analysis of the purpose of the code, the portion has been deleted from <code>review_bids_controller.rb</code> because there was no usage observed with the result of the execution.

====review_bid.rb====
Per the previous iteration's feedback, the review_bid.rb model file presented some DRY violations. These violations are highlighted in the method below:
<pre>
def self.reviewer_self_topic(reviewer_id)
participant = AssignmentParticipant.find_by(id: reviewer_id)
team_id = participant.team.try(:id)
self_topic = SignedUpTeam.where(team_id: team_id).pluck(:topic_id).first

return self_topic
end
</pre>
This method returns the topic corresponding to the reviewer that is passed into the method. Line 1 gets the participant from the reviewer_id parameter. Line 2 extracts the team that the participant is in. Line 3 will get the topic which is assigned to the participant's team. This topic is returned. Our goal while refactoring was to find a similar method that can replace this method.

While searching through the project for similar methods which perform the same action, we came across this method in the signed_up_team.rb model:
<pre>
def self.topic_id(assignment_id, user_id)
# team_id variable represents the team_id for this user in this assignment
team_id = TeamsUser.team_id(assignment_id, user_id)
topic_id_by_team_id(team_id) if team_id
end
</pre>
This method gets the team_id from the user_id and isolates the topic from said team_id. This method performs a similar function to reviewer_self_topic. We decided to replace all uses of reviewer_self_topic with topic_id. By reusing an existing method, we are DRYing out this code. By removing reviewer_self_topic from the code base and using an existing method, less code is being reused.
 
The previous uses of reviewer_self_topic such as:
<pre>
def self.reviewer_bidding_data(reviewer,assignment_id)
self_topic = self.reviewer_self_topic(reviewer)
bidding_data = {'tid'=> [], 'otid' => self_topic, 'priority' => [], 'time'=>[]}
bids = ReviewBid.where(participant_id: reviewer)
</pre>
were instead rewritten to adapt the existing topic_id method:
<pre>
def self.reviewer_bidding_data(reviewer_id,assignment_id)
reviewer_user_id = AssignmentParticipant.find(reviewer_id).user_id
self_topic = SignedUpTeam.topic_id(assignment_id, reviewer_user_id)
bidding_data = {'tid'=> [], 'otid' => self_topic, 'priority' => [], 'time'=>[]}
bids = ReviewBid.where(participant_id: reviewer_id)
</pre>

====Refactoring Views (_all_actions.html.erb and _topic_names.html.erb)====
The last element of the prior implementation that caused DRY violations had to do with its implementation of views. Two primary views are necessary for the review bids functionality: one which shows a table where students can bid on which topic they wish to review (rendered by a file called 'show.html.erb'), and a list of assigned reviews which a student must select from (rendered by a file called 'others_work.html.erb').

In order to render the elements in 'show.html.erb' in particular, a number of partials must be utilized:
*a partial for the table's header (_table_header.html.erb)
*a partial for rendering each of the table's rows, describing a topic to bid on (_table_line.html.erb)
*a partial to contain information related to what actions are available for a student to do for a review, like edit or view (_all_actions.html.erb)
*a partial for finding and rendering the name of each topic one must review (_topic_names.html.erb)

Both of these views and all four of these partials were contained within the "review_bids" directory within the views directory. The DRY violation incurred by these views has to do with these partials: in particular, '_topic_names.html.erb' and '_all_actions.html.erb' are near identical code copies to those within the sign_up_sheet views, used for rendering and displaying information for topic bidding and normal reviews.

In order to DRY out this portion of the code, we decided that having a separate set of views for the results related to review bids were unnecessary. Within expertiza's structure, most of the views related to students choosing which topics or reviews to work on are all contained within the sign_up_sheet views. The review_vids views were copying the partials within the sign_up_sheet views because they were using the same fundamental components. Thus, we thought the best solution would be to move the two primary views for review_bids ('show.html.erb' and 'others_work.html.erb') into the sign_up_sheet views, and leave behind all redundant partials. In doing so, the redundancy of '_all_actions.html.erb' and '_topic_names.html.erb' were eliminated.

It could be argued that this implementation could be further DRY-ed by leaving behind the other two partials, '_table_header.html.erb' and '_table_line.html.erb', as they are structurally similar to those used for topic bidding. However, this is not actually permissible, as the header and lines for topics when bidding on them for reviews are significantly different than when bidding on them for assignments. More information needs to be available when bidding on topics for an assignment, such as the number of slots and number on the waitlist, but such an idea is not applicable for bids on reviews. While it would be possible to have a single file which produces one view or another depending upon if it's bidding for topics or reviews, we believed this violated the Expert Pattern, by attempting to create a single view for two distinct purposes. Instead, we thought it better to simply have two separate partials, each which precisely gave the desired functionality. Thus, the other two partials were brought over, being renamed '_review_bids_table_header.html.erb' and '_review_bids_table_line.html.erb'.

==Tests==

Throughout the refactoring and merging of [https://github.com/expertiza/expertiza/pull/1822 E2085] to [https://github.com/expertiza/expertiza/pull/2150 E2151], there were some updates that has been made to the test suits to maintain corrosponding testing automation. Since our implementations and changes are focusing on DRY principal optimization, most of the test are based on the previous test that interact with the system and observe the behavior of the user views.

===Video Demonstration===
Visit [https://www.youtube.com/watch?v=wxXALQNWFcc Feature Demo] to see a video demonstrating the functionality of the review bidding feature that is working properly after our DRY principle optimization.

===RSpec Testing===
One of the primary design goals when doing this refactoring was to ensure that all test cases still passed after all changes were made.

There was a single change that needed to be made to the RSpec files for the review bidding algorithm, which was the removal of a test relying on the 'create' method of the review_bids_controller. As previously mentioned, this function was removed because it had become obsolete. Thus, the test case for this function (which simply checked if it redirected to the right page) was no longer necessary, and thus was deleted.

Otherwise, the test cases for both the review_bids_controller and the review_bids model were manually run and found to have no errors, as shown below.

[[File:Bidding_test_results.png|800px]]

For details on what these specific tests were, please refer to the previous work referenced before.

==Conclusions and Future Work==
While the current state of the review bidding code is operational, and is certainly more DRY and stable than the previous incarnation of this work, there are still several issues with the current state of this feature that make it difficult to recommend an immediate merge into expertiza.

===Enhancement of Algorithm Service Stability===

Across the process of refining the refactored front end page view along with more robust edge cases testing towards the algorithm service, we have discovered an underlying issue that could impact the availability of the review bidding algorithm service. Through cause isolation, we have observed a high correlation to flask application crash with the edge case scenario input where the algorithm service is trying to match a student who was not signed up with any topic with other students.

Because the scope of our project was focused on refactoring, we did not have the time or prior warning to need to address these edge cases. However, we will provide the error log for the reference of any future teams who may work on the stability improvement of the algorithm service.

<pre>
"POST /match_topics HTTP/1.1" 500 -
{'tid': [5133, 5134, 5135, None], 'users': {'45514': {'otid': 5135, 'tid': [5133, 5134], 'time': ['2021-11-30T17:57:20.000-05:00', '2021-11-30T17:57:24.000-05:00'], 'priority': [2, 1]}, '45513': {'otid': 5134, 'tid': [5133, 5135], 'time': ['2021-11-30T17:57:41.000-05:00', '2021-11-30T17:57:43.000-05:00'], 'priority': [2, 1]}, '45512': {'otid': 5133, 'tid': [5134, 5135], 'time': ['2021-11-30T17:57:58.000-05:00', '2021-11-30T17:58:00.000-05:00'], 'priority': [2, 1]}, '45515': {'otid': None, 'tid': [], 'time': [0], 'priority': [3, 1, 4, 2]}}, 'max_accepted_proposals': 3}
[2021-11-28 18:10:40,576] ERROR in app: Exception on /match_topics [POST]
Traceback (most recent call last):
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/usr/.local/lib/python3.5/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/usr/IntelligentAssignment/app/app.py", line 19, in topic_matching
return jsonify(assigned_topics_model.get_student_topic_matches())
File "/home/usr/IntelligentAssignment/app/topics_matcher.py", line 57, in get_student_topic_matches
student.accept_proposal()
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in accept_proposal
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in <lambda>
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
ValueError: None is not in list
</pre>

As we can see from the additional information provided above, the error is due to an unhandled edge case of attempting to access missing value. Our further observation also concluded that such missing value is due to the fact that one or more to-be-matched students did not have any topic signed up at the time of algorithm execution. To address the issue, a general enhancement can be implemented on the algorithm service side with Flask to increase the crash recovery stability. Further refinement of interaction flow with user in the front end can also reduce the possibility of algorithm service error.

Additional to the unhandled cases from the algorithm service, we also noticed an inconsistency of the system due to the performance limitation of the Heroku deployment.

[[File:Out_of_scope_External-Issue_E2151.gif|frameless|x1000px]]

As we can see from the above recorded screen capture animation, with the exact same input, the output and algorithm service result are unstable. Because of the performance limitation of the Heroku, sometime the Flask app that deployed were overloaded. Furthermore, we have also attempted to deploy on VCL, however, we notice the limitation of VCL deployment that will become offline once the VCL reservation expired. After all, because of the focus of our tasks is on the DRY principal optimization, the external algorithm service can be improved in the future. To access and deploy the algorithm on local environment or another environment, setup guide and resource are available on [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment] repository.

===Lack of Comprehensive Testing===

A notable contradiction that was noticed when working with this project was that the previously-mentioned failed edge cases existed, yet all tests for this feature passed. This is illustrative of the fact that this feature requires better and more exhaustive testing. With the scope limitation of our tasks that focusing on the DRY principal enhancement of the Expertiza internally, the external algorithm service module was not within our focus. However, with our experience of deployment with the algorithm service, we can offer some suggestions for the future team who will address those edge cases of the algorithm service.

When dealing with an issue with so many factors as how students bid on reviews, an extensive suite of edge cases should be considered. Any future project group working on this function should consider the following edge cases to cover:
* Does a student who does not bid on any reviews still get reviews?
* Does a student who did not submit anything for their assignment still have a review to bid on?
* What if there are less assignments to have reviews for than the minimum number required for all students to have the minimum number of reviews?

===Conclusion===
While we believe the current state of this feature is greatly improved, with many of its DRY violations removed and several noteworthy bugs corrected, we believe the feature of allowing students to bid on what to review still needs further development before it is deployed. While the basic functionality is working under ideal situations, the issues mentioned above should be addressed before this feature is put in the hands of actual instructors, as if this feature fails, it could ruin the entire reviewing process for an assignment.

Still, we think there is notable promise to this feature, and it would be a very engaging and interesting feature if deployed successfully, so we believe develop on it should continue until it is ready for deployment.

==Github and Related Link==
Github repository corresponding to this task is publicly available at [https://github.com/WeiRui-Wang/expertiza E2151 GitHub].

Github pull request and file changes can be viewed at [https://github.com/expertiza/expertiza/pull/2150/files E2151 Pull Request].

Github of source code of externally hosted bidding algorithm service deployment manual and files can be access at [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment].

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: Geoff Garrido, Ayush Luthra, and WeiRui Wang. Our project mentor was Luis Delossantos (lgdeloss@ncsu.edu)

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-30T01:54:34Z

Aluthra: /* review_bid.rb */

==Topic Overview==
===Statement of Problem===
Reviewing is an important stage of learning and improving, both as a reviewer and a reviewee, and so allowing reviewers more freedom and choice in what they review would result in better reviews and more engaged reviewers. However, currently the only way to choose a topic to review is a first-come-first-served basis, penalizing students who do not sign up quickly by not allowing them to review the topics they would be most interesting in engaging with.

A better alternative would be to implement a bidding system, similar to that used for topics, which would allow students to establish preferences for which topics they would most like to review. A good bidding algorithm implementing with this idea would ensure that all students (who care enough to bid) receive a topic to review that they actively want.

A prior project group created a prototype for this function, allowing students to bid on what topics they wanted to review. While the functionality was as desired, the feature was not able to be merged due to having too many DRY violations. Thus, the goal of this topic was to bring this prior implementation into the current version of the expertiza beta branch, and remove these DRY violations, so that it may be merged.

As development continued, it became clear this prototype also had several bugs and issues that made it unable to work under several conditions. Thus, a secondary goal was to fix any issues which existed such that the feature would generally work with the functionality the prior implementation demonstrated.

===Prior Work===
The prior work which is the basis for this topic is linked here [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review is linked here]. This work was done in Fall of 2020, and as previously mentioned, achieved much of the desired functionality for this task. This prior work will be referenced several times within this documentation, as it has a more comprehensive explanation of the underlying design principles behind the views, controller, model, algorithm, and RSpec tests implemented. These details will only be briefly summarized here.

As previously noted, this prior implementation was rejected due to DRY violations, and so correcting these violations--and explaining how those corrections were done--will be the major focus of this project documentation.

==Explanation of Feature==

===Existing Bidding Functionality for Topics===
Currently in expertiza, students may bid on which topics they would like to work on. An image showcasing this functionality is given below:

[[File:expected_review_UI.PNG]]

Students may choose a number of topics to bid on, in order of priority, and once the deadline is reached, all students are given a topic.

Behind the scenes, this works by utilizing a modified version of the Gale-Shapely Algorithm. This algorithm seeks to find a 'stable' matching of each student to each topic, such that in the end, all students have a topic they wanted (at least to some degree), and that the preferences of students is respected as well as possible. To help break situations where there are ties in preferences, each topic assigns a Priority to each student. This Priority is determined by A. whether the student is part of a group bidding for this topic (groups getting higher priority), and B. the time at which the bid for the topic was made (earlier bidders are given Priority). Once this has been determined and the deadline for bidding has ended, the following steps occur:
# Each student is matched with each topic they most prefer.
# If the number of students assigned to a topic is above the maximum allowed for that topic, the students with the highest Priority are assigned the topic
# All maximally filled topics are removed from the pool of available topics
# All unmatched students begin this cycle again, now choosing the most preferred topic remaining
# Repeat until all students are assigned a topic
# If any matches are 'unstable' (two students would have their preferences better met if they were swapped), perform any swaps until matches are as stable as possible.

This algorithm has a time complexity of O(n^2). For more details on this algorithm and its implementation, refer to the previous year's work, where this algorithm was implemented.

===Desired Functionality for Reviews===

The desired feature for reviews should have an almost identical structure to this, with reviews instead of topics. Students should be shown a list of the topics that were worked on, and may bid on which of those topics they would like to review the most. The Priority system could be handled in the same way, with the only caveat being that a student should not be able to review their own work.

In terms of what this feature looks like, it can be enabled as an option when editing/creating a new assignment, under the "Topics" section, using interface drop down show below.

[[File:Bidding_interface.png|center|800px]]

Once this option is enabled, during the review phase of an assignment, students will be able to access a new view listing all the topics of all the submitted assignments (except for their own). They will then be able to bid on as many topics as they like, assigning them a preference, in a manner similar to the interface for topic bidding. The table that allows for this drag-and-drop priorities is shown below.

[[File:Bidding_options.png|center|900px]]

Whenever the instructor decides they would like to close the window for review bidding, they may return to the options for the given assignment and navigate to "Topics", where they will see a button marked "Run Review Algorithm". This will make a call to the external webservice to determine which students are assigned what reviews, and will then assign those reviews to all students. Once this button is pressed, it locks out the ability for students to perform any more bidding, and cannot be pressed again. The button allowing this option is shown below.

[[File:Bidding_run_algorithm.png|center|900px]]

After this point, students may navigate to the "Others Work" option under their assignment and see all of their assigned reviews. They will see up to the maximum number of allowed reviews, but will be informed of how many they are required to do (for example, a student may see 4 reviews, but will be told they are only required to complete 2).

===Use Case Diagram===
The following Use Case Diagram demonstrates the primary functionality explained in the previous section for different types of users.

[[File:Use_Case_Diagram_Rev1_E2151.jpg|center]]

==Plan and List of Work Done==

===General Design Goals===

* Move all prior work for this topic into a new version of the expertiza beta branch, without loss in functionality.
* Ensure all prior/expected functionality is working correctly, and if not, make any repairs or changes necessary.
* DRY out prior implementation's controller and model methods which have functionality that already exists within the expertiza system.
* Remove repetitive views and partials that exist by merging views for review bids into the set of views for sign up sheets, from which it uses partials.
* Ensure all present test cases still work properly after making the above changes.

===Specific Tasks Completed===

* Transferred over all code files relevant to the review bidding feature
* Re-deployed the external website which handles the review bidding algorithm
* Added that website's url to the webservices url's within expertiza
* Enhanced the stability of the webservice for the bidding algorithm by testing with edge cases and optimize the endpoint by restoring the original access endpoint reference
* Analyzed the necessity of the phase finding related logic code section
* Optimized phase finding related logic by removing unused code segment
* Removed an instance variable within the ''review_bids_controller'' which would have made any practical deployment fail, removing the unnecessary functionality it was created for
* Updated method calls to ''assignment.find_current_stage'' to their new form ''assignment.current_stage''
* Removed the review_bid model method ''self.reviewer_self_topic'' and replaced its usage with the already-existing ''SignedUpTeam.topic_id'' method
* Removed DRY-violating code used to calculate the current assignment phase which was never utilized by the controller or views
* Moved several view files from the ''review_bids'' view directory into the ''sign_up_sheet'' view directory. Partials which were duplicates in the two directories (''_all_actions.html.erb'' and ''_topic_names.html.erb'' were no longer necessary and deleted
* Refactored the naming of the variable that represents the id of the reviewer object from 'reviewer' to 'reviewer_id' for clarity
* Removed the unnecessary 'create' function within the review_bids_controller
* Removed the sparse testing files under the 'test' directory in keeping with typical expertiza structure
* Removed a test case from the unit tests which relied upon the removed 'create' method of the review_bids_controller

==Initial Implementation==

A considerable amount of time and effort was necessary to review, understand, and re-implement the previous implementation of the Review Bidding feature. While the feature was only created a year ago, several underlying functions and methods have since been refactored, and so needed to be changed in the implementation. Additionally, certain parts of how the implementation was designed did not seem to be properly implemented, and so had to be revised.

===Code Transfer===
First, all files that were added in the original pull request were copied and moved to the new version of the expertiza beta branch. This includes the following files:

*'''controllers/review_bids_controller.rb''': the primary controller that handles calls to review bids functionality from views, managing them itself or directing them to the appropriate model
*'''helpers/review_bids_helper.rb''': contains helper methods for rendering the rows of topics available to choose a review for
*'''models/review_bid.rb''': contains the methods that each individual review bid knows and manages, including assigning a review to a student, getting info on the topic it applies to, etc.
*'''views/review_bids/_all_actions.html.erb''': a view partial used to render the actions available to a student on the review bidding view. Nearly identical to the _all_actions file for topic bidding.
*'''views/review_bids/_table_header.html.erb''': a view partial used to render the head of the table for review bids. Though structurally similar to that of topics reviews, contains significantly less information, as less needs to be known by a student for particular topics to review.
*'''views/review_bids/_table_line.html.erb''': a view partial used to render individual rows of the review bids table. Like the table header, is structurally similar to that of topic bidding, but functions differently.
*'''views/review_bids/show.html.erb''': the view which displays all the available topics to bid on reviewing. Uses the aforementioned partials to render this bidding table.
*'''views/review_bids/others_work.html.erb''': displays the available review work which has the user has bid for.
*'''db/migrate/20201107203204_create_review_bids.rb''': the migration which creates the review bid objects within the expertiza database.
*'''spec/controllers/review_bids_controller_spec.rb''': the unit tests for the review bids controller
*'''spec/models/review_bid_spec.rb''': the unit tests for the review bids model
*'''test/controllers/review_bids_controller_test.rb''': a more elaborate feature test for the review bids functionality
*'''test/fixtures/review_bids.yml''': fixtures that are utilized for the review_bids_controller_test
*'''test/models/review_bid_test.rb''': A more elaborate feature test of the review bid model.

In addition to adding the previous files, the following files were also added to existing pages as modifications:

*'''controllers/student_review_controller.rb''': the student review controller's "list" method now redirects to the review_bid_controller's "index" method if the assignment in question has a review_choosing_algorithm set to "Bidding" (the option an instructor sets when they wish to allow students to bid on reviews).
*'''student_task/view.html.erb''': Instead of simply linking directly to the student_review controller's "list" method, the view instead checks if the assignment this view pertains to has their review_choosing_algorithm set to Bidding. If it does, and the assignment allows students to choose the topic to review (meaning the algorithm to assign bids has not yet been run), they are redirected to the 'show' method of the review_bids_controller. Otherwise, they are redirected to the 'index' method (showing them the info on their currently available topics to review).
*'''config/webservices.yml''': added a new url for the website that is called to run the review bidding algorithm
*'''config/routes.rb''': routes added for review_bids.
*'''views/assignments/edit/_topics.html.erb''': Added a new option box which allows an instructor to choose to allow bidding to occur using the review bidding algorithm, rather than the default 'Simple Choose.' Once this option is selected, it also offers a button that, when clicked, runs the review bidding algorithm on all currently bid upon reviews.

===Website Reimplementation===
In the previous implementation, the algorithm which handles the assigning of reviews was hosted on an external website. This was done for the sake of improved performance. Thus, to get the feature in working condition, we pulled the code necessary to run the algorithm and hosted it on our own website (linked at the end of this documentation). We also made a note in the pull request that the place where this website is referenced (in the "assign_reviews_algorithm" method of the review_bids_controller) should be replaced by a proper external web tool, as is done for similar web tools in other parts of expertiza.

===Modifications to Methods===
Certain methods utilized in the original version of the webpages were changed or renamed in later versions, and so, needed to be modified. The primary example of this is the method that assignments use to determine what stage of assignment completion they are in. The previous year's implementation called this method as "assignment.find_current_stage(topic_id)," but this failed when trying to execute. After comparing how this page worked to how normal review pages work, we determined the new call to this method is "assignment.current_stage(topic_id)." All calls to this method were replaced in all files.

Another change to methods was the removal of a method from the review_bids_controller named 'create'. This method primarily served as a redirect for an aspect of the feature that was unnecessary and so removed (see the following section), that being the button to show another review. With this feature removed, this method was no longer used, and so it was removed from the controller.

===Errors Corrected===
A notable period of time was also spent on determining the purpose of a particular local area that caused issues when trying to view the bids, that being a variable named "@number_of_reviews_to_show". This variable apparently kept track of how many reviews a student needed to be shown, so that they could request another. The issue was that this was created as a local variable within the review_bids_controller, and therefore would never work outside of a manual testing environment, as the variable would need to be different from every student and every assignment. After some investigation, we realized that all this variable did was limit the number of already-assigned reviews the student was able to see (as the review bidding algorithm already assigned all possible reviews). Thus, rather than try to recreate this functionality, we simply removed it.

This means that, when the review bidding algorithm runs, a student is able to see all of the reviews assigned to them, up to the maximum number of reviews for the assignment, and therefore have no reason to ask for any more. It may be the case that this feature could be improved by adding a possibility for students to request any reviews not assigned to any other student, or that still have available slots, using the "normal" review assignment scheme. For the time being, we did not consider this part of the baseline necessary functionality, and so did not implement it.

==Final Implementation==

===DRY Principle and Code Duplication Refactoring===
Per comments and feedbacks on the previous implementation [https://github.com/expertiza/expertiza/pull/1822 E2085], there were 3 sections of the code that potentially were DRY violations. For each of those violations, we have had made appropriate and provided the justification on the results.

====review_bids_controller.rb====
For the refactoring of review bids controller.eb, the original issue is the DRY principle violation on the internal logic of finding the phase of assignments.
The original implementation to be refactored and optimized are shown as follow.
<pre>
# Finding the current phase that we are in
due_dates = AssignmentDueDate.where(parent_id: @assignment.id)
@very_last_due_date = AssignmentDueDate.where(parent_id: @assignment.id).order("due_at DESC").limit(1)
next_due_date = @very_last_due_date[0]
for due_date in due_dates
if due_date.due_at > Time.now
next_due_date = due_date if due_date.due_at < next_due_date.due_at
end
end
@review_phase = next_due_date.deadline_type_id
</pre>
Through some discovery process, we have found some related logics that is already implemented from the system which enable the refactoring and reuse of the method. Some code snippets are demonstrated as follow.
<pre>
def self.current_due_date(due_dates)
#Get the current due date from list of due dates
due_dates.each do |due_date|
if due_date.due_at > Time.now
current_due_date = due_date
return current_due_date
end
end
#in case current due date not found
return nil
end
</pre>
<pre>
def self.teammate_review_allowed(student)
# time when teammate review is allowed
due_date = current_due_date(student.assignment.due_dates)
student.assignment.find_current_stage == 'Finished' ||
due_date &&
(due_date.teammate_review_allowed_id == 3 ||
due_date.teammate_review_allowed_id == 2) # late(2) or yes(3)
end
</pre>
As we can observed above, both methods of <code>current_due_date</code> and <code>teammate_review_allowed</code> provides the capability of getting the <code>review_phase</code> without the repetition. Additionally, we can see that those existing implementation still have some difference in implementation and purpose to be fully reusable by the logic that find current phase of a topic because the methods provided are helper methods for broader abstraction.

Furthermore, the section that we attempted to refactoring out have a similar logic that is used to get the current stage as shown below.
<pre>
def find_current_stage(topic_id = nil)
next_due_date = DueDate.get_next_due_date(self.id, topic_id)
return 'Finished' if next_due_date.nil?
next_due_date
end
</pre>
However, it is worth pointing out that the method is not fully substitutable, particularly the internal functionality that finds the <code>@very_last_due_date</code>. A similar logic does exist that is meant to use to acquire current due date and assist with finding current stage.
<pre>
def current_stage_name(topic_id = nil)
if self.staggered_deadline?
return (topic_id.nil? ? 'Unknown' : current_stage(topic_id))
end

due_date = find_current_stage(topic_id)
unless due_date == 'Finished' || due_date.nil? || due_date.deadline_name.nil?
return due_date.deadline_name
end
current_stage(topic_id)
end
</pre>
With all the findings, we have come to conclusion that the portion of the code to be dry is in similar, but required full restructure, form of existing methods. The internal logic flow itself is not fully matching the reusable methods consists across the controllers. Because of the clarity of limited reusability of existing methods, we have come to conclusion that the portions to be refactored are not duplication of any existing methods. Additionally, after a more detailed discovery, it has come to realization that the code portion of getting the current phase of the next due date was, in fact, meant to find the last due date. After further analysis of the purpose of the code, the portion has been deleted from <code>review_bids_controller.rb</code> because there was no usage observed with the result of the execution.

====review_bid.rb====
Per the previous iteration's feedback, the review_bid.rb model file presented some DRY violations. These violations are highlighted in the method below:
<pre>
def self.reviewer_self_topic(reviewer_id)
participant = AssignmentParticipant.find_by(id: reviewer_id)
team_id = participant.team.try(:id)
self_topic = SignedUpTeam.where(team_id: team_id).pluck(:topic_id).first

return self_topic
end
</pre>
This method returns the topic corresponding to the reviewer that is passed into the method. Line 1 gets the participant from the reviewer_id parameter. Line 2 extracts the team that the participant is in. Line 3 will get the topic which is assigned to the participant's team. This topic is returned. Our goal while refactoring was to find a similar method that can replace this method.

While searching through the project for similar methods which perform the same action, we came across this method in the signed_up_team.rb model:
<pre>
def self.topic_id(assignment_id, user_id)
# team_id variable represents the team_id for this user in this assignment
team_id = TeamsUser.team_id(assignment_id, user_id)
topic_id_by_team_id(team_id) if team_id
end
</pre>
This method gets the team_id from the user_id and isolates the topic from said team_id. This method performs a similar function to reviewer_self_topic. We decided to replace all uses of reviewer_self_topic with topic_id. By reusing an existing method, we are DRYing out this code. By removing reviewer_self_topic from the code base and using an existing method, less code is being reused.
 
The previous uses of reviewer_self_topic such as:
<pre>
def self.reviewer_bidding_data(reviewer,assignment_id)
self_topic = self.reviewer_self_topic(reviewer)
bidding_data = {'tid'=> [], 'otid' => self_topic, 'priority' => [], 'time'=>[]}
bids = ReviewBid.where(participant_id: reviewer)
</pre>
<pre>

</pre>

====Refactoring Views (_all_actions.html.erb and _topic_names.html.erb)====
The last element of the prior implementation that caused DRY violations had to do with its implementation of views. Two primary views are necessary for the review bids functionality: one which shows a table where students can bid on which topic they wish to review (rendered by a file called 'show.html.erb'), and a list of assigned reviews which a student must select from (rendered by a file called 'others_work.html.erb').

In order to render the elements in 'show.html.erb' in particular, a number of partials must be utilized:
*a partial for the table's header (_table_header.html.erb)
*a partial for rendering each of the table's rows, describing a topic to bid on (_table_line.html.erb)
*a partial to contain information related to what actions are available for a student to do for a review, like edit or view (_all_actions.html.erb)
*a partial for finding and rendering the name of each topic one must review (_topic_names.html.erb)

Both of these views and all four of these partials were contained within the "review_bids" directory within the views directory. The DRY violation incurred by these views has to do with these partials: in particular, '_topic_names.html.erb' and '_all_actions.html.erb' are near identical code copies to those within the sign_up_sheet views, used for rendering and displaying information for topic bidding and normal reviews.

In order to DRY out this portion of the code, we decided that having a separate set of views for the results related to review bids were unnecessary. Within expertiza's structure, most of the views related to students choosing which topics or reviews to work on are all contained within the sign_up_sheet views. The review_vids views were copying the partials within the sign_up_sheet views because they were using the same fundamental components. Thus, we thought the best solution would be to move the two primary views for review_bids ('show.html.erb' and 'others_work.html.erb') into the sign_up_sheet views, and leave behind all redundant partials. In doing so, the redundancy of '_all_actions.html.erb' and '_topic_names.html.erb' were eliminated.

It could be argued that this implementation could be further DRY-ed by leaving behind the other two partials, '_table_header.html.erb' and '_table_line.html.erb', as they are structurally similar to those used for topic bidding. However, this is not actually permissible, as the header and lines for topics when bidding on them for reviews are significantly different than when bidding on them for assignments. More information needs to be available when bidding on topics for an assignment, such as the number of slots and number on the waitlist, but such an idea is not applicable for bids on reviews. While it would be possible to have a single file which produces one view or another depending upon if it's bidding for topics or reviews, we believed this violated the Expert Pattern, by attempting to create a single view for two distinct purposes. Instead, we thought it better to simply have two separate partials, each which precisely gave the desired functionality. Thus, the other two partials were brought over, being renamed '_review_bids_table_header.html.erb' and '_review_bids_table_line.html.erb'.

==Tests==

Throughout the refactoring and merging of [https://github.com/expertiza/expertiza/pull/1822 E2085] to [https://github.com/expertiza/expertiza/pull/2150 E2151], there were some updates that has been made to the test suits to maintain corrosponding testing automation. Since our implementations and changes are focusing on DRY principal optimization, most of the test are based on the previous test that interact with the system and observe the behavior of the user views.

===Video Demonstration===
Visit [https://www.youtube.com/watch?v=wxXALQNWFcc Feature Demo] to see a video demonstrating the functionality of the review bidding feature that is working properly after our DRY principle optimization.

===RSpec Testing===
One of the primary design goals when doing this refactoring was to ensure that all test cases still passed after all changes were made.

There was a single change that needed to be made to the RSpec files for the review bidding algorithm, which was the removal of a test relying on the 'create' method of the review_bids_controller. As previously mentioned, this function was removed because it had become obsolete. Thus, the test case for this function (which simply checked if it redirected to the right page) was no longer necessary, and thus was deleted.

Otherwise, the test cases for both the review_bids_controller and the review_bids model were manually run and found to have no errors, as shown below.

[[File:Bidding_test_results.png|800px]]

For details on what these specific tests were, please refer to the previous work referenced before.

==Conclusions and Future Work==
While the current state of the review bidding code is operational, and is certainly more DRY and stable than the previous incarnation of this work, there are still several issues with the current state of this feature that make it difficult to recommend an immediate merge into expertiza.

===Enhancement of Algorithm Service Stability===

Across the process of refining the refactored front end page view along with more robust edge cases testing towards the algorithm service, we have discovered an underlying issue that could impact the availability of the review bidding algorithm service. Through cause isolation, we have observed a high correlation to flask application crash with the edge case scenario input where the algorithm service is trying to match a student who was not signed up with any topic with other students.

Because the scope of our project was focused on refactoring, we did not have the time or prior warning to need to address these edge cases. However, we will provide the error log for the reference of any future teams who may work on the stability improvement of the algorithm service.

<pre>
"POST /match_topics HTTP/1.1" 500 -
{'tid': [5133, 5134, 5135, None], 'users': {'45514': {'otid': 5135, 'tid': [5133, 5134], 'time': ['2021-11-30T17:57:20.000-05:00', '2021-11-30T17:57:24.000-05:00'], 'priority': [2, 1]}, '45513': {'otid': 5134, 'tid': [5133, 5135], 'time': ['2021-11-30T17:57:41.000-05:00', '2021-11-30T17:57:43.000-05:00'], 'priority': [2, 1]}, '45512': {'otid': 5133, 'tid': [5134, 5135], 'time': ['2021-11-30T17:57:58.000-05:00', '2021-11-30T17:58:00.000-05:00'], 'priority': [2, 1]}, '45515': {'otid': None, 'tid': [], 'time': [0], 'priority': [3, 1, 4, 2]}}, 'max_accepted_proposals': 3}
[2021-11-28 18:10:40,576] ERROR in app: Exception on /match_topics [POST]
Traceback (most recent call last):
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/usr/.local/lib/python3.5/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/usr/IntelligentAssignment/app/app.py", line 19, in topic_matching
return jsonify(assigned_topics_model.get_student_topic_matches())
File "/home/usr/IntelligentAssignment/app/topics_matcher.py", line 57, in get_student_topic_matches
student.accept_proposal()
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in accept_proposal
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in <lambda>
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
ValueError: None is not in list
</pre>

As we can see from the additional information provided above, the error is due to an unhandled edge case of attempting to access missing value. Our further observation also concluded that such missing value is due to the fact that one or more to-be-matched students did not have any topic signed up at the time of algorithm execution. To address the issue, a general enhancement can be implemented on the algorithm service side with Flask to increase the crash recovery stability. Further refinement of interaction flow with user in the front end can also reduce the possibility of algorithm service error.

Additional to the unhandled cases from the algorithm service, we also noticed an inconsistency of the system due to the performance limitation of the Heroku deployment.

[[File:Out_of_scope_External-Issue_E2151.gif|frameless|x1000px]]

As we can see from the above recorded screen capture animation, with the exact same input, the output and algorithm service result are unstable. Because of the performance limitation of the Heroku, sometime the Flask app that deployed were overloaded. Furthermore, we have also attempted to deploy on VCL, however, we notice the limitation of VCL deployment that will become offline once the VCL reservation expired. After all, because of the focus of our tasks is on the DRY principal optimization, the external algorithm service can be improved in the future. To access and deploy the algorithm on local environment or another environment, setup guide and resource are available on [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment] repository.

===Lack of Comprehensive Testing===

A notable contradiction that was noticed when working with this project was that the previously-mentioned failed edge cases existed, yet all tests for this feature passed. This is illustrative of the fact that this feature requires better and more exhaustive testing. With the scope limitation of our tasks that focusing on the DRY principal enhancement of the Expertiza internally, the external algorithm service module was not within our focus. However, with our experience of deployment with the algorithm service, we can offer some suggestions for the future team who will address those edge cases of the algorithm service.

When dealing with an issue with so many factors as how students bid on reviews, an extensive suite of edge cases should be considered. Any future project group working on this function should consider the following edge cases to cover:
* Does a student who does not bid on any reviews still get reviews?
* Does a student who did not submit anything for their assignment still have a review to bid on?
* What if there are less assignments to have reviews for than the minimum number required for all students to have the minimum number of reviews?

===Conclusion===
While we believe the current state of this feature is greatly improved, with many of its DRY violations removed and several noteworthy bugs corrected, we believe the feature of allowing students to bid on what to review still needs further development before it is deployed. While the basic functionality is working under ideal situations, the issues mentioned above should be addressed before this feature is put in the hands of actual instructors, as if this feature fails, it could ruin the entire reviewing process for an assignment.

Still, we think there is notable promise to this feature, and it would be a very engaging and interesting feature if deployed successfully, so we believe develop on it should continue until it is ready for deployment.

==Github and Related Link==
Github repository corresponding to this task is publicly available at [https://github.com/WeiRui-Wang/expertiza E2151 GitHub].

Github pull request and file changes can be viewed at [https://github.com/expertiza/expertiza/pull/2150/files E2151 Pull Request].

Github of source code of externally hosted bidding algorithm service deployment manual and files can be access at [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment].

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: Geoff Garrido, Ayush Luthra, and WeiRui Wang. Our project mentor was Luis Delossantos (lgdeloss@ncsu.edu)

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-30T01:36:37Z

Aluthra: /* review_bid.rb */

==Topic Overview==
===Statement of Problem===
Reviewing is an important stage of learning and improving, both as a reviewer and a reviewee, and so allowing reviewers more freedom and choice in what they review would result in better reviews and more engaged reviewers. However, currently the only way to choose a topic to review is a first-come-first-served basis, penalizing students who do not sign up quickly by not allowing them to review the topics they would be most interesting in engaging with.

A better alternative would be to implement a bidding system, similar to that used for topics, which would allow students to establish preferences for which topics they would most like to review. A good bidding algorithm implementing with this idea would ensure that all students (who care enough to bid) receive a topic to review that they actively want.

A prior project group created a prototype for this function, allowing students to bid on what topics they wanted to review. While the functionality was as desired, the feature was not able to be merged due to having too many DRY violations. Thus, the goal of this topic was to bring this prior implementation into the current version of the expertiza beta branch, and remove these DRY violations, so that it may be merged.

As development continued, it became clear this prototype also had several bugs and issues that made it unable to work under several conditions. Thus, a secondary goal was to fix any issues which existed such that the feature would generally work with the functionality the prior implementation demonstrated.

===Prior Work===
The prior work which is the basis for this topic is linked here [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review is linked here]. This work was done in Fall of 2020, and as previously mentioned, achieved much of the desired functionality for this task. This prior work will be referenced several times within this documentation, as it has a more comprehensive explanation of the underlying design principles behind the views, controller, model, algorithm, and RSpec tests implemented. These details will only be briefly summarized here.

As previously noted, this prior implementation was rejected due to DRY violations, and so correcting these violations--and explaining how those corrections were done--will be the major focus of this project documentation.

==Explanation of Feature==

===Existing Bidding Functionality for Topics===
Currently in expertiza, students may bid on which topics they would like to work on. An image showcasing this functionality is given below:

[[File:expected_review_UI.PNG]]

Students may choose a number of topics to bid on, in order of priority, and once the deadline is reached, all students are given a topic.

Behind the scenes, this works by utilizing a modified version of the Gale-Shapely Algorithm. This algorithm seeks to find a 'stable' matching of each student to each topic, such that in the end, all students have a topic they wanted (at least to some degree), and that the preferences of students is respected as well as possible. To help break situations where there are ties in preferences, each topic assigns a Priority to each student. This Priority is determined by A. whether the student is part of a group bidding for this topic (groups getting higher priority), and B. the time at which the bid for the topic was made (earlier bidders are given Priority). Once this has been determined and the deadline for bidding has ended, the following steps occur:
# Each student is matched with each topic they most prefer.
# If the number of students assigned to a topic is above the maximum allowed for that topic, the students with the highest Priority are assigned the topic
# All maximally filled topics are removed from the pool of available topics
# All unmatched students begin this cycle again, now choosing the most preferred topic remaining
# Repeat until all students are assigned a topic
# If any matches are 'unstable' (two students would have their preferences better met if they were swapped), perform any swaps until matches are as stable as possible.

This algorithm has a time complexity of O(n^2). For more details on this algorithm and its implementation, refer to the previous year's work, where this algorithm was implemented.

===Desired Functionality for Reviews===

The desired feature for reviews should have an almost identical structure to this, with reviews instead of topics. Students should be shown a list of the topics that were worked on, and may bid on which of those topics they would like to review the most. The Priority system could be handled in the same way, with the only caveat being that a student should not be able to review their own work.

In terms of what this feature looks like, it can be enabled as an option when editing/creating a new assignment, under the "Topics" section, using interface drop down show below.

[[File:Bidding_interface.png|center|800px]]

Once this option is enabled, during the review phase of an assignment, students will be able to access a new view listing all the topics of all the submitted assignments (except for their own). They will then be able to bid on as many topics as they like, assigning them a preference, in a manner similar to the interface for topic bidding. The table that allows for this drag-and-drop priorities is shown below.

[[File:Bidding_options.png|center|900px]]

Whenever the instructor decides they would like to close the window for review bidding, they may return to the options for the given assignment and navigate to "Topics", where they will see a button marked "Run Review Algorithm". This will make a call to the external webservice to determine which students are assigned what reviews, and will then assign those reviews to all students. Once this button is pressed, it locks out the ability for students to perform any more bidding, and cannot be pressed again. The button allowing this option is shown below.

[[File:Bidding_run_algorithm.png|center|900px]]

After this point, students may navigate to the "Others Work" option under their assignment and see all of their assigned reviews. They will see up to the maximum number of allowed reviews, but will be informed of how many they are required to do (for example, a student may see 4 reviews, but will be told they are only required to complete 2).

===Use Case Diagram===
The following Use Case Diagram demonstrates the primary functionality explained in the previous section for different types of users.

[[File:Use_Case_Diagram_Rev1_E2151.jpg|center]]

==Plan and List of Work Done==

===General Design Goals===

* Move all prior work for this topic into a new version of the expertiza beta branch, without loss in functionality.
* Ensure all prior/expected functionality is working correctly, and if not, make any repairs or changes necessary.
* DRY out prior implementation's controller and model methods which have functionality that already exists within the expertiza system.
* Remove repetitive views and partials that exist by merging views for review bids into the set of views for sign up sheets, from which it uses partials.
* Ensure all present test cases still work properly after making the above changes.

===Specific Tasks Completed===

* Transferred over all code files relevant to the review bidding feature
* Re-deployed the external website which handles the review bidding algorithm
* Added that website's url to the webservices url's within expertiza
* Enhanced the stability of the webservice for the bidding algorithm by testing with edge cases and optimize the endpoint by restoring the original access endpoint reference
* Analyzed the necessity of the phase finding related logic code section
* Optimized phase finding related logic by removing unused code segment
* Removed an instance variable within the ''review_bids_controller'' which would have made any practical deployment fail, removing the unnecessary functionality it was created for
* Updated method calls to ''assignment.find_current_stage'' to their new form ''assignment.current_stage''
* Removed the review_bid model method ''self.reviewer_self_topic'' and replaced its usage with the already-existing ''SignedUpTeam.topic_id'' method
* Removed DRY-violating code used to calculate the current assignment phase which was never utilized by the controller or views
* Moved several view files from the ''review_bids'' view directory into the ''sign_up_sheet'' view directory. Partials which were duplicates in the two directories (''_all_actions.html.erb'' and ''_topic_names.html.erb'' were no longer necessary and deleted
* Refactored the naming of the variable that represents the id of the reviewer object from 'reviewer' to 'reviewer_id' for clarity
* Removed the unnecessary 'create' function within the review_bids_controller
* Removed the sparse testing files under the 'test' directory in keeping with typical expertiza structure
* Removed a test case from the unit tests which relied upon the removed 'create' method of the review_bids_controller

==Initial Implementation==

A considerable amount of time and effort was necessary to review, understand, and re-implement the previous implementation of the Review Bidding feature. While the feature was only created a year ago, several underlying functions and methods have since been refactored, and so needed to be changed in the implementation. Additionally, certain parts of how the implementation was designed did not seem to be properly implemented, and so had to be revised.

===Code Transfer===
First, all files that were added in the original pull request were copied and moved to the new version of the expertiza beta branch. This includes the following files:

*'''controllers/review_bids_controller.rb''': the primary controller that handles calls to review bids functionality from views, managing them itself or directing them to the appropriate model
*'''helpers/review_bids_helper.rb''': contains helper methods for rendering the rows of topics available to choose a review for
*'''models/review_bid.rb''': contains the methods that each individual review bid knows and manages, including assigning a review to a student, getting info on the topic it applies to, etc.
*'''views/review_bids/_all_actions.html.erb''': a view partial used to render the actions available to a student on the review bidding view. Nearly identical to the _all_actions file for topic bidding.
*'''views/review_bids/_table_header.html.erb''': a view partial used to render the head of the table for review bids. Though structurally similar to that of topics reviews, contains significantly less information, as less needs to be known by a student for particular topics to review.
*'''views/review_bids/_table_line.html.erb''': a view partial used to render individual rows of the review bids table. Like the table header, is structurally similar to that of topic bidding, but functions differently.
*'''views/review_bids/show.html.erb''': the view which displays all the available topics to bid on reviewing. Uses the aforementioned partials to render this bidding table.
*'''views/review_bids/others_work.html.erb''': displays the available review work which has the user has bid for.
*'''db/migrate/20201107203204_create_review_bids.rb''': the migration which creates the review bid objects within the expertiza database.
*'''spec/controllers/review_bids_controller_spec.rb''': the unit tests for the review bids controller
*'''spec/models/review_bid_spec.rb''': the unit tests for the review bids model
*'''test/controllers/review_bids_controller_test.rb''': a more elaborate feature test for the review bids functionality
*'''test/fixtures/review_bids.yml''': fixtures that are utilized for the review_bids_controller_test
*'''test/models/review_bid_test.rb''': A more elaborate feature test of the review bid model.

In addition to adding the previous files, the following files were also added to existing pages as modifications:

*'''controllers/student_review_controller.rb''': the student review controller's "list" method now redirects to the review_bid_controller's "index" method if the assignment in question has a review_choosing_algorithm set to "Bidding" (the option an instructor sets when they wish to allow students to bid on reviews).
*'''student_task/view.html.erb''': Instead of simply linking directly to the student_review controller's "list" method, the view instead checks if the assignment this view pertains to has their review_choosing_algorithm set to Bidding. If it does, and the assignment allows students to choose the topic to review (meaning the algorithm to assign bids has not yet been run), they are redirected to the 'show' method of the review_bids_controller. Otherwise, they are redirected to the 'index' method (showing them the info on their currently available topics to review).
*'''config/webservices.yml''': added a new url for the website that is called to run the review bidding algorithm
*'''config/routes.rb''': routes added for review_bids.
*'''views/assignments/edit/_topics.html.erb''': Added a new option box which allows an instructor to choose to allow bidding to occur using the review bidding algorithm, rather than the default 'Simple Choose.' Once this option is selected, it also offers a button that, when clicked, runs the review bidding algorithm on all currently bid upon reviews.

===Website Reimplementation===
In the previous implementation, the algorithm which handles the assigning of reviews was hosted on an external website. This was done for the sake of improved performance. Thus, to get the feature in working condition, we pulled the code necessary to run the algorithm and hosted it on our own website (linked at the end of this documentation). We also made a note in the pull request that the place where this website is referenced (in the "assign_reviews_algorithm" method of the review_bids_controller) should be replaced by a proper external web tool, as is done for similar web tools in other parts of expertiza.

===Modifications to Methods===
Certain methods utilized in the original version of the webpages were changed or renamed in later versions, and so, needed to be modified. The primary example of this is the method that assignments use to determine what stage of assignment completion they are in. The previous year's implementation called this method as "assignment.find_current_stage(topic_id)," but this failed when trying to execute. After comparing how this page worked to how normal review pages work, we determined the new call to this method is "assignment.current_stage(topic_id)." All calls to this method were replaced in all files.

Another change to methods was the removal of a method from the review_bids_controller named 'create'. This method primarily served as a redirect for an aspect of the feature that was unnecessary and so removed (see the following section), that being the button to show another review. With this feature removed, this method was no longer used, and so it was removed from the controller.

===Errors Corrected===
A notable period of time was also spent on determining the purpose of a particular local area that caused issues when trying to view the bids, that being a variable named "@number_of_reviews_to_show". This variable apparently kept track of how many reviews a student needed to be shown, so that they could request another. The issue was that this was created as a local variable within the review_bids_controller, and therefore would never work outside of a manual testing environment, as the variable would need to be different from every student and every assignment. After some investigation, we realized that all this variable did was limit the number of already-assigned reviews the student was able to see (as the review bidding algorithm already assigned all possible reviews). Thus, rather than try to recreate this functionality, we simply removed it.

This means that, when the review bidding algorithm runs, a student is able to see all of the reviews assigned to them, up to the maximum number of reviews for the assignment, and therefore have no reason to ask for any more. It may be the case that this feature could be improved by adding a possibility for students to request any reviews not assigned to any other student, or that still have available slots, using the "normal" review assignment scheme. For the time being, we did not consider this part of the baseline necessary functionality, and so did not implement it.

==Final Implementation==

===DRY Principle and Code Duplication Refactoring===
Per comments and feedbacks on the previous implementation [https://github.com/expertiza/expertiza/pull/1822 E2085], there were 3 sections of the code that potentially were DRY violations. For each of those violations, we have had made appropriate and provided the justification on the results.

====review_bids_controller.rb====
For the refactoring of review bids controller.eb, the original issue is the DRY principle violation on the internal logic of finding the phase of assignments.
The original implementation to be refactored and optimized are shown as follow.
<pre>
# Finding the current phase that we are in
due_dates = AssignmentDueDate.where(parent_id: @assignment.id)
@very_last_due_date = AssignmentDueDate.where(parent_id: @assignment.id).order("due_at DESC").limit(1)
next_due_date = @very_last_due_date[0]
for due_date in due_dates
if due_date.due_at > Time.now
next_due_date = due_date if due_date.due_at < next_due_date.due_at
end
end
@review_phase = next_due_date.deadline_type_id
</pre>
Through some discovery process, we have found some related logics that is already implemented from the system which enable the refactoring and reuse of the method. Some code snippets are demonstrated as follow.
<pre>
def self.current_due_date(due_dates)
#Get the current due date from list of due dates
due_dates.each do |due_date|
if due_date.due_at > Time.now
current_due_date = due_date
return current_due_date
end
end
#in case current due date not found
return nil
end
</pre>
<pre>
def self.teammate_review_allowed(student)
# time when teammate review is allowed
due_date = current_due_date(student.assignment.due_dates)
student.assignment.find_current_stage == 'Finished' ||
due_date &&
(due_date.teammate_review_allowed_id == 3 ||
due_date.teammate_review_allowed_id == 2) # late(2) or yes(3)
end
</pre>
As we can observed above, both methods of <code>current_due_date</code> and <code>teammate_review_allowed</code> provides the capability of getting the <code>review_phase</code> without the repetition. Additionally, we can see that those existing implementation still have some difference in implementation and purpose to be fully reusable by the logic that find current phase of a topic because the methods provided are helper methods for broader abstraction.

Furthermore, the section that we attempted to refactoring out have a similar logic that is used to get the current stage as shown below.
<pre>
def find_current_stage(topic_id = nil)
next_due_date = DueDate.get_next_due_date(self.id, topic_id)
return 'Finished' if next_due_date.nil?
next_due_date
end
</pre>
However, it is worth pointing out that the method is not fully substitutable, particularly the internal functionality that finds the <code>@very_last_due_date</code>. A similar logic does exist that is meant to use to acquire current due date and assist with finding current stage.
<pre>
def current_stage_name(topic_id = nil)
if self.staggered_deadline?
return (topic_id.nil? ? 'Unknown' : current_stage(topic_id))
end

due_date = find_current_stage(topic_id)
unless due_date == 'Finished' || due_date.nil? || due_date.deadline_name.nil?
return due_date.deadline_name
end
current_stage(topic_id)
end
</pre>
With all the findings, we have come to conclusion that the portion of the code to be dry is in similar, but required full restructure, form of existing methods. The internal logic flow itself is not fully matching the reusable methods consists across the controllers. Because of the clarity of limited reusability of existing methods, we have come to conclusion that the portions to be refactored are not duplication of any existing methods. Additionally, after a more detailed discovery, it has come to realization that the code portion of getting the current phase of the next due date was, in fact, meant to find the last due date. After further analysis of the purpose of the code, the portion has been deleted from <code>review_bids_controller.rb</code> because there was no usage observed with the result of the execution.

====review_bid.rb====
Per the previous iteration's feedback, the review_bid.rb model file presented some DRY violations. These violations are highlighted in the method below:
<pre>
def self.reviewer_self_topic(reviewer_id)
participant = AssignmentParticipant.find_by(id: reviewer_id)
team_id = participant.team.try(:id)
self_topic = SignedUpTeam.where(team_id: team_id).pluck(:topic_id).first

return self_topic
end
</pre>
This method returns the topic corresponding to the reviewer that is passed into the method. Line 1 gets the participant from the reviewer_id parameter. Line 2 extracts the team that the participant is in. Line 3 will get the topic which is assigned to the participant's team. This topic is returned. Our goal while refactoring was to find a similar method that can replace this method.

While searching through the project for similar methods which perform the same action, we came across this method in the signed_up_team.rb model:
<pre>
def self.topic_id(assignment_id, user_id)
# team_id variable represents the team_id for this user in this assignment
team_id = TeamsUser.team_id(assignment_id, user_id)
topic_id_by_team_id(team_id) if team_id
end
</pre>
This method gets the team_id from the user_id and isolates the topic from said team_id. This method performs a similar function to reviewer_self_topic. We decided to replace all uses of reviewer_self_topic with topic_id.
 
By reusing an existing method, we are DRYing out this code. By removing reviewer_self_topic from the code base and using an existing method, less code is being reused.

====Refactoring Views (_all_actions.html.erb and _topic_names.html.erb)====
The last element of the prior implementation that caused DRY violations had to do with its implementation of views. Two primary views are necessary for the review bids functionality: one which shows a table where students can bid on which topic they wish to review (rendered by a file called 'show.html.erb'), and a list of assigned reviews which a student must select from (rendered by a file called 'others_work.html.erb').

In order to render the elements in 'show.html.erb' in particular, a number of partials must be utilized:
*a partial for the table's header (_table_header.html.erb)
*a partial for rendering each of the table's rows, describing a topic to bid on (_table_line.html.erb)
*a partial to contain information related to what actions are available for a student to do for a review, like edit or view (_all_actions.html.erb)
*a partial for finding and rendering the name of each topic one must review (_topic_names.html.erb)

Both of these views and all four of these partials were contained within the "review_bids" directory within the views directory. The DRY violation incurred by these views has to do with these partials: in particular, '_topic_names.html.erb' and '_all_actions.html.erb' are near identical code copies to those within the sign_up_sheet views, used for rendering and displaying information for topic bidding and normal reviews.

In order to DRY out this portion of the code, we decided that having a separate set of views for the results related to review bids were unnecessary. Within expertiza's structure, most of the views related to students choosing which topics or reviews to work on are all contained within the sign_up_sheet views. The review_vids views were copying the partials within the sign_up_sheet views because they were using the same fundamental components. Thus, we thought the best solution would be to move the two primary views for review_bids ('show.html.erb' and 'others_work.html.erb') into the sign_up_sheet views, and leave behind all redundant partials. In doing so, the redundancy of '_all_actions.html.erb' and '_topic_names.html.erb' were eliminated.

It could be argued that this implementation could be further DRY-ed by leaving behind the other two partials, '_table_header.html.erb' and '_table_line.html.erb', as they are structurally similar to those used for topic bidding. However, this is not actually permissible, as the header and lines for topics when bidding on them for reviews are significantly different than when bidding on them for assignments. More information needs to be available when bidding on topics for an assignment, such as the number of slots and number on the waitlist, but such an idea is not applicable for bids on reviews. While it would be possible to have a single file which produces one view or another depending upon if it's bidding for topics or reviews, we believed this violated the Expert Pattern, by attempting to create a single view for two distinct purposes. Instead, we thought it better to simply have two separate partials, each which precisely gave the desired functionality. Thus, the other two partials were brought over, being renamed '_review_bids_table_header.html.erb' and '_review_bids_table_line.html.erb'.

==Tests==

Throughout the refactoring and merging of [https://github.com/expertiza/expertiza/pull/1822 E2085] to [https://github.com/expertiza/expertiza/pull/2150 E2151], there were some updates that has been made to the test suits to maintain corrosponding testing automation.

===Video Demonstration===
Visit [https://www.youtube.com/watch?v=wxXALQNWFcc Feature Demo] to see a video demonstrating the functionality of the review bidding feature that is working properly after our DRY principle optimization.

===RSpec Testing===
One of the primary design goals when doing this refactoring was to ensure that all test cases still passed after all changes were made.

There was a single change that needed to be made to the RSpec files for the review bidding algorithm, which was the removal of a test relying on the 'create' method of the review_bids_controller. As previously mentioned, this function was removed because it had become obsolete. Thus, the test case for this function (which simply checked if it redirected to the right page) was no longer necessary, and thus was deleted.

Otherwise, the test cases for both the review_bids_controller and the review_bids model were manually run and found to have no errors, as shown below.

[[File:Bidding_test_results.png|800px]]

For details on what these specific tests were, please refer to the previous work referenced before.

==Conclusions and Future Work==
While the current state of the review bidding code is operational, and is certainly more DRY and stable than the previous incarnation of this work, there are still several issues with the current state of this feature that make it difficult to recommend an immediate merge into expertiza.

===Enhancement of Algorithm Service Stability===

Across the process of refining the refactored front end page view along with more robust edge cases testing towards the algorithm service, we have discovered an underlying issue that could impact the availability of the review bidding algorithm service. Through cause isolation, we have observed a high correlation to flask application crash with the edge case scenario input where the algorithm service is trying to match a student who was not signed up with any topic with other students.

Because the scope of our project was focused on refactoring, we did not have the time or prior warning to need to address these edge cases. However, we will provide the error log for the reference of any future teams who may work on the stability improvement of the algorithm service.

<pre>
"POST /match_topics HTTP/1.1" 500 -
{'tid': [5133, 5134, 5135, None], 'users': {'45514': {'otid': 5135, 'tid': [5133, 5134], 'time': ['2021-11-30T17:57:20.000-05:00', '2021-11-30T17:57:24.000-05:00'], 'priority': [2, 1]}, '45513': {'otid': 5134, 'tid': [5133, 5135], 'time': ['2021-11-30T17:57:41.000-05:00', '2021-11-30T17:57:43.000-05:00'], 'priority': [2, 1]}, '45512': {'otid': 5133, 'tid': [5134, 5135], 'time': ['2021-11-30T17:57:58.000-05:00', '2021-11-30T17:58:00.000-05:00'], 'priority': [2, 1]}, '45515': {'otid': None, 'tid': [], 'time': [0], 'priority': [3, 1, 4, 2]}}, 'max_accepted_proposals': 3}
[2021-11-28 18:10:40,576] ERROR in app: Exception on /match_topics [POST]
Traceback (most recent call last):
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/usr/.local/lib/python3.5/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/usr/IntelligentAssignment/app/app.py", line 19, in topic_matching
return jsonify(assigned_topics_model.get_student_topic_matches())
File "/home/usr/IntelligentAssignment/app/topics_matcher.py", line 57, in get_student_topic_matches
student.accept_proposal()
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in accept_proposal
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in <lambda>
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
ValueError: None is not in list
</pre>

As we can see from the additional information provided above, the error is due to an unhandled edge case of attempting to access missing value. Our further observation also concluded that such missing value is due to the fact that one or more to-be-matched students did not have any topic signed up at the time of algorithm execution. To address the issue, a general enhancement can be implemented on the algorithm service side with Flask to increase the crash recovery stability. Further refinement of interaction flow with user in the front end can also reduce the possibility of algorithm service error.

Additional to the unhandled cases from the algorithm service, we also noticed an inconsistency of the system due to the performance limitation of the Heroku deployment.

[[File:Out_of_scope_External-Issue_E2151.gif|frameless|x1000px]]

As we can see from the above recorded screen capture animation, with the exact same input, the output and algorithm service result are unstable. Because of the performance limitation of the Heroku, sometime the Flask app that deployed were overloaded. Furthermore, we have also attempted to deploy on VCL, however, we notice the limitation of VCL deployment that will become offline once the VCL reservation expired. After all, because of the focus of our tasks is on the DRY principal optimization, the external algorithm service can be improved in the future. To access and deploy the algorithm on local environment or another environment, setup guide and resource are available on [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment] repository.

===Lack of Comprehensive Testing===

A notable contradiction that was noticed when working with this project was that the previously-mentioned failed edge cases existed, yet all tests for this feature passed. This is illustrative of the fact that this feature requires better and more exhaustive testing. With the scope limitation of our tasks that focusing on the DRY principal enhancement of the Expertiza internally, the external algorithm service module was not within our focus. However, with our experience of deployment with the algorithm service, we can offer some suggestions for the future team who will address those edge cases of the algorithm service.

When dealing with an issue with so many factors as how students bid on reviews, an extensive suite of edge cases should be considered. Any future project group working on this function should consider the following edge cases to cover:
* Does a student who does not bid on any reviews still get reviews?
* Does a student who did not submit anything for their assignment still have a review to bid on?
* What if there are less assignments to have reviews for than the minimum number required for all students to have the minimum number of reviews?

===Conclusion===
While we believe the current state of this feature is greatly improved, with many of its DRY violations removed and several noteworthy bugs corrected, we believe the feature of allowing students to bid on what to review still needs further development before it is deployed. While the basic functionality is working under ideal situations, the issues mentioned above should be addressed before this feature is put in the hands of actual instructors, as if this feature fails, it could ruin the entire reviewing process for an assignment.

Still, we think there is notable promise to this feature, and it would be a very engaging and interesting feature if deployed successfully, so we believe develop on it should continue until it is ready for deployment.

==Github and Related Link==
Github repository corresponding to this task is publicly available at [https://github.com/WeiRui-Wang/expertiza E2151 GitHub].

Github pull request and file changes can be viewed at [https://github.com/expertiza/expertiza/pull/2150/files E2151 Pull Request].

Github of source code of externally hosted bidding algorithm service deployment manual and files can be access at [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment].

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: Geoff Garrido, Ayush Luthra, and WeiRui Wang. Our project mentor was Luis Delossantos (lgdeloss@ncsu.edu)

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-30T01:34:35Z

Aluthra: /* review_bid.rb */

==Topic Overview==
===Statement of Problem===
Reviewing is an important stage of learning and improving, both as a reviewer and a reviewee, and so allowing reviewers more freedom and choice in what they review would result in better reviews and more engaged reviewers. However, currently the only way to choose a topic to review is a first-come-first-served basis, penalizing students who do not sign up quickly by not allowing them to review the topics they would be most interesting in engaging with.

A better alternative would be to implement a bidding system, similar to that used for topics, which would allow students to establish preferences for which topics they would most like to review. A good bidding algorithm implementing with this idea would ensure that all students (who care enough to bid) receive a topic to review that they actively want.

A prior project group created a prototype for this function, allowing students to bid on what topics they wanted to review. While the functionality was as desired, the feature was not able to be merged due to having too many DRY violations. Thus, the goal of this topic was to bring this prior implementation into the current version of the expertiza beta branch, and remove these DRY violations, so that it may be merged.

As development continued, it became clear this prototype also had several bugs and issues that made it unable to work under several conditions. Thus, a secondary goal was to fix any issues which existed such that the feature would generally work with the functionality the prior implementation demonstrated.

===Prior Work===
The prior work which is the basis for this topic is linked here [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review is linked here]. This work was done in Fall of 2020, and as previously mentioned, achieved much of the desired functionality for this task. This prior work will be referenced several times within this documentation, as it has a more comprehensive explanation of the underlying design principles behind the views, controller, model, algorithm, and RSpec tests implemented. These details will only be briefly summarized here.

As previously noted, this prior implementation was rejected due to DRY violations, and so correcting these violations--and explaining how those corrections were done--will be the major focus of this project documentation.

==Explanation of Feature==

===Existing Bidding Functionality for Topics===
Currently in expertiza, students may bid on which topics they would like to work on. An image showcasing this functionality is given below:

[[File:expected_review_UI.PNG]]

Students may choose a number of topics to bid on, in order of priority, and once the deadline is reached, all students are given a topic.

Behind the scenes, this works by utilizing a modified version of the Gale-Shapely Algorithm. This algorithm seeks to find a 'stable' matching of each student to each topic, such that in the end, all students have a topic they wanted (at least to some degree), and that the preferences of students is respected as well as possible. To help break situations where there are ties in preferences, each topic assigns a Priority to each student. This Priority is determined by A. whether the student is part of a group bidding for this topic (groups getting higher priority), and B. the time at which the bid for the topic was made (earlier bidders are given Priority). Once this has been determined and the deadline for bidding has ended, the following steps occur:
# Each student is matched with each topic they most prefer.
# If the number of students assigned to a topic is above the maximum allowed for that topic, the students with the highest Priority are assigned the topic
# All maximally filled topics are removed from the pool of available topics
# All unmatched students begin this cycle again, now choosing the most preferred topic remaining
# Repeat until all students are assigned a topic
# If any matches are 'unstable' (two students would have their preferences better met if they were swapped), perform any swaps until matches are as stable as possible.

This algorithm has a time complexity of O(n^2). For more details on this algorithm and its implementation, refer to the previous year's work, where this algorithm was implemented.

===Desired Functionality for Reviews===

The desired feature for reviews should have an almost identical structure to this, with reviews instead of topics. Students should be shown a list of the topics that were worked on, and may bid on which of those topics they would like to review the most. The Priority system could be handled in the same way, with the only caveat being that a student should not be able to review their own work.

In terms of what this feature looks like, it can be enabled as an option when editing/creating a new assignment, under the "Topics" section, using interface drop down show below.

[[File:Bidding_interface.png|center|800px]]

Once this option is enabled, during the review phase of an assignment, students will be able to access a new view listing all the topics of all the submitted assignments (except for their own). They will then be able to bid on as many topics as they like, assigning them a preference, in a manner similar to the interface for topic bidding. The table that allows for this drag-and-drop priorities is shown below.

[[File:Bidding_options.png|center|900px]]

Whenever the instructor decides they would like to close the window for review bidding, they may return to the options for the given assignment and navigate to "Topics", where they will see a button marked "Run Review Algorithm". This will make a call to the external webservice to determine which students are assigned what reviews, and will then assign those reviews to all students. Once this button is pressed, it locks out the ability for students to perform any more bidding, and cannot be pressed again. The button allowing this option is shown below.

[[File:Bidding_run_algorithm.png|center|900px]]

After this point, students may navigate to the "Others Work" option under their assignment and see all of their assigned reviews. They will see up to the maximum number of allowed reviews, but will be informed of how many they are required to do (for example, a student may see 4 reviews, but will be told they are only required to complete 2).

===Use Case Diagram===
The following Use Case Diagram demonstrates the primary functionality explained in the previous section for different types of users.

[[File:Use_Case_Diagram_Rev1_E2151.jpg|center]]

==Plan and List of Work Done==

===General Design Goals===

* Move all prior work for this topic into a new version of the expertiza beta branch, without loss in functionality.
* Ensure all prior/expected functionality is working correctly, and if not, make any repairs or changes necessary.
* DRY out prior implementation's controller and model methods which have functionality that already exists within the expertiza system.
* Remove repetitive views and partials that exist by merging views for review bids into the set of views for sign up sheets, from which it uses partials.
* Ensure all present test cases still work properly after making the above changes.

===Specific Tasks Completed===

* Transferred over all code files relevant to the review bidding feature
* Re-deployed the external website which handles the review bidding algorithm
* Added that website's url to the webservices url's within expertiza
* Enhanced the stability of the webservice for the bidding algorithm by testing with edge cases and optimize the endpoint by restoring the original access endpoint reference
* Analyzed the necessity of the phase finding related logic code section
* Optimized phase finding related logic by removing unused code segment
* Removed an instance variable within the ''review_bids_controller'' which would have made any practical deployment fail, removing the unnecessary functionality it was created for
* Updated method calls to ''assignment.find_current_stage'' to their new form ''assignment.current_stage''
* Removed the review_bid model method ''self.reviewer_self_topic'' and replaced its usage with the already-existing ''SignedUpTeam.topic_id'' method
* Removed DRY-violating code used to calculate the current assignment phase which was never utilized by the controller or views
* Moved several view files from the ''review_bids'' view directory into the ''sign_up_sheet'' view directory. Partials which were duplicates in the two directories (''_all_actions.html.erb'' and ''_topic_names.html.erb'' were no longer necessary and deleted
* Refactored the naming of the variable that represents the id of the reviewer object from 'reviewer' to 'reviewer_id' for clarity
* Removed the unnecessary 'create' function within the review_bids_controller
* Removed the sparse testing files under the 'test' directory in keeping with typical expertiza structure
* Removed a test case from the unit tests which relied upon the removed 'create' method of the review_bids_controller

==Initial Implementation==

A considerable amount of time and effort was necessary to review, understand, and re-implement the previous implementation of the Review Bidding feature. While the feature was only created a year ago, several underlying functions and methods have since been refactored, and so needed to be changed in the implementation. Additionally, certain parts of how the implementation was designed did not seem to be properly implemented, and so had to be revised.

===Code Transfer===
First, all files that were added in the original pull request were copied and moved to the new version of the expertiza beta branch. This includes the following files:

*'''controllers/review_bids_controller.rb''': the primary controller that handles calls to review bids functionality from views, managing them itself or directing them to the appropriate model
*'''helpers/review_bids_helper.rb''': contains helper methods for rendering the rows of topics available to choose a review for
*'''models/review_bid.rb''': contains the methods that each individual review bid knows and manages, including assigning a review to a student, getting info on the topic it applies to, etc.
*'''views/review_bids/_all_actions.html.erb''': a view partial used to render the actions available to a student on the review bidding view. Nearly identical to the _all_actions file for topic bidding.
*'''views/review_bids/_table_header.html.erb''': a view partial used to render the head of the table for review bids. Though structurally similar to that of topics reviews, contains significantly less information, as less needs to be known by a student for particular topics to review.
*'''views/review_bids/_table_line.html.erb''': a view partial used to render individual rows of the review bids table. Like the table header, is structurally similar to that of topic bidding, but functions differently.
*'''views/review_bids/show.html.erb''': the view which displays all the available topics to bid on reviewing. Uses the aforementioned partials to render this bidding table.
*'''views/review_bids/others_work.html.erb''': displays the available review work which has the user has bid for.
*'''db/migrate/20201107203204_create_review_bids.rb''': the migration which creates the review bid objects within the expertiza database.
*'''spec/controllers/review_bids_controller_spec.rb''': the unit tests for the review bids controller
*'''spec/models/review_bid_spec.rb''': the unit tests for the review bids model
*'''test/controllers/review_bids_controller_test.rb''': a more elaborate feature test for the review bids functionality
*'''test/fixtures/review_bids.yml''': fixtures that are utilized for the review_bids_controller_test
*'''test/models/review_bid_test.rb''': A more elaborate feature test of the review bid model.

In addition to adding the previous files, the following files were also added to existing pages as modifications:

*'''controllers/student_review_controller.rb''': the student review controller's "list" method now redirects to the review_bid_controller's "index" method if the assignment in question has a review_choosing_algorithm set to "Bidding" (the option an instructor sets when they wish to allow students to bid on reviews).
*'''student_task/view.html.erb''': Instead of simply linking directly to the student_review controller's "list" method, the view instead checks if the assignment this view pertains to has their review_choosing_algorithm set to Bidding. If it does, and the assignment allows students to choose the topic to review (meaning the algorithm to assign bids has not yet been run), they are redirected to the 'show' method of the review_bids_controller. Otherwise, they are redirected to the 'index' method (showing them the info on their currently available topics to review).
*'''config/webservices.yml''': added a new url for the website that is called to run the review bidding algorithm
*'''config/routes.rb''': routes added for review_bids.
*'''views/assignments/edit/_topics.html.erb''': Added a new option box which allows an instructor to choose to allow bidding to occur using the review bidding algorithm, rather than the default 'Simple Choose.' Once this option is selected, it also offers a button that, when clicked, runs the review bidding algorithm on all currently bid upon reviews.

===Website Reimplementation===
In the previous implementation, the algorithm which handles the assigning of reviews was hosted on an external website. This was done for the sake of improved performance. Thus, to get the feature in working condition, we pulled the code necessary to run the algorithm and hosted it on our own website (linked at the end of this documentation). We also made a note in the pull request that the place where this website is referenced (in the "assign_reviews_algorithm" method of the review_bids_controller) should be replaced by a proper external web tool, as is done for similar web tools in other parts of expertiza.

===Modifications to Methods===
Certain methods utilized in the original version of the webpages were changed or renamed in later versions, and so, needed to be modified. The primary example of this is the method that assignments use to determine what stage of assignment completion they are in. The previous year's implementation called this method as "assignment.find_current_stage(topic_id)," but this failed when trying to execute. After comparing how this page worked to how normal review pages work, we determined the new call to this method is "assignment.current_stage(topic_id)." All calls to this method were replaced in all files.

Another change to methods was the removal of a method from the review_bids_controller named 'create'. This method primarily served as a redirect for an aspect of the feature that was unnecessary and so removed (see the following section), that being the button to show another review. With this feature removed, this method was no longer used, and so it was removed from the controller.

===Errors Corrected===
A notable period of time was also spent on determining the purpose of a particular local area that caused issues when trying to view the bids, that being a variable named "@number_of_reviews_to_show". This variable apparently kept track of how many reviews a student needed to be shown, so that they could request another. The issue was that this was created as a local variable within the review_bids_controller, and therefore would never work outside of a manual testing environment, as the variable would need to be different from every student and every assignment. After some investigation, we realized that all this variable did was limit the number of already-assigned reviews the student was able to see (as the review bidding algorithm already assigned all possible reviews). Thus, rather than try to recreate this functionality, we simply removed it.

This means that, when the review bidding algorithm runs, a student is able to see all of the reviews assigned to them, up to the maximum number of reviews for the assignment, and therefore have no reason to ask for any more. It may be the case that this feature could be improved by adding a possibility for students to request any reviews not assigned to any other student, or that still have available slots, using the "normal" review assignment scheme. For the time being, we did not consider this part of the baseline necessary functionality, and so did not implement it.

==Final Implementation==

===DRY Principle and Code Duplication Refactoring===
Per comments and feedbacks on the previous implementation [https://github.com/expertiza/expertiza/pull/1822 E2085], there were 3 sections of the code that potentially were DRY violations. For each of those violations, we have had made appropriate and provided the justification on the results.

====review_bids_controller.rb====
For the refactoring of review bids controller.eb, the original issue is the DRY principle violation on the internal logic of finding the phase of assignments.
The original implementation to be refactored and optimized are shown as follow.
<pre>
# Finding the current phase that we are in
due_dates = AssignmentDueDate.where(parent_id: @assignment.id)
@very_last_due_date = AssignmentDueDate.where(parent_id: @assignment.id).order("due_at DESC").limit(1)
next_due_date = @very_last_due_date[0]
for due_date in due_dates
if due_date.due_at > Time.now
next_due_date = due_date if due_date.due_at < next_due_date.due_at
end
end
@review_phase = next_due_date.deadline_type_id
</pre>
Through some discovery process, we have found some related logics that is already implemented from the system which enable the refactoring and reuse of the method. Some code snippets are demonstrated as follow.
<pre>
def self.current_due_date(due_dates)
#Get the current due date from list of due dates
due_dates.each do |due_date|
if due_date.due_at > Time.now
current_due_date = due_date
return current_due_date
end
end
#in case current due date not found
return nil
end
</pre>
<pre>
def self.teammate_review_allowed(student)
# time when teammate review is allowed
due_date = current_due_date(student.assignment.due_dates)
student.assignment.find_current_stage == 'Finished' ||
due_date &&
(due_date.teammate_review_allowed_id == 3 ||
due_date.teammate_review_allowed_id == 2) # late(2) or yes(3)
end
</pre>
As we can observed above, both methods of <code>current_due_date</code> and <code>teammate_review_allowed</code> provides the capability of getting the <code>review_phase</code> without the repetition. Additionally, we can see that those existing implementation still have some difference in implementation and purpose to be fully reusable by the logic that find current phase of a topic because the methods provided are helper methods for broader abstraction.

Furthermore, the section that we attempted to refactoring out have a similar logic that is used to get the current stage as shown below.
<pre>
def find_current_stage(topic_id = nil)
next_due_date = DueDate.get_next_due_date(self.id, topic_id)
return 'Finished' if next_due_date.nil?
next_due_date
end
</pre>
However, it is worth pointing out that the method is not fully substitutable, particularly the internal functionality that finds the <code>@very_last_due_date</code>. A similar logic does exist that is meant to use to acquire current due date and assist with finding current stage.
<pre>
def current_stage_name(topic_id = nil)
if self.staggered_deadline?
return (topic_id.nil? ? 'Unknown' : current_stage(topic_id))
end

due_date = find_current_stage(topic_id)
unless due_date == 'Finished' || due_date.nil? || due_date.deadline_name.nil?
return due_date.deadline_name
end
current_stage(topic_id)
end
</pre>
With all the findings, we have come to conclusion that the portion of the code to be dry is in similar, but required full restructure, form of existing methods. The internal logic flow itself is not fully matching the reusable methods consists across the controllers. Because of the clarity of limited reusability of existing methods, we have come to conclusion that the portions to be refactored are not duplication of any existing methods. Additionally, after a more detailed discovery, it has come to realization that the code portion of getting the current phase of the next due date was, in fact, meant to find the last due date. After further analysis of the purpose of the code, the portion has been deleted from <code>review_bids_controller.rb</code> because there was no usage observed with the result of the execution.

====review_bid.rb====
Per the previous iteration's feedback, the review_bid.rb model file presented some DRY violations. These violations are highlighted in the method below:
<pre>
def self.reviewer_self_topic(reviewer_id)
participant = AssignmentParticipant.find_by(id: reviewer_id)
team_id = participant.team.try(:id)
self_topic = SignedUpTeam.where(team_id: team_id).pluck(:topic_id).first

return self_topic
end
</pre>
This method returns the topic corresponding to the reviewer that is passed into the method. Line 1 gets the participant from the reviewer_id parameter. Line 2 extracts the team that the participant is in. Line 3 will get the topic which is assigned to the participant's team. This topic is returned. Our goal while refactoring was to find a similar method that can replace this method.

While searching through the project for similar methods which perform the same action, we came across this method in the signed_up_team.rb model:
<pre>
def self.topic_id(assignment_id, user_id)
# team_id variable represents the team_id for this user in this assignment
team_id = TeamsUser.team_id(assignment_id, user_id)
topic_id_by_team_id(team_id) if team_id
end
</pre>
This method gets the team_id from the user_id and isolates the topic from said team_id. This method performs a similar function to reviewer_self_topic. We decided to replace all uses of reviewer_self_topic with topic_id.
 

====Refactoring Views (_all_actions.html.erb and _topic_names.html.erb)====
The last element of the prior implementation that caused DRY violations had to do with its implementation of views. Two primary views are necessary for the review bids functionality: one which shows a table where students can bid on which topic they wish to review (rendered by a file called 'show.html.erb'), and a list of assigned reviews which a student must select from (rendered by a file called 'others_work.html.erb').

In order to render the elements in 'show.html.erb' in particular, a number of partials must be utilized:
*a partial for the table's header (_table_header.html.erb)
*a partial for rendering each of the table's rows, describing a topic to bid on (_table_line.html.erb)
*a partial to contain information related to what actions are available for a student to do for a review, like edit or view (_all_actions.html.erb)
*a partial for finding and rendering the name of each topic one must review (_topic_names.html.erb)

Both of these views and all four of these partials were contained within the "review_bids" directory within the views directory. The DRY violation incurred by these views has to do with these partials: in particular, '_topic_names.html.erb' and '_all_actions.html.erb' are near identical code copies to those within the sign_up_sheet views, used for rendering and displaying information for topic bidding and normal reviews.

In order to DRY out this portion of the code, we decided that having a separate set of views for the results related to review bids were unnecessary. Within expertiza's structure, most of the views related to students choosing which topics or reviews to work on are all contained within the sign_up_sheet views. The review_vids views were copying the partials within the sign_up_sheet views because they were using the same fundamental components. Thus, we thought the best solution would be to move the two primary views for review_bids ('show.html.erb' and 'others_work.html.erb') into the sign_up_sheet views, and leave behind all redundant partials. In doing so, the redundancy of '_all_actions.html.erb' and '_topic_names.html.erb' were eliminated.

It could be argued that this implementation could be further DRY-ed by leaving behind the other two partials, '_table_header.html.erb' and '_table_line.html.erb', as they are structurally similar to those used for topic bidding. However, this is not actually permissible, as the header and lines for topics when bidding on them for reviews are significantly different than when bidding on them for assignments. More information needs to be available when bidding on topics for an assignment, such as the number of slots and number on the waitlist, but such an idea is not applicable for bids on reviews. While it would be possible to have a single file which produces one view or another depending upon if it's bidding for topics or reviews, we believed this violated the Expert Pattern, by attempting to create a single view for two distinct purposes. Instead, we thought it better to simply have two separate partials, each which precisely gave the desired functionality. Thus, the other two partials were brought over, being renamed '_review_bids_table_header.html.erb' and '_review_bids_table_line.html.erb'.

==Tests==

Throughout the refactoring and merging of [https://github.com/expertiza/expertiza/pull/1822 E2085] to [https://github.com/expertiza/expertiza/pull/2150 E2151], there were some updates that has been made to the test suits to maintain corrosponding testing automation.

===Video Demonstration===
Visit [https://www.youtube.com/watch?v=wxXALQNWFcc Feature Demo] to see a video demonstrating the functionality of the review bidding feature that is working properly after our DRY principle optimization.

===RSpec Testing===
One of the primary design goals when doing this refactoring was to ensure that all test cases still passed after all changes were made.

There was a single change that needed to be made to the RSpec files for the review bidding algorithm, which was the removal of a test relying on the 'create' method of the review_bids_controller. As previously mentioned, this function was removed because it had become obsolete. Thus, the test case for this function (which simply checked if it redirected to the right page) was no longer necessary, and thus was deleted.

Otherwise, the test cases for both the review_bids_controller and the review_bids model were manually run and found to have no errors, as shown below.

[[File:Bidding_test_results.png|800px]]

For details on what these specific tests were, please refer to the previous work referenced before.

==Conclusions and Future Work==
While the current state of the review bidding code is operational, and is certainly more DRY and stable than the previous incarnation of this work, there are still several issues with the current state of this feature that make it difficult to recommend an immediate merge into expertiza.

===Enhancement of Algorithm Service Stability===

Across the process of refining the refactored front end page view along with more robust edge cases testing towards the algorithm service, we have discovered an underlying issue that could impact the availability of the review bidding algorithm service. Through cause isolation, we have observed a high correlation to flask application crash with the edge case scenario input where the algorithm service is trying to match a student who was not signed up with any topic with other students.

Because the scope of our project was focused on refactoring, we did not have the time or prior warning to need to address these edge cases. However, we will provide the error log for the reference of any future teams who may work on the stability improvement of the algorithm service.

<pre>
"POST /match_topics HTTP/1.1" 500 -
{'tid': [5133, 5134, 5135, None], 'users': {'45514': {'otid': 5135, 'tid': [5133, 5134], 'time': ['2021-11-30T17:57:20.000-05:00', '2021-11-30T17:57:24.000-05:00'], 'priority': [2, 1]}, '45513': {'otid': 5134, 'tid': [5133, 5135], 'time': ['2021-11-30T17:57:41.000-05:00', '2021-11-30T17:57:43.000-05:00'], 'priority': [2, 1]}, '45512': {'otid': 5133, 'tid': [5134, 5135], 'time': ['2021-11-30T17:57:58.000-05:00', '2021-11-30T17:58:00.000-05:00'], 'priority': [2, 1]}, '45515': {'otid': None, 'tid': [], 'time': [0], 'priority': [3, 1, 4, 2]}}, 'max_accepted_proposals': 3}
[2021-11-28 18:10:40,576] ERROR in app: Exception on /match_topics [POST]
Traceback (most recent call last):
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/usr/.local/lib/python3.5/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/usr/IntelligentAssignment/app/app.py", line 19, in topic_matching
return jsonify(assigned_topics_model.get_student_topic_matches())
File "/home/usr/IntelligentAssignment/app/topics_matcher.py", line 57, in get_student_topic_matches
student.accept_proposal()
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in accept_proposal
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in <lambda>
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
ValueError: None is not in list
</pre>

As we can see from the additional information provided above, the error is due to an unhandled edge case of attempting to access missing value. Our further observation also concluded that such missing value is due to the fact that one or more to-be-matched students did not have any topic signed up at the time of algorithm execution. To address the issue, a general enhancement can be implemented on the algorithm service side with Flask to increase the crash recovery stability. Further refinement of interaction flow with user in the front end can also reduce the possibility of algorithm service error.

Additional to the unhandled cases from the algorithm service, we also noticed an inconsistency of the system due to the performance limitation of the Heroku deployment.

[[File:Out_of_scope_External-Issue_E2151.gif|frameless|x1000px]]

As we can see from the above recorded screen capture animation, with the exact same input, the output and algorithm service result are unstable. Because of the performance limitation of the Heroku, sometime the Flask app that deployed were overloaded. Furthermore, we have also attempted to deploy on VCL, however, we notice the limitation of VCL deployment that will become offline once the VCL reservation expired. After all, because of the focus of our tasks is on the DRY principal optimization, the external algorithm service can be improved in the future. To access and deploy the algorithm on local environment or another environment, setup guide and resource are available on [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment] repository.

===Lack of Comprehensive Testing===

A notable contradiction that was noticed when working with this project was that the previously-mentioned failed edge cases existed, yet all tests for this feature passed. This is illustrative of the fact that this feature requires better and more exhaustive testing. With the scope limitation of our tasks that focusing on the DRY principal enhancement of the Expertiza internally, the external algorithm service module was not within our focus. However, with our experience of deployment with the algorithm service, we can offer some suggestions for the future team who will address those edge cases of the algorithm service.

When dealing with an issue with so many factors as how students bid on reviews, an extensive suite of edge cases should be considered. Any future project group working on this function should consider the following edge cases to cover:
* Does a student who does not bid on any reviews still get reviews?
* Does a student who did not submit anything for their assignment still have a review to bid on?
* What if there are less assignments to have reviews for than the minimum number required for all students to have the minimum number of reviews?

===Conclusion===
While we believe the current state of this feature is greatly improved, with many of its DRY violations removed and several noteworthy bugs corrected, we believe the feature of allowing students to bid on what to review still needs further development before it is deployed. While the basic functionality is working under ideal situations, the issues mentioned above should be addressed before this feature is put in the hands of actual instructors, as if this feature fails, it could ruin the entire reviewing process for an assignment.

Still, we think there is notable promise to this feature, and it would be a very engaging and interesting feature if deployed successfully, so we believe develop on it should continue until it is ready for deployment.

==Github and Related Link==
Github repository corresponding to this task is publicly available at [https://github.com/WeiRui-Wang/expertiza E2151 GitHub].

Github pull request and file changes can be viewed at [https://github.com/expertiza/expertiza/pull/2150/files E2151 Pull Request].

Github of source code of externally hosted bidding algorithm service deployment manual and files can be access at [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment].

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: Geoff Garrido, Ayush Luthra, and WeiRui Wang. Our project mentor was Luis Delossantos (lgdeloss@ncsu.edu)

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-30T01:22:23Z

Aluthra: /* review_bid.rb */

==Topic Overview==
===Statement of Problem===
Reviewing is an important stage of learning and improving, both as a reviewer and a reviewee, and so allowing reviewers more freedom and choice in what they review would result in better reviews and more engaged reviewers. However, currently the only way to choose a topic to review is a first-come-first-served basis, penalizing students who do not sign up quickly by not allowing them to review the topics they would be most interesting in engaging with.

A better alternative would be to implement a bidding system, similar to that used for topics, which would allow students to establish preferences for which topics they would most like to review. A good bidding algorithm implementing with this idea would ensure that all students (who care enough to bid) receive a topic to review that they actively want.

A prior project group created a prototype for this function, allowing students to bid on what topics they wanted to review. While the functionality was as desired, the feature was not able to be merged due to having too many DRY violations. Thus, the goal of this topic was to bring this prior implementation into the current version of the expertiza beta branch, and remove these DRY violations, so that it may be merged.

As development continued, it became clear this prototype also had several bugs and issues that made it unable to work under several conditions. Thus, a secondary goal was to fix any issues which existed such that the feature would generally work with the functionality the prior implementation demonstrated.

===Prior Work===
The prior work which is the basis for this topic is linked here [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review is linked here]. This work was done in Fall of 2020, and as previously mentioned, achieved much of the desired functionality for this task. This prior work will be referenced several times within this documentation, as it has a more comprehensive explanation of the underlying design principles behind the views, controller, model, algorithm, and RSpec tests implemented. These details will only be briefly summarized here.

As previously noted, this prior implementation was rejected due to DRY violations, and so correcting these violations--and explaining how those corrections were done--will be the major focus of this project documentation.

==Explanation of Feature==

===Existing Bidding Functionality for Topics===
Currently in expertiza, students may bid on which topics they would like to work on. An image showcasing this functionality is given below:

[[File:expected_review_UI.PNG]]

Students may choose a number of topics to bid on, in order of priority, and once the deadline is reached, all students are given a topic.

Behind the scenes, this works by utilizing a modified version of the Gale-Shapely Algorithm. This algorithm seeks to find a 'stable' matching of each student to each topic, such that in the end, all students have a topic they wanted (at least to some degree), and that the preferences of students is respected as well as possible. To help break situations where there are ties in preferences, each topic assigns a Priority to each student. This Priority is determined by A. whether the student is part of a group bidding for this topic (groups getting higher priority), and B. the time at which the bid for the topic was made (earlier bidders are given Priority). Once this has been determined and the deadline for bidding has ended, the following steps occur:
# Each student is matched with each topic they most prefer.
# If the number of students assigned to a topic is above the maximum allowed for that topic, the students with the highest Priority are assigned the topic
# All maximally filled topics are removed from the pool of available topics
# All unmatched students begin this cycle again, now choosing the most preferred topic remaining
# Repeat until all students are assigned a topic
# If any matches are 'unstable' (two students would have their preferences better met if they were swapped), perform any swaps until matches are as stable as possible.

This algorithm has a time complexity of O(n^2). For more details on this algorithm and its implementation, refer to the previous year's work, where this algorithm was implemented.

===Desired Functionality for Reviews===

The desired feature for reviews should have an almost identical structure to this, with reviews instead of topics. Students should be shown a list of the topics that were worked on, and may bid on which of those topics they would like to review the most. The Priority system could be handled in the same way, with the only caveat being that a student should not be able to review their own work.

In terms of what this feature looks like, it can be enabled as an option when editing/creating a new assignment, under the "Topics" section, using interface drop down show below.

[[File:Bidding_interface.png|center|800px]]

Once this option is enabled, during the review phase of an assignment, students will be able to access a new view listing all the topics of all the submitted assignments (except for their own). They will then be able to bid on as many topics as they like, assigning them a preference, in a manner similar to the interface for topic bidding. The table that allows for this drag-and-drop priorities is shown below.

[[File:Bidding_options.png|center|900px]]

Whenever the instructor decides they would like to close the window for review bidding, they may return to the options for the given assignment and navigate to "Topics", where they will see a button marked "Run Review Algorithm". This will make a call to the external webservice to determine which students are assigned what reviews, and will then assign those reviews to all students. Once this button is pressed, it locks out the ability for students to perform any more bidding, and cannot be pressed again. The button allowing this option is shown below.

[[File:Bidding_run_algorithm.png|center|900px]]

After this point, students may navigate to the "Others Work" option under their assignment and see all of their assigned reviews. They will see up to the maximum number of allowed reviews, but will be informed of how many they are required to do (for example, a student may see 4 reviews, but will be told they are only required to complete 2).

===Use Case Diagram===
The following Use Case Diagram demonstrates the primary functionality explained in the previous section for different types of users.

[[File:Use_Case_Diagram_Rev1_E2151.jpg|center]]

==Plan and List of Work Done==

===General Design Goals===

* Move all prior work for this topic into a new version of the expertiza beta branch, without loss in functionality.
* Ensure all prior/expected functionality is working correctly, and if not, make any repairs or changes necessary.
* DRY out prior implementation's controller and model methods which have functionality that already exists within the expertiza system.
* Remove repetitive views and partials that exist by merging views for review bids into the set of views for sign up sheets, from which it uses partials.
* Ensure all present test cases still work properly after making the above changes.

===Specific Tasks Completed===

* Transferred over all code files relevant to the review bidding feature
* Re-deployed the external website which handles the review bidding algorithm
* Added that website's url to the webservices url's within expertiza
* Enhanced the stability of the webservice for the bidding algorithm by testing with edge cases and optimize the endpoint by restoring the original access endpoint reference
* Analyzed the necessity of the phase finding related logic code section
* Optimized phase finding related logic by removing unused code segment
* Removed an instance variable within the ''review_bids_controller'' which would have made any practical deployment fail, removing the unnecessary functionality it was created for
* Updated method calls to ''assignment.find_current_stage'' to their new form ''assignment.current_stage''
* Removed the review_bid model method ''self.reviewer_self_topic'' and replaced its usage with the already-existing ''SignedUpTeam.topic_id'' method
* Removed DRY-violating code used to calculate the current assignment phase which was never utilized by the controller or views
* Moved several view files from the ''review_bids'' view directory into the ''sign_up_sheet'' view directory. Partials which were duplicates in the two directories (''_all_actions.html.erb'' and ''_topic_names.html.erb'' were no longer necessary and deleted
* Refactored the naming of the variable that represents the id of the reviewer object from 'reviewer' to 'reviewer_id' for clarity
* Removed the unnecessary 'create' function within the review_bids_controller
* Removed the sparse testing files under the 'test' directory in keeping with typical expertiza structure
* Removed a test case from the unit tests which relied upon the removed 'create' method of the review_bids_controller

==Initial Implementation==

A considerable amount of time and effort was necessary to review, understand, and re-implement the previous implementation of the Review Bidding feature. While the feature was only created a year ago, several underlying functions and methods have since been refactored, and so needed to be changed in the implementation. Additionally, certain parts of how the implementation was designed did not seem to be properly implemented, and so had to be revised.

===Code Transfer===
First, all files that were added in the original pull request were copied and moved to the new version of the expertiza beta branch. This includes the following files:

*'''controllers/review_bids_controller.rb''': the primary controller that handles calls to review bids functionality from views, managing them itself or directing them to the appropriate model
*'''helpers/review_bids_helper.rb''': contains helper methods for rendering the rows of topics available to choose a review for
*'''models/review_bid.rb''': contains the methods that each individual review bid knows and manages, including assigning a review to a student, getting info on the topic it applies to, etc.
*'''views/review_bids/_all_actions.html.erb''': a view partial used to render the actions available to a student on the review bidding view. Nearly identical to the _all_actions file for topic bidding.
*'''views/review_bids/_table_header.html.erb''': a view partial used to render the head of the table for review bids. Though structurally similar to that of topics reviews, contains significantly less information, as less needs to be known by a student for particular topics to review.
*'''views/review_bids/_table_line.html.erb''': a view partial used to render individual rows of the review bids table. Like the table header, is structurally similar to that of topic bidding, but functions differently.
*'''views/review_bids/show.html.erb''': the view which displays all the available topics to bid on reviewing. Uses the aforementioned partials to render this bidding table.
*'''views/review_bids/others_work.html.erb''': displays the available review work which has the user has bid for.
*'''db/migrate/20201107203204_create_review_bids.rb''': the migration which creates the review bid objects within the expertiza database.
*'''spec/controllers/review_bids_controller_spec.rb''': the unit tests for the review bids controller
*'''spec/models/review_bid_spec.rb''': the unit tests for the review bids model
*'''test/controllers/review_bids_controller_test.rb''': a more elaborate feature test for the review bids functionality
*'''test/fixtures/review_bids.yml''': fixtures that are utilized for the review_bids_controller_test
*'''test/models/review_bid_test.rb''': A more elaborate feature test of the review bid model.

In addition to adding the previous files, the following files were also added to existing pages as modifications:

*'''controllers/student_review_controller.rb''': the student review controller's "list" method now redirects to the review_bid_controller's "index" method if the assignment in question has a review_choosing_algorithm set to "Bidding" (the option an instructor sets when they wish to allow students to bid on reviews).
*'''student_task/view.html.erb''': Instead of simply linking directly to the student_review controller's "list" method, the view instead checks if the assignment this view pertains to has their review_choosing_algorithm set to Bidding. If it does, and the assignment allows students to choose the topic to review (meaning the algorithm to assign bids has not yet been run), they are redirected to the 'show' method of the review_bids_controller. Otherwise, they are redirected to the 'index' method (showing them the info on their currently available topics to review).
*'''config/webservices.yml''': added a new url for the website that is called to run the review bidding algorithm
*'''config/routes.rb''': routes added for review_bids.
*'''views/assignments/edit/_topics.html.erb''': Added a new option box which allows an instructor to choose to allow bidding to occur using the review bidding algorithm, rather than the default 'Simple Choose.' Once this option is selected, it also offers a button that, when clicked, runs the review bidding algorithm on all currently bid upon reviews.

===Website Reimplementation===
In the previous implementation, the algorithm which handles the assigning of reviews was hosted on an external website. This was done for the sake of improved performance. Thus, to get the feature in working condition, we pulled the code necessary to run the algorithm and hosted it on our own website (linked at the end of this documentation). We also made a note in the pull request that the place where this website is referenced (in the "assign_reviews_algorithm" method of the review_bids_controller) should be replaced by a proper external web tool, as is done for similar web tools in other parts of expertiza.

===Modifications to Methods===
Certain methods utilized in the original version of the webpages were changed or renamed in later versions, and so, needed to be modified. The primary example of this is the method that assignments use to determine what stage of assignment completion they are in. The previous year's implementation called this method as "assignment.find_current_stage(topic_id)," but this failed when trying to execute. After comparing how this page worked to how normal review pages work, we determined the new call to this method is "assignment.current_stage(topic_id)." All calls to this method were replaced in all files.

Another change to methods was the removal of a method from the review_bids_controller named 'create'. This method primarily served as a redirect for an aspect of the feature that was unnecessary and so removed (see the following section), that being the button to show another review. With this feature removed, this method was no longer used, and so it was removed from the controller.

===Errors Corrected===
A notable period of time was also spent on determining the purpose of a particular local area that caused issues when trying to view the bids, that being a variable named "@number_of_reviews_to_show". This variable apparently kept track of how many reviews a student needed to be shown, so that they could request another. The issue was that this was created as a local variable within the review_bids_controller, and therefore would never work outside of a manual testing environment, as the variable would need to be different from every student and every assignment. After some investigation, we realized that all this variable did was limit the number of already-assigned reviews the student was able to see (as the review bidding algorithm already assigned all possible reviews). Thus, rather than try to recreate this functionality, we simply removed it.

This means that, when the review bidding algorithm runs, a student is able to see all of the reviews assigned to them, up to the maximum number of reviews for the assignment, and therefore have no reason to ask for any more. It may be the case that this feature could be improved by adding a possibility for students to request any reviews not assigned to any other student, or that still have available slots, using the "normal" review assignment scheme. For the time being, we did not consider this part of the baseline necessary functionality, and so did not implement it.

==Refinement Implementation==

===DRY Principle and Code Duplication Refactoring===
Per comments and feedbacks on the previous implementation [https://github.com/expertiza/expertiza/pull/1822 E2085], there were 3 sections of the code that potentially were DRY violations. For each of those violations, we have had made appropriate and provided the justification on the results.

====review_bids_controller.rb====
For the refactoring of review bids controller.eb, the original issue is the DRY principle violation on the internal logic of finding the phase of assignments.
The original implementation to be refactored and optimized are shown as follow.
<pre>
# Finding the current phase that we are in
due_dates = AssignmentDueDate.where(parent_id: @assignment.id)
@very_last_due_date = AssignmentDueDate.where(parent_id: @assignment.id).order("due_at DESC").limit(1)
next_due_date = @very_last_due_date[0]
for due_date in due_dates
if due_date.due_at > Time.now
next_due_date = due_date if due_date.due_at < next_due_date.due_at
end
end
@review_phase = next_due_date.deadline_type_id
</pre>
Through some discovery process, we have found some related logics that is already implemented from the system which enable the refactoring and reuse of the method. Some code snippets are demonstrated as follow.
<pre>
def self.current_due_date(due_dates)
#Get the current due date from list of due dates
due_dates.each do |due_date|
if due_date.due_at > Time.now
current_due_date = due_date
return current_due_date
end
end
#in case current due date not found
return nil
end
</pre>
<pre>
def self.teammate_review_allowed(student)
# time when teammate review is allowed
due_date = current_due_date(student.assignment.due_dates)
student.assignment.find_current_stage == 'Finished' ||
due_date &&
(due_date.teammate_review_allowed_id == 3 ||
due_date.teammate_review_allowed_id == 2) # late(2) or yes(3)
end
</pre>
As we can observed above, both methods of <code>current_due_date</code> and <code>teammate_review_allowed</code> provides the capability of getting the <code>review_phase</code> without the repetition. Additionally, we can see that those existing implementation still have some difference in implementation and purpose to be fully reusable by the logic that find current phase of a topic because the methods provided are helper methods for broader abstraction.

Furthermore, the section that we attempted to refactoring out have a similar logic that is used to get the current stage as shown below.
<pre>
def find_current_stage(topic_id = nil)
next_due_date = DueDate.get_next_due_date(self.id, topic_id)
return 'Finished' if next_due_date.nil?
next_due_date
end
</pre>
However, it is worth pointing out that the method is not fully substitutable, particularly the internal functionality that finds the <code>@very_last_due_date</code>. A similar logic does exist that is meant to use to acquire current due date and assist with finding current stage.
<pre>
def current_stage_name(topic_id = nil)
if self.staggered_deadline?
return (topic_id.nil? ? 'Unknown' : current_stage(topic_id))
end

due_date = find_current_stage(topic_id)
unless due_date == 'Finished' || due_date.nil? || due_date.deadline_name.nil?
return due_date.deadline_name
end
current_stage(topic_id)
end
</pre>
With all the findings, we have come to conclusion that the portion of the code to be dry is in similar, but required full restructure, form of existing methods. The internal logic flow itself is not fully matching the reusable methods consists across the controllers. Because of the clarity of limited reusability of existing methods, we have come to conclusion that the portions to be refactored are not duplication of any existing methods. Additionally, after a more detailed discovery, it has come to realization that the code portion of getting the current phase of the next due date was, in fact, meant to find the last due date. After further analysis of the purpose of the code, the portion has been deleted from <code>review_bids_controller.rb</code> because there was no usage observed with the result of the execution.

====review_bid.rb====
Per the previous iteration's feedback, the review_bid.rb model file presented some DRY violations. These violations are highlighted in the method below:
<pre>
# method to get reviewers own project topic
# return self_topic
def self.reviewer_self_topic(reviewer_id)
participant = AssignmentParticipant.find_by(id: reviewer_id)
team_id = participant.team.try(:id)
self_topic = SignedUpTeam.where(team_id: team_id).pluck(:topic_id).first

return self_topic
end
</pre>
 

====Refactoring Views (_all_actions.html.erb and _topic_names.html.erb)====
The last element of the prior implementation that caused DRY violations had to do with its implementation of views. Two primary views are necessary for the review bids functionality: one which shows a table where students can bid on which topic they wish to review (rendered by a file called 'show.html.erb'), and a list of assigned reviews which a student must select from (rendered by a file called 'others_work.html.erb').

In order to render the elements in 'show.html.erb' in particular, a number of partials must be utilized:
*a partial for the table's header (_table_header.html.erb)
*a partial for rendering each of the table's rows, describing a topic to bid on (_table_line.html.erb)
*a partial to contain information related to what actions are available for a student to do for a review, like edit or view (_all_actions.html.erb)
*a partial for finding and rendering the name of each topic one must review (_topic_names.html.erb)

Both of these views and all four of these partials were contained within the "review_bids" directory within the views directory. The DRY violation incurred by these views has to do with these partials: in particular, '_topic_names.html.erb' and '_all_actions.html.erb' are near identical code copies to those within the sign_up_sheet views, used for rendering and displaying information for topic bidding and normal reviews.

In order to DRY out this portion of the code, we decided that having a separate set of views for the results related to review bids were unnecessary. Within expertiza's structure, most of the views related to students choosing which topics or reviews to work on are all contained within the sign_up_sheet views. The review_vids views were copying the partials within the sign_up_sheet views because they were using the same fundamental components. Thus, we thought the best solution would be to move the two primary views for review_bids ('show.html.erb' and 'others_work.html.erb') into the sign_up_sheet views, and leave behind all redundant partials. In doing so, the redundancy of '_all_actions.html.erb' and '_topic_names.html.erb' were eliminated.

It could be argued that this implementation could be further DRY-ed by leaving behind the other two partials, '_table_header.html.erb' and '_table_line.html.erb', as they are structurally similar to those used for topic bidding. However, this is not actually permissible, as the header and lines for topics when bidding on them for reviews are significantly different than when bidding on them for assignments. More information needs to be available when bidding on topics for an assignment, such as the number of slots and number on the waitlist, but such an idea is not applicable for bids on reviews. While it would be possible to have a single file which produces one view or another depending upon if it's bidding for topics or reviews, we believed this violated the Expert Pattern, by attempting to create a single view for two distinct purposes. Instead, we thought it better to simply have two separate partials, each which precisely gave the desired functionality. Thus, the other two partials were brought over, being renamed '_review_bids_table_header.html.erb' and '_review_bids_table_line.html.erb'.

==Tests==

Throughout the refactoring and merging of [https://github.com/expertiza/expertiza/pull/1822 E2085] to [https://github.com/expertiza/expertiza/pull/2150 E2151], there were some updates that has been made to the test suits to maintain corrosponding testing automation.

===Video Demonstration===
[https://www.youtube.com/watch?v=wxXALQNWFcc Click this link] to see a video demonstrating the functionality of the review bidding feature.

===RSpec Testing===
One of the primary design goals when doing this refactoring was to ensure that all test cases still passed after all changes were made.

There was a single change that needed to be made to the RSpec files for the review bidding algorithm, which was the removal of a test relying on the 'create' method of the review_bids_controller. As previously mentioned, this function was removed because it had become obsolete. Thus, the test case for this function (which simply checked if it redirected to the right page) was no longer necessary, and thus was deleted.

Otherwise, the test cases for both the review_bids_controller and the review_bids model were manually run and found to have no errors, as shown below.

[[File:Bidding_test_results.png|800px]]

For details on what these specific tests were, please refer to the previous work referenced before.

==Conclusions and Future Work==
While the current state of the review bidding code is operational, and is certainly more DRY and stable than the previous incarnation of this work, there are still several issues with the current state of this feature that make it difficult to recommend an immediate merge into expertiza.

===Enhancement of Algorithm Service Stability===

Across the process of refining the refactored front end page view along with more robust edge cases testing towards the algorithm service, we have discovered an underlying issue that could impact the availability of the review bidding algorithm service. Through cause isolation, we have observed a high correlation to flask application crash with the edge case scenario input where the algorithm service is trying to match a student who was not signed up with any topic with other students.

Because the scope of our project was focused on refactoring, we did not have the time or prior warning to need to address these edge cases. However, we will provide the error log for the reference of any future teams who may work on the stability improvement of the algorithm service.

<pre>
"POST /match_topics HTTP/1.1" 500 -
{'tid': [5133, 5134, 5135, None], 'users': {'45514': {'otid': 5135, 'tid': [5133, 5134], 'time': ['2021-11-30T17:57:20.000-05:00', '2021-11-30T17:57:24.000-05:00'], 'priority': [2, 1]}, '45513': {'otid': 5134, 'tid': [5133, 5135], 'time': ['2021-11-30T17:57:41.000-05:00', '2021-11-30T17:57:43.000-05:00'], 'priority': [2, 1]}, '45512': {'otid': 5133, 'tid': [5134, 5135], 'time': ['2021-11-30T17:57:58.000-05:00', '2021-11-30T17:58:00.000-05:00'], 'priority': [2, 1]}, '45515': {'otid': None, 'tid': [], 'time': [0], 'priority': [3, 1, 4, 2]}}, 'max_accepted_proposals': 3}
[2021-11-28 18:10:40,576] ERROR in app: Exception on /match_topics [POST]
Traceback (most recent call last):
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/usr/.local/lib/python3.5/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/usr/IntelligentAssignment/app/app.py", line 19, in topic_matching
return jsonify(assigned_topics_model.get_student_topic_matches())
File "/home/usr/IntelligentAssignment/app/topics_matcher.py", line 57, in get_student_topic_matches
student.accept_proposal()
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in accept_proposal
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in <lambda>
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
ValueError: None is not in list
</pre>

As we can see from the additional information provided above, the error is due to an unhandled edge case of attempting to access missing value. Our further observation also concluded that such missing value is due to the fact that one or more to-be-matched students did not have any topic signed up at the time of algorithm execution. To address the issue, a general enhancement can be implemented on the algorithm service side with Flask to increase the crash recovery stability. Further refinement of interaction flow with user in the front end can also reduce the possibility of algorithm service error.

Additional to the unhandled cases from the algorithm service, we also noticed an inconsistency of the system due to the performance limitation of the Heroku deployment.

[[File:Out_of_scope_External-Issue_E2151.gif|frameless|x1000px]]

As we can see from the above recorded screen capture animation, with the exact same input, the output and algorithm service result are unstable. Because of the performance limitation of the Heroku, sometime the Flask app that deployed were overloaded. Furthermore, we have also attempted to deploy on VCL, however, we notice the limitation of VCL deployment that will become offline once the VCL reservation expired. After all, because of the focus of our tasks is on the DRY principal optimization, the external algorithm service can be improved in the future. To access and deploy the algorithm on local environment or another environment, setup guide and resource are available on [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment] repository.

===Lack of Comprehensive Testing===

A notable contradiction that was noticed when working with this project was that the previously-mentioned failed edge cases existed, yet all tests for this feature passed. This is illustrative of the fact that this feature requires better and more exhaustive testing.

With the scope limitation of our tasks that focusing on the DRY principal enhancement of the Expertiza internally, the external algorithm service module was not within our focus. However, with our experience of deployment with the algorithm service, we can offer some suggestions for the future team who will address those edge cases of the algorithm service.

When dealing with an issue with so many factors as how students bid on reviews, an extensive suite of edge cases should be considered. Any future project group working on this function should consider the following edge cases to cover:
* Does a student who does not bid on any reviews still get reviews?
* Does a student who did not submit anything for their assignment still have a review to bid on?
* What if there are less assignments to have reviews for than the minimum number required for all students to have the minimum number of reviews?

===Conclusion===
While we believe the current state of this feature is greatly improved, with many of its DRY violations removed and several noteworthy bugs corrected, we believe the feature of allowing students to bid on what to review still needs further development before it is deployed. While the basic functionality is working under ideal situations, the issues mentioned above should be addressed before this feature is put in the hands of actual instructors, as if this feature fails, it could ruin the entire reviewing process for an assignment.

Still, we think there is notable promise to this feature, and it would be a very engaging and interesting feature if deployed successfully, so we believe develop on it should continue until it is ready for deployment.

==Github and Related Link==
Github repository corresponding to this task is publicly available at [https://github.com/WeiRui-Wang/expertiza E2151 GitHub].

Github pull request and file changes can be viewed at [https://github.com/expertiza/expertiza/pull/2150/files E2151 Pull Request].

Github of source code of externally hosted bidding algorithm service deployment manual and files can be access at [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment].

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: Geoff Garrido, Ayush Luthra, and WeiRui Wang. Our project mentor was Luis Delossantos (lgdeloss@ncsu.edu)

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-30T01:20:36Z

Aluthra: /* review_bid.rb */

==Topic Overview==
===Statement of Problem===
Reviewing is an important stage of learning and improving, both as a reviewer and a reviewee, and so allowing reviewers more freedom and choice in what they review would result in better reviews and more engaged reviewers. However, currently the only way to choose a topic to review is a first-come-first-served basis, penalizing students who do not sign up quickly by not allowing them to review the topics they would be most interesting in engaging with.

A better alternative would be to implement a bidding system, similar to that used for topics, which would allow students to establish preferences for which topics they would most like to review. A good bidding algorithm implementing with this idea would ensure that all students (who care enough to bid) receive a topic to review that they actively want.

A prior project group created a prototype for this function, allowing students to bid on what topics they wanted to review. While the functionality was as desired, the feature was not able to be merged due to having too many DRY violations. Thus, the goal of this topic was to bring this prior implementation into the current version of the expertiza beta branch, and remove these DRY violations, so that it may be merged.

As development continued, it became clear this prototype also had several bugs and issues that made it unable to work under several conditions. Thus, a secondary goal was to fix any issues which existed such that the feature would generally work with the functionality the prior implementation demonstrated.

===Prior Work===
The prior work which is the basis for this topic is linked here [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review is linked here]. This work was done in Fall of 2020, and as previously mentioned, achieved much of the desired functionality for this task. This prior work will be referenced several times within this documentation, as it has a more comprehensive explanation of the underlying design principles behind the views, controller, model, algorithm, and RSpec tests implemented. These details will only be briefly summarized here.

As previously noted, this prior implementation was rejected due to DRY violations, and so correcting these violations--and explaining how those corrections were done--will be the major focus of this project documentation.

==Explanation of Feature==

===Existing Bidding Functionality for Topics===
Currently in expertiza, students may bid on which topics they would like to work on. An image showcasing this functionality is given below:

[[File:expected_review_UI.PNG]]

Students may choose a number of topics to bid on, in order of priority, and once the deadline is reached, all students are given a topic.

Behind the scenes, this works by utilizing a modified version of the Gale-Shapely Algorithm. This algorithm seeks to find a 'stable' matching of each student to each topic, such that in the end, all students have a topic they wanted (at least to some degree), and that the preferences of students is respected as well as possible. To help break situations where there are ties in preferences, each topic assigns a Priority to each student. This Priority is determined by A. whether the student is part of a group bidding for this topic (groups getting higher priority), and B. the time at which the bid for the topic was made (earlier bidders are given Priority). Once this has been determined and the deadline for bidding has ended, the following steps occur:
# Each student is matched with each topic they most prefer.
# If the number of students assigned to a topic is above the maximum allowed for that topic, the students with the highest Priority are assigned the topic
# All maximally filled topics are removed from the pool of available topics
# All unmatched students begin this cycle again, now choosing the most preferred topic remaining
# Repeat until all students are assigned a topic
# If any matches are 'unstable' (two students would have their preferences better met if they were swapped), perform any swaps until matches are as stable as possible.

This algorithm has a time complexity of O(n^2). For more details on this algorithm and its implementation, refer to the previous year's work, where this algorithm was implemented.

===Desired Functionality for Reviews===

The desired feature for reviews should have an almost identical structure to this, with reviews instead of topics. Students should be shown a list of the topics that were worked on, and may bid on which of those topics they would like to review the most. The Priority system could be handled in the same way, with the only caveat being that a student should not be able to review their own work.

In terms of what this feature looks like, it can be enabled as an option when editing/creating a new assignment, under the "Topics" section, using interface drop down show below.

[[File:Bidding_interface.png|center|800px]]

Once this option is enabled, during the review phase of an assignment, students will be able to access a new view listing all the topics of all the submitted assignments (except for their own). They will then be able to bid on as many topics as they like, assigning them a preference, in a manner similar to the interface for topic bidding. The table that allows for this drag-and-drop priorities is shown below.

[[File:Bidding_options.png|center|900px]]

Whenever the instructor decides they would like to close the window for review bidding, they may return to the options for the given assignment and navigate to "Topics", where they will see a button marked "Run Review Algorithm". This will make a call to the external webservice to determine which students are assigned what reviews, and will then assign those reviews to all students. Once this button is pressed, it locks out the ability for students to perform any more bidding, and cannot be pressed again. The button allowing this option is shown below.

[[File:Bidding_run_algorithm.png|center|900px]]

After this point, students may navigate to the "Others Work" option under their assignment and see all of their assigned reviews. They will see up to the maximum number of allowed reviews, but will be informed of how many they are required to do (for example, a student may see 4 reviews, but will be told they are only required to complete 2).

===Use Case Diagram===
The following Use Case Diagram demonstrates the primary functionality explained in the previous section for different types of users.

[[File:Use_Case_Diagram_Rev1_E2151.jpg|center]]

==Plan and List of Work Done==

===General Design Goals===

* Move all prior work for this topic into a new version of the expertiza beta branch, without loss in functionality.
* Ensure all prior/expected functionality is working correctly, and if not, make any repairs or changes necessary.
* DRY out prior implementation's controller and model methods which have functionality that already exists within the expertiza system.
* Remove repetitive views and partials that exist by merging views for review bids into the set of views for sign up sheets, from which it uses partials.
* Ensure all present test cases still work properly after making the above changes.

===Specific Tasks Completed===

* Transferred over all code files relevant to the review bidding feature
* Re-deployed the external website which handles the review bidding algorithm
* Added that website's url to the webservices url's within expertiza
* Enhanced the stability of the webservice for the bidding algorithm by testing with edge cases and optimize the endpoint by restoring the original access endpoint reference* Analyzed the necessity of the phase finding related logic code section
* Optimized phase finding related logic by removing unused code segment
* Removed an instance variable within the ''review_bids_controller'' which would have made any practical deployment fail, removing the unnecessary functionality it was created for
* Updated method calls to ''assignment.find_current_stage'' to their new form ''assignment.current_stage''
* Removed the review_bid model method ''self.reviewer_self_topic'' and replaced its usage with the already-existing ''SignedUpTeam.topic_id'' method
* Removed DRY-violating code used to calculate the current assignment phase which was never utilized by the controller or views
* Moved several view files from the ''review_bids'' view directory into the ''sign_up_sheet'' view directory. Partials which were duplicates in the two directories (''_all_actions.html.erb'' and ''_topic_names.html.erb'' were no longer necessary and deleted
* Refactored the naming of the variable that represents the id of the reviewer object from 'reviewer' to 'reviewer_id' for clarity
* Removed the unnecessary 'create' function within the review_bids_controller
* Removed the sparse testing files under the 'test' directory in keeping with typical expertiza structure
* Removed a test case from the unit tests which relied upon the removed 'create' method of the review_bids_controller

==Initial Implementation==

A considerable amount of time and effort was necessary to review, understand, and re-implement the previous implementation of the Review Bidding feature. While the feature was only created a year ago, several underlying functions and methods have since been refactored, and so needed to be changed in the implementation. Additionally, certain parts of how the implementation was designed did not seem to be properly implemented, and so had to be revised.

===Code Transfer===
First, all files that were added in the original pull request were copied and moved to the new version of the expertiza beta branch. This includes the following files:

*'''controllers/review_bids_controller.rb''': the primary controller that handles calls to review bids functionality from views, managing them itself or directing them to the appropriate model
*'''helpers/review_bids_helper.rb''': contains helper methods for rendering the rows of topics available to choose a review for
*'''models/review_bid.rb''': contains the methods that each individual review bid knows and manages, including assigning a review to a student, getting info on the topic it applies to, etc.
*'''views/review_bids/_all_actions.html.erb''': a view partial used to render the actions available to a student on the review bidding view. Nearly identical to the _all_actions file for topic bidding.
*'''views/review_bids/_table_header.html.erb''': a view partial used to render the head of the table for review bids. Though structurally similar to that of topics reviews, contains significantly less information, as less needs to be known by a student for particular topics to review.
*'''views/review_bids/_table_line.html.erb''': a view partial used to render individual rows of the review bids table. Like the table header, is structurally similar to that of topic bidding, but functions differently.
*'''views/review_bids/show.html.erb''': the view which displays all the available topics to bid on reviewing. Uses the aforementioned partials to render this bidding table.
*'''views/review_bids/others_work.html.erb''': displays the available review work which has the user has bid for.
*'''db/migrate/20201107203204_create_review_bids.rb''': the migration which creates the review bid objects within the expertiza database.
*'''spec/controllers/review_bids_controller_spec.rb''': the unit tests for the review bids controller
*'''spec/models/review_bid_spec.rb''': the unit tests for the review bids model
*'''test/controllers/review_bids_controller_test.rb''': a more elaborate feature test for the review bids functionality
*'''test/fixtures/review_bids.yml''': fixtures that are utilized for the review_bids_controller_test
*'''test/models/review_bid_test.rb''': A more elaborate feature test of the review bid model.

In addition to adding the previous files, the following files were also added to existing pages as modifications:

*'''controllers/student_review_controller.rb''': the student review controller's "list" method now redirects to the review_bid_controller's "index" method if the assignment in question has a review_choosing_algorithm set to "Bidding" (the option an instructor sets when they wish to allow students to bid on reviews).
*'''student_task/view.html.erb''': Instead of simply linking directly to the student_review controller's "list" method, the view instead checks if the assignment this view pertains to has their review_choosing_algorithm set to Bidding. If it does, and the assignment allows students to choose the topic to review (meaning the algorithm to assign bids has not yet been run), they are redirected to the 'show' method of the review_bids_controller. Otherwise, they are redirected to the 'index' method (showing them the info on their currently available topics to review).
*'''config/webservices.yml''': added a new url for the website that is called to run the review bidding algorithm
*'''config/routes.rb''': routes added for review_bids.
*'''views/assignments/edit/_topics.html.erb''': Added a new option box which allows an instructor to choose to allow bidding to occur using the review bidding algorithm, rather than the default 'Simple Choose.' Once this option is selected, it also offers a button that, when clicked, runs the review bidding algorithm on all currently bid upon reviews.

===Website Reimplementation===
In the previous implementation, the algorithm which handles the assigning of reviews was hosted on an external website. This was done for the sake of improved performance. Thus, to get the feature in working condition, we pulled the code necessary to run the algorithm and hosted it on our own website (linked at the end of this documentation). We also made a note in the pull request that the place where this website is referenced (in the "assign_reviews_algorithm" method of the review_bids_controller) should be replaced by a proper external web tool, as is done for similar web tools in other parts of expertiza.

===Modifications to Methods===
Certain methods utilized in the original version of the webpages were changed or renamed in later versions, and so, needed to be modified. The primary example of this is the method that assignments use to determine what stage of assignment completion they are in. The previous year's implementation called this method as "assignment.find_current_stage(topic_id)," but this failed when trying to execute. After comparing how this page worked to how normal review pages work, we determined the new call to this method is "assignment.current_stage(topic_id)." All calls to this method were replaced in all files.

Another change to methods was the removal of a method from the review_bids_controller named 'create'. This method primarily served as a redirect for an aspect of the feature that was unnecessary and so removed (see the following section), that being the button to show another review. With this feature removed, this method was no longer used, and so it was removed from the controller.

===Errors Corrected===
A notable period of time was also spent on determining the purpose of a particular local area that caused issues when trying to view the bids, that being a variable named "@number_of_reviews_to_show". This variable apparently kept track of how many reviews a student needed to be shown, so that they could request another. The issue was that this was created as a local variable within the review_bids_controller, and therefore would never work outside of a manual testing environment, as the variable would need to be different from every student and every assignment. After some investigation, we realized that all this variable did was limit the number of already-assigned reviews the student was able to see (as the review bidding algorithm already assigned all possible reviews). Thus, rather than try to recreate this functionality, we simply removed it.

This means that, when the review bidding algorithm runs, a student is able to see all of the reviews assigned to them, up to the maximum number of reviews for the assignment, and therefore have no reason to ask for any more. It may be the case that this feature could be improved by adding a possibility for students to request any reviews not assigned to any other student, or that still have available slots, using the "normal" review assignment scheme. For the time being, we did not consider this part of the baseline necessary functionality, and so did not implement it.

==Refinement Implementation==

===DRY Principle and Code Duplication Refactoring===
Per comments and feedbacks on the previous implementation [https://github.com/expertiza/expertiza/pull/1822 E2085], there were 3 sections of the code that potentially were DRY violations. For each of those violations, we have had made appropriate and provided the justification on the results.

====review_bids_controller.rb====
For the refactoring of review bids controller.eb, the original issue is the DRY principle violation on the internal logic of finding the phase of assignments.
The original implementation to be refactored and optimized are shown as follow.
<pre>
# Finding the current phase that we are in
due_dates = AssignmentDueDate.where(parent_id: @assignment.id)
@very_last_due_date = AssignmentDueDate.where(parent_id: @assignment.id).order("due_at DESC").limit(1)
next_due_date = @very_last_due_date[0]
for due_date in due_dates
if due_date.due_at > Time.now
next_due_date = due_date if due_date.due_at < next_due_date.due_at
end
end
@review_phase = next_due_date.deadline_type_id
</pre>
Through some discovery process, we have found some related logics that is already implemented from the system which enable the refactoring and reuse of the method. Some code snippets are demonstrated as follow.
<pre>
def self.current_due_date(due_dates)
#Get the current due date from list of due dates
due_dates.each do |due_date|
if due_date.due_at > Time.now
current_due_date = due_date
return current_due_date
end
end
#in case current due date not found
return nil
end
</pre>
<pre>
def self.teammate_review_allowed(student)
# time when teammate review is allowed
due_date = current_due_date(student.assignment.due_dates)
student.assignment.find_current_stage == 'Finished' ||
due_date &&
(due_date.teammate_review_allowed_id == 3 ||
due_date.teammate_review_allowed_id == 2) # late(2) or yes(3)
end
</pre>
As we can observed above, both methods of <code>current_due_date</code> and <code>teammate_review_allowed</code> provides the capability of getting the <code>review_phase</code> without the repetition. Additionally, we can see that those existing implementation still have some difference in implementation and purpose to be fully reusable by the logic that find current phase of a topic because the methods provided are helper methods for broader abstraction.

Furthermore, the section that we attempted to refactoring out have a similar logic that is used to get the current stage as shown below.
<pre>
def find_current_stage(topic_id = nil)
next_due_date = DueDate.get_next_due_date(self.id, topic_id)
return 'Finished' if next_due_date.nil?
next_due_date
end
</pre>
However, it is worth pointing out that the method is not fully substitutable, particularly the internal functionality that finds the <code>@very_last_due_date</code>. A similar logic does exist that is meant to use to acquire current due date and assist with finding current stage.
<pre>
def current_stage_name(topic_id = nil)
if self.staggered_deadline?
return (topic_id.nil? ? 'Unknown' : current_stage(topic_id))
end

due_date = find_current_stage(topic_id)
unless due_date == 'Finished' || due_date.nil? || due_date.deadline_name.nil?
return due_date.deadline_name
end
current_stage(topic_id)
end
</pre>
With all the findings, we have come to conclusion that the portion of the code to be dry is in similar, but required full restructure, form of existing methods. The internal logic flow itself is not fully matching the reusable methods consists across the controllers. Because of the clarity of limited reusability of existing methods, we have come to conclusion that the portions to be refactored are not duplication of any existing methods. Additionally, after a more detailed discovery, it has come to realization that the code portion of getting the current phase of the next due date was, in fact, meant to find the last due date. After further analysis of the purpose of the code, the portion has been deleted from <code>review_bids_controller.rb</code> because there was no usage observed with the result of the execution.

====review_bid.rb====
Per the previous iteration's feedback, the review_bid.rb model file presented some DRY violations. These violations are highlighted in the method below:
 
[[File:Review_bid_original.png]]
 
 

====Refactoring Views (_all_actions.html.erb and _topic_names.html.erb)====
The last element of the prior implementation that caused DRY violations had to do with its implementation of views. Two primary views are necessary for the review bids functionality: one which shows a table where students can bid on which topic they wish to review (rendered by a file called 'show.html.erb'), and a list of assigned reviews which a student must select from (rendered by a file called 'others_work.html.erb').

In order to render the elements in 'show.html.erb' in particular, a number of partials must be utilized:
*a partial for the table's header (_table_header.html.erb)
*a partial for rendering each of the table's rows, describing a topic to bid on (_table_line.html.erb)
*a partial to contain information related to what actions are available for a student to do for a review, like edit or view (_all_actions.html.erb)
*a partial for finding and rendering the name of each topic one must review (_topic_names.html.erb)

Both of these views and all four of these partials were contained within the "review_bids" directory within the views directory. The DRY violation incurred by these views has to do with these partials: in particular, '_topic_names.html.erb' and '_all_actions.html.erb' are near identical code copies to those within the sign_up_sheet views, used for rendering and displaying information for topic bidding and normal reviews.

In order to DRY out this portion of the code, we decided that having a separate set of views for the results related to review bids were unnecessary. Within expertiza's structure, most of the views related to students choosing which topics or reviews to work on are all contained within the sign_up_sheet views. The review_vids views were copying the partials within the sign_up_sheet views because they were using the same fundamental components. Thus, we thought the best solution would be to move the two primary views for review_bids ('show.html.erb' and 'others_work.html.erb') into the sign_up_sheet views, and leave behind all redundant partials. In doing so, the redundancy of '_all_actions.html.erb' and '_topic_names.html.erb' were eliminated.

It could be argued that this implementation could be further DRY-ed by leaving behind the other two partials, '_table_header.html.erb' and '_table_line.html.erb', as they are structurally similar to those used for topic bidding. However, this is not actually permissible, as the header and lines for topics when bidding on them for reviews are significantly different than when bidding on them for assignments. More information needs to be available when bidding on topics for an assignment, such as the number of slots and number on the waitlist, but such an idea is not applicable for bids on reviews. While it would be possible to have a single file which produces one view or another depending upon if it's bidding for topics or reviews, we believed this violated the Expert Pattern, by attempting to create a single view for two distinct purposes. Instead, we thought it better to simply have two separate partials, each which precisely gave the desired functionality. Thus, the other two partials were brought over, being renamed '_review_bids_table_header.html.erb' and '_review_bids_table_line.html.erb'.

==Tests==

Throughout the refactoring and merging of [https://github.com/expertiza/expertiza/pull/1822 E2085] to [https://github.com/expertiza/expertiza/pull/2150 E2151], there were some updates that has been made to the test suits to maintain corrosponding testing automation.

===Video Demonstration===
[https://www.youtube.com/watch?v=wxXALQNWFcc Click this link] to see a video demonstrating the functionality of the review bidding feature.

===RSpec Testing===
One of the primary design goals when doing this refactoring was to ensure that all test cases still passed after all changes were made.

There was a single change that needed to be made to the RSpec files for the review bidding algorithm, which was the removal of a test relying on the 'create' method of the review_bids_controller. As previously mentioned, this function was removed because it had become obsolete. Thus, the test case for this function (which simply checked if it redirected to the right page) was no longer necessary, and thus was deleted.

Otherwise, the test cases for both the review_bids_controller and the review_bids model were manually run and found to have no errors, as shown below.

[[File:Bidding_test_results.png|800px]]

For details on what these specific tests were, please refer to the previous work referenced before.

==Conclusions and Future Work==
While the current state of the review bidding code is operational, and is certainly more DRY and stable than the previous incarnation of this work, there are still several issues with the current state of this feature that make it difficult to recommend an immediate merge into expertiza.

===Enhancement of Algorithm Service Stability===

Across the process of refining the refactored front end page view along with more robust edge cases testing towards the algorithm service, we have discovered an underlying issue that could impact the availability of the review bidding algorithm service. Through cause isolation, we have observed a high correlation to flask application crash with the edge case scenario input where the algorithm service is trying to match a student who was not signed up with any topic with other students.

Because the scope of our project was focused on refactoring, we did not have the time or prior warning to need to address these edge cases. However, we will provide the error log for the reference of any future teams who may work on the stability improvement of the algorithm service.

<pre>
"POST /match_topics HTTP/1.1" 500 -
{'tid': [5133, 5134, 5135, None], 'users': {'45514': {'otid': 5135, 'tid': [5133, 5134], 'time': ['2021-11-30T17:57:20.000-05:00', '2021-11-30T17:57:24.000-05:00'], 'priority': [2, 1]}, '45513': {'otid': 5134, 'tid': [5133, 5135], 'time': ['2021-11-30T17:57:41.000-05:00', '2021-11-30T17:57:43.000-05:00'], 'priority': [2, 1]}, '45512': {'otid': 5133, 'tid': [5134, 5135], 'time': ['2021-11-30T17:57:58.000-05:00', '2021-11-30T17:58:00.000-05:00'], 'priority': [2, 1]}, '45515': {'otid': None, 'tid': [], 'time': [0], 'priority': [3, 1, 4, 2]}}, 'max_accepted_proposals': 3}
[2021-11-28 18:10:40,576] ERROR in app: Exception on /match_topics [POST]
Traceback (most recent call last):
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/usr/.local/lib/python3.5/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/usr/IntelligentAssignment/app/app.py", line 19, in topic_matching
return jsonify(assigned_topics_model.get_student_topic_matches())
File "/home/usr/IntelligentAssignment/app/topics_matcher.py", line 57, in get_student_topic_matches
student.accept_proposal()
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in accept_proposal
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in <lambda>
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
ValueError: None is not in list
</pre>

As we can see from the additional information provided above, the error is due to an unhandled edge case of attempting to access missing value. Our further observation also concluded that such missing value is due to the fact that one or more to-be-matched students did not have any topic signed up at the time of algorithm execution. To address the issue, a general enhancement can be implemented on the algorithm service side with Flask to increase the crash recovery stability. Further refinement of interaction flow with user in the front end can also reduce the possibility of algorithm service error.

Additional to the unhandled cases from the algorithm service, we also noticed an inconsistency of the system due to the performance limitation of the Heroku deployment.

[[File:Out_of_scope_External-Issue_E2151.gif|frameless|x1000px]]

As we can see from the above recorded screen capture animation, with the exact same input, the output and algorithm service result are unstable. Because of the performance limitation of the Heroku, sometime the Flask app that deployed were overloaded. Furthermore, we have also attempted to deploy on VCL, however, we notice the limitation of VCL deployment that will become offline once the VCL reservation expired. After all, because of the focus of our tasks is on the DRY principal optimization, the external algorithm service can be improved in the future. To access and deploy the algorithm on local environment or another environment, setup guide and resource are available on [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment] repository.

===Lack of Comprehensive Testing===

A notable contradiction that was noticed when working with this project was that the previously-mentioned failed edge cases existed, yet all tests for this feature passed. This is illustrative of the fact that this feature requires better and more exhaustive testing.

When dealing with an issue with so many factors as how students bid on reviews, an extensive suite of edge cases should be considered. Any future project group working on this function should consider the following edge cases to cover:
* Does a student who does not bid on any reviews still get reviews?
* Does a student who did not submit anything for their assignment still have a review to bid on?
* What if there are less assignments to have reviews for than the minimum number required for all students to have the minimum number of reviews?

===Conclusion===
While we believe the current state of this feature is greatly improved, with many of its DRY violations removed and several noteworthy bugs corrected, we believe the feature of allowing students to bid on what to review still needs further development before it is deployed. While the basic functionality is working under ideal situations, the issues mentioned above should be addressed before this feature is put in the hands of actual instructors, as if this feature fails, it could ruin the entire reviewing process for an assignment.

Still, we think there is notable promise to this feature, and it would be a very engaging and interesting feature if deployed successfully, so we believe develop on it should continue until it is ready for deployment.

==Github and Related Link==
Github repository corresponding to this task is publicly available at [https://github.com/WeiRui-Wang/expertiza E2151 GitHub].

Github pull request and file changes can be viewed at [https://github.com/expertiza/expertiza/pull/2150/files E2151 Pull Request].

Github of source code of externally hosted bidding algorithm service deployment manual and files can be access at [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment].

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: Geoff Garrido, Ayush Luthra, and WeiRui Wang. Our project mentor was Luis Delossantos (lgdeloss@ncsu.edu)

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-30T01:11:53Z

Aluthra: /* review_bid.rb */

==Topic Overview==
===Statement of Problem===
Reviewing is an important stage of learning and improving, both as a reviewer and a reviewee, and so allowing reviewers more freedom and choice in what they review would result in better reviews and more engaged reviewers. However, currently the only way to choose a topic to review is a first-come-first-served basis, penalizing students who do not sign up quickly by not allowing them to review the topics they would be most interesting in engaging with.

A better alternative would be to implement a bidding system, similar to that used for topics, which would allow students to establish preferences for which topics they would most like to review. A good bidding algorithm implementing with this idea would ensure that all students (who care enough to bid) receive a topic to review that they actively want.

A prior project group created a prototype for this function, allowing students to bid on what topics they wanted to review. While the functionality was as desired, the feature was not able to be merged due to having too many DRY violations. Thus, the goal of this topic was to bring this prior implementation into the current version of the expertiza beta branch, and remove these DRY violations, so that it may be merged.

As development continued, it became clear this prototype also had several bugs and issues that made it unable to work under several conditions. Thus, a secondary goal was to fix any issues which existed such that the feature would generally work with the functionality the prior implementation demonstrated.

===Prior Work===
The prior work which is the basis for this topic is linked here [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review is linked here]. This work was done in Fall of 2020, and as previously mentioned, achieved much of the desired functionality for this task. This prior work will be referenced several times within this documentation, as it has a more comprehensive explanation of the underlying design principles behind the views, controller, model, algorithm, and RSpec tests implemented. These details will only be briefly summarized here.

As previously noted, this prior implementation was rejected due to DRY violations, and so correcting these violations--and explaining how those corrections were done--will be the major focus of this project documentation.

==Explanation of Feature==

===Existing Bidding Functionality for Topics===
Currently in expertiza, students may bid on which topics they would like to work on. An image showcasing this functionality is given below:

[[File:expected_review_UI.PNG]]

Students may choose a number of topics to bid on, in order of priority, and once the deadline is reached, all students are given a topic.

Behind the scenes, this works by utilizing a modified version of the Gale-Shapely Algorithm. This algorithm seeks to find a 'stable' matching of each student to each topic, such that in the end, all students have a topic they wanted (at least to some degree), and that the preferences of students is respected as well as possible. To help break situations where there are ties in preferences, each topic assigns a Priority to each student. This Priority is determined by A. whether the student is part of a group bidding for this topic (groups getting higher priority), and B. the time at which the bid for the topic was made (earlier bidders are given Priority). Once this has been determined and the deadline for bidding has ended, the following steps occur:
# Each student is matched with each topic they most prefer.
# If the number of students assigned to a topic is above the maximum allowed for that topic, the students with the highest Priority are assigned the topic
# All maximally filled topics are removed from the pool of available topics
# All unmatched students begin this cycle again, now choosing the most preferred topic remaining
# Repeat until all students are assigned a topic
# If any matches are 'unstable' (two students would have their preferences better met if they were swapped), perform any swaps until matches are as stable as possible.

This algorithm has a time complexity of O(n^2). For more details on this algorithm and its implementation, refer to the previous year's work, where this algorithm was implemented.

===Desired Functionality for Reviews===

The desired feature for reviews should have an almost identical structure to this, with reviews instead of topics. Students should be shown a list of the topics that were worked on, and may bid on which of those topics they would like to review the most. The Priority system could be handled in the same way, with the only caveat being that a student should not be able to review their own work.

In terms of what this feature looks like, it can be enabled as an option when editing/creating a new assignment, under the "Topics" section, using interface drop down show below.

[[File:Bidding_interface.png|center|800px]]

Once this option is enabled, during the review phase of an assignment, students will be able to access a new view listing all the topics of all the submitted assignments (except for their own). They will then be able to bid on as many topics as they like, assigning them a preference, in a manner similar to the interface for topic bidding. The table that allows for this drag-and-drop priorities is shown below.

[[File:Bidding_options.png|center|900px]]

Whenever the instructor decides they would like to close the window for review bidding, they may return to the options for the given assignment and navigate to "Topics", where they will see a button marked "Run Review Algorithm". This will make a call to the external webservice to determine which students are assigned what reviews, and will then assign those reviews to all students. Once this button is pressed, it locks out the ability for students to perform any more bidding, and cannot be pressed again. The button allowing this option is shown below.

[[File:Bidding_run_algorithm.png|center|900px]]

After this point, students may navigate to the "Others Work" option under their assignment and see all of their assigned reviews. They will see up to the maximum number of allowed reviews, but will be informed of how many they are required to do (for example, a student may see 4 reviews, but will be told they are only required to complete 2).

===Use Case Diagram===
The following Use Case Diagram demonstrates the primary functionality explained in the previous section for different types of users.

[[File:Use_Case_Diagram_Rev1_E2151.jpg|center]]

==Plan and List of Work Done==

===General Design Goals===

* Move all prior work for this topic into a new version of the expertiza beta branch, without loss in functionality.
* Ensure all prior/expected functionality is working correctly, and if not, make any repairs or changes necessary.
* DRY out prior implementation's controller and model methods which have functionality that already exists within the expertiza system.
* Remove repetitive views and partials that exist by merging views for review bids into the set of views for sign up sheets, from which it uses partials.
* Ensure all present test cases still work properly after making the above changes.

===Specific Tasks Completed===

* Transferred over all code files relevant to the review bidding feature
* Re-deployed the external website which handles the review bidding algorithm
* Added that website's url to the webservices url's within expertiza
* Enhanced the stability of the webservice for the bidding algorithm by testing with edge cases and optimize the endpoint by restoring the original access endpoint reference* Analyzed the necessity of the phase finding related logic code section
* Optimized phase finding related logic by removing unused code segment
* Removed an instance variable within the ''review_bids_controller'' which would have made any practical deployment fail, removing the unnecessary functionality it was created for
* Updated method calls to ''assignment.find_current_stage'' to their new form ''assignment.current_stage''
* Removed the review_bid model method ''self.reviewer_self_topic'' and replaced its usage with the already-existing ''SignedUpTeam.topic_id'' method
* Removed DRY-violating code used to calculate the current assignment phase which was never utilized by the controller or views
* Moved several view files from the ''review_bids'' view directory into the ''sign_up_sheet'' view directory. Partials which were duplicates in the two directories (''_all_actions.html.erb'' and ''_topic_names.html.erb'' were no longer necessary and deleted
* Refactored the naming of the variable that represents the id of the reviewer object from 'reviewer' to 'reviewer_id' for clarity
* Removed the unnecessary 'create' function within the review_bids_controller
* Removed the sparse testing files under the 'test' directory in keeping with typical expertiza structure
* Removed a test case from the unit tests which relied upon the removed 'create' method of the review_bids_controller

==Initial Implementation==

A considerable amount of time and effort was necessary to review, understand, and re-implement the previous implementation of the Review Bidding feature. While the feature was only created a year ago, several underlying functions and methods have since been refactored, and so needed to be changed in the implementation. Additionally, certain parts of how the implementation was designed did not seem to be properly implemented, and so had to be revised.

===Code Transfer===
First, all files that were added in the original pull request were copied and moved to the new version of the expertiza beta branch. This includes the following files:

*'''controllers/review_bids_controller.rb''': the primary controller that handles calls to review bids functionality from views, managing them itself or directing them to the appropriate model
*'''helpers/review_bids_helper.rb''': contains helper methods for rendering the rows of topics available to choose a review for
*'''models/review_bid.rb''': contains the methods that each individual review bid knows and manages, including assigning a review to a student, getting info on the topic it applies to, etc.
*'''views/review_bids/_all_actions.html.erb''': a view partial used to render the actions available to a student on the review bidding view. Nearly identical to the _all_actions file for topic bidding.
*'''views/review_bids/_table_header.html.erb''': a view partial used to render the head of the table for review bids. Though structurally similar to that of topics reviews, contains significantly less information, as less needs to be known by a student for particular topics to review.
*'''views/review_bids/_table_line.html.erb''': a view partial used to render individual rows of the review bids table. Like the table header, is structurally similar to that of topic bidding, but functions differently.
*'''views/review_bids/show.html.erb''': the view which displays all the available topics to bid on reviewing. Uses the aforementioned partials to render this bidding table.
*'''views/review_bids/others_work.html.erb''': displays the available review work which has the user has bid for.
*'''db/migrate/20201107203204_create_review_bids.rb''': the migration which creates the review bid objects within the expertiza database.
*'''spec/controllers/review_bids_controller_spec.rb''': the unit tests for the review bids controller
*'''spec/models/review_bid_spec.rb''': the unit tests for the review bids model
*'''test/controllers/review_bids_controller_test.rb''': a more elaborate feature test for the review bids functionality
*'''test/fixtures/review_bids.yml''': fixtures that are utilized for the review_bids_controller_test
*'''test/models/review_bid_test.rb''': A more elaborate feature test of the review bid model.

In addition to adding the previous files, the following files were also added to existing pages as modifications:

*'''controllers/student_review_controller.rb''': the student review controller's "list" method now redirects to the review_bid_controller's "index" method if the assignment in question has a review_choosing_algorithm set to "Bidding" (the option an instructor sets when they wish to allow students to bid on reviews).
*'''student_task/view.html.erb''': Instead of simply linking directly to the student_review controller's "list" method, the view instead checks if the assignment this view pertains to has their review_choosing_algorithm set to Bidding. If it does, and the assignment allows students to choose the topic to review (meaning the algorithm to assign bids has not yet been run), they are redirected to the 'show' method of the review_bids_controller. Otherwise, they are redirected to the 'index' method (showing them the info on their currently available topics to review).
*'''config/webservices.yml''': added a new url for the website that is called to run the review bidding algorithm
*'''config/routes.rb''': routes added for review_bids.
*'''views/assignments/edit/_topics.html.erb''': Added a new option box which allows an instructor to choose to allow bidding to occur using the review bidding algorithm, rather than the default 'Simple Choose.' Once this option is selected, it also offers a button that, when clicked, runs the review bidding algorithm on all currently bid upon reviews.

===Website Reimplementation===
In the previous implementation, the algorithm which handles the assigning of reviews was hosted on an external website. This was done for the sake of improved performance. Thus, to get the feature in working condition, we pulled the code necessary to run the algorithm and hosted it on our own website (linked at the end of this documentation). We also made a note in the pull request that the place where this website is referenced (in the "assign_reviews_algorithm" method of the review_bids_controller) should be replaced by a proper external web tool, as is done for similar web tools in other parts of expertiza.

===Modifications to Methods===
Certain methods utilized in the original version of the webpages were changed or renamed in later versions, and so, needed to be modified. The primary example of this is the method that assignments use to determine what stage of assignment completion they are in. The previous year's implementation called this method as "assignment.find_current_stage(topic_id)," but this failed when trying to execute. After comparing how this page worked to how normal review pages work, we determined the new call to this method is "assignment.current_stage(topic_id)." All calls to this method were replaced in all files.

Another change to methods was the removal of a method from the review_bids_controller named 'create'. This method primarily served as a redirect for an aspect of the feature that was unnecessary and so removed (see the following section), that being the button to show another review. With this feature removed, this method was no longer used, and so it was removed from the controller.

===Errors Corrected===
A notable period of time was also spent on determining the purpose of a particular local area that caused issues when trying to view the bids, that being a variable named "@number_of_reviews_to_show". This variable apparently kept track of how many reviews a student needed to be shown, so that they could request another. The issue was that this was created as a local variable within the review_bids_controller, and therefore would never work outside of a manual testing environment, as the variable would need to be different from every student and every assignment. After some investigation, we realized that all this variable did was limit the number of already-assigned reviews the student was able to see (as the review bidding algorithm already assigned all possible reviews). Thus, rather than try to recreate this functionality, we simply removed it.

This means that, when the review bidding algorithm runs, a student is able to see all of the reviews assigned to them, up to the maximum number of reviews for the assignment, and therefore have no reason to ask for any more. It may be the case that this feature could be improved by adding a possibility for students to request any reviews not assigned to any other student, or that still have available slots, using the "normal" review assignment scheme. For the time being, we did not consider this part of the baseline necessary functionality, and so did not implement it.

==Refinement Implementation==

===DRY Principle and Code Duplication Refactoring===
Per comments and feedbacks on the previous implementation [https://github.com/expertiza/expertiza/pull/1822 E2085], there were 3 sections of the code that potentially were DRY violations. For each of those violations, we have had made appropriate and provided the justification on the results.

====review_bids_controller.rb====
For the refactoring of review bids controller.eb, the original issue is the DRY principle violation on the internal logic of finding the phase of assignments.
The original implementation to be refactored and optimized are shown as follow.
<pre>
# Finding the current phase that we are in
due_dates = AssignmentDueDate.where(parent_id: @assignment.id)
@very_last_due_date = AssignmentDueDate.where(parent_id: @assignment.id).order("due_at DESC").limit(1)
next_due_date = @very_last_due_date[0]
for due_date in due_dates
if due_date.due_at > Time.now
next_due_date = due_date if due_date.due_at < next_due_date.due_at
end
end
@review_phase = next_due_date.deadline_type_id
</pre>
Through some discovery process, we have found some related logics that is already implemented from the system which enable the refactoring and reuse of the method. Some code snippets are demonstrated as follow.
<pre>
def self.current_due_date(due_dates)
#Get the current due date from list of due dates
due_dates.each do |due_date|
if due_date.due_at > Time.now
current_due_date = due_date
return current_due_date
end
end
#in case current due date not found
return nil
end
</pre>
<pre>
def self.teammate_review_allowed(student)
# time when teammate review is allowed
due_date = current_due_date(student.assignment.due_dates)
student.assignment.find_current_stage == 'Finished' ||
due_date &&
(due_date.teammate_review_allowed_id == 3 ||
due_date.teammate_review_allowed_id == 2) # late(2) or yes(3)
end
</pre>
As we can observed above, both methods of <code>current_due_date</code> and <code>teammate_review_allowed</code> provides the capability of getting the <code>review_phase</code> without the repetition. Additionally, we can see that those existing implementation still have some difference in implementation and purpose to be fully reusable by the logic that find current phase of a topic because the methods provided are helper methods for broader abstraction.

Furthermore, the section that we attempted to refactoring out have a similar logic that is used to get the current stage as shown below.
<pre>
def find_current_stage(topic_id = nil)
next_due_date = DueDate.get_next_due_date(self.id, topic_id)
return 'Finished' if next_due_date.nil?
next_due_date
end
</pre>
However, it is worth pointing out that the method is not fully substitutable, particularly the internal functionality that finds the <code>@very_last_due_date</code>. A similar logic does exist that is meant to use to acquire current due date and assist with finding current stage.
<pre>
def current_stage_name(topic_id = nil)
if self.staggered_deadline?
return (topic_id.nil? ? 'Unknown' : current_stage(topic_id))
end

due_date = find_current_stage(topic_id)
unless due_date == 'Finished' || due_date.nil? || due_date.deadline_name.nil?
return due_date.deadline_name
end
current_stage(topic_id)
end
</pre>
With all the findings, we have come to conclusion that the portion of the code to be dry is in similar, but required full restructure, form of existing methods. The internal logic flow itself is not fully matching the reusable methods consists across the controllers. Because of the clarity of limited reusability of existing methods, we have come to conclusion that the portions to be refactored are not duplication of any existing methods. Additionally, after a more detailed discovery, it has come to realization that the code portion of getting the current phase of the next due date was, in fact, meant to find the last due date. After further analysis of the purpose of the code, the portion has been deleted from <code>review_bids_controller.rb</code> because there was no usage observed with the result of the execution.

====review_bid.rb====
Per the previous iteration's feedback, the review_bid.rb model file presented some DRY violations. These violations are highlighted in the method below:
 
[[File:Review_bid_original.png]]
 

====Refactoring Views (_all_actions.html.erb and _topic_names.html.erb)====
The last element of the prior implementation that caused DRY violations had to do with its implementation of views. Two primary views are necessary for the review bids functionality: one which shows a table where students can bid on which topic they wish to review (rendered by a file called 'show.html.erb'), and a list of assigned reviews which a student must select from (rendered by a file called 'others_work.html.erb').

In order to render the elements in 'show.html.erb' in particular, a number of partials must be utilized:
*a partial for the table's header (_table_header.html.erb)
*a partial for rendering each of the table's rows, describing a topic to bid on (_table_line.html.erb)
*a partial to contain information related to what actions are available for a student to do for a review, like edit or view (_all_actions.html.erb)
*a partial for finding and rendering the name of each topic one must review (_topic_names.html.erb)

Both of these views and all four of these partials were contained within the "review_bids" directory within the views directory. The DRY violation incurred by these views has to do with these partials: in particular, '_topic_names.html.erb' and '_all_actions.html.erb' are near identical code copies to those within the sign_up_sheet views, used for rendering and displaying information for topic bidding and normal reviews.

In order to DRY out this portion of the code, we decided that having a separate set of views for the results related to review bids were unnecessary. Within expertiza's structure, most of the views related to students choosing which topics or reviews to work on are all contained within the sign_up_sheet views. The review_vids views were copying the partials within the sign_up_sheet views because they were using the same fundamental components. Thus, we thought the best solution would be to move the two primary views for review_bids ('show.html.erb' and 'others_work.html.erb') into the sign_up_sheet views, and leave behind all redundant partials. In doing so, the redundancy of '_all_actions.html.erb' and '_topic_names.html.erb' were eliminated.

It could be argued that this implementation could be further DRY-ed by leaving behind the other two partials, '_table_header.html.erb' and '_table_line.html.erb', as they are structurally similar to those used for topic bidding. However, this is not actually permissible, as the header and lines for topics when bidding on them for reviews are significantly different than when bidding on them for assignments. More information needs to be available when bidding on topics for an assignment, such as the number of slots and number on the waitlist, but such an idea is not applicable for bids on reviews. While it would be possible to have a single file which produces one view or another depending upon if it's bidding for topics or reviews, we believed this violated the Expert Pattern, by attempting to create a single view for two distinct purposes. Instead, we thought it better to simply have two separate partials, each which precisely gave the desired functionality. Thus, the other two partials were brought over, being renamed '_review_bids_table_header.html.erb' and '_review_bids_table_line.html.erb'.

==Tests==

Throughout the refactoring and merging of [https://github.com/expertiza/expertiza/pull/1822 E2085] to [https://github.com/expertiza/expertiza/pull/2150 E2151], there were some updates that has been made to the test suits to maintain corrosponding testing automation.

===Video Demonstration===
[https://www.youtube.com/watch?v=wxXALQNWFcc Click this link] to see a video demonstrating the functionality of the review bidding feature.

===RSpec Testing===
One of the primary design goals when doing this refactoring was to ensure that all test cases still passed after all changes were made.

There was a single change that needed to be made to the RSpec files for the review bidding algorithm, which was the removal of a test relying on the 'create' method of the review_bids_controller. As previously mentioned, this function was removed because it had become obsolete. Thus, the test case for this function (which simply checked if it redirected to the right page) was no longer necessary, and thus was deleted.

Otherwise, the test cases for both the review_bids_controller and the review_bids model were manually run and found to have no errors, as shown below.

[[File:Bidding_test_results.png|800px]]

For details on what these specific tests were, please refer to the previous work referenced before.

==Conclusions and Future Work==
While the current state of the review bidding code is operational, and is certainly more DRY and stable than the previous incarnation of this work, there are still several issues with the current state of this feature that make it difficult to recommend an immediate merge into expertiza.

===Enhancement of Algorithm Service Stability===

Across the process of refining the refactored front end page view along with more robust edge cases testing towards the algorithm service, we have discovered an underlying issue that could impact the availability of the review bidding algorithm service. Through cause isolation, we have observed a high correlation to flask application crash with the edge case scenario input where the algorithm service is trying to match a student who was not signed up with any topic with other students.

Because the scope of our project was focused on refactoring, we did not have the time or prior warning to need to address these edge cases. However, we will provide the error log for the reference of any future teams who may work on the stability improvement of the algorithm service.

<pre>
"POST /match_topics HTTP/1.1" 500 -
{'tid': [5133, 5134, 5135, None], 'users': {'45514': {'otid': 5135, 'tid': [5133, 5134], 'time': ['2021-11-30T17:57:20.000-05:00', '2021-11-30T17:57:24.000-05:00'], 'priority': [2, 1]}, '45513': {'otid': 5134, 'tid': [5133, 5135], 'time': ['2021-11-30T17:57:41.000-05:00', '2021-11-30T17:57:43.000-05:00'], 'priority': [2, 1]}, '45512': {'otid': 5133, 'tid': [5134, 5135], 'time': ['2021-11-30T17:57:58.000-05:00', '2021-11-30T17:58:00.000-05:00'], 'priority': [2, 1]}, '45515': {'otid': None, 'tid': [], 'time': [0], 'priority': [3, 1, 4, 2]}}, 'max_accepted_proposals': 3}
[2021-11-28 18:10:40,576] ERROR in app: Exception on /match_topics [POST]
Traceback (most recent call last):
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/usr/.local/lib/python3.5/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/usr/IntelligentAssignment/app/app.py", line 19, in topic_matching
return jsonify(assigned_topics_model.get_student_topic_matches())
File "/home/usr/IntelligentAssignment/app/topics_matcher.py", line 57, in get_student_topic_matches
student.accept_proposal()
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in accept_proposal
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in <lambda>
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
ValueError: None is not in list
</pre>

As we can see from the additional information provided above, the error is due to an unhandled edge case of attempting to access missing value. Our further observation also concluded that such missing value is due to the fact that one or more to-be-matched students did not have any topic signed up at the time of algorithm execution. To address the issue, a general enhancement can be implemented on the algorithm service side with Flask to increase the crash recovery stability. Further refinement of interaction flow with user in the front end can also reduce the possibility of algorithm service error.

Additional to the unhandled cases from the algorithm service, we also noticed an inconsistency of the system due to the performance limitation of the Heroku deployment.

[[File:Out_of_scope_External-Issue_E2151.gif|frameless|x1000px]]

As we can see from the above recorded screen capture animation, with the exact same input, the output and algorithm service result are unstable. Because of the performance limitation of the Heroku, sometime the Flask app that deployed were overloaded. Furthermore, we have also attempted to deploy on VCL, however, we notice the limitation of VCL deployment that will become offline once the VCL reservation expired. After all, because of the focus of our tasks is on the DRY principal optimization, the external algorithm service can be improved in the future. To access and deploy the algorithm on local environment or another environment, setup guide and resource are available on [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment] repository.

===Lack of Comprehensive Testing===

A notable contradiction that was noticed when working with this project was that the previously-mentioned failed edge cases existed, yet all tests for this feature passed. This is illustrative of the fact that this feature requires better and more exhaustive testing.

When dealing with an issue with so many factors as how students bid on reviews, an extensive suite of edge cases should be considered. Any future project group working on this function should consider the following edge cases to cover:
* Does a student who does not bid on any reviews still get reviews?
* Does a student who did not submit anything for their assignment still have a review to bid on?
* What if there are less assignments to have reviews for than the minimum number required for all students to have the minimum number of reviews?

===Conclusion===
While we believe the current state of this feature is greatly improved, with many of its DRY violations removed and several noteworthy bugs corrected, we believe the feature of allowing students to bid on what to review still needs further development before it is deployed. While the basic functionality is working under ideal situations, the issues mentioned above should be addressed before this feature is put in the hands of actual instructors, as if this feature fails, it could ruin the entire reviewing process for an assignment.

Still, we think there is notable promise to this feature, and it would be a very engaging and interesting feature if deployed successfully, so we believe develop on it should continue until it is ready for deployment.

==Github==
The Github corresponding to this task is publicly available here: https://github.com/WeiRui-Wang/expertiza

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: Geoff Garrido, Ayush Luthra, and WeiRui Wang. Our project mentor was Luis Delossantos (lgdeloss@ncsu.edu)

File:Review bid original.png

2021-11-30T01:11:01Z

Aluthra:

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-30T01:10:04Z

Aluthra: /* review_bid.rb */

==Topic Overview==
===Statement of Problem===
Reviewing is an important stage of learning and improving, both as a reviewer and a reviewee, and so allowing reviewers more freedom and choice in what they review would result in better reviews and more engaged reviewers. However, currently the only way to choose a topic to review is a first-come-first-served basis, penalizing students who do not sign up quickly by not allowing them to review the topics they would be most interesting in engaging with.

A better alternative would be to implement a bidding system, similar to that used for topics, which would allow students to establish preferences for which topics they would most like to review. A good bidding algorithm implementing with this idea would ensure that all students (who care enough to bid) receive a topic to review that they actively want.

A prior project group created a prototype for this function, allowing students to bid on what topics they wanted to review. While the functionality was as desired, the feature was not able to be merged due to having too many DRY violations. Thus, the goal of this topic was to bring this prior implementation into the current version of the expertiza beta branch, and remove these DRY violations, so that it may be merged.

As development continued, it became clear this prototype also had several bugs and issues that made it unable to work under several conditions. Thus, a secondary goal was to fix any issues which existed such that the feature would generally work with the functionality the prior implementation demonstrated.

===Prior Work===
The prior work which is the basis for this topic is linked here [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review is linked here]. This work was done in Fall of 2020, and as previously mentioned, achieved much of the desired functionality for this task. This prior work will be referenced several times within this documentation, as it has a more comprehensive explanation of the underlying design principles behind the views, controller, model, algorithm, and RSpec tests implemented. These details will only be briefly summarized here.

As previously noted, this prior implementation was rejected due to DRY violations, and so correcting these violations--and explaining how those corrections were done--will be the major focus of this project documentation.

==Explanation of Feature==

===Existing Bidding Functionality for Topics===
Currently in expertiza, students may bid on which topics they would like to work on. An image showcasing this functionality is given below:

[[File:expected_review_UI.PNG]]

Students may choose a number of topics to bid on, in order of priority, and once the deadline is reached, all students are given a topic.

Behind the scenes, this works by utilizing a modified version of the Gale-Shapely Algorithm. This algorithm seeks to find a 'stable' matching of each student to each topic, such that in the end, all students have a topic they wanted (at least to some degree), and that the preferences of students is respected as well as possible. To help break situations where there are ties in preferences, each topic assigns a Priority to each student. This Priority is determined by A. whether the student is part of a group bidding for this topic (groups getting higher priority), and B. the time at which the bid for the topic was made (earlier bidders are given Priority). Once this has been determined and the deadline for bidding has ended, the following steps occur:
# Each student is matched with each topic they most prefer.
# If the number of students assigned to a topic is above the maximum allowed for that topic, the students with the highest Priority are assigned the topic
# All maximally filled topics are removed from the pool of available topics
# All unmatched students begin this cycle again, now choosing the most preferred topic remaining
# Repeat until all students are assigned a topic
# If any matches are 'unstable' (two students would have their preferences better met if they were swapped), perform any swaps until matches are as stable as possible.

This algorithm has a time complexity of O(n^2). For more details on this algorithm and its implementation, refer to the previous year's work, where this algorithm was implemented.

===Desired Functionality for Reviews===

The desired feature for reviews should have an almost identical structure to this, with reviews instead of topics. Students should be shown a list of the topics that were worked on, and may bid on which of those topics they would like to review the most. The Priority system could be handled in the same way, with the only caveat being that a student should not be able to review their own work.

In terms of what this feature looks like, it can be enabled as an option when editing/creating a new assignment, under the "Topics" section, using interface drop down show below.

[[File:Bidding_interface.png|center|800px]]

Once this option is enabled, during the review phase of an assignment, students will be able to access a new view listing all the topics of all the submitted assignments (except for their own). They will then be able to bid on as many topics as they like, assigning them a preference, in a manner similar to the interface for topic bidding. The table that allows for this drag-and-drop priorities is shown below.

[[File:Bidding_options.png|center|900px]]

Whenever the instructor decides they would like to close the window for review bidding, they may return to the options for the given assignment and navigate to "Topics", where they will see a button marked "Run Review Algorithm". This will make a call to the external webservice to determine which students are assigned what reviews, and will then assign those reviews to all students. Once this button is pressed, it locks out the ability for students to perform any more bidding, and cannot be pressed again. The button allowing this option is shown below.

[[File:Bidding_run_algorithm.png|center|900px]]

After this point, students may navigate to the "Others Work" option under their assignment and see all of their assigned reviews. They will see up to the maximum number of allowed reviews, but will be informed of how many they are required to do (for example, a student may see 4 reviews, but will be told they are only required to complete 2).

===Use Case Diagram===
The following Use Case Diagram demonstrates the primary functionality explained in the previous section for different types of users.

[[File:Use_Case_Diagram_Rev1_E2151.jpg|center]]

==Plan and List of Work Done==

===General Design Goals===

* Move all prior work for this topic into a new version of the expertiza beta branch, without loss in functionality.
* Ensure all prior/expected functionality is working correctly, and if not, make any repairs or changes necessary.
* DRY out prior implementation's controller and model methods which have functionality that already exists within the expertiza system.
* Remove repetitive views and partials that exist by merging views for review bids into the set of views for sign up sheets, from which it uses partials.
* Ensure all present test cases still work properly after making the above changes.

===Specific Tasks Completed===

* Transferred over all code files relevant to the review bidding feature
* Re-deployed the external website which handles the review bidding algorithm
* Added that website's url to the webservices url's within expertiza
* Enhanced the stability of the webservice for the bidding algorithm by testing with edge cases and optimize the endpoint by restoring the original access endpoint reference* Analyzed the necessity of the phase finding related logic code section
* Optimized phase finding related logic by removing unused code segment
* Removed an instance variable within the ''review_bids_controller'' which would have made any practical deployment fail, removing the unnecessary functionality it was created for
* Updated method calls to ''assignment.find_current_stage'' to their new form ''assignment.current_stage''
* Removed the review_bid model method ''self.reviewer_self_topic'' and replaced its usage with the already-existing ''SignedUpTeam.topic_id'' method
* Removed DRY-violating code used to calculate the current assignment phase which was never utilized by the controller or views
* Moved several view files from the ''review_bids'' view directory into the ''sign_up_sheet'' view directory. Partials which were duplicates in the two directories (''_all_actions.html.erb'' and ''_topic_names.html.erb'' were no longer necessary and deleted
* Refactored the naming of the variable that represents the id of the reviewer object from 'reviewer' to 'reviewer_id' for clarity
* Removed the unnecessary 'create' function within the review_bids_controller
* Removed the sparse testing files under the 'test' directory in keeping with typical expertiza structure
* Removed a test case from the unit tests which relied upon the removed 'create' method of the review_bids_controller

==Initial Implementation==

A considerable amount of time and effort was necessary to review, understand, and re-implement the previous implementation of the Review Bidding feature. While the feature was only created a year ago, several underlying functions and methods have since been refactored, and so needed to be changed in the implementation. Additionally, certain parts of how the implementation was designed did not seem to be properly implemented, and so had to be revised.

===Code Transfer===
First, all files that were added in the original pull request were copied and moved to the new version of the expertiza beta branch. This includes the following files:

*'''controllers/review_bids_controller.rb''': the primary controller that handles calls to review bids functionality from views, managing them itself or directing them to the appropriate model
*'''helpers/review_bids_helper.rb''': contains helper methods for rendering the rows of topics available to choose a review for
*'''models/review_bid.rb''': contains the methods that each individual review bid knows and manages, including assigning a review to a student, getting info on the topic it applies to, etc.
*'''views/review_bids/_all_actions.html.erb''': a view partial used to render the actions available to a student on the review bidding view. Nearly identical to the _all_actions file for topic bidding.
*'''views/review_bids/_table_header.html.erb''': a view partial used to render the head of the table for review bids. Though structurally similar to that of topics reviews, contains significantly less information, as less needs to be known by a student for particular topics to review.
*'''views/review_bids/_table_line.html.erb''': a view partial used to render individual rows of the review bids table. Like the table header, is structurally similar to that of topic bidding, but functions differently.
*'''views/review_bids/show.html.erb''': the view which displays all the available topics to bid on reviewing. Uses the aforementioned partials to render this bidding table.
*'''views/review_bids/others_work.html.erb''': displays the available review work which has the user has bid for.
*'''db/migrate/20201107203204_create_review_bids.rb''': the migration which creates the review bid objects within the expertiza database.
*'''spec/controllers/review_bids_controller_spec.rb''': the unit tests for the review bids controller
*'''spec/models/review_bid_spec.rb''': the unit tests for the review bids model
*'''test/controllers/review_bids_controller_test.rb''': a more elaborate feature test for the review bids functionality
*'''test/fixtures/review_bids.yml''': fixtures that are utilized for the review_bids_controller_test
*'''test/models/review_bid_test.rb''': A more elaborate feature test of the review bid model.

In addition to adding the previous files, the following files were also added to existing pages as modifications:

*'''controllers/student_review_controller.rb''': the student review controller's "list" method now redirects to the review_bid_controller's "index" method if the assignment in question has a review_choosing_algorithm set to "Bidding" (the option an instructor sets when they wish to allow students to bid on reviews).
*'''student_task/view.html.erb''': Instead of simply linking directly to the student_review controller's "list" method, the view instead checks if the assignment this view pertains to has their review_choosing_algorithm set to Bidding. If it does, and the assignment allows students to choose the topic to review (meaning the algorithm to assign bids has not yet been run), they are redirected to the 'show' method of the review_bids_controller. Otherwise, they are redirected to the 'index' method (showing them the info on their currently available topics to review).
*'''config/webservices.yml''': added a new url for the website that is called to run the review bidding algorithm
*'''config/routes.rb''': routes added for review_bids.
*'''views/assignments/edit/_topics.html.erb''': Added a new option box which allows an instructor to choose to allow bidding to occur using the review bidding algorithm, rather than the default 'Simple Choose.' Once this option is selected, it also offers a button that, when clicked, runs the review bidding algorithm on all currently bid upon reviews.

===Website Reimplementation===
In the previous implementation, the algorithm which handles the assigning of reviews was hosted on an external website. This was done for the sake of improved performance. Thus, to get the feature in working condition, we pulled the code necessary to run the algorithm and hosted it on our own website (linked at the end of this documentation). We also made a note in the pull request that the place where this website is referenced (in the "assign_reviews_algorithm" method of the review_bids_controller) should be replaced by a proper external web tool, as is done for similar web tools in other parts of expertiza.

===Modifications to Methods===
Certain methods utilized in the original version of the webpages were changed or renamed in later versions, and so, needed to be modified. The primary example of this is the method that assignments use to determine what stage of assignment completion they are in. The previous year's implementation called this method as "assignment.find_current_stage(topic_id)," but this failed when trying to execute. After comparing how this page worked to how normal review pages work, we determined the new call to this method is "assignment.current_stage(topic_id)." All calls to this method were replaced in all files.

Another change to methods was the removal of a method from the review_bids_controller named 'create'. This method primarily served as a redirect for an aspect of the feature that was unnecessary and so removed (see the following section), that being the button to show another review. With this feature removed, this method was no longer used, and so it was removed from the controller.

===Errors Corrected===
A notable period of time was also spent on determining the purpose of a particular local area that caused issues when trying to view the bids, that being a variable named "@number_of_reviews_to_show". This variable apparently kept track of how many reviews a student needed to be shown, so that they could request another. The issue was that this was created as a local variable within the review_bids_controller, and therefore would never work outside of a manual testing environment, as the variable would need to be different from every student and every assignment. After some investigation, we realized that all this variable did was limit the number of already-assigned reviews the student was able to see (as the review bidding algorithm already assigned all possible reviews). Thus, rather than try to recreate this functionality, we simply removed it.

This means that, when the review bidding algorithm runs, a student is able to see all of the reviews assigned to them, up to the maximum number of reviews for the assignment, and therefore have no reason to ask for any more. It may be the case that this feature could be improved by adding a possibility for students to request any reviews not assigned to any other student, or that still have available slots, using the "normal" review assignment scheme. For the time being, we did not consider this part of the baseline necessary functionality, and so did not implement it.

==Refinement Implementation==

===DRY Principle and Code Duplication Refactoring===
Per comments and feedbacks on the previous implementation [https://github.com/expertiza/expertiza/pull/1822 E2085], there were 3 sections of the code that potentially were DRY violations. For each of those violations, we have had made appropriate and provided the justification on the results.

====review_bids_controller.rb====
For the refactoring of review bids controller.eb, the original issue is the DRY principle violation on the internal logic of finding the phase of assignments.
The original implementation to be refactored and optimized are shown as follow.
<pre>
# Finding the current phase that we are in
due_dates = AssignmentDueDate.where(parent_id: @assignment.id)
@very_last_due_date = AssignmentDueDate.where(parent_id: @assignment.id).order("due_at DESC").limit(1)
next_due_date = @very_last_due_date[0]
for due_date in due_dates
if due_date.due_at > Time.now
next_due_date = due_date if due_date.due_at < next_due_date.due_at
end
end
@review_phase = next_due_date.deadline_type_id
</pre>
Through some discovery process, we have found some related logics that is already implemented from the system which enable the refactoring and reuse of the method. Some code snippets are demonstrated as follow.
<pre>
def self.current_due_date(due_dates)
#Get the current due date from list of due dates
due_dates.each do |due_date|
if due_date.due_at > Time.now
current_due_date = due_date
return current_due_date
end
end
#in case current due date not found
return nil
end
</pre>
<pre>
def self.teammate_review_allowed(student)
# time when teammate review is allowed
due_date = current_due_date(student.assignment.due_dates)
student.assignment.find_current_stage == 'Finished' ||
due_date &&
(due_date.teammate_review_allowed_id == 3 ||
due_date.teammate_review_allowed_id == 2) # late(2) or yes(3)
end
</pre>
As we can observed above, both methods of <code>current_due_date</code> and <code>teammate_review_allowed</code> provides the capability of getting the <code>review_phase</code> without the repetition. Additionally, we can see that those existing implementation still have some difference in implementation and purpose to be fully reusable by the logic that find current phase of a topic because the methods provided are helper methods for broader abstraction.

Furthermore, the section that we attempted to refactoring out have a similar logic that is used to get the current stage as shown below.
<pre>
def find_current_stage(topic_id = nil)
next_due_date = DueDate.get_next_due_date(self.id, topic_id)
return 'Finished' if next_due_date.nil?
next_due_date
end
</pre>
However, it is worth pointing out that the method is not fully substitutable, particularly the internal functionality that finds the <code>@very_last_due_date</code>. A similar logic does exist that is meant to use to acquire current due date and assist with finding current stage.
<pre>
def current_stage_name(topic_id = nil)
if self.staggered_deadline?
return (topic_id.nil? ? 'Unknown' : current_stage(topic_id))
end

due_date = find_current_stage(topic_id)
unless due_date == 'Finished' || due_date.nil? || due_date.deadline_name.nil?
return due_date.deadline_name
end
current_stage(topic_id)
end
</pre>
With all the findings, we have come to conclusion that the portion of the code to be dry is in similar, but required full restructure, form of existing methods. The internal logic flow itself is not fully matching the reusable methods consists across the controllers. Because of the clarity of limited reusability of existing methods, we have come to conclusion that the portions to be refactored are not duplication of any existing methods. Additionally, after a more detailed discovery, it has come to realization that the code portion of getting the current phase of the next due date was, in fact, meant to find the last due date. After further analysis of the purpose of the code, the portion has been deleted from <code>review_bids_controller.rb</code> because there was no usage observed with the result of the execution.

====review_bid.rb====
Per the previous iteration's feedback, the review_bid.rb model file presented some DRY violations. These violations are highlighted in the method below:
 
[[File:review_bid_original.png]]
 

====Refactoring Views (_all_actions.html.erb and _topic_names.html.erb)====
The last element of the prior implementation that caused DRY violations had to do with its implementation of views. Two primary views are necessary for the review bids functionality: one which shows a table where students can bid on which topic they wish to review (rendered by a file called 'show.html.erb'), and a list of assigned reviews which a student must select from (rendered by a file called 'others_work.html.erb').

In order to render the elements in 'show.html.erb' in particular, a number of partials must be utilized:
*a partial for the table's header (_table_header.html.erb)
*a partial for rendering each of the table's rows, describing a topic to bid on (_table_line.html.erb)
*a partial to contain information related to what actions are available for a student to do for a review, like edit or view (_all_actions.html.erb)
*a partial for finding and rendering the name of each topic one must review (_topic_names.html.erb)

Both of these views and all four of these partials were contained within the "review_bids" directory within the views directory. The DRY violation incurred by these views has to do with these partials: in particular, '_topic_names.html.erb' and '_all_actions.html.erb' are near identical code copies to those within the sign_up_sheet views, used for rendering and displaying information for topic bidding and normal reviews.

In order to DRY out this portion of the code, we decided that having a separate set of views for the results related to review bids were unnecessary. Within expertiza's structure, most of the views related to students choosing which topics or reviews to work on are all contained within the sign_up_sheet views. The review_vids views were copying the partials within the sign_up_sheet views because they were using the same fundamental components. Thus, we thought the best solution would be to move the two primary views for review_bids ('show.html.erb' and 'others_work.html.erb') into the sign_up_sheet views, and leave behind all redundant partials. In doing so, the redundancy of '_all_actions.html.erb' and '_topic_names.html.erb' were eliminated.

It could be argued that this implementation could be further DRY-ed by leaving behind the other two partials, '_table_header.html.erb' and '_table_line.html.erb', as they are structurally similar to those used for topic bidding. However, this is not actually permissible, as the header and lines for topics when bidding on them for reviews are significantly different than when bidding on them for assignments. More information needs to be available when bidding on topics for an assignment, such as the number of slots and number on the waitlist, but such an idea is not applicable for bids on reviews. While it would be possible to have a single file which produces one view or another depending upon if it's bidding for topics or reviews, we believed this violated the Expert Pattern, by attempting to create a single view for two distinct purposes. Instead, we thought it better to simply have two separate partials, each which precisely gave the desired functionality. Thus, the other two partials were brought over, being renamed '_review_bids_table_header.html.erb' and '_review_bids_table_line.html.erb'.

==Tests==

Throughout the refactoring and merging of [https://github.com/expertiza/expertiza/pull/1822 E2085] to [https://github.com/expertiza/expertiza/pull/2150 E2151], there were some updates that has been made to the test suits to maintain corrosponding testing automation.

===Video Demonstration===
[https://www.youtube.com/watch?v=wxXALQNWFcc Click this link] to see a video demonstrating the functionality of the review bidding feature.

===RSpec Testing===
One of the primary design goals when doing this refactoring was to ensure that all test cases still passed after all changes were made.

There was a single change that needed to be made to the RSpec files for the review bidding algorithm, which was the removal of a test relying on the 'create' method of the review_bids_controller. As previously mentioned, this function was removed because it had become obsolete. Thus, the test case for this function (which simply checked if it redirected to the right page) was no longer necessary, and thus was deleted.

Otherwise, the test cases for both the review_bids_controller and the review_bids model were manually run and found to have no errors, as shown below.

[[File:Bidding_test_results.png|800px]]

For details on what these specific tests were, please refer to the previous work referenced before.

==Conclusions and Future Work==
While the current state of the review bidding code is operational, and is certainly more DRY and stable than the previous incarnation of this work, there are still several issues with the current state of this feature that make it difficult to recommend an immediate merge into expertiza.

===Enhancement of Algorithm Service Stability===

Across the process of refining the refactored front end page view along with more robust edge cases testing towards the algorithm service, we have discovered an underlying issue that could impact the availability of the review bidding algorithm service. Through cause isolation, we have observed a high correlation to flask application crash with the edge case scenario input where the algorithm service is trying to match a student who was not signed up with any topic with other students.

Because the scope of our project was focused on refactoring, we did not have the time or prior warning to need to address these edge cases. However, we will provide the error log for the reference of any future teams who may work on the stability improvement of the algorithm service.

<pre>
"POST /match_topics HTTP/1.1" 500 -
{'tid': [5133, 5134, 5135, None], 'users': {'45514': {'otid': 5135, 'tid': [5133, 5134], 'time': ['2021-11-30T17:57:20.000-05:00', '2021-11-30T17:57:24.000-05:00'], 'priority': [2, 1]}, '45513': {'otid': 5134, 'tid': [5133, 5135], 'time': ['2021-11-30T17:57:41.000-05:00', '2021-11-30T17:57:43.000-05:00'], 'priority': [2, 1]}, '45512': {'otid': 5133, 'tid': [5134, 5135], 'time': ['2021-11-30T17:57:58.000-05:00', '2021-11-30T17:58:00.000-05:00'], 'priority': [2, 1]}, '45515': {'otid': None, 'tid': [], 'time': [0], 'priority': [3, 1, 4, 2]}}, 'max_accepted_proposals': 3}
[2021-11-28 18:10:40,576] ERROR in app: Exception on /match_topics [POST]
Traceback (most recent call last):
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/usr/.local/lib/python3.5/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/usr/IntelligentAssignment/app/app.py", line 19, in topic_matching
return jsonify(assigned_topics_model.get_student_topic_matches())
File "/home/usr/IntelligentAssignment/app/topics_matcher.py", line 57, in get_student_topic_matches
student.accept_proposal()
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in accept_proposal
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in <lambda>
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
ValueError: None is not in list
</pre>

As we can see from the additional information provided above, the error is due to an unhandled edge case of attempting to access missing value. Our further observation also concluded that such missing value is due to the fact that one or more to-be-matched students did not have any topic signed up at the time of algorithm execution. To address the issue, a general enhancement can be implemented on the algorithm service side with Flask to increase the crash recovery stability. Further refinement of interaction flow with user in the front end can also reduce the possibility of algorithm service error.

Additional to the unhandled cases from the algorithm service, we also noticed an inconsistency of the system due to the performance limitation of the Heroku deployment.

[[File:Out_of_scope_External-Issue_E2151.gif|frameless|x1000px]]

As we can see from the above recorded screen capture animation, with the exact same input, the output and algorithm service result are unstable. Because of the performance limitation of the Heroku, sometime the Flask app that deployed were overloaded. Furthermore, we have also attempted to deploy on VCL, however, we notice the limitation of VCL deployment that will become offline once the VCL reservation expired. After all, because of the focus of our tasks is on the DRY principal optimization, the external algorithm service can be improved in the future. To access and deploy the algorithm on local environment or another environment, setup guide and resource are available on [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment] repository.

===Lack of Comprehensive Testing===

A notable contradiction that was noticed when working with this project was that the previously-mentioned failed edge cases existed, yet all tests for this feature passed. This is illustrative of the fact that this feature requires better and more exhaustive testing.

When dealing with an issue with so many factors as how students bid on reviews, an extensive suite of edge cases should be considered. Any future project group working on this function should consider the following edge cases to cover:
* Does a student who does not bid on any reviews still get reviews?
* Does a student who did not submit anything for their assignment still have a review to bid on?
* What if there are less assignments to have reviews for than the minimum number required for all students to have the minimum number of reviews?

===Conclusion===
While we believe the current state of this feature is greatly improved, with many of its DRY violations removed and several noteworthy bugs corrected, we believe the feature of allowing students to bid on what to review still needs further development before it is deployed. While the basic functionality is working under ideal situations, the issues mentioned above should be addressed before this feature is put in the hands of actual instructors, as if this feature fails, it could ruin the entire reviewing process for an assignment.

Still, we think there is notable promise to this feature, and it would be a very engaging and interesting feature if deployed successfully, so we believe develop on it should continue until it is ready for deployment.

==Github==
The Github corresponding to this task is publicly available here: https://github.com/WeiRui-Wang/expertiza

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: Geoff Garrido, Ayush Luthra, and WeiRui Wang. Our project mentor was Luis Delossantos (lgdeloss@ncsu.edu)

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-30T01:09:37Z

Aluthra: /* review_bid.rb */

==Topic Overview==
===Statement of Problem===
Reviewing is an important stage of learning and improving, both as a reviewer and a reviewee, and so allowing reviewers more freedom and choice in what they review would result in better reviews and more engaged reviewers. However, currently the only way to choose a topic to review is a first-come-first-served basis, penalizing students who do not sign up quickly by not allowing them to review the topics they would be most interesting in engaging with.

A better alternative would be to implement a bidding system, similar to that used for topics, which would allow students to establish preferences for which topics they would most like to review. A good bidding algorithm implementing with this idea would ensure that all students (who care enough to bid) receive a topic to review that they actively want.

A prior project group created a prototype for this function, allowing students to bid on what topics they wanted to review. While the functionality was as desired, the feature was not able to be merged due to having too many DRY violations. Thus, the goal of this topic was to bring this prior implementation into the current version of the expertiza beta branch, and remove these DRY violations, so that it may be merged.

As development continued, it became clear this prototype also had several bugs and issues that made it unable to work under several conditions. Thus, a secondary goal was to fix any issues which existed such that the feature would generally work with the functionality the prior implementation demonstrated.

===Prior Work===
The prior work which is the basis for this topic is linked here [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review is linked here]. This work was done in Fall of 2020, and as previously mentioned, achieved much of the desired functionality for this task. This prior work will be referenced several times within this documentation, as it has a more comprehensive explanation of the underlying design principles behind the views, controller, model, algorithm, and RSpec tests implemented. These details will only be briefly summarized here.

As previously noted, this prior implementation was rejected due to DRY violations, and so correcting these violations--and explaining how those corrections were done--will be the major focus of this project documentation.

==Explanation of Feature==

===Existing Bidding Functionality for Topics===
Currently in expertiza, students may bid on which topics they would like to work on. An image showcasing this functionality is given below:

[[File:expected_review_UI.PNG]]

Students may choose a number of topics to bid on, in order of priority, and once the deadline is reached, all students are given a topic.

Behind the scenes, this works by utilizing a modified version of the Gale-Shapely Algorithm. This algorithm seeks to find a 'stable' matching of each student to each topic, such that in the end, all students have a topic they wanted (at least to some degree), and that the preferences of students is respected as well as possible. To help break situations where there are ties in preferences, each topic assigns a Priority to each student. This Priority is determined by A. whether the student is part of a group bidding for this topic (groups getting higher priority), and B. the time at which the bid for the topic was made (earlier bidders are given Priority). Once this has been determined and the deadline for bidding has ended, the following steps occur:
# Each student is matched with each topic they most prefer.
# If the number of students assigned to a topic is above the maximum allowed for that topic, the students with the highest Priority are assigned the topic
# All maximally filled topics are removed from the pool of available topics
# All unmatched students begin this cycle again, now choosing the most preferred topic remaining
# Repeat until all students are assigned a topic
# If any matches are 'unstable' (two students would have their preferences better met if they were swapped), perform any swaps until matches are as stable as possible.

This algorithm has a time complexity of O(n^2). For more details on this algorithm and its implementation, refer to the previous year's work, where this algorithm was implemented.

===Desired Functionality for Reviews===

The desired feature for reviews should have an almost identical structure to this, with reviews instead of topics. Students should be shown a list of the topics that were worked on, and may bid on which of those topics they would like to review the most. The Priority system could be handled in the same way, with the only caveat being that a student should not be able to review their own work.

In terms of what this feature looks like, it can be enabled as an option when editing/creating a new assignment, under the "Topics" section, using interface drop down show below.

[[File:Bidding_interface.png|center|800px]]

Once this option is enabled, during the review phase of an assignment, students will be able to access a new view listing all the topics of all the submitted assignments (except for their own). They will then be able to bid on as many topics as they like, assigning them a preference, in a manner similar to the interface for topic bidding. The table that allows for this drag-and-drop priorities is shown below.

[[File:Bidding_options.png|center|900px]]

Whenever the instructor decides they would like to close the window for review bidding, they may return to the options for the given assignment and navigate to "Topics", where they will see a button marked "Run Review Algorithm". This will make a call to the external webservice to determine which students are assigned what reviews, and will then assign those reviews to all students. Once this button is pressed, it locks out the ability for students to perform any more bidding, and cannot be pressed again. The button allowing this option is shown below.

[[File:Bidding_run_algorithm.png|center|900px]]

After this point, students may navigate to the "Others Work" option under their assignment and see all of their assigned reviews. They will see up to the maximum number of allowed reviews, but will be informed of how many they are required to do (for example, a student may see 4 reviews, but will be told they are only required to complete 2).

===Use Case Diagram===
The following Use Case Diagram demonstrates the primary functionality explained in the previous section for different types of users.

[[File:Use_Case_Diagram_Rev1_E2151.jpg|center]]

==Plan and List of Work Done==

===General Design Goals===

* Move all prior work for this topic into a new version of the expertiza beta branch, without loss in functionality.
* Ensure all prior/expected functionality is working correctly, and if not, make any repairs or changes necessary.
* DRY out prior implementation's controller and model methods which have functionality that already exists within the expertiza system.
* Remove repetitive views and partials that exist by merging views for review bids into the set of views for sign up sheets, from which it uses partials.
* Ensure all present test cases still work properly after making the above changes.

===Specific Tasks Completed===

* Transferred over all code files relevant to the review bidding feature
* Re-deployed the external website which handles the review bidding algorithm
* Added that website's url to the webservices url's within expertiza
* Enhanced the stability of the webservice for the bidding algorithm by testing with edge cases and optimize the endpoint by restoring the original access endpoint reference* Analyzed the necessity of the phase finding related logic code section
* Optimized phase finding related logic by removing unused code segment
* Removed an instance variable within the ''review_bids_controller'' which would have made any practical deployment fail, removing the unnecessary functionality it was created for
* Updated method calls to ''assignment.find_current_stage'' to their new form ''assignment.current_stage''
* Removed the review_bid model method ''self.reviewer_self_topic'' and replaced its usage with the already-existing ''SignedUpTeam.topic_id'' method
* Removed DRY-violating code used to calculate the current assignment phase which was never utilized by the controller or views
* Moved several view files from the ''review_bids'' view directory into the ''sign_up_sheet'' view directory. Partials which were duplicates in the two directories (''_all_actions.html.erb'' and ''_topic_names.html.erb'' were no longer necessary and deleted
* Refactored the naming of the variable that represents the id of the reviewer object from 'reviewer' to 'reviewer_id' for clarity
* Removed the unnecessary 'create' function within the review_bids_controller
* Removed the sparse testing files under the 'test' directory in keeping with typical expertiza structure
* Removed a test case from the unit tests which relied upon the removed 'create' method of the review_bids_controller

==Initial Implementation==

A considerable amount of time and effort was necessary to review, understand, and re-implement the previous implementation of the Review Bidding feature. While the feature was only created a year ago, several underlying functions and methods have since been refactored, and so needed to be changed in the implementation. Additionally, certain parts of how the implementation was designed did not seem to be properly implemented, and so had to be revised.

===Code Transfer===
First, all files that were added in the original pull request were copied and moved to the new version of the expertiza beta branch. This includes the following files:

*'''controllers/review_bids_controller.rb''': the primary controller that handles calls to review bids functionality from views, managing them itself or directing them to the appropriate model
*'''helpers/review_bids_helper.rb''': contains helper methods for rendering the rows of topics available to choose a review for
*'''models/review_bid.rb''': contains the methods that each individual review bid knows and manages, including assigning a review to a student, getting info on the topic it applies to, etc.
*'''views/review_bids/_all_actions.html.erb''': a view partial used to render the actions available to a student on the review bidding view. Nearly identical to the _all_actions file for topic bidding.
*'''views/review_bids/_table_header.html.erb''': a view partial used to render the head of the table for review bids. Though structurally similar to that of topics reviews, contains significantly less information, as less needs to be known by a student for particular topics to review.
*'''views/review_bids/_table_line.html.erb''': a view partial used to render individual rows of the review bids table. Like the table header, is structurally similar to that of topic bidding, but functions differently.
*'''views/review_bids/show.html.erb''': the view which displays all the available topics to bid on reviewing. Uses the aforementioned partials to render this bidding table.
*'''views/review_bids/others_work.html.erb''': displays the available review work which has the user has bid for.
*'''db/migrate/20201107203204_create_review_bids.rb''': the migration which creates the review bid objects within the expertiza database.
*'''spec/controllers/review_bids_controller_spec.rb''': the unit tests for the review bids controller
*'''spec/models/review_bid_spec.rb''': the unit tests for the review bids model
*'''test/controllers/review_bids_controller_test.rb''': a more elaborate feature test for the review bids functionality
*'''test/fixtures/review_bids.yml''': fixtures that are utilized for the review_bids_controller_test
*'''test/models/review_bid_test.rb''': A more elaborate feature test of the review bid model.

In addition to adding the previous files, the following files were also added to existing pages as modifications:

*'''controllers/student_review_controller.rb''': the student review controller's "list" method now redirects to the review_bid_controller's "index" method if the assignment in question has a review_choosing_algorithm set to "Bidding" (the option an instructor sets when they wish to allow students to bid on reviews).
*'''student_task/view.html.erb''': Instead of simply linking directly to the student_review controller's "list" method, the view instead checks if the assignment this view pertains to has their review_choosing_algorithm set to Bidding. If it does, and the assignment allows students to choose the topic to review (meaning the algorithm to assign bids has not yet been run), they are redirected to the 'show' method of the review_bids_controller. Otherwise, they are redirected to the 'index' method (showing them the info on their currently available topics to review).
*'''config/webservices.yml''': added a new url for the website that is called to run the review bidding algorithm
*'''config/routes.rb''': routes added for review_bids.
*'''views/assignments/edit/_topics.html.erb''': Added a new option box which allows an instructor to choose to allow bidding to occur using the review bidding algorithm, rather than the default 'Simple Choose.' Once this option is selected, it also offers a button that, when clicked, runs the review bidding algorithm on all currently bid upon reviews.

===Website Reimplementation===
In the previous implementation, the algorithm which handles the assigning of reviews was hosted on an external website. This was done for the sake of improved performance. Thus, to get the feature in working condition, we pulled the code necessary to run the algorithm and hosted it on our own website (linked at the end of this documentation). We also made a note in the pull request that the place where this website is referenced (in the "assign_reviews_algorithm" method of the review_bids_controller) should be replaced by a proper external web tool, as is done for similar web tools in other parts of expertiza.

===Modifications to Methods===
Certain methods utilized in the original version of the webpages were changed or renamed in later versions, and so, needed to be modified. The primary example of this is the method that assignments use to determine what stage of assignment completion they are in. The previous year's implementation called this method as "assignment.find_current_stage(topic_id)," but this failed when trying to execute. After comparing how this page worked to how normal review pages work, we determined the new call to this method is "assignment.current_stage(topic_id)." All calls to this method were replaced in all files.

Another change to methods was the removal of a method from the review_bids_controller named 'create'. This method primarily served as a redirect for an aspect of the feature that was unnecessary and so removed (see the following section), that being the button to show another review. With this feature removed, this method was no longer used, and so it was removed from the controller.

===Errors Corrected===
A notable period of time was also spent on determining the purpose of a particular local area that caused issues when trying to view the bids, that being a variable named "@number_of_reviews_to_show". This variable apparently kept track of how many reviews a student needed to be shown, so that they could request another. The issue was that this was created as a local variable within the review_bids_controller, and therefore would never work outside of a manual testing environment, as the variable would need to be different from every student and every assignment. After some investigation, we realized that all this variable did was limit the number of already-assigned reviews the student was able to see (as the review bidding algorithm already assigned all possible reviews). Thus, rather than try to recreate this functionality, we simply removed it.

This means that, when the review bidding algorithm runs, a student is able to see all of the reviews assigned to them, up to the maximum number of reviews for the assignment, and therefore have no reason to ask for any more. It may be the case that this feature could be improved by adding a possibility for students to request any reviews not assigned to any other student, or that still have available slots, using the "normal" review assignment scheme. For the time being, we did not consider this part of the baseline necessary functionality, and so did not implement it.

==Refinement Implementation==

===DRY Principle and Code Duplication Refactoring===
Per comments and feedbacks on the previous implementation [https://github.com/expertiza/expertiza/pull/1822 E2085], there were 3 sections of the code that potentially were DRY violations. For each of those violations, we have had made appropriate and provided the justification on the results.

====review_bids_controller.rb====
For the refactoring of review bids controller.eb, the original issue is the DRY principle violation on the internal logic of finding the phase of assignments.
The original implementation to be refactored and optimized are shown as follow.
<pre>
# Finding the current phase that we are in
due_dates = AssignmentDueDate.where(parent_id: @assignment.id)
@very_last_due_date = AssignmentDueDate.where(parent_id: @assignment.id).order("due_at DESC").limit(1)
next_due_date = @very_last_due_date[0]
for due_date in due_dates
if due_date.due_at > Time.now
next_due_date = due_date if due_date.due_at < next_due_date.due_at
end
end
@review_phase = next_due_date.deadline_type_id
</pre>
Through some discovery process, we have found some related logics that is already implemented from the system which enable the refactoring and reuse of the method. Some code snippets are demonstrated as follow.
<pre>
def self.current_due_date(due_dates)
#Get the current due date from list of due dates
due_dates.each do |due_date|
if due_date.due_at > Time.now
current_due_date = due_date
return current_due_date
end
end
#in case current due date not found
return nil
end
</pre>
<pre>
def self.teammate_review_allowed(student)
# time when teammate review is allowed
due_date = current_due_date(student.assignment.due_dates)
student.assignment.find_current_stage == 'Finished' ||
due_date &&
(due_date.teammate_review_allowed_id == 3 ||
due_date.teammate_review_allowed_id == 2) # late(2) or yes(3)
end
</pre>
As we can observed above, both methods of <code>current_due_date</code> and <code>teammate_review_allowed</code> provides the capability of getting the <code>review_phase</code> without the repetition. Additionally, we can see that those existing implementation still have some difference in implementation and purpose to be fully reusable by the logic that find current phase of a topic because the methods provided are helper methods for broader abstraction.

Furthermore, the section that we attempted to refactoring out have a similar logic that is used to get the current stage as shown below.
<pre>
def find_current_stage(topic_id = nil)
next_due_date = DueDate.get_next_due_date(self.id, topic_id)
return 'Finished' if next_due_date.nil?
next_due_date
end
</pre>
However, it is worth pointing out that the method is not fully substitutable, particularly the internal functionality that finds the <code>@very_last_due_date</code>. A similar logic does exist that is meant to use to acquire current due date and assist with finding current stage.
<pre>
def current_stage_name(topic_id = nil)
if self.staggered_deadline?
return (topic_id.nil? ? 'Unknown' : current_stage(topic_id))
end

due_date = find_current_stage(topic_id)
unless due_date == 'Finished' || due_date.nil? || due_date.deadline_name.nil?
return due_date.deadline_name
end
current_stage(topic_id)
end
</pre>
With all the findings, we have come to conclusion that the portion of the code to be dry is in similar, but required full restructure, form of existing methods. The internal logic flow itself is not fully matching the reusable methods consists across the controllers. Because of the clarity of limited reusability of existing methods, we have come to conclusion that the portions to be refactored are not duplication of any existing methods. Additionally, after a more detailed discovery, it has come to realization that the code portion of getting the current phase of the next due date was, in fact, meant to find the last due date. After further analysis of the purpose of the code, the portion has been deleted from <code>review_bids_controller.rb</code> because there was no usage observed with the result of the execution.

====review_bid.rb====
Per the previous iteration's feedback, the review_bid.rb model file presented some DRY violations. These violations are highlighted in the method below:
 
[[File:File.jpg]]
<br

====Refactoring Views (_all_actions.html.erb and _topic_names.html.erb)====
The last element of the prior implementation that caused DRY violations had to do with its implementation of views. Two primary views are necessary for the review bids functionality: one which shows a table where students can bid on which topic they wish to review (rendered by a file called 'show.html.erb'), and a list of assigned reviews which a student must select from (rendered by a file called 'others_work.html.erb').

In order to render the elements in 'show.html.erb' in particular, a number of partials must be utilized:
*a partial for the table's header (_table_header.html.erb)
*a partial for rendering each of the table's rows, describing a topic to bid on (_table_line.html.erb)
*a partial to contain information related to what actions are available for a student to do for a review, like edit or view (_all_actions.html.erb)
*a partial for finding and rendering the name of each topic one must review (_topic_names.html.erb)

Both of these views and all four of these partials were contained within the "review_bids" directory within the views directory. The DRY violation incurred by these views has to do with these partials: in particular, '_topic_names.html.erb' and '_all_actions.html.erb' are near identical code copies to those within the sign_up_sheet views, used for rendering and displaying information for topic bidding and normal reviews.

In order to DRY out this portion of the code, we decided that having a separate set of views for the results related to review bids were unnecessary. Within expertiza's structure, most of the views related to students choosing which topics or reviews to work on are all contained within the sign_up_sheet views. The review_vids views were copying the partials within the sign_up_sheet views because they were using the same fundamental components. Thus, we thought the best solution would be to move the two primary views for review_bids ('show.html.erb' and 'others_work.html.erb') into the sign_up_sheet views, and leave behind all redundant partials. In doing so, the redundancy of '_all_actions.html.erb' and '_topic_names.html.erb' were eliminated.

It could be argued that this implementation could be further DRY-ed by leaving behind the other two partials, '_table_header.html.erb' and '_table_line.html.erb', as they are structurally similar to those used for topic bidding. However, this is not actually permissible, as the header and lines for topics when bidding on them for reviews are significantly different than when bidding on them for assignments. More information needs to be available when bidding on topics for an assignment, such as the number of slots and number on the waitlist, but such an idea is not applicable for bids on reviews. While it would be possible to have a single file which produces one view or another depending upon if it's bidding for topics or reviews, we believed this violated the Expert Pattern, by attempting to create a single view for two distinct purposes. Instead, we thought it better to simply have two separate partials, each which precisely gave the desired functionality. Thus, the other two partials were brought over, being renamed '_review_bids_table_header.html.erb' and '_review_bids_table_line.html.erb'.

==Tests==

Throughout the refactoring and merging of [https://github.com/expertiza/expertiza/pull/1822 E2085] to [https://github.com/expertiza/expertiza/pull/2150 E2151], there were some updates that has been made to the test suits to maintain corrosponding testing automation.

===Video Demonstration===
[https://www.youtube.com/watch?v=wxXALQNWFcc Click this link] to see a video demonstrating the functionality of the review bidding feature.

===RSpec Testing===
One of the primary design goals when doing this refactoring was to ensure that all test cases still passed after all changes were made.

There was a single change that needed to be made to the RSpec files for the review bidding algorithm, which was the removal of a test relying on the 'create' method of the review_bids_controller. As previously mentioned, this function was removed because it had become obsolete. Thus, the test case for this function (which simply checked if it redirected to the right page) was no longer necessary, and thus was deleted.

Otherwise, the test cases for both the review_bids_controller and the review_bids model were manually run and found to have no errors, as shown below.

[[File:Bidding_test_results.png|800px]]

For details on what these specific tests were, please refer to the previous work referenced before.

==Conclusions and Future Work==
While the current state of the review bidding code is operational, and is certainly more DRY and stable than the previous incarnation of this work, there are still several issues with the current state of this feature that make it difficult to recommend an immediate merge into expertiza.

===Enhancement of Algorithm Service Stability===

Across the process of refining the refactored front end page view along with more robust edge cases testing towards the algorithm service, we have discovered an underlying issue that could impact the availability of the review bidding algorithm service. Through cause isolation, we have observed a high correlation to flask application crash with the edge case scenario input where the algorithm service is trying to match a student who was not signed up with any topic with other students.

Because the scope of our project was focused on refactoring, we did not have the time or prior warning to need to address these edge cases. However, we will provide the error log for the reference of any future teams who may work on the stability improvement of the algorithm service.

<pre>
"POST /match_topics HTTP/1.1" 500 -
{'tid': [5133, 5134, 5135, None], 'users': {'45514': {'otid': 5135, 'tid': [5133, 5134], 'time': ['2021-11-30T17:57:20.000-05:00', '2021-11-30T17:57:24.000-05:00'], 'priority': [2, 1]}, '45513': {'otid': 5134, 'tid': [5133, 5135], 'time': ['2021-11-30T17:57:41.000-05:00', '2021-11-30T17:57:43.000-05:00'], 'priority': [2, 1]}, '45512': {'otid': 5133, 'tid': [5134, 5135], 'time': ['2021-11-30T17:57:58.000-05:00', '2021-11-30T17:58:00.000-05:00'], 'priority': [2, 1]}, '45515': {'otid': None, 'tid': [], 'time': [0], 'priority': [3, 1, 4, 2]}}, 'max_accepted_proposals': 3}
[2021-11-28 18:10:40,576] ERROR in app: Exception on /match_topics [POST]
Traceback (most recent call last):
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/usr/.local/lib/python3.5/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/usr/.local/lib/python3.5/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/usr/IntelligentAssignment/app/app.py", line 19, in topic_matching
return jsonify(assigned_topics_model.get_student_topic_matches())
File "/home/usr/IntelligentAssignment/app/topics_matcher.py", line 57, in get_student_topic_matches
student.accept_proposal()
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in accept_proposal
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
File "/home/usr/IntelligentAssignment/app/student.py", line 24, in <lambda>
self.proposals.sort(key=lambda proposal: self.choices.index(proposal))
ValueError: None is not in list
</pre>

As we can see from the additional information provided above, the error is due to an unhandled edge case of attempting to access missing value. Our further observation also concluded that such missing value is due to the fact that one or more to-be-matched students did not have any topic signed up at the time of algorithm execution. To address the issue, a general enhancement can be implemented on the algorithm service side with Flask to increase the crash recovery stability. Further refinement of interaction flow with user in the front end can also reduce the possibility of algorithm service error.

Additional to the unhandled cases from the algorithm service, we also noticed an inconsistency of the system due to the performance limitation of the Heroku deployment.

[[File:Out_of_scope_External-Issue_E2151.gif|frameless|x1000px]]

As we can see from the above recorded screen capture animation, with the exact same input, the output and algorithm service result are unstable. Because of the performance limitation of the Heroku, sometime the Flask app that deployed were overloaded. Furthermore, we have also attempted to deploy on VCL, however, we notice the limitation of VCL deployment that will become offline once the VCL reservation expired. After all, because of the focus of our tasks is on the DRY principal optimization, the external algorithm service can be improved in the future. To access and deploy the algorithm on local environment or another environment, setup guide and resource are available on [https://github.com/uahamedncsu/IntelligentAssignment IntelligentAssignment] repository.

===Lack of Comprehensive Testing===

A notable contradiction that was noticed when working with this project was that the previously-mentioned failed edge cases existed, yet all tests for this feature passed. This is illustrative of the fact that this feature requires better and more exhaustive testing.

When dealing with an issue with so many factors as how students bid on reviews, an extensive suite of edge cases should be considered. Any future project group working on this function should consider the following edge cases to cover:
* Does a student who does not bid on any reviews still get reviews?
* Does a student who did not submit anything for their assignment still have a review to bid on?
* What if there are less assignments to have reviews for than the minimum number required for all students to have the minimum number of reviews?

===Conclusion===
While we believe the current state of this feature is greatly improved, with many of its DRY violations removed and several noteworthy bugs corrected, we believe the feature of allowing students to bid on what to review still needs further development before it is deployed. While the basic functionality is working under ideal situations, the issues mentioned above should be addressed before this feature is put in the hands of actual instructors, as if this feature fails, it could ruin the entire reviewing process for an assignment.

Still, we think there is notable promise to this feature, and it would be a very engaging and interesting feature if deployed successfully, so we believe develop on it should continue until it is ready for deployment.

==Github==
The Github corresponding to this task is publicly available here: https://github.com/WeiRui-Wang/expertiza

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: Geoff Garrido, Ayush Luthra, and WeiRui Wang. Our project mentor was Luis Delossantos (lgdeloss@ncsu.edu)

CSC/ECE 517 Fall 2020 - E2085. Allow reviewers to bid on what to review

2021-11-28T20:34:59Z

Aluthra: /* Method: assign_review_topics */

==Project Overview==
=== ''Background'' ===
Assigning reviews to users is a complicated process. Currently, reviews are still assigned using a first-come-first-served basis. However, reviews could be bid on like how topics are bid on. This would involve matching multiple students to review a submission up to the maximum reviewers for a submission.The manual selection process can be replaced with an automated bidding approach for this purpose.

=== ''Objectives'' ===
# Implement top trading cycles on a web service.
# Add front end code to allow bidding on topics and call the appropriate web service from the lottery controller.
# Add working test cases to test the new changes.
# Code should be inclined to the Beta branch
# Link to run bidding assignment should be on the page associated with the assignment actions.
# Code such as getter methods should be added to the model and not the controller.
#The number of reviews created by default needs to be checked. One way of achieving this is creating minimum reviews and then allowing users to request more reviews if required.

=== ''Previous Work'' ===
There have been two previous implementations:
* Fall 2019 [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1986._Allow_reviewers_to_bid_on_what_to_review here]
* Spring 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Spring_2020_-_E2021._Allow_reviewers_to_bid_on_what_to_review here]

=== ''Relevant Links'' ===
''Github Repository for Expertiza'': https://github.com/uahamedncsu/expertiza/tree/beta 
''Github Repository for Web service'': https://github.com/uahamedncsu/IntelligentAssignment 
''Pull Request'': https://github.com/expertiza/expertiza/pull/1822 
''Expertiza Server'': http://152.7.99.70:8081/ 
''Web Service Server'': https://app-csc517.herokuapp.com/match_topics 
''Video Demo'': https://www.screencast.com/t/GJNOcfa9Ln 

=== ''Files Modified'' ===
This is a list of all the files added or changed in completing this project. 
 
''Changed'': 
*config/routes.rb 

*config/webservices.yml

*app/controllers/student_review_controller.rb 

*app/views/assignments/edit/_topics.html.erb 

*app/views/student_task/list.html.erb 

*app/views/student_task/view.html.erb 

*spec/factories/factories.rb 

 
''Added'': 
*app/controllers/review_bids_controller.rb 

*app/helpers/review_bids_helper.rb 

*app/models/review_bid.rb 

*app/views/review_bids/_all_actions.html.erb 

*app/views/review_bids/_table_header.html.erb 

*app/views/review_bids/_table_line.html.erb 

*app/views/review_bids/_topic_names.html.erb 

*app/views/review_bids/others_work.html.erb 

*app/views/review_bids/show.html.erb 

*db/migrate/20201107203204_create_review_bids.rb 

*spec/controllers/review_bids_controller_spec.rb 

*spec/models/review_bid_spec.rb 

==Design==

=== ''User Interface'' ===

Currently, the users are offered the entire list of topics to review and are asked to select a topic to begin a review. Topics that cannot be reviewed or have already been reviewed the maximum number of times are greyed out and cannot be selected. There is also a check box in which the student can select "I don't care which topic I review" and will be randomly assigned a topic from the list. The current User Interface (UI) for this is shown below.
 
 
[[File:current_review_UI.PNG]]

 

As discussed in the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Project_Overview Project Overview] section, a bidding system similar to the bidding system used in topic selection is preferable to the current review selection system. The UI for this process will mirror the UI for the topic bidding system. The new UI will allow users to drag a topic from a list of all the topics on the left, to their priority list on the right. The ordering of the right list indicates the priority of the users preference. The colors indicate the likelihood of being assigned that topic. By utilizing the same UI, the consistency of the user experience and the DRY-ness of the code are increased. An example of the topic selection bidding UI is shown below.
 
 
[[File:expected_review_UI.PNG]]

 

Currently, an instructor has the option while creating or editing an assignment to allow a reviewers to select the topics they want to review. This is done by checking '''Has teams?''' and '''Has topics?''' in the '''General''' tab of an assignment then checking '''Allow reviewer to choose which topic to review?''' in the '''Topics''' tab. This interface can be used for allowing review bidding either by editing the current check box or adding an additional check box for '''Allow reviewer to bid on topics to review?'''.
 
 
[[File:Instructor_ui_for_review_bidding.png|1000px]]

=== ''Analyzing the problem'' ===

Currently, the student can either pick a topic or is randomly assigned a topic, from among the ones that are currently eligible to be reviewed. In this scenario, it is possible that some good topics are already picked by a lot of students and if someone is late to do the selecting, they miss out on it since it is grayed out.
 
To solve the above problem of not getting the topics they like, the participants must be assigned topics based on a list of topics they pre-select.
This is where an algorithm will come in handy. For picking a suitable algorithm, we should first identify the type of the problem.
 
Type of problem 
Each student should be able to bid on multiple assignments of their choice and the same assignment can be assigned to multiple participants for review. 
This requires a Many to Many mapping.

[[File:many2.png]]

The algorithm of choice for this type of stable allocations is Gale-Shapely algorithm. 

In the real world, there are cases where we face this problem:
<ul><li> Student choosing their future universities [https://www.jstor.org/stable/3132114?seq=1&cid=pdf-reference#references_tab_content '''School Choice Problem'''] (One-to-Many)</li>
<li> Men and women seeking each other to create families [https://en.wikipedia.org/wiki/Stable_marriage_problem '''Stable Marriage Problem'''] (One-to-One)</li>
<li> Internet services that need to be connected with users with the smallest amount of time </li></ul>

===''Algorithm Implementation for matching students and topics''===
The Gale-Shapely Algorithm has steps to make sure it is a Stable marriage/allocation.
An unstable allocation is when the participant does not like the topic assigned to them.
But it can happen that the participant 1 might like the review topic assigned to participant 2 and participant 2 likes the topic assigned to participant 1.
This is the unstable case. To make it stable, the topics need to be exchanged between these 2 participants so that both of them get the topic they wanted.

[[File:Snippy2.png]]
Source: [https://www.journalajrcos.com/index.php/AJRCOS/article/view/24749/46281 Stable Marriage Algorithm]

===''Assignment of priority ''===
When a student places a bid, the timestamp is recorded. Priority will be given to the earlier bidder. If the student had chosen only very few bids, then this person will be given a higher priority. Any topics that were not selected by the participant are added to bottom of their preference list in a random preference order to create a situation where the number of students and number of topics is equal. 

===''Algorithm Steps ''===
<li> For the first round, the Algorithm will assign topics to each student based on the Priority and each topic is assigned to a limited number of students based on the Limit. It picks one topic and begins assigning students to it. (Note: The Algorithm makes sure the participant is not in a group that worked on this particular topic.)
<li>If the one topic is assigned to enough participants already, then it is removed from the queue and the same process is repeated for the next topic.
<li> As this goes on, all students are matched even if all matches are not stable at this point.
<li> The Algorithm terminates when all Topics that have not reached the maximum assignment level have tried to accept every student.

 
 
The time complexity of this algorithm is O(n^2) 

===''Simulation ''===

[[File:gale-shapeley-1.png]]
[[File:gale-shapeley-2.png]]

In the simulation above, we have equal number of Students (S1, S2, S3) and Review topics to assign (R1, R2, R3). The students preference of topics is to left of student. The Review topics does not have any preferences. To have preferences, the timestamps are used to determine which student bid on each topic first to create such a Preference list for topics.

# Phase 1: Both Students S1 and S2 have selected R2 as their highest preference. Since there is a conflict, The preference of R2 is checked which shows that S2 had bid earlier than S1. Thus, the topic R2 is assigned to student S2.
# Phase 2: Now, S1 and S3 both want to review R1 but R1 was first bid on by S1 as shown in the preference. Hence, R1 is assigned to S1.
# Phase 3: The only topic remaining in this case is R3 which is also on the preference list of Student S3. Therefore S3 is assigned the topic R3.
#By the end of the 3 phases, All students are assigned to a Review. The same process needs to be repeated till the required number of reviews are assigned to each student. In each iteration, if a review is assigned to max number of students already, the review is removed from consideration.

==Implementation==
This section details the implementation of the code and will highlight each component of the design in the code and its intended purpose, as well as any modifications or testing. As the implementation progresses this section will be updated.

=== ''Overview'' ===

This project was split into 2 objectives, the Expertiza MVC implementation and the webservice algorithm. The Expertiza MVC follows the MVC standard such that the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Controller controller] handles directing the users to the correct views, as well as handling the communication with the webservice. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Model model] performs all necessary computations and data handling like the creation and storing of bids on reviews and assigning reviews to students. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Views views] render the corresponding webpage based on the actions of the user. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Webservice webservice] contains the algorithm outlined in the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Algorithm_Implementation_for_matching_students_and_topics Algorithm Implementation] section of Design. Our Expertiza controller sends this webservice bidding information and the algorithm returns the matched topics to be assigned.

 

=== ''Controller'' ===
The controller contains 6 methods that perform all the necessary functions. The first method, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_action_allowed? "action_allowed?"] performs authentication checks The second, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_index "index"], is used to redirect the user to a list of assigned reviews after the algorithm has been run. The third, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_show "show"], is used in conjunction with fourth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_set_priority "set_priority"], to update and store the bids when users make changes in the bidding page. The fifth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"], utilizes the sixth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_run_bidding_algorithm "run_bidding_algorithm"], to connect to the webservice, send and receive the correct bidding information for the algorithm.
 

====== Method: action_allowed? ======
This method performs authentication by allowing only certain types of users to access controller methods. The first portion of this method allows users 'Instructor', 'Teaching Assistant', 'Administrator', 'Super-Administrator', and 'Student' to access the methods 'show', 'set_priority', and 'index'. These are the methods that render the bidding pages and review listings. The other methods are restricted for 'Student' users.
 
[[File:Controller action allowed..PNG]]
 

====== Method: index ======
This method collects all the necessary information about the user, Assignment details, and Review Mapping details to render the 'review_bids/others_work', which lists the reviews that are assigned to the user. This page also includes a button to request more reviews which will add reviews up to the maximum number of reviews. To see the view rendered by this method, please go to [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#View:_others_work.html.erb "others_work"].
 
[[File:Controller_index.PNG]]
 

====== Method: show ======
This method renders the bidding page in which participants can select from a table of topics the topics they would prefer to review. It collects the necessary information about the user, Assignment, Topics, and current Review Bids to render this page. It is used in conjunction with [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_set_priority set_priority] where the users bidding information is stored, and the page updated. To see the bidding page that this function renders, visit the view [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#View:_show.html.erb "show"] 
[[File:Controller_show.PNG]]

 

====== Method: set_priority ======
In the view file [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#view:_show.html.erb "show"] there exists a line the calls this method when the page is updated or accessed. This method stores the user's bidding information in the schema database. This allows the model to access this information when collecting all the bidding information, as well as allowing users to refresh or return to this page with their selections staying in the bid table.
 
[[File:Controller_set_priority.PNG]]

 

====== Method: assign_bidding ======
This method utilizes methods in the model, as well as the method [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_run_bidding_algorithm "run_bidding_algorithm"] to collect the bidding information of all students of a particular Assignment, send it to the webservice algorithm, store the matched topics that are returned, and pass these matched topics to the model to be assigned. After this method is run, the students should have assigned reviews and bidding should be turned off.
 
[[File:Controller_assign_bidding.PNG]]
 

====== Method: run_bidding_algorithm ======
This method connects to the webservice using RestClient API, which is part of the 'require's at the top of the Controller. It will return a false if the connection fails for ay reason such as connection is not established or the algorithm has an internal error, otherwise it returns the matched topics from the algorithm as a 'json'. 
[[File:Controller_requirements.PNG]] 
[[File:Controller_run_bidding_algorithm.PNG]]
 

=== ''Model '' ===
The model contains 5 methods that are utilized by the controller to save bidding information per user or all users, retrieve bidding information per user or for all users, and assign topics to review to participants.
 

====== Method: get_bidding_data ======
This method is called by the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"] controller method. It receives the Assignment ID and a list of reviewers, which are student IDs. It creates a Hash to store bidding data into and iterates through all of the reviewers, calling [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_reviewer_bidding_data "reviewer_bidding_data"] to get each students bidding data and store it in the Hash. It returns a Hash with all the bidding data needed for the Webservice Algorithm to run.
 
[[File:Model_get_bidding_data.PNG]]
 

====== Method: assign_review_topics ======
This method is called by the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"] controller method. Given the Assignment ID, list of reviewers, and the matched topics returned by the webservice algorithm, it first clears any previously assigned reviews for that Assignment to prevent multiple assigns, then iterates through each reviewer and assigns them to the topics that the algorithm selected for them.
 
[[File:Model_assign_review_topics.PNG]]
 

====== Method: assign_topic_to_reviewer ======
This method is called by the method [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_review_topics "assign_review_topics"] and assigns a single topic to a single user to review given an Assignment ID.
[[File:Model_assign_topic_to_reviewer.PNG]]
 

====== Method: reviewer_self_topic ======
This method returns the topic id of the reviewer that is passed in. This method is utilized to retrieve the reviewers' Topic ID, which is needed for the webservice algorithm to prevent users from being assigned to review their own topic. This is also prevented in views, but this is an additional safety precaution. 
[[File:Model_reviewer_self_topic.PNG]]
 

====== Method: reviewer_bidding_data ======
This method is called by [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_get_bidding_data "get_bidding_data"]. It creates a hash to store all the reviewer's bidding information. It iterates through all the bids for a reviewer given the Assignment ID and it returns a Hash of the bidding data.
 
[[File:Model_reviewer_bidding_data.PNG]]
 

=== ''Views'' ===
There are two main views for this project. They are for bidding and for showing assigned reviews after the bidding algorithm has been run.
====== View: others_work.html.erb ======
This file renders the view where students see a list of the reviews they've been assigned, with appropriate links and related messages. It includes a "Request Additional Review" button that will allow students to view more reviews up to the max number of reviews allowed.
[[File:Views_others_work_pt1.PNG]] 
[[File:Views_others_work_pt2.PNG]] 
[[File:Views_others_work_pt3.PNG]] 
[[File:Views_others_work_pt4.PNG]] 
 
 
View rendered by above: 
[[File:Views_others_work_rendered.PNG]]

====== View: show.html.erb ======
This view file renders the Review Bidding page where participants can select which topics to review. It renders partials which are included below, and includes coloring effects from helper methods indicating likelihood of receiving that particular topic to review. 
[[File:Views_show_pt1.PNG]] 
[[File:Views_show_pt2.PNG]] 
 
 
Page rendered by above: (This rendering is an instructor view, the student view is the same without the "Edit Topics" column.) 
[[File:Views_show_rendered.PNG|900px]]
 

=== ''Webservice'' ===
===== ''app'' =====

The entry point to the Web service is the app.py file where we have the POST Method match_topics.
This receives the input in a json format from Expertiza. This json will have a list of all topics, list of all students and the topics each student selected and also the maximum number of review topics per student. Also, it has the timestamp information of the bidding time to give preference to early bidders. The input json is passed to the JsonParser class to create a Dictionary.

 
[[File:tab.jpg]]

 

===== ''json parser'' =====
The json parser converts the json into a Dictionary. The parser also creates a preferences dictionary for the Student to store their topic preferences and
one for Topics to store list of topics which have been selected by students and also the priority for the selection using the Timestamps.
The json parser does one more task of finding Popular topics. This is needed because topics that are in demand should be preserved for students who bid on it and to avoid assigning it to students who did not bid on anything. High demand topics are assigned a lower priority for such students so that there are other topics they can be assigned.

[[File:json1.png]]

 

[[File:json2.png]]

 

[[File:json3.png]]

 

===== ''Class Topic'' =====
 
The Topic class is the one that does the proposing to the students during the runtime.
There are 2 cases. In case 1 when the particular topic is asked by more students than the limit of number of students the one topic can be assigned to. In this case, it uses the priorities it already has in the dictionary to make sure the topic proposes only to those students who chose it and those who bid earlier.
In Case 2, There are less students who asked for the topic. Hence, the topic can propose to all the students who asked for it and still be under the limit. In the end of this step. All topics make their proposals to be assigned to some students.

[[File:json11.png]]

 

===== ''Class Student'' =====
 
Once the Topics propose to students, the Student has to accept some of the proposals.
First, each student sorts the proposals based on the students preferences. Now, the student accepts all the proposals they can upto the number of slots remaining for them. The max slots for the student is determined by the max_proposals_accepted field in the json. Once the acceptance is complete, the Topics are all updated so that the topics can adjust their remaining slots accordingly.

[[File:json12.png]]

===== ''Topic Matcher'' =====

Next, The Topics Matcher class is called which is the main class that runs the Algorithm.
Before running the algorithm, it creates a list of Student and Topic objects by using instances of Student and Topic class. Each student and Topic now has its own class so that it can be used to make proposals for assignment of topics.
 

[[File:json4.png]]

 
The get_student_topic_matches() method kicks off the Algorithm.
Each topic proposes to students based on the topic priorities as covered in the Topics section. It uses a max_topic_assignment_limit (say 4 for example) to make sure the same topic is not assigned to too many students.
Once the proposals are made, we iterate over students so that they can accept proposals as long as they have slots remaining to be assigned to.
 
The round is now complete. To determine if we need a next round, we ask the topics if they are done proposing using is_topics_done_proposing() method. As long as each topic has not reached its assignment limit and there are still students to propose to, the proposing continues for the next round.
Once every topic is done, it breaks out of the loop and we now have the list of students with equal number of assigned topics ( 3-4 for example) and this list is returned back to Expertiza as the Stable matching to do the assignment.

[[File:json5.png]]

===== ''Test Input'' =====

Sample Test case: 
Case: There are a few very popular topics and the students who bid for them and bid early should receive it.
Popular Topics: 3969, 3971, 3972

<code>{
"tid": [
3969,3970,3971,3972,3973,3974,3975,3976,3977
],
"users": {
"36239": {
"tid": [
3970, 3972
],
"otid": 3977,
"priority": [
2,1
],
"time": [
"Thu, 12 Nov 2020 12:01:06 EST -05:00","Thu, 12 Nov 2020 12:01:07 EST -05:00"
]
},
"36240": {
"tid": [],
"otid": 3976,
"priority": [],
"time": []
},
"36241": {
"tid": [
3969,3971,3972
],
"otid": 3975,
"priority": [
1,3,2
],
"time": [
"Thu, 12 Nov 2020 12:00:22 EST -05:00","Thu, 12 Nov 2020 12:00:25 EST -05:00","Thu, 12 Nov 2020 12:00:27 EST -05:00"
]
},
"36242": {
"tid": [
3969,3971,3973
],
"otid": 3974,
"priority": [
3,2,1
],
"time": [
"Wed, 11 Nov 2020 12:15:43 EST -05:00","Thu, 12 Nov 2020 11:59:40 EST -05:00","Thu, 12 Nov 2020 13:07:53 EST -05:00"
]
},
"36243": {
"tid": [
3971,3969,3970,3976
],
"otid": 3972,
"priority": [
4,3,2,1,5
],
"time": [
"Wed, 11 Nov 2020 11:34:50 EST -05:00","Wed, 11 Nov 2020 12:30:16 EST -05:00","Wed, 11 Nov 2020 12:30:19 EST -05:00","Thu, 12 Nov 2020 13:02:02 EST -05:00"
]
}
},
"max_accepted_proposals": 3
} </code>

 

===== ''Output'' =====

Since there is competition for these topics, the algorithm recognizes this and makes sure to assign them only to the students who bid for it.
Other students who picked other topics or did not pick anything are assigned different topics.

<code> {
"36239": [
3970,
3972,
3975
],
"36240": [
3973,
3974,
3972
],
"36241": [
3969,
3971,
3972
],
"36242": [
3969,
3971,
3973
],
"36243": [
3969,
3970,
3971
]
}</code>

==Test Plan==
This section outlines the test plan for this project including testing scenarios/edge cases, the manual testing plan, and the automatic/RSpec testing plan.

=== ''Testing Scenarios'' ===
This section outlines the various bidding scenarios that could occur. These scenarios should be tested in either '''Manual Testing''' or '''Automatic Testing'''.

==== Basic Bidding Scenario ====
*Reviewers bid on multiple projects at different times
** Bidding algorithm should give priority to reviewers that bid first and should assign based on reviewers ranking

==== Edge Cases ====
* All reviewers bid on the same project
** If different time stamps: bidding algorithm should give priority to reviewers that bid first
** If same time stamps: bidding algorithm should assign reviews randomly
* None of the reviews bid on any projects
** Bidding algorithm should assign reviews randomly
* Reviewer bids on their own project
** Bidding algorithm should not allow a reviewer to bid on their own project and should have validation to prevent assigning a reviewer their own project
* None of the reviewers bid on a specific project
** Bidding algorithm should still assign reviewers to this project

=== ''Manual Testing'' ===
Manual testing should be preformed on an Expertiza server and webserver platform 

==== Manual Testing Plan for Expertiza Server ====
* test that an instructor can allow review bidding for an assignment
* test that server routes correctly when review bidding is allowed
** test that task box links to bidding page before algorithm is run
** test that assignment->others work links to bidding page before algorithm is run
** test that task box links to review page with bidding information after algorithm is run
** test that assignment->others work links to review page with bidding information after algorithm is run
* test that the bidding UI is implemented as expected
* test that a participant can bid on projects to review
** test that a participant can move projects into the bidding column
** test that a participant can reorder projects in the bidding column
** test that a participant's bidding preferences save
** test that a participant can't bid for their own topic
* test that run algorithm button calls webserver with correct information and returns review assignments for each user

==== Directions for Manual Testing of Expertiza Server ====
1. Go to [http://152.7.99.70:8081/ testing server] 
2. Log in as username: '''instructor6''' password: '''password''' 
3. Go to '''Manage''' -> '''Assignments''' -> '''E2085 Manual Testing''' -> '''edit''' (pencil symbol)
:# In the '''General''' tab make sure '''Has teams?''' and '''Has topics?''' are checked
:# In the '''Topics''' tab check '''Allow review to choose which topic to review?''' then click '''save'''
:# In the '''Topics''' tab select '''Bidding''' then click '''save'''
4. Go to '''Assignments'''
:# '''E2085 Manual Testing review''' task should be listed in the main assignments box
::# This UI is based on '''instructor6''' being a '''participant''' assigned to topic '''Topic4'''
::# The UI blocks user from bidding on their own project
::# Check that user can bid on topics and reorder bids
::# Check that users topic preferences save on page refresh
::# Impersonate other participants and repeat bidding UI tests ('''student7601''', '''student7602''', '''student7603''', '''student7604''', '''student7605''', '''student7606''', '''student7607''', '''student7608''', '''student7609'')
5. Go to '''Manage''' -> '''Assignments''' -> '''E2085 Manual Testing''' -> '''edit''' (pencil symbol)
:# In the '''Topics''' tab click '''Run Review Algorithm''' button
6. Go to '''Assignments'''
:# '''E2085 Manual Testing review''' task should be listed in the main assignments box
::# '''E2085 Manual Testing review''' should now link to the reviewing page
::# The number of required reviews should already be displayed
::# A '''Request Another Review''' button should be present unless reviews listed is equal to the number of reviews allowed
::# '''Begin''' a review and test '''save''' and '''submit'''

==== Manual Testing Demo Video for Expertiza Server ====
Video demonstration of manual testing: [https://www.screencast.com/t/0rSg1zDW here]

=== ''Automatic/RSpec Testing '' ===
RSpec tests will need to be written to related controller(s) and model(s). 

==== RSpec Testing Plan ====
These RSpec tests:
* test basic functionality of review bids controller
* test basic functionality of review bid model

==== Directions for RSpec Testing ====
:# [https://docs.google.com/document/d/1tXmwju6R7KQbvycku-bdXxa6rXSUN4BMyvjY3ROmMSw/edit Set up Expertiza] on personal device using [https://github.com/uahamedncsu/expertiza forked repo]
:# Run the following commands:

# review bids controller:
bundle exec rspec spec/controllers/review_bids_controller_spec.rb

# review bid model:
bundle exec rspec spec/models/review_bid_spec.rb

==== RSpec Testing Demo Video ====
Video demonstration of RSpec tests passing: [https://www.screencast.com/t/uSWZHSX8s here]
 

==Team==
Ummu Kolusum Yasmin Ahamed Adam (uahamed) 
Ryan Grainger (rpgraing) 
Luis Delossantos (lgdeloss) 
Colleen "Bria" Engen (ceengen) 
'''Mentor:''' Saurabh Shingte (svshingt)

 

==Resources==
''Previous Implementations'': [[http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1986._Allow_reviewers_to_bid_on_what_to_review 1]], [[https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Spring_2020_-_E2021._Allow_reviewers_to_bid_on_what_to_review 2]] 
''Bidding Interface Implementation Pull Request'': [[https://github.com/expertiza/expertiza/pull/778 3]]

=== ''Relevant Links'' ===
''Github Repository for Expertiza'': https://github.com/uahamedncsu/expertiza/tree/beta 
''Github Repository for Web service'': https://github.com/uahamedncsu/IntelligentAssignment 
''Pull Request'': https://github.com/expertiza/expertiza/pull/1822 
''Expertiza Server'': http://152.7.99.70:8081/ 
''Web Service Server'': https://app-csc517.herokuapp.com/match_topics 
''Video Demo'': https://www.screencast.com/t/GJNOcfa9Ln

CSC/ECE 517 Fall 2020 - E2085. Allow reviewers to bid on what to review

2021-11-28T20:34:44Z

Aluthra: /* Method: get_bidding_data */

==Project Overview==
=== ''Background'' ===
Assigning reviews to users is a complicated process. Currently, reviews are still assigned using a first-come-first-served basis. However, reviews could be bid on like how topics are bid on. This would involve matching multiple students to review a submission up to the maximum reviewers for a submission.The manual selection process can be replaced with an automated bidding approach for this purpose.

=== ''Objectives'' ===
# Implement top trading cycles on a web service.
# Add front end code to allow bidding on topics and call the appropriate web service from the lottery controller.
# Add working test cases to test the new changes.
# Code should be inclined to the Beta branch
# Link to run bidding assignment should be on the page associated with the assignment actions.
# Code such as getter methods should be added to the model and not the controller.
#The number of reviews created by default needs to be checked. One way of achieving this is creating minimum reviews and then allowing users to request more reviews if required.

=== ''Previous Work'' ===
There have been two previous implementations:
* Fall 2019 [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1986._Allow_reviewers_to_bid_on_what_to_review here]
* Spring 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Spring_2020_-_E2021._Allow_reviewers_to_bid_on_what_to_review here]

=== ''Relevant Links'' ===
''Github Repository for Expertiza'': https://github.com/uahamedncsu/expertiza/tree/beta 
''Github Repository for Web service'': https://github.com/uahamedncsu/IntelligentAssignment 
''Pull Request'': https://github.com/expertiza/expertiza/pull/1822 
''Expertiza Server'': http://152.7.99.70:8081/ 
''Web Service Server'': https://app-csc517.herokuapp.com/match_topics 
''Video Demo'': https://www.screencast.com/t/GJNOcfa9Ln 

=== ''Files Modified'' ===
This is a list of all the files added or changed in completing this project. 
 
''Changed'': 
*config/routes.rb 

*config/webservices.yml

*app/controllers/student_review_controller.rb 

*app/views/assignments/edit/_topics.html.erb 

*app/views/student_task/list.html.erb 

*app/views/student_task/view.html.erb 

*spec/factories/factories.rb 

 
''Added'': 
*app/controllers/review_bids_controller.rb 

*app/helpers/review_bids_helper.rb 

*app/models/review_bid.rb 

*app/views/review_bids/_all_actions.html.erb 

*app/views/review_bids/_table_header.html.erb 

*app/views/review_bids/_table_line.html.erb 

*app/views/review_bids/_topic_names.html.erb 

*app/views/review_bids/others_work.html.erb 

*app/views/review_bids/show.html.erb 

*db/migrate/20201107203204_create_review_bids.rb 

*spec/controllers/review_bids_controller_spec.rb 

*spec/models/review_bid_spec.rb 

==Design==

=== ''User Interface'' ===

Currently, the users are offered the entire list of topics to review and are asked to select a topic to begin a review. Topics that cannot be reviewed or have already been reviewed the maximum number of times are greyed out and cannot be selected. There is also a check box in which the student can select "I don't care which topic I review" and will be randomly assigned a topic from the list. The current User Interface (UI) for this is shown below.
 
 
[[File:current_review_UI.PNG]]

 

As discussed in the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Project_Overview Project Overview] section, a bidding system similar to the bidding system used in topic selection is preferable to the current review selection system. The UI for this process will mirror the UI for the topic bidding system. The new UI will allow users to drag a topic from a list of all the topics on the left, to their priority list on the right. The ordering of the right list indicates the priority of the users preference. The colors indicate the likelihood of being assigned that topic. By utilizing the same UI, the consistency of the user experience and the DRY-ness of the code are increased. An example of the topic selection bidding UI is shown below.
 
 
[[File:expected_review_UI.PNG]]

 

Currently, an instructor has the option while creating or editing an assignment to allow a reviewers to select the topics they want to review. This is done by checking '''Has teams?''' and '''Has topics?''' in the '''General''' tab of an assignment then checking '''Allow reviewer to choose which topic to review?''' in the '''Topics''' tab. This interface can be used for allowing review bidding either by editing the current check box or adding an additional check box for '''Allow reviewer to bid on topics to review?'''.
 
 
[[File:Instructor_ui_for_review_bidding.png|1000px]]

=== ''Analyzing the problem'' ===

Currently, the student can either pick a topic or is randomly assigned a topic, from among the ones that are currently eligible to be reviewed. In this scenario, it is possible that some good topics are already picked by a lot of students and if someone is late to do the selecting, they miss out on it since it is grayed out.
 
To solve the above problem of not getting the topics they like, the participants must be assigned topics based on a list of topics they pre-select.
This is where an algorithm will come in handy. For picking a suitable algorithm, we should first identify the type of the problem.
 
Type of problem 
Each student should be able to bid on multiple assignments of their choice and the same assignment can be assigned to multiple participants for review. 
This requires a Many to Many mapping.

[[File:many2.png]]

The algorithm of choice for this type of stable allocations is Gale-Shapely algorithm. 

In the real world, there are cases where we face this problem:
<ul><li> Student choosing their future universities [https://www.jstor.org/stable/3132114?seq=1&cid=pdf-reference#references_tab_content '''School Choice Problem'''] (One-to-Many)</li>
<li> Men and women seeking each other to create families [https://en.wikipedia.org/wiki/Stable_marriage_problem '''Stable Marriage Problem'''] (One-to-One)</li>
<li> Internet services that need to be connected with users with the smallest amount of time </li></ul>

===''Algorithm Implementation for matching students and topics''===
The Gale-Shapely Algorithm has steps to make sure it is a Stable marriage/allocation.
An unstable allocation is when the participant does not like the topic assigned to them.
But it can happen that the participant 1 might like the review topic assigned to participant 2 and participant 2 likes the topic assigned to participant 1.
This is the unstable case. To make it stable, the topics need to be exchanged between these 2 participants so that both of them get the topic they wanted.

[[File:Snippy2.png]]
Source: [https://www.journalajrcos.com/index.php/AJRCOS/article/view/24749/46281 Stable Marriage Algorithm]

===''Assignment of priority ''===
When a student places a bid, the timestamp is recorded. Priority will be given to the earlier bidder. If the student had chosen only very few bids, then this person will be given a higher priority. Any topics that were not selected by the participant are added to bottom of their preference list in a random preference order to create a situation where the number of students and number of topics is equal. 

===''Algorithm Steps ''===
<li> For the first round, the Algorithm will assign topics to each student based on the Priority and each topic is assigned to a limited number of students based on the Limit. It picks one topic and begins assigning students to it. (Note: The Algorithm makes sure the participant is not in a group that worked on this particular topic.)
<li>If the one topic is assigned to enough participants already, then it is removed from the queue and the same process is repeated for the next topic.
<li> As this goes on, all students are matched even if all matches are not stable at this point.
<li> The Algorithm terminates when all Topics that have not reached the maximum assignment level have tried to accept every student.

 
 
The time complexity of this algorithm is O(n^2) 

===''Simulation ''===

[[File:gale-shapeley-1.png]]
[[File:gale-shapeley-2.png]]

In the simulation above, we have equal number of Students (S1, S2, S3) and Review topics to assign (R1, R2, R3). The students preference of topics is to left of student. The Review topics does not have any preferences. To have preferences, the timestamps are used to determine which student bid on each topic first to create such a Preference list for topics.

# Phase 1: Both Students S1 and S2 have selected R2 as their highest preference. Since there is a conflict, The preference of R2 is checked which shows that S2 had bid earlier than S1. Thus, the topic R2 is assigned to student S2.
# Phase 2: Now, S1 and S3 both want to review R1 but R1 was first bid on by S1 as shown in the preference. Hence, R1 is assigned to S1.
# Phase 3: The only topic remaining in this case is R3 which is also on the preference list of Student S3. Therefore S3 is assigned the topic R3.
#By the end of the 3 phases, All students are assigned to a Review. The same process needs to be repeated till the required number of reviews are assigned to each student. In each iteration, if a review is assigned to max number of students already, the review is removed from consideration.

==Implementation==
This section details the implementation of the code and will highlight each component of the design in the code and its intended purpose, as well as any modifications or testing. As the implementation progresses this section will be updated.

=== ''Overview'' ===

This project was split into 2 objectives, the Expertiza MVC implementation and the webservice algorithm. The Expertiza MVC follows the MVC standard such that the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Controller controller] handles directing the users to the correct views, as well as handling the communication with the webservice. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Model model] performs all necessary computations and data handling like the creation and storing of bids on reviews and assigning reviews to students. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Views views] render the corresponding webpage based on the actions of the user. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Webservice webservice] contains the algorithm outlined in the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Algorithm_Implementation_for_matching_students_and_topics Algorithm Implementation] section of Design. Our Expertiza controller sends this webservice bidding information and the algorithm returns the matched topics to be assigned.

 

=== ''Controller'' ===
The controller contains 6 methods that perform all the necessary functions. The first method, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_action_allowed? "action_allowed?"] performs authentication checks The second, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_index "index"], is used to redirect the user to a list of assigned reviews after the algorithm has been run. The third, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_show "show"], is used in conjunction with fourth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_set_priority "set_priority"], to update and store the bids when users make changes in the bidding page. The fifth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"], utilizes the sixth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_run_bidding_algorithm "run_bidding_algorithm"], to connect to the webservice, send and receive the correct bidding information for the algorithm.
 

====== Method: action_allowed? ======
This method performs authentication by allowing only certain types of users to access controller methods. The first portion of this method allows users 'Instructor', 'Teaching Assistant', 'Administrator', 'Super-Administrator', and 'Student' to access the methods 'show', 'set_priority', and 'index'. These are the methods that render the bidding pages and review listings. The other methods are restricted for 'Student' users.
 
[[File:Controller action allowed..PNG]]
 

====== Method: index ======
This method collects all the necessary information about the user, Assignment details, and Review Mapping details to render the 'review_bids/others_work', which lists the reviews that are assigned to the user. This page also includes a button to request more reviews which will add reviews up to the maximum number of reviews. To see the view rendered by this method, please go to [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#View:_others_work.html.erb "others_work"].
 
[[File:Controller_index.PNG]]
 

====== Method: show ======
This method renders the bidding page in which participants can select from a table of topics the topics they would prefer to review. It collects the necessary information about the user, Assignment, Topics, and current Review Bids to render this page. It is used in conjunction with [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_set_priority set_priority] where the users bidding information is stored, and the page updated. To see the bidding page that this function renders, visit the view [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#View:_show.html.erb "show"] 
[[File:Controller_show.PNG]]

 

====== Method: set_priority ======
In the view file [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#view:_show.html.erb "show"] there exists a line the calls this method when the page is updated or accessed. This method stores the user's bidding information in the schema database. This allows the model to access this information when collecting all the bidding information, as well as allowing users to refresh or return to this page with their selections staying in the bid table.
 
[[File:Controller_set_priority.PNG]]

 

====== Method: assign_bidding ======
This method utilizes methods in the model, as well as the method [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_run_bidding_algorithm "run_bidding_algorithm"] to collect the bidding information of all students of a particular Assignment, send it to the webservice algorithm, store the matched topics that are returned, and pass these matched topics to the model to be assigned. After this method is run, the students should have assigned reviews and bidding should be turned off.
 
[[File:Controller_assign_bidding.PNG]]
 

====== Method: run_bidding_algorithm ======
This method connects to the webservice using RestClient API, which is part of the 'require's at the top of the Controller. It will return a false if the connection fails for ay reason such as connection is not established or the algorithm has an internal error, otherwise it returns the matched topics from the algorithm as a 'json'. 
[[File:Controller_requirements.PNG]] 
[[File:Controller_run_bidding_algorithm.PNG]]
 

=== ''Model '' ===
The model contains 5 methods that are utilized by the controller to save bidding information per user or all users, retrieve bidding information per user or for all users, and assign topics to review to participants.
 

====== Method: get_bidding_data ======
This method is called by the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"] controller method. It receives the Assignment ID and a list of reviewers, which are student IDs. It creates a Hash to store bidding data into and iterates through all of the reviewers, calling [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_reviewer_bidding_data "reviewer_bidding_data"] to get each students bidding data and store it in the Hash. It returns a Hash with all the bidding data needed for the Webservice Algorithm to run.
 
[[File:Model_get_bidding_data.PNG]]
 

====== Method: assign_review_topics ======
This method is called by the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"] controller method. Given the Assignment ID, list of reviewers, and the matched topics returned by the webservice algorithm, it first clears any previously assigned reviews for that Assignment to prevent multiple assigns, then iterates through each reviewer and assigns them to the topics that the algorithm selected for them.
[[File:Model_assign_review_topics.PNG]]
 

====== Method: assign_topic_to_reviewer ======
This method is called by the method [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_review_topics "assign_review_topics"] and assigns a single topic to a single user to review given an Assignment ID.
[[File:Model_assign_topic_to_reviewer.PNG]]
 

====== Method: reviewer_self_topic ======
This method returns the topic id of the reviewer that is passed in. This method is utilized to retrieve the reviewers' Topic ID, which is needed for the webservice algorithm to prevent users from being assigned to review their own topic. This is also prevented in views, but this is an additional safety precaution. 
[[File:Model_reviewer_self_topic.PNG]]
 

====== Method: reviewer_bidding_data ======
This method is called by [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_get_bidding_data "get_bidding_data"]. It creates a hash to store all the reviewer's bidding information. It iterates through all the bids for a reviewer given the Assignment ID and it returns a Hash of the bidding data.
 
[[File:Model_reviewer_bidding_data.PNG]]
 

=== ''Views'' ===
There are two main views for this project. They are for bidding and for showing assigned reviews after the bidding algorithm has been run.
====== View: others_work.html.erb ======
This file renders the view where students see a list of the reviews they've been assigned, with appropriate links and related messages. It includes a "Request Additional Review" button that will allow students to view more reviews up to the max number of reviews allowed.
[[File:Views_others_work_pt1.PNG]] 
[[File:Views_others_work_pt2.PNG]] 
[[File:Views_others_work_pt3.PNG]] 
[[File:Views_others_work_pt4.PNG]] 
 
 
View rendered by above: 
[[File:Views_others_work_rendered.PNG]]

====== View: show.html.erb ======
This view file renders the Review Bidding page where participants can select which topics to review. It renders partials which are included below, and includes coloring effects from helper methods indicating likelihood of receiving that particular topic to review. 
[[File:Views_show_pt1.PNG]] 
[[File:Views_show_pt2.PNG]] 
 
 
Page rendered by above: (This rendering is an instructor view, the student view is the same without the "Edit Topics" column.) 
[[File:Views_show_rendered.PNG|900px]]
 

=== ''Webservice'' ===
===== ''app'' =====

The entry point to the Web service is the app.py file where we have the POST Method match_topics.
This receives the input in a json format from Expertiza. This json will have a list of all topics, list of all students and the topics each student selected and also the maximum number of review topics per student. Also, it has the timestamp information of the bidding time to give preference to early bidders. The input json is passed to the JsonParser class to create a Dictionary.

 
[[File:tab.jpg]]

 

===== ''json parser'' =====
The json parser converts the json into a Dictionary. The parser also creates a preferences dictionary for the Student to store their topic preferences and
one for Topics to store list of topics which have been selected by students and also the priority for the selection using the Timestamps.
The json parser does one more task of finding Popular topics. This is needed because topics that are in demand should be preserved for students who bid on it and to avoid assigning it to students who did not bid on anything. High demand topics are assigned a lower priority for such students so that there are other topics they can be assigned.

[[File:json1.png]]

 

[[File:json2.png]]

 

[[File:json3.png]]

 

===== ''Class Topic'' =====
 
The Topic class is the one that does the proposing to the students during the runtime.
There are 2 cases. In case 1 when the particular topic is asked by more students than the limit of number of students the one topic can be assigned to. In this case, it uses the priorities it already has in the dictionary to make sure the topic proposes only to those students who chose it and those who bid earlier.
In Case 2, There are less students who asked for the topic. Hence, the topic can propose to all the students who asked for it and still be under the limit. In the end of this step. All topics make their proposals to be assigned to some students.

[[File:json11.png]]

 

===== ''Class Student'' =====
 
Once the Topics propose to students, the Student has to accept some of the proposals.
First, each student sorts the proposals based on the students preferences. Now, the student accepts all the proposals they can upto the number of slots remaining for them. The max slots for the student is determined by the max_proposals_accepted field in the json. Once the acceptance is complete, the Topics are all updated so that the topics can adjust their remaining slots accordingly.

[[File:json12.png]]

===== ''Topic Matcher'' =====

Next, The Topics Matcher class is called which is the main class that runs the Algorithm.
Before running the algorithm, it creates a list of Student and Topic objects by using instances of Student and Topic class. Each student and Topic now has its own class so that it can be used to make proposals for assignment of topics.
 

[[File:json4.png]]

 
The get_student_topic_matches() method kicks off the Algorithm.
Each topic proposes to students based on the topic priorities as covered in the Topics section. It uses a max_topic_assignment_limit (say 4 for example) to make sure the same topic is not assigned to too many students.
Once the proposals are made, we iterate over students so that they can accept proposals as long as they have slots remaining to be assigned to.
 
The round is now complete. To determine if we need a next round, we ask the topics if they are done proposing using is_topics_done_proposing() method. As long as each topic has not reached its assignment limit and there are still students to propose to, the proposing continues for the next round.
Once every topic is done, it breaks out of the loop and we now have the list of students with equal number of assigned topics ( 3-4 for example) and this list is returned back to Expertiza as the Stable matching to do the assignment.

[[File:json5.png]]

===== ''Test Input'' =====

Sample Test case: 
Case: There are a few very popular topics and the students who bid for them and bid early should receive it.
Popular Topics: 3969, 3971, 3972

<code>{
"tid": [
3969,3970,3971,3972,3973,3974,3975,3976,3977
],
"users": {
"36239": {
"tid": [
3970, 3972
],
"otid": 3977,
"priority": [
2,1
],
"time": [
"Thu, 12 Nov 2020 12:01:06 EST -05:00","Thu, 12 Nov 2020 12:01:07 EST -05:00"
]
},
"36240": {
"tid": [],
"otid": 3976,
"priority": [],
"time": []
},
"36241": {
"tid": [
3969,3971,3972
],
"otid": 3975,
"priority": [
1,3,2
],
"time": [
"Thu, 12 Nov 2020 12:00:22 EST -05:00","Thu, 12 Nov 2020 12:00:25 EST -05:00","Thu, 12 Nov 2020 12:00:27 EST -05:00"
]
},
"36242": {
"tid": [
3969,3971,3973
],
"otid": 3974,
"priority": [
3,2,1
],
"time": [
"Wed, 11 Nov 2020 12:15:43 EST -05:00","Thu, 12 Nov 2020 11:59:40 EST -05:00","Thu, 12 Nov 2020 13:07:53 EST -05:00"
]
},
"36243": {
"tid": [
3971,3969,3970,3976
],
"otid": 3972,
"priority": [
4,3,2,1,5
],
"time": [
"Wed, 11 Nov 2020 11:34:50 EST -05:00","Wed, 11 Nov 2020 12:30:16 EST -05:00","Wed, 11 Nov 2020 12:30:19 EST -05:00","Thu, 12 Nov 2020 13:02:02 EST -05:00"
]
}
},
"max_accepted_proposals": 3
} </code>

 

===== ''Output'' =====

Since there is competition for these topics, the algorithm recognizes this and makes sure to assign them only to the students who bid for it.
Other students who picked other topics or did not pick anything are assigned different topics.

<code> {
"36239": [
3970,
3972,
3975
],
"36240": [
3973,
3974,
3972
],
"36241": [
3969,
3971,
3972
],
"36242": [
3969,
3971,
3973
],
"36243": [
3969,
3970,
3971
]
}</code>

==Test Plan==
This section outlines the test plan for this project including testing scenarios/edge cases, the manual testing plan, and the automatic/RSpec testing plan.

=== ''Testing Scenarios'' ===
This section outlines the various bidding scenarios that could occur. These scenarios should be tested in either '''Manual Testing''' or '''Automatic Testing'''.

==== Basic Bidding Scenario ====
*Reviewers bid on multiple projects at different times
** Bidding algorithm should give priority to reviewers that bid first and should assign based on reviewers ranking

==== Edge Cases ====
* All reviewers bid on the same project
** If different time stamps: bidding algorithm should give priority to reviewers that bid first
** If same time stamps: bidding algorithm should assign reviews randomly
* None of the reviews bid on any projects
** Bidding algorithm should assign reviews randomly
* Reviewer bids on their own project
** Bidding algorithm should not allow a reviewer to bid on their own project and should have validation to prevent assigning a reviewer their own project
* None of the reviewers bid on a specific project
** Bidding algorithm should still assign reviewers to this project

=== ''Manual Testing'' ===
Manual testing should be preformed on an Expertiza server and webserver platform 

==== Manual Testing Plan for Expertiza Server ====
* test that an instructor can allow review bidding for an assignment
* test that server routes correctly when review bidding is allowed
** test that task box links to bidding page before algorithm is run
** test that assignment->others work links to bidding page before algorithm is run
** test that task box links to review page with bidding information after algorithm is run
** test that assignment->others work links to review page with bidding information after algorithm is run
* test that the bidding UI is implemented as expected
* test that a participant can bid on projects to review
** test that a participant can move projects into the bidding column
** test that a participant can reorder projects in the bidding column
** test that a participant's bidding preferences save
** test that a participant can't bid for their own topic
* test that run algorithm button calls webserver with correct information and returns review assignments for each user

==== Directions for Manual Testing of Expertiza Server ====
1. Go to [http://152.7.99.70:8081/ testing server] 
2. Log in as username: '''instructor6''' password: '''password''' 
3. Go to '''Manage''' -> '''Assignments''' -> '''E2085 Manual Testing''' -> '''edit''' (pencil symbol)
:# In the '''General''' tab make sure '''Has teams?''' and '''Has topics?''' are checked
:# In the '''Topics''' tab check '''Allow review to choose which topic to review?''' then click '''save'''
:# In the '''Topics''' tab select '''Bidding''' then click '''save'''
4. Go to '''Assignments'''
:# '''E2085 Manual Testing review''' task should be listed in the main assignments box
::# This UI is based on '''instructor6''' being a '''participant''' assigned to topic '''Topic4'''
::# The UI blocks user from bidding on their own project
::# Check that user can bid on topics and reorder bids
::# Check that users topic preferences save on page refresh
::# Impersonate other participants and repeat bidding UI tests ('''student7601''', '''student7602''', '''student7603''', '''student7604''', '''student7605''', '''student7606''', '''student7607''', '''student7608''', '''student7609'')
5. Go to '''Manage''' -> '''Assignments''' -> '''E2085 Manual Testing''' -> '''edit''' (pencil symbol)
:# In the '''Topics''' tab click '''Run Review Algorithm''' button
6. Go to '''Assignments'''
:# '''E2085 Manual Testing review''' task should be listed in the main assignments box
::# '''E2085 Manual Testing review''' should now link to the reviewing page
::# The number of required reviews should already be displayed
::# A '''Request Another Review''' button should be present unless reviews listed is equal to the number of reviews allowed
::# '''Begin''' a review and test '''save''' and '''submit'''

==== Manual Testing Demo Video for Expertiza Server ====
Video demonstration of manual testing: [https://www.screencast.com/t/0rSg1zDW here]

=== ''Automatic/RSpec Testing '' ===
RSpec tests will need to be written to related controller(s) and model(s). 

==== RSpec Testing Plan ====
These RSpec tests:
* test basic functionality of review bids controller
* test basic functionality of review bid model

==== Directions for RSpec Testing ====
:# [https://docs.google.com/document/d/1tXmwju6R7KQbvycku-bdXxa6rXSUN4BMyvjY3ROmMSw/edit Set up Expertiza] on personal device using [https://github.com/uahamedncsu/expertiza forked repo]
:# Run the following commands:

# review bids controller:
bundle exec rspec spec/controllers/review_bids_controller_spec.rb

# review bid model:
bundle exec rspec spec/models/review_bid_spec.rb

==== RSpec Testing Demo Video ====
Video demonstration of RSpec tests passing: [https://www.screencast.com/t/uSWZHSX8s here]
 

==Team==
Ummu Kolusum Yasmin Ahamed Adam (uahamed) 
Ryan Grainger (rpgraing) 
Luis Delossantos (lgdeloss) 
Colleen "Bria" Engen (ceengen) 
'''Mentor:''' Saurabh Shingte (svshingt)

 

==Resources==
''Previous Implementations'': [[http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1986._Allow_reviewers_to_bid_on_what_to_review 1]], [[https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Spring_2020_-_E2021._Allow_reviewers_to_bid_on_what_to_review 2]] 
''Bidding Interface Implementation Pull Request'': [[https://github.com/expertiza/expertiza/pull/778 3]]

=== ''Relevant Links'' ===
''Github Repository for Expertiza'': https://github.com/uahamedncsu/expertiza/tree/beta 
''Github Repository for Web service'': https://github.com/uahamedncsu/IntelligentAssignment 
''Pull Request'': https://github.com/expertiza/expertiza/pull/1822 
''Expertiza Server'': http://152.7.99.70:8081/ 
''Web Service Server'': https://app-csc517.herokuapp.com/match_topics 
''Video Demo'': https://www.screencast.com/t/GJNOcfa9Ln

CSC/ECE 517 Fall 2020 - E2085. Allow reviewers to bid on what to review

2021-11-28T20:32:47Z

Aluthra: /* Method: assign_bidding */

==Project Overview==
=== ''Background'' ===
Assigning reviews to users is a complicated process. Currently, reviews are still assigned using a first-come-first-served basis. However, reviews could be bid on like how topics are bid on. This would involve matching multiple students to review a submission up to the maximum reviewers for a submission.The manual selection process can be replaced with an automated bidding approach for this purpose.

=== ''Objectives'' ===
# Implement top trading cycles on a web service.
# Add front end code to allow bidding on topics and call the appropriate web service from the lottery controller.
# Add working test cases to test the new changes.
# Code should be inclined to the Beta branch
# Link to run bidding assignment should be on the page associated with the assignment actions.
# Code such as getter methods should be added to the model and not the controller.
#The number of reviews created by default needs to be checked. One way of achieving this is creating minimum reviews and then allowing users to request more reviews if required.

=== ''Previous Work'' ===
There have been two previous implementations:
* Fall 2019 [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1986._Allow_reviewers_to_bid_on_what_to_review here]
* Spring 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Spring_2020_-_E2021._Allow_reviewers_to_bid_on_what_to_review here]

=== ''Relevant Links'' ===
''Github Repository for Expertiza'': https://github.com/uahamedncsu/expertiza/tree/beta 
''Github Repository for Web service'': https://github.com/uahamedncsu/IntelligentAssignment 
''Pull Request'': https://github.com/expertiza/expertiza/pull/1822 
''Expertiza Server'': http://152.7.99.70:8081/ 
''Web Service Server'': https://app-csc517.herokuapp.com/match_topics 
''Video Demo'': https://www.screencast.com/t/GJNOcfa9Ln 

=== ''Files Modified'' ===
This is a list of all the files added or changed in completing this project. 
 
''Changed'': 
*config/routes.rb 

*config/webservices.yml

*app/controllers/student_review_controller.rb 

*app/views/assignments/edit/_topics.html.erb 

*app/views/student_task/list.html.erb 

*app/views/student_task/view.html.erb 

*spec/factories/factories.rb 

 
''Added'': 
*app/controllers/review_bids_controller.rb 

*app/helpers/review_bids_helper.rb 

*app/models/review_bid.rb 

*app/views/review_bids/_all_actions.html.erb 

*app/views/review_bids/_table_header.html.erb 

*app/views/review_bids/_table_line.html.erb 

*app/views/review_bids/_topic_names.html.erb 

*app/views/review_bids/others_work.html.erb 

*app/views/review_bids/show.html.erb 

*db/migrate/20201107203204_create_review_bids.rb 

*spec/controllers/review_bids_controller_spec.rb 

*spec/models/review_bid_spec.rb 

==Design==

=== ''User Interface'' ===

Currently, the users are offered the entire list of topics to review and are asked to select a topic to begin a review. Topics that cannot be reviewed or have already been reviewed the maximum number of times are greyed out and cannot be selected. There is also a check box in which the student can select "I don't care which topic I review" and will be randomly assigned a topic from the list. The current User Interface (UI) for this is shown below.
 
 
[[File:current_review_UI.PNG]]

 

As discussed in the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Project_Overview Project Overview] section, a bidding system similar to the bidding system used in topic selection is preferable to the current review selection system. The UI for this process will mirror the UI for the topic bidding system. The new UI will allow users to drag a topic from a list of all the topics on the left, to their priority list on the right. The ordering of the right list indicates the priority of the users preference. The colors indicate the likelihood of being assigned that topic. By utilizing the same UI, the consistency of the user experience and the DRY-ness of the code are increased. An example of the topic selection bidding UI is shown below.
 
 
[[File:expected_review_UI.PNG]]

 

Currently, an instructor has the option while creating or editing an assignment to allow a reviewers to select the topics they want to review. This is done by checking '''Has teams?''' and '''Has topics?''' in the '''General''' tab of an assignment then checking '''Allow reviewer to choose which topic to review?''' in the '''Topics''' tab. This interface can be used for allowing review bidding either by editing the current check box or adding an additional check box for '''Allow reviewer to bid on topics to review?'''.
 
 
[[File:Instructor_ui_for_review_bidding.png|1000px]]

=== ''Analyzing the problem'' ===

Currently, the student can either pick a topic or is randomly assigned a topic, from among the ones that are currently eligible to be reviewed. In this scenario, it is possible that some good topics are already picked by a lot of students and if someone is late to do the selecting, they miss out on it since it is grayed out.
 
To solve the above problem of not getting the topics they like, the participants must be assigned topics based on a list of topics they pre-select.
This is where an algorithm will come in handy. For picking a suitable algorithm, we should first identify the type of the problem.
 
Type of problem 
Each student should be able to bid on multiple assignments of their choice and the same assignment can be assigned to multiple participants for review. 
This requires a Many to Many mapping.

[[File:many2.png]]

The algorithm of choice for this type of stable allocations is Gale-Shapely algorithm. 

In the real world, there are cases where we face this problem:
<ul><li> Student choosing their future universities [https://www.jstor.org/stable/3132114?seq=1&cid=pdf-reference#references_tab_content '''School Choice Problem'''] (One-to-Many)</li>
<li> Men and women seeking each other to create families [https://en.wikipedia.org/wiki/Stable_marriage_problem '''Stable Marriage Problem'''] (One-to-One)</li>
<li> Internet services that need to be connected with users with the smallest amount of time </li></ul>

===''Algorithm Implementation for matching students and topics''===
The Gale-Shapely Algorithm has steps to make sure it is a Stable marriage/allocation.
An unstable allocation is when the participant does not like the topic assigned to them.
But it can happen that the participant 1 might like the review topic assigned to participant 2 and participant 2 likes the topic assigned to participant 1.
This is the unstable case. To make it stable, the topics need to be exchanged between these 2 participants so that both of them get the topic they wanted.

[[File:Snippy2.png]]
Source: [https://www.journalajrcos.com/index.php/AJRCOS/article/view/24749/46281 Stable Marriage Algorithm]

===''Assignment of priority ''===
When a student places a bid, the timestamp is recorded. Priority will be given to the earlier bidder. If the student had chosen only very few bids, then this person will be given a higher priority. Any topics that were not selected by the participant are added to bottom of their preference list in a random preference order to create a situation where the number of students and number of topics is equal. 

===''Algorithm Steps ''===
<li> For the first round, the Algorithm will assign topics to each student based on the Priority and each topic is assigned to a limited number of students based on the Limit. It picks one topic and begins assigning students to it. (Note: The Algorithm makes sure the participant is not in a group that worked on this particular topic.)
<li>If the one topic is assigned to enough participants already, then it is removed from the queue and the same process is repeated for the next topic.
<li> As this goes on, all students are matched even if all matches are not stable at this point.
<li> The Algorithm terminates when all Topics that have not reached the maximum assignment level have tried to accept every student.

 
 
The time complexity of this algorithm is O(n^2) 

===''Simulation ''===

[[File:gale-shapeley-1.png]]
[[File:gale-shapeley-2.png]]

In the simulation above, we have equal number of Students (S1, S2, S3) and Review topics to assign (R1, R2, R3). The students preference of topics is to left of student. The Review topics does not have any preferences. To have preferences, the timestamps are used to determine which student bid on each topic first to create such a Preference list for topics.

# Phase 1: Both Students S1 and S2 have selected R2 as their highest preference. Since there is a conflict, The preference of R2 is checked which shows that S2 had bid earlier than S1. Thus, the topic R2 is assigned to student S2.
# Phase 2: Now, S1 and S3 both want to review R1 but R1 was first bid on by S1 as shown in the preference. Hence, R1 is assigned to S1.
# Phase 3: The only topic remaining in this case is R3 which is also on the preference list of Student S3. Therefore S3 is assigned the topic R3.
#By the end of the 3 phases, All students are assigned to a Review. The same process needs to be repeated till the required number of reviews are assigned to each student. In each iteration, if a review is assigned to max number of students already, the review is removed from consideration.

==Implementation==
This section details the implementation of the code and will highlight each component of the design in the code and its intended purpose, as well as any modifications or testing. As the implementation progresses this section will be updated.

=== ''Overview'' ===

This project was split into 2 objectives, the Expertiza MVC implementation and the webservice algorithm. The Expertiza MVC follows the MVC standard such that the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Controller controller] handles directing the users to the correct views, as well as handling the communication with the webservice. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Model model] performs all necessary computations and data handling like the creation and storing of bids on reviews and assigning reviews to students. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Views views] render the corresponding webpage based on the actions of the user. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Webservice webservice] contains the algorithm outlined in the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Algorithm_Implementation_for_matching_students_and_topics Algorithm Implementation] section of Design. Our Expertiza controller sends this webservice bidding information and the algorithm returns the matched topics to be assigned.

 

=== ''Controller'' ===
The controller contains 6 methods that perform all the necessary functions. The first method, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_action_allowed? "action_allowed?"] performs authentication checks The second, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_index "index"], is used to redirect the user to a list of assigned reviews after the algorithm has been run. The third, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_show "show"], is used in conjunction with fourth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_set_priority "set_priority"], to update and store the bids when users make changes in the bidding page. The fifth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"], utilizes the sixth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_run_bidding_algorithm "run_bidding_algorithm"], to connect to the webservice, send and receive the correct bidding information for the algorithm.
 

====== Method: action_allowed? ======
This method performs authentication by allowing only certain types of users to access controller methods. The first portion of this method allows users 'Instructor', 'Teaching Assistant', 'Administrator', 'Super-Administrator', and 'Student' to access the methods 'show', 'set_priority', and 'index'. These are the methods that render the bidding pages and review listings. The other methods are restricted for 'Student' users.
 
[[File:Controller action allowed..PNG]]
 

====== Method: index ======
This method collects all the necessary information about the user, Assignment details, and Review Mapping details to render the 'review_bids/others_work', which lists the reviews that are assigned to the user. This page also includes a button to request more reviews which will add reviews up to the maximum number of reviews. To see the view rendered by this method, please go to [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#View:_others_work.html.erb "others_work"].
 
[[File:Controller_index.PNG]]
 

====== Method: show ======
This method renders the bidding page in which participants can select from a table of topics the topics they would prefer to review. It collects the necessary information about the user, Assignment, Topics, and current Review Bids to render this page. It is used in conjunction with [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_set_priority set_priority] where the users bidding information is stored, and the page updated. To see the bidding page that this function renders, visit the view [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#View:_show.html.erb "show"] 
[[File:Controller_show.PNG]]

 

====== Method: set_priority ======
In the view file [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#view:_show.html.erb "show"] there exists a line the calls this method when the page is updated or accessed. This method stores the user's bidding information in the schema database. This allows the model to access this information when collecting all the bidding information, as well as allowing users to refresh or return to this page with their selections staying in the bid table.
 
[[File:Controller_set_priority.PNG]]

 

====== Method: assign_bidding ======
This method utilizes methods in the model, as well as the method [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_run_bidding_algorithm "run_bidding_algorithm"] to collect the bidding information of all students of a particular Assignment, send it to the webservice algorithm, store the matched topics that are returned, and pass these matched topics to the model to be assigned. After this method is run, the students should have assigned reviews and bidding should be turned off.
 
[[File:Controller_assign_bidding.PNG]]
 

====== Method: run_bidding_algorithm ======
This method connects to the webservice using RestClient API, which is part of the 'require's at the top of the Controller. It will return a false if the connection fails for ay reason such as connection is not established or the algorithm has an internal error, otherwise it returns the matched topics from the algorithm as a 'json'. 
[[File:Controller_requirements.PNG]] 
[[File:Controller_run_bidding_algorithm.PNG]]
 

=== ''Model '' ===
The model contains 5 methods that are utilized by the controller to save bidding information per user or all users, retrieve bidding information per user or for all users, and assign topics to review to participants.
 

====== Method: get_bidding_data ======
This method is called by the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"] controller method. It receives the Assignment ID and a list of reviewers, which are student IDs. It creates a Hash to store bidding data into and iterates through all of the reviewers, calling [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_reviewer_bidding_data "reviewer_bidding_data"] to get each students bidding data and store it in the Hash. It returns a Hash with all the bidding data needed for the Webservice Algorithm to run.
[[File:Model_get_bidding_data.PNG]]
 

====== Method: assign_review_topics ======
This method is called by the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"] controller method. Given the Assignment ID, list of reviewers, and the matched topics returned by the webservice algorithm, it first clears any previously assigned reviews for that Assignment to prevent multiple assigns, then iterates through each reviewer and assigns them to the topics that the algorithm selected for them.
[[File:Model_assign_review_topics.PNG]]
 

====== Method: assign_topic_to_reviewer ======
This method is called by the method [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_review_topics "assign_review_topics"] and assigns a single topic to a single user to review given an Assignment ID.
[[File:Model_assign_topic_to_reviewer.PNG]]
 

====== Method: reviewer_self_topic ======
This method returns the topic id of the reviewer that is passed in. This method is utilized to retrieve the reviewers' Topic ID, which is needed for the webservice algorithm to prevent users from being assigned to review their own topic. This is also prevented in views, but this is an additional safety precaution. 
[[File:Model_reviewer_self_topic.PNG]]
 

====== Method: reviewer_bidding_data ======
This method is called by [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_get_bidding_data "get_bidding_data"]. It creates a hash to store all the reviewer's bidding information. It iterates through all the bids for a reviewer given the Assignment ID and it returns a Hash of the bidding data.
 
[[File:Model_reviewer_bidding_data.PNG]]
 

=== ''Views'' ===
There are two main views for this project. They are for bidding and for showing assigned reviews after the bidding algorithm has been run.
====== View: others_work.html.erb ======
This file renders the view where students see a list of the reviews they've been assigned, with appropriate links and related messages. It includes a "Request Additional Review" button that will allow students to view more reviews up to the max number of reviews allowed.
[[File:Views_others_work_pt1.PNG]] 
[[File:Views_others_work_pt2.PNG]] 
[[File:Views_others_work_pt3.PNG]] 
[[File:Views_others_work_pt4.PNG]] 
 
 
View rendered by above: 
[[File:Views_others_work_rendered.PNG]]

====== View: show.html.erb ======
This view file renders the Review Bidding page where participants can select which topics to review. It renders partials which are included below, and includes coloring effects from helper methods indicating likelihood of receiving that particular topic to review. 
[[File:Views_show_pt1.PNG]] 
[[File:Views_show_pt2.PNG]] 
 
 
Page rendered by above: (This rendering is an instructor view, the student view is the same without the "Edit Topics" column.) 
[[File:Views_show_rendered.PNG|900px]]
 

=== ''Webservice'' ===
===== ''app'' =====

The entry point to the Web service is the app.py file where we have the POST Method match_topics.
This receives the input in a json format from Expertiza. This json will have a list of all topics, list of all students and the topics each student selected and also the maximum number of review topics per student. Also, it has the timestamp information of the bidding time to give preference to early bidders. The input json is passed to the JsonParser class to create a Dictionary.

 
[[File:tab.jpg]]

 

===== ''json parser'' =====
The json parser converts the json into a Dictionary. The parser also creates a preferences dictionary for the Student to store their topic preferences and
one for Topics to store list of topics which have been selected by students and also the priority for the selection using the Timestamps.
The json parser does one more task of finding Popular topics. This is needed because topics that are in demand should be preserved for students who bid on it and to avoid assigning it to students who did not bid on anything. High demand topics are assigned a lower priority for such students so that there are other topics they can be assigned.

[[File:json1.png]]

 

[[File:json2.png]]

 

[[File:json3.png]]

 

===== ''Class Topic'' =====
 
The Topic class is the one that does the proposing to the students during the runtime.
There are 2 cases. In case 1 when the particular topic is asked by more students than the limit of number of students the one topic can be assigned to. In this case, it uses the priorities it already has in the dictionary to make sure the topic proposes only to those students who chose it and those who bid earlier.
In Case 2, There are less students who asked for the topic. Hence, the topic can propose to all the students who asked for it and still be under the limit. In the end of this step. All topics make their proposals to be assigned to some students.

[[File:json11.png]]

 

===== ''Class Student'' =====
 
Once the Topics propose to students, the Student has to accept some of the proposals.
First, each student sorts the proposals based on the students preferences. Now, the student accepts all the proposals they can upto the number of slots remaining for them. The max slots for the student is determined by the max_proposals_accepted field in the json. Once the acceptance is complete, the Topics are all updated so that the topics can adjust their remaining slots accordingly.

[[File:json12.png]]

===== ''Topic Matcher'' =====

Next, The Topics Matcher class is called which is the main class that runs the Algorithm.
Before running the algorithm, it creates a list of Student and Topic objects by using instances of Student and Topic class. Each student and Topic now has its own class so that it can be used to make proposals for assignment of topics.
 

[[File:json4.png]]

 
The get_student_topic_matches() method kicks off the Algorithm.
Each topic proposes to students based on the topic priorities as covered in the Topics section. It uses a max_topic_assignment_limit (say 4 for example) to make sure the same topic is not assigned to too many students.
Once the proposals are made, we iterate over students so that they can accept proposals as long as they have slots remaining to be assigned to.
 
The round is now complete. To determine if we need a next round, we ask the topics if they are done proposing using is_topics_done_proposing() method. As long as each topic has not reached its assignment limit and there are still students to propose to, the proposing continues for the next round.
Once every topic is done, it breaks out of the loop and we now have the list of students with equal number of assigned topics ( 3-4 for example) and this list is returned back to Expertiza as the Stable matching to do the assignment.

[[File:json5.png]]

===== ''Test Input'' =====

Sample Test case: 
Case: There are a few very popular topics and the students who bid for them and bid early should receive it.
Popular Topics: 3969, 3971, 3972

<code>{
"tid": [
3969,3970,3971,3972,3973,3974,3975,3976,3977
],
"users": {
"36239": {
"tid": [
3970, 3972
],
"otid": 3977,
"priority": [
2,1
],
"time": [
"Thu, 12 Nov 2020 12:01:06 EST -05:00","Thu, 12 Nov 2020 12:01:07 EST -05:00"
]
},
"36240": {
"tid": [],
"otid": 3976,
"priority": [],
"time": []
},
"36241": {
"tid": [
3969,3971,3972
],
"otid": 3975,
"priority": [
1,3,2
],
"time": [
"Thu, 12 Nov 2020 12:00:22 EST -05:00","Thu, 12 Nov 2020 12:00:25 EST -05:00","Thu, 12 Nov 2020 12:00:27 EST -05:00"
]
},
"36242": {
"tid": [
3969,3971,3973
],
"otid": 3974,
"priority": [
3,2,1
],
"time": [
"Wed, 11 Nov 2020 12:15:43 EST -05:00","Thu, 12 Nov 2020 11:59:40 EST -05:00","Thu, 12 Nov 2020 13:07:53 EST -05:00"
]
},
"36243": {
"tid": [
3971,3969,3970,3976
],
"otid": 3972,
"priority": [
4,3,2,1,5
],
"time": [
"Wed, 11 Nov 2020 11:34:50 EST -05:00","Wed, 11 Nov 2020 12:30:16 EST -05:00","Wed, 11 Nov 2020 12:30:19 EST -05:00","Thu, 12 Nov 2020 13:02:02 EST -05:00"
]
}
},
"max_accepted_proposals": 3
} </code>

 

===== ''Output'' =====

Since there is competition for these topics, the algorithm recognizes this and makes sure to assign them only to the students who bid for it.
Other students who picked other topics or did not pick anything are assigned different topics.

<code> {
"36239": [
3970,
3972,
3975
],
"36240": [
3973,
3974,
3972
],
"36241": [
3969,
3971,
3972
],
"36242": [
3969,
3971,
3973
],
"36243": [
3969,
3970,
3971
]
}</code>

==Test Plan==
This section outlines the test plan for this project including testing scenarios/edge cases, the manual testing plan, and the automatic/RSpec testing plan.

=== ''Testing Scenarios'' ===
This section outlines the various bidding scenarios that could occur. These scenarios should be tested in either '''Manual Testing''' or '''Automatic Testing'''.

==== Basic Bidding Scenario ====
*Reviewers bid on multiple projects at different times
** Bidding algorithm should give priority to reviewers that bid first and should assign based on reviewers ranking

==== Edge Cases ====
* All reviewers bid on the same project
** If different time stamps: bidding algorithm should give priority to reviewers that bid first
** If same time stamps: bidding algorithm should assign reviews randomly
* None of the reviews bid on any projects
** Bidding algorithm should assign reviews randomly
* Reviewer bids on their own project
** Bidding algorithm should not allow a reviewer to bid on their own project and should have validation to prevent assigning a reviewer their own project
* None of the reviewers bid on a specific project
** Bidding algorithm should still assign reviewers to this project

=== ''Manual Testing'' ===
Manual testing should be preformed on an Expertiza server and webserver platform 

==== Manual Testing Plan for Expertiza Server ====
* test that an instructor can allow review bidding for an assignment
* test that server routes correctly when review bidding is allowed
** test that task box links to bidding page before algorithm is run
** test that assignment->others work links to bidding page before algorithm is run
** test that task box links to review page with bidding information after algorithm is run
** test that assignment->others work links to review page with bidding information after algorithm is run
* test that the bidding UI is implemented as expected
* test that a participant can bid on projects to review
** test that a participant can move projects into the bidding column
** test that a participant can reorder projects in the bidding column
** test that a participant's bidding preferences save
** test that a participant can't bid for their own topic
* test that run algorithm button calls webserver with correct information and returns review assignments for each user

==== Directions for Manual Testing of Expertiza Server ====
1. Go to [http://152.7.99.70:8081/ testing server] 
2. Log in as username: '''instructor6''' password: '''password''' 
3. Go to '''Manage''' -> '''Assignments''' -> '''E2085 Manual Testing''' -> '''edit''' (pencil symbol)
:# In the '''General''' tab make sure '''Has teams?''' and '''Has topics?''' are checked
:# In the '''Topics''' tab check '''Allow review to choose which topic to review?''' then click '''save'''
:# In the '''Topics''' tab select '''Bidding''' then click '''save'''
4. Go to '''Assignments'''
:# '''E2085 Manual Testing review''' task should be listed in the main assignments box
::# This UI is based on '''instructor6''' being a '''participant''' assigned to topic '''Topic4'''
::# The UI blocks user from bidding on their own project
::# Check that user can bid on topics and reorder bids
::# Check that users topic preferences save on page refresh
::# Impersonate other participants and repeat bidding UI tests ('''student7601''', '''student7602''', '''student7603''', '''student7604''', '''student7605''', '''student7606''', '''student7607''', '''student7608''', '''student7609'')
5. Go to '''Manage''' -> '''Assignments''' -> '''E2085 Manual Testing''' -> '''edit''' (pencil symbol)
:# In the '''Topics''' tab click '''Run Review Algorithm''' button
6. Go to '''Assignments'''
:# '''E2085 Manual Testing review''' task should be listed in the main assignments box
::# '''E2085 Manual Testing review''' should now link to the reviewing page
::# The number of required reviews should already be displayed
::# A '''Request Another Review''' button should be present unless reviews listed is equal to the number of reviews allowed
::# '''Begin''' a review and test '''save''' and '''submit'''

==== Manual Testing Demo Video for Expertiza Server ====
Video demonstration of manual testing: [https://www.screencast.com/t/0rSg1zDW here]

=== ''Automatic/RSpec Testing '' ===
RSpec tests will need to be written to related controller(s) and model(s). 

==== RSpec Testing Plan ====
These RSpec tests:
* test basic functionality of review bids controller
* test basic functionality of review bid model

==== Directions for RSpec Testing ====
:# [https://docs.google.com/document/d/1tXmwju6R7KQbvycku-bdXxa6rXSUN4BMyvjY3ROmMSw/edit Set up Expertiza] on personal device using [https://github.com/uahamedncsu/expertiza forked repo]
:# Run the following commands:

# review bids controller:
bundle exec rspec spec/controllers/review_bids_controller_spec.rb

# review bid model:
bundle exec rspec spec/models/review_bid_spec.rb

==== RSpec Testing Demo Video ====
Video demonstration of RSpec tests passing: [https://www.screencast.com/t/uSWZHSX8s here]
 

==Team==
Ummu Kolusum Yasmin Ahamed Adam (uahamed) 
Ryan Grainger (rpgraing) 
Luis Delossantos (lgdeloss) 
Colleen "Bria" Engen (ceengen) 
'''Mentor:''' Saurabh Shingte (svshingt)

 

==Resources==
''Previous Implementations'': [[http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1986._Allow_reviewers_to_bid_on_what_to_review 1]], [[https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Spring_2020_-_E2021._Allow_reviewers_to_bid_on_what_to_review 2]] 
''Bidding Interface Implementation Pull Request'': [[https://github.com/expertiza/expertiza/pull/778 3]]

=== ''Relevant Links'' ===
''Github Repository for Expertiza'': https://github.com/uahamedncsu/expertiza/tree/beta 
''Github Repository for Web service'': https://github.com/uahamedncsu/IntelligentAssignment 
''Pull Request'': https://github.com/expertiza/expertiza/pull/1822 
''Expertiza Server'': http://152.7.99.70:8081/ 
''Web Service Server'': https://app-csc517.herokuapp.com/match_topics 
''Video Demo'': https://www.screencast.com/t/GJNOcfa9Ln

CSC/ECE 517 Fall 2020 - E2085. Allow reviewers to bid on what to review

2021-11-28T20:32:34Z

Aluthra: /* Method: set_priority */

==Project Overview==
=== ''Background'' ===
Assigning reviews to users is a complicated process. Currently, reviews are still assigned using a first-come-first-served basis. However, reviews could be bid on like how topics are bid on. This would involve matching multiple students to review a submission up to the maximum reviewers for a submission.The manual selection process can be replaced with an automated bidding approach for this purpose.

=== ''Objectives'' ===
# Implement top trading cycles on a web service.
# Add front end code to allow bidding on topics and call the appropriate web service from the lottery controller.
# Add working test cases to test the new changes.
# Code should be inclined to the Beta branch
# Link to run bidding assignment should be on the page associated with the assignment actions.
# Code such as getter methods should be added to the model and not the controller.
#The number of reviews created by default needs to be checked. One way of achieving this is creating minimum reviews and then allowing users to request more reviews if required.

=== ''Previous Work'' ===
There have been two previous implementations:
* Fall 2019 [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1986._Allow_reviewers_to_bid_on_what_to_review here]
* Spring 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Spring_2020_-_E2021._Allow_reviewers_to_bid_on_what_to_review here]

=== ''Relevant Links'' ===
''Github Repository for Expertiza'': https://github.com/uahamedncsu/expertiza/tree/beta 
''Github Repository for Web service'': https://github.com/uahamedncsu/IntelligentAssignment 
''Pull Request'': https://github.com/expertiza/expertiza/pull/1822 
''Expertiza Server'': http://152.7.99.70:8081/ 
''Web Service Server'': https://app-csc517.herokuapp.com/match_topics 
''Video Demo'': https://www.screencast.com/t/GJNOcfa9Ln 

=== ''Files Modified'' ===
This is a list of all the files added or changed in completing this project. 
 
''Changed'': 
*config/routes.rb 

*config/webservices.yml

*app/controllers/student_review_controller.rb 

*app/views/assignments/edit/_topics.html.erb 

*app/views/student_task/list.html.erb 

*app/views/student_task/view.html.erb 

*spec/factories/factories.rb 

 
''Added'': 
*app/controllers/review_bids_controller.rb 

*app/helpers/review_bids_helper.rb 

*app/models/review_bid.rb 

*app/views/review_bids/_all_actions.html.erb 

*app/views/review_bids/_table_header.html.erb 

*app/views/review_bids/_table_line.html.erb 

*app/views/review_bids/_topic_names.html.erb 

*app/views/review_bids/others_work.html.erb 

*app/views/review_bids/show.html.erb 

*db/migrate/20201107203204_create_review_bids.rb 

*spec/controllers/review_bids_controller_spec.rb 

*spec/models/review_bid_spec.rb 

==Design==

=== ''User Interface'' ===

Currently, the users are offered the entire list of topics to review and are asked to select a topic to begin a review. Topics that cannot be reviewed or have already been reviewed the maximum number of times are greyed out and cannot be selected. There is also a check box in which the student can select "I don't care which topic I review" and will be randomly assigned a topic from the list. The current User Interface (UI) for this is shown below.
 
 
[[File:current_review_UI.PNG]]

 

As discussed in the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Project_Overview Project Overview] section, a bidding system similar to the bidding system used in topic selection is preferable to the current review selection system. The UI for this process will mirror the UI for the topic bidding system. The new UI will allow users to drag a topic from a list of all the topics on the left, to their priority list on the right. The ordering of the right list indicates the priority of the users preference. The colors indicate the likelihood of being assigned that topic. By utilizing the same UI, the consistency of the user experience and the DRY-ness of the code are increased. An example of the topic selection bidding UI is shown below.
 
 
[[File:expected_review_UI.PNG]]

 

Currently, an instructor has the option while creating or editing an assignment to allow a reviewers to select the topics they want to review. This is done by checking '''Has teams?''' and '''Has topics?''' in the '''General''' tab of an assignment then checking '''Allow reviewer to choose which topic to review?''' in the '''Topics''' tab. This interface can be used for allowing review bidding either by editing the current check box or adding an additional check box for '''Allow reviewer to bid on topics to review?'''.
 
 
[[File:Instructor_ui_for_review_bidding.png|1000px]]

=== ''Analyzing the problem'' ===

Currently, the student can either pick a topic or is randomly assigned a topic, from among the ones that are currently eligible to be reviewed. In this scenario, it is possible that some good topics are already picked by a lot of students and if someone is late to do the selecting, they miss out on it since it is grayed out.
 
To solve the above problem of not getting the topics they like, the participants must be assigned topics based on a list of topics they pre-select.
This is where an algorithm will come in handy. For picking a suitable algorithm, we should first identify the type of the problem.
 
Type of problem 
Each student should be able to bid on multiple assignments of their choice and the same assignment can be assigned to multiple participants for review. 
This requires a Many to Many mapping.

[[File:many2.png]]

The algorithm of choice for this type of stable allocations is Gale-Shapely algorithm. 

In the real world, there are cases where we face this problem:
<ul><li> Student choosing their future universities [https://www.jstor.org/stable/3132114?seq=1&cid=pdf-reference#references_tab_content '''School Choice Problem'''] (One-to-Many)</li>
<li> Men and women seeking each other to create families [https://en.wikipedia.org/wiki/Stable_marriage_problem '''Stable Marriage Problem'''] (One-to-One)</li>
<li> Internet services that need to be connected with users with the smallest amount of time </li></ul>

===''Algorithm Implementation for matching students and topics''===
The Gale-Shapely Algorithm has steps to make sure it is a Stable marriage/allocation.
An unstable allocation is when the participant does not like the topic assigned to them.
But it can happen that the participant 1 might like the review topic assigned to participant 2 and participant 2 likes the topic assigned to participant 1.
This is the unstable case. To make it stable, the topics need to be exchanged between these 2 participants so that both of them get the topic they wanted.

[[File:Snippy2.png]]
Source: [https://www.journalajrcos.com/index.php/AJRCOS/article/view/24749/46281 Stable Marriage Algorithm]

===''Assignment of priority ''===
When a student places a bid, the timestamp is recorded. Priority will be given to the earlier bidder. If the student had chosen only very few bids, then this person will be given a higher priority. Any topics that were not selected by the participant are added to bottom of their preference list in a random preference order to create a situation where the number of students and number of topics is equal. 

===''Algorithm Steps ''===
<li> For the first round, the Algorithm will assign topics to each student based on the Priority and each topic is assigned to a limited number of students based on the Limit. It picks one topic and begins assigning students to it. (Note: The Algorithm makes sure the participant is not in a group that worked on this particular topic.)
<li>If the one topic is assigned to enough participants already, then it is removed from the queue and the same process is repeated for the next topic.
<li> As this goes on, all students are matched even if all matches are not stable at this point.
<li> The Algorithm terminates when all Topics that have not reached the maximum assignment level have tried to accept every student.

 
 
The time complexity of this algorithm is O(n^2) 

===''Simulation ''===

[[File:gale-shapeley-1.png]]
[[File:gale-shapeley-2.png]]

In the simulation above, we have equal number of Students (S1, S2, S3) and Review topics to assign (R1, R2, R3). The students preference of topics is to left of student. The Review topics does not have any preferences. To have preferences, the timestamps are used to determine which student bid on each topic first to create such a Preference list for topics.

# Phase 1: Both Students S1 and S2 have selected R2 as their highest preference. Since there is a conflict, The preference of R2 is checked which shows that S2 had bid earlier than S1. Thus, the topic R2 is assigned to student S2.
# Phase 2: Now, S1 and S3 both want to review R1 but R1 was first bid on by S1 as shown in the preference. Hence, R1 is assigned to S1.
# Phase 3: The only topic remaining in this case is R3 which is also on the preference list of Student S3. Therefore S3 is assigned the topic R3.
#By the end of the 3 phases, All students are assigned to a Review. The same process needs to be repeated till the required number of reviews are assigned to each student. In each iteration, if a review is assigned to max number of students already, the review is removed from consideration.

==Implementation==
This section details the implementation of the code and will highlight each component of the design in the code and its intended purpose, as well as any modifications or testing. As the implementation progresses this section will be updated.

=== ''Overview'' ===

This project was split into 2 objectives, the Expertiza MVC implementation and the webservice algorithm. The Expertiza MVC follows the MVC standard such that the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Controller controller] handles directing the users to the correct views, as well as handling the communication with the webservice. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Model model] performs all necessary computations and data handling like the creation and storing of bids on reviews and assigning reviews to students. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Views views] render the corresponding webpage based on the actions of the user. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Webservice webservice] contains the algorithm outlined in the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Algorithm_Implementation_for_matching_students_and_topics Algorithm Implementation] section of Design. Our Expertiza controller sends this webservice bidding information and the algorithm returns the matched topics to be assigned.

 

=== ''Controller'' ===
The controller contains 6 methods that perform all the necessary functions. The first method, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_action_allowed? "action_allowed?"] performs authentication checks The second, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_index "index"], is used to redirect the user to a list of assigned reviews after the algorithm has been run. The third, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_show "show"], is used in conjunction with fourth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_set_priority "set_priority"], to update and store the bids when users make changes in the bidding page. The fifth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"], utilizes the sixth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_run_bidding_algorithm "run_bidding_algorithm"], to connect to the webservice, send and receive the correct bidding information for the algorithm.
 

====== Method: action_allowed? ======
This method performs authentication by allowing only certain types of users to access controller methods. The first portion of this method allows users 'Instructor', 'Teaching Assistant', 'Administrator', 'Super-Administrator', and 'Student' to access the methods 'show', 'set_priority', and 'index'. These are the methods that render the bidding pages and review listings. The other methods are restricted for 'Student' users.
 
[[File:Controller action allowed..PNG]]
 

====== Method: index ======
This method collects all the necessary information about the user, Assignment details, and Review Mapping details to render the 'review_bids/others_work', which lists the reviews that are assigned to the user. This page also includes a button to request more reviews which will add reviews up to the maximum number of reviews. To see the view rendered by this method, please go to [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#View:_others_work.html.erb "others_work"].
 
[[File:Controller_index.PNG]]
 

====== Method: show ======
This method renders the bidding page in which participants can select from a table of topics the topics they would prefer to review. It collects the necessary information about the user, Assignment, Topics, and current Review Bids to render this page. It is used in conjunction with [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_set_priority set_priority] where the users bidding information is stored, and the page updated. To see the bidding page that this function renders, visit the view [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#View:_show.html.erb "show"] 
[[File:Controller_show.PNG]]

 

====== Method: set_priority ======
In the view file [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#view:_show.html.erb "show"] there exists a line the calls this method when the page is updated or accessed. This method stores the user's bidding information in the schema database. This allows the model to access this information when collecting all the bidding information, as well as allowing users to refresh or return to this page with their selections staying in the bid table.
 
[[File:Controller_set_priority.PNG]]

 

====== Method: assign_bidding ======
This method utilizes methods in the model, as well as the method [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_run_bidding_algorithm "run_bidding_algorithm"] to collect the bidding information of all students of a particular Assignment, send it to the webservice algorithm, store the matched topics that are returned, and pass these matched topics to the model to be assigned. After this method is run, the students should have assigned reviews and bidding should be turned off.
[[File:Controller_assign_bidding.PNG]]
 

====== Method: run_bidding_algorithm ======
This method connects to the webservice using RestClient API, which is part of the 'require's at the top of the Controller. It will return a false if the connection fails for ay reason such as connection is not established or the algorithm has an internal error, otherwise it returns the matched topics from the algorithm as a 'json'. 
[[File:Controller_requirements.PNG]] 
[[File:Controller_run_bidding_algorithm.PNG]]
 

=== ''Model '' ===
The model contains 5 methods that are utilized by the controller to save bidding information per user or all users, retrieve bidding information per user or for all users, and assign topics to review to participants.
 

====== Method: get_bidding_data ======
This method is called by the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"] controller method. It receives the Assignment ID and a list of reviewers, which are student IDs. It creates a Hash to store bidding data into and iterates through all of the reviewers, calling [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_reviewer_bidding_data "reviewer_bidding_data"] to get each students bidding data and store it in the Hash. It returns a Hash with all the bidding data needed for the Webservice Algorithm to run.
[[File:Model_get_bidding_data.PNG]]
 

====== Method: assign_review_topics ======
This method is called by the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"] controller method. Given the Assignment ID, list of reviewers, and the matched topics returned by the webservice algorithm, it first clears any previously assigned reviews for that Assignment to prevent multiple assigns, then iterates through each reviewer and assigns them to the topics that the algorithm selected for them.
[[File:Model_assign_review_topics.PNG]]
 

====== Method: assign_topic_to_reviewer ======
This method is called by the method [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_review_topics "assign_review_topics"] and assigns a single topic to a single user to review given an Assignment ID.
[[File:Model_assign_topic_to_reviewer.PNG]]
 

====== Method: reviewer_self_topic ======
This method returns the topic id of the reviewer that is passed in. This method is utilized to retrieve the reviewers' Topic ID, which is needed for the webservice algorithm to prevent users from being assigned to review their own topic. This is also prevented in views, but this is an additional safety precaution. 
[[File:Model_reviewer_self_topic.PNG]]
 

====== Method: reviewer_bidding_data ======
This method is called by [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_get_bidding_data "get_bidding_data"]. It creates a hash to store all the reviewer's bidding information. It iterates through all the bids for a reviewer given the Assignment ID and it returns a Hash of the bidding data.
 
[[File:Model_reviewer_bidding_data.PNG]]
 

=== ''Views'' ===
There are two main views for this project. They are for bidding and for showing assigned reviews after the bidding algorithm has been run.
====== View: others_work.html.erb ======
This file renders the view where students see a list of the reviews they've been assigned, with appropriate links and related messages. It includes a "Request Additional Review" button that will allow students to view more reviews up to the max number of reviews allowed.
[[File:Views_others_work_pt1.PNG]] 
[[File:Views_others_work_pt2.PNG]] 
[[File:Views_others_work_pt3.PNG]] 
[[File:Views_others_work_pt4.PNG]] 
 
 
View rendered by above: 
[[File:Views_others_work_rendered.PNG]]

====== View: show.html.erb ======
This view file renders the Review Bidding page where participants can select which topics to review. It renders partials which are included below, and includes coloring effects from helper methods indicating likelihood of receiving that particular topic to review. 
[[File:Views_show_pt1.PNG]] 
[[File:Views_show_pt2.PNG]] 
 
 
Page rendered by above: (This rendering is an instructor view, the student view is the same without the "Edit Topics" column.) 
[[File:Views_show_rendered.PNG|900px]]
 

=== ''Webservice'' ===
===== ''app'' =====

The entry point to the Web service is the app.py file where we have the POST Method match_topics.
This receives the input in a json format from Expertiza. This json will have a list of all topics, list of all students and the topics each student selected and also the maximum number of review topics per student. Also, it has the timestamp information of the bidding time to give preference to early bidders. The input json is passed to the JsonParser class to create a Dictionary.

 
[[File:tab.jpg]]

 

===== ''json parser'' =====
The json parser converts the json into a Dictionary. The parser also creates a preferences dictionary for the Student to store their topic preferences and
one for Topics to store list of topics which have been selected by students and also the priority for the selection using the Timestamps.
The json parser does one more task of finding Popular topics. This is needed because topics that are in demand should be preserved for students who bid on it and to avoid assigning it to students who did not bid on anything. High demand topics are assigned a lower priority for such students so that there are other topics they can be assigned.

[[File:json1.png]]

 

[[File:json2.png]]

 

[[File:json3.png]]

 

===== ''Class Topic'' =====
 
The Topic class is the one that does the proposing to the students during the runtime.
There are 2 cases. In case 1 when the particular topic is asked by more students than the limit of number of students the one topic can be assigned to. In this case, it uses the priorities it already has in the dictionary to make sure the topic proposes only to those students who chose it and those who bid earlier.
In Case 2, There are less students who asked for the topic. Hence, the topic can propose to all the students who asked for it and still be under the limit. In the end of this step. All topics make their proposals to be assigned to some students.

[[File:json11.png]]

 

===== ''Class Student'' =====
 
Once the Topics propose to students, the Student has to accept some of the proposals.
First, each student sorts the proposals based on the students preferences. Now, the student accepts all the proposals they can upto the number of slots remaining for them. The max slots for the student is determined by the max_proposals_accepted field in the json. Once the acceptance is complete, the Topics are all updated so that the topics can adjust their remaining slots accordingly.

[[File:json12.png]]

===== ''Topic Matcher'' =====

Next, The Topics Matcher class is called which is the main class that runs the Algorithm.
Before running the algorithm, it creates a list of Student and Topic objects by using instances of Student and Topic class. Each student and Topic now has its own class so that it can be used to make proposals for assignment of topics.
 

[[File:json4.png]]

 
The get_student_topic_matches() method kicks off the Algorithm.
Each topic proposes to students based on the topic priorities as covered in the Topics section. It uses a max_topic_assignment_limit (say 4 for example) to make sure the same topic is not assigned to too many students.
Once the proposals are made, we iterate over students so that they can accept proposals as long as they have slots remaining to be assigned to.
 
The round is now complete. To determine if we need a next round, we ask the topics if they are done proposing using is_topics_done_proposing() method. As long as each topic has not reached its assignment limit and there are still students to propose to, the proposing continues for the next round.
Once every topic is done, it breaks out of the loop and we now have the list of students with equal number of assigned topics ( 3-4 for example) and this list is returned back to Expertiza as the Stable matching to do the assignment.

[[File:json5.png]]

===== ''Test Input'' =====

Sample Test case: 
Case: There are a few very popular topics and the students who bid for them and bid early should receive it.
Popular Topics: 3969, 3971, 3972

<code>{
"tid": [
3969,3970,3971,3972,3973,3974,3975,3976,3977
],
"users": {
"36239": {
"tid": [
3970, 3972
],
"otid": 3977,
"priority": [
2,1
],
"time": [
"Thu, 12 Nov 2020 12:01:06 EST -05:00","Thu, 12 Nov 2020 12:01:07 EST -05:00"
]
},
"36240": {
"tid": [],
"otid": 3976,
"priority": [],
"time": []
},
"36241": {
"tid": [
3969,3971,3972
],
"otid": 3975,
"priority": [
1,3,2
],
"time": [
"Thu, 12 Nov 2020 12:00:22 EST -05:00","Thu, 12 Nov 2020 12:00:25 EST -05:00","Thu, 12 Nov 2020 12:00:27 EST -05:00"
]
},
"36242": {
"tid": [
3969,3971,3973
],
"otid": 3974,
"priority": [
3,2,1
],
"time": [
"Wed, 11 Nov 2020 12:15:43 EST -05:00","Thu, 12 Nov 2020 11:59:40 EST -05:00","Thu, 12 Nov 2020 13:07:53 EST -05:00"
]
},
"36243": {
"tid": [
3971,3969,3970,3976
],
"otid": 3972,
"priority": [
4,3,2,1,5
],
"time": [
"Wed, 11 Nov 2020 11:34:50 EST -05:00","Wed, 11 Nov 2020 12:30:16 EST -05:00","Wed, 11 Nov 2020 12:30:19 EST -05:00","Thu, 12 Nov 2020 13:02:02 EST -05:00"
]
}
},
"max_accepted_proposals": 3
} </code>

 

===== ''Output'' =====

Since there is competition for these topics, the algorithm recognizes this and makes sure to assign them only to the students who bid for it.
Other students who picked other topics or did not pick anything are assigned different topics.

<code> {
"36239": [
3970,
3972,
3975
],
"36240": [
3973,
3974,
3972
],
"36241": [
3969,
3971,
3972
],
"36242": [
3969,
3971,
3973
],
"36243": [
3969,
3970,
3971
]
}</code>

==Test Plan==
This section outlines the test plan for this project including testing scenarios/edge cases, the manual testing plan, and the automatic/RSpec testing plan.

=== ''Testing Scenarios'' ===
This section outlines the various bidding scenarios that could occur. These scenarios should be tested in either '''Manual Testing''' or '''Automatic Testing'''.

==== Basic Bidding Scenario ====
*Reviewers bid on multiple projects at different times
** Bidding algorithm should give priority to reviewers that bid first and should assign based on reviewers ranking

==== Edge Cases ====
* All reviewers bid on the same project
** If different time stamps: bidding algorithm should give priority to reviewers that bid first
** If same time stamps: bidding algorithm should assign reviews randomly
* None of the reviews bid on any projects
** Bidding algorithm should assign reviews randomly
* Reviewer bids on their own project
** Bidding algorithm should not allow a reviewer to bid on their own project and should have validation to prevent assigning a reviewer their own project
* None of the reviewers bid on a specific project
** Bidding algorithm should still assign reviewers to this project

=== ''Manual Testing'' ===
Manual testing should be preformed on an Expertiza server and webserver platform 

==== Manual Testing Plan for Expertiza Server ====
* test that an instructor can allow review bidding for an assignment
* test that server routes correctly when review bidding is allowed
** test that task box links to bidding page before algorithm is run
** test that assignment->others work links to bidding page before algorithm is run
** test that task box links to review page with bidding information after algorithm is run
** test that assignment->others work links to review page with bidding information after algorithm is run
* test that the bidding UI is implemented as expected
* test that a participant can bid on projects to review
** test that a participant can move projects into the bidding column
** test that a participant can reorder projects in the bidding column
** test that a participant's bidding preferences save
** test that a participant can't bid for their own topic
* test that run algorithm button calls webserver with correct information and returns review assignments for each user

==== Directions for Manual Testing of Expertiza Server ====
1. Go to [http://152.7.99.70:8081/ testing server] 
2. Log in as username: '''instructor6''' password: '''password''' 
3. Go to '''Manage''' -> '''Assignments''' -> '''E2085 Manual Testing''' -> '''edit''' (pencil symbol)
:# In the '''General''' tab make sure '''Has teams?''' and '''Has topics?''' are checked
:# In the '''Topics''' tab check '''Allow review to choose which topic to review?''' then click '''save'''
:# In the '''Topics''' tab select '''Bidding''' then click '''save'''
4. Go to '''Assignments'''
:# '''E2085 Manual Testing review''' task should be listed in the main assignments box
::# This UI is based on '''instructor6''' being a '''participant''' assigned to topic '''Topic4'''
::# The UI blocks user from bidding on their own project
::# Check that user can bid on topics and reorder bids
::# Check that users topic preferences save on page refresh
::# Impersonate other participants and repeat bidding UI tests ('''student7601''', '''student7602''', '''student7603''', '''student7604''', '''student7605''', '''student7606''', '''student7607''', '''student7608''', '''student7609'')
5. Go to '''Manage''' -> '''Assignments''' -> '''E2085 Manual Testing''' -> '''edit''' (pencil symbol)
:# In the '''Topics''' tab click '''Run Review Algorithm''' button
6. Go to '''Assignments'''
:# '''E2085 Manual Testing review''' task should be listed in the main assignments box
::# '''E2085 Manual Testing review''' should now link to the reviewing page
::# The number of required reviews should already be displayed
::# A '''Request Another Review''' button should be present unless reviews listed is equal to the number of reviews allowed
::# '''Begin''' a review and test '''save''' and '''submit'''

==== Manual Testing Demo Video for Expertiza Server ====
Video demonstration of manual testing: [https://www.screencast.com/t/0rSg1zDW here]

=== ''Automatic/RSpec Testing '' ===
RSpec tests will need to be written to related controller(s) and model(s). 

==== RSpec Testing Plan ====
These RSpec tests:
* test basic functionality of review bids controller
* test basic functionality of review bid model

==== Directions for RSpec Testing ====
:# [https://docs.google.com/document/d/1tXmwju6R7KQbvycku-bdXxa6rXSUN4BMyvjY3ROmMSw/edit Set up Expertiza] on personal device using [https://github.com/uahamedncsu/expertiza forked repo]
:# Run the following commands:

# review bids controller:
bundle exec rspec spec/controllers/review_bids_controller_spec.rb

# review bid model:
bundle exec rspec spec/models/review_bid_spec.rb

==== RSpec Testing Demo Video ====
Video demonstration of RSpec tests passing: [https://www.screencast.com/t/uSWZHSX8s here]
 

==Team==
Ummu Kolusum Yasmin Ahamed Adam (uahamed) 
Ryan Grainger (rpgraing) 
Luis Delossantos (lgdeloss) 
Colleen "Bria" Engen (ceengen) 
'''Mentor:''' Saurabh Shingte (svshingt)

 

==Resources==
''Previous Implementations'': [[http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1986._Allow_reviewers_to_bid_on_what_to_review 1]], [[https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Spring_2020_-_E2021._Allow_reviewers_to_bid_on_what_to_review 2]] 
''Bidding Interface Implementation Pull Request'': [[https://github.com/expertiza/expertiza/pull/778 3]]

=== ''Relevant Links'' ===
''Github Repository for Expertiza'': https://github.com/uahamedncsu/expertiza/tree/beta 
''Github Repository for Web service'': https://github.com/uahamedncsu/IntelligentAssignment 
''Pull Request'': https://github.com/expertiza/expertiza/pull/1822 
''Expertiza Server'': http://152.7.99.70:8081/ 
''Web Service Server'': https://app-csc517.herokuapp.com/match_topics 
''Video Demo'': https://www.screencast.com/t/GJNOcfa9Ln

CSC/ECE 517 Fall 2020 - E2085. Allow reviewers to bid on what to review

2021-11-28T20:32:20Z

Aluthra: /* Method: index */

==Project Overview==
=== ''Background'' ===
Assigning reviews to users is a complicated process. Currently, reviews are still assigned using a first-come-first-served basis. However, reviews could be bid on like how topics are bid on. This would involve matching multiple students to review a submission up to the maximum reviewers for a submission.The manual selection process can be replaced with an automated bidding approach for this purpose.

=== ''Objectives'' ===
# Implement top trading cycles on a web service.
# Add front end code to allow bidding on topics and call the appropriate web service from the lottery controller.
# Add working test cases to test the new changes.
# Code should be inclined to the Beta branch
# Link to run bidding assignment should be on the page associated with the assignment actions.
# Code such as getter methods should be added to the model and not the controller.
#The number of reviews created by default needs to be checked. One way of achieving this is creating minimum reviews and then allowing users to request more reviews if required.

=== ''Previous Work'' ===
There have been two previous implementations:
* Fall 2019 [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1986._Allow_reviewers_to_bid_on_what_to_review here]
* Spring 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Spring_2020_-_E2021._Allow_reviewers_to_bid_on_what_to_review here]

=== ''Relevant Links'' ===
''Github Repository for Expertiza'': https://github.com/uahamedncsu/expertiza/tree/beta 
''Github Repository for Web service'': https://github.com/uahamedncsu/IntelligentAssignment 
''Pull Request'': https://github.com/expertiza/expertiza/pull/1822 
''Expertiza Server'': http://152.7.99.70:8081/ 
''Web Service Server'': https://app-csc517.herokuapp.com/match_topics 
''Video Demo'': https://www.screencast.com/t/GJNOcfa9Ln 

=== ''Files Modified'' ===
This is a list of all the files added or changed in completing this project. 
 
''Changed'': 
*config/routes.rb 

*config/webservices.yml

*app/controllers/student_review_controller.rb 

*app/views/assignments/edit/_topics.html.erb 

*app/views/student_task/list.html.erb 

*app/views/student_task/view.html.erb 

*spec/factories/factories.rb 

 
''Added'': 
*app/controllers/review_bids_controller.rb 

*app/helpers/review_bids_helper.rb 

*app/models/review_bid.rb 

*app/views/review_bids/_all_actions.html.erb 

*app/views/review_bids/_table_header.html.erb 

*app/views/review_bids/_table_line.html.erb 

*app/views/review_bids/_topic_names.html.erb 

*app/views/review_bids/others_work.html.erb 

*app/views/review_bids/show.html.erb 

*db/migrate/20201107203204_create_review_bids.rb 

*spec/controllers/review_bids_controller_spec.rb 

*spec/models/review_bid_spec.rb 

==Design==

=== ''User Interface'' ===

Currently, the users are offered the entire list of topics to review and are asked to select a topic to begin a review. Topics that cannot be reviewed or have already been reviewed the maximum number of times are greyed out and cannot be selected. There is also a check box in which the student can select "I don't care which topic I review" and will be randomly assigned a topic from the list. The current User Interface (UI) for this is shown below.
 
 
[[File:current_review_UI.PNG]]

 

As discussed in the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Project_Overview Project Overview] section, a bidding system similar to the bidding system used in topic selection is preferable to the current review selection system. The UI for this process will mirror the UI for the topic bidding system. The new UI will allow users to drag a topic from a list of all the topics on the left, to their priority list on the right. The ordering of the right list indicates the priority of the users preference. The colors indicate the likelihood of being assigned that topic. By utilizing the same UI, the consistency of the user experience and the DRY-ness of the code are increased. An example of the topic selection bidding UI is shown below.
 
 
[[File:expected_review_UI.PNG]]

 

Currently, an instructor has the option while creating or editing an assignment to allow a reviewers to select the topics they want to review. This is done by checking '''Has teams?''' and '''Has topics?''' in the '''General''' tab of an assignment then checking '''Allow reviewer to choose which topic to review?''' in the '''Topics''' tab. This interface can be used for allowing review bidding either by editing the current check box or adding an additional check box for '''Allow reviewer to bid on topics to review?'''.
 
 
[[File:Instructor_ui_for_review_bidding.png|1000px]]

=== ''Analyzing the problem'' ===

Currently, the student can either pick a topic or is randomly assigned a topic, from among the ones that are currently eligible to be reviewed. In this scenario, it is possible that some good topics are already picked by a lot of students and if someone is late to do the selecting, they miss out on it since it is grayed out.
 
To solve the above problem of not getting the topics they like, the participants must be assigned topics based on a list of topics they pre-select.
This is where an algorithm will come in handy. For picking a suitable algorithm, we should first identify the type of the problem.
 
Type of problem 
Each student should be able to bid on multiple assignments of their choice and the same assignment can be assigned to multiple participants for review. 
This requires a Many to Many mapping.

[[File:many2.png]]

The algorithm of choice for this type of stable allocations is Gale-Shapely algorithm. 

In the real world, there are cases where we face this problem:
<ul><li> Student choosing their future universities [https://www.jstor.org/stable/3132114?seq=1&cid=pdf-reference#references_tab_content '''School Choice Problem'''] (One-to-Many)</li>
<li> Men and women seeking each other to create families [https://en.wikipedia.org/wiki/Stable_marriage_problem '''Stable Marriage Problem'''] (One-to-One)</li>
<li> Internet services that need to be connected with users with the smallest amount of time </li></ul>

===''Algorithm Implementation for matching students and topics''===
The Gale-Shapely Algorithm has steps to make sure it is a Stable marriage/allocation.
An unstable allocation is when the participant does not like the topic assigned to them.
But it can happen that the participant 1 might like the review topic assigned to participant 2 and participant 2 likes the topic assigned to participant 1.
This is the unstable case. To make it stable, the topics need to be exchanged between these 2 participants so that both of them get the topic they wanted.

[[File:Snippy2.png]]
Source: [https://www.journalajrcos.com/index.php/AJRCOS/article/view/24749/46281 Stable Marriage Algorithm]

===''Assignment of priority ''===
When a student places a bid, the timestamp is recorded. Priority will be given to the earlier bidder. If the student had chosen only very few bids, then this person will be given a higher priority. Any topics that were not selected by the participant are added to bottom of their preference list in a random preference order to create a situation where the number of students and number of topics is equal. 

===''Algorithm Steps ''===
<li> For the first round, the Algorithm will assign topics to each student based on the Priority and each topic is assigned to a limited number of students based on the Limit. It picks one topic and begins assigning students to it. (Note: The Algorithm makes sure the participant is not in a group that worked on this particular topic.)
<li>If the one topic is assigned to enough participants already, then it is removed from the queue and the same process is repeated for the next topic.
<li> As this goes on, all students are matched even if all matches are not stable at this point.
<li> The Algorithm terminates when all Topics that have not reached the maximum assignment level have tried to accept every student.

 
 
The time complexity of this algorithm is O(n^2) 

===''Simulation ''===

[[File:gale-shapeley-1.png]]
[[File:gale-shapeley-2.png]]

In the simulation above, we have equal number of Students (S1, S2, S3) and Review topics to assign (R1, R2, R3). The students preference of topics is to left of student. The Review topics does not have any preferences. To have preferences, the timestamps are used to determine which student bid on each topic first to create such a Preference list for topics.

# Phase 1: Both Students S1 and S2 have selected R2 as their highest preference. Since there is a conflict, The preference of R2 is checked which shows that S2 had bid earlier than S1. Thus, the topic R2 is assigned to student S2.
# Phase 2: Now, S1 and S3 both want to review R1 but R1 was first bid on by S1 as shown in the preference. Hence, R1 is assigned to S1.
# Phase 3: The only topic remaining in this case is R3 which is also on the preference list of Student S3. Therefore S3 is assigned the topic R3.
#By the end of the 3 phases, All students are assigned to a Review. The same process needs to be repeated till the required number of reviews are assigned to each student. In each iteration, if a review is assigned to max number of students already, the review is removed from consideration.

==Implementation==
This section details the implementation of the code and will highlight each component of the design in the code and its intended purpose, as well as any modifications or testing. As the implementation progresses this section will be updated.

=== ''Overview'' ===

This project was split into 2 objectives, the Expertiza MVC implementation and the webservice algorithm. The Expertiza MVC follows the MVC standard such that the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Controller controller] handles directing the users to the correct views, as well as handling the communication with the webservice. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Model model] performs all necessary computations and data handling like the creation and storing of bids on reviews and assigning reviews to students. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Views views] render the corresponding webpage based on the actions of the user. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Webservice webservice] contains the algorithm outlined in the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Algorithm_Implementation_for_matching_students_and_topics Algorithm Implementation] section of Design. Our Expertiza controller sends this webservice bidding information and the algorithm returns the matched topics to be assigned.

 

=== ''Controller'' ===
The controller contains 6 methods that perform all the necessary functions. The first method, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_action_allowed? "action_allowed?"] performs authentication checks The second, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_index "index"], is used to redirect the user to a list of assigned reviews after the algorithm has been run. The third, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_show "show"], is used in conjunction with fourth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_set_priority "set_priority"], to update and store the bids when users make changes in the bidding page. The fifth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"], utilizes the sixth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_run_bidding_algorithm "run_bidding_algorithm"], to connect to the webservice, send and receive the correct bidding information for the algorithm.
 

====== Method: action_allowed? ======
This method performs authentication by allowing only certain types of users to access controller methods. The first portion of this method allows users 'Instructor', 'Teaching Assistant', 'Administrator', 'Super-Administrator', and 'Student' to access the methods 'show', 'set_priority', and 'index'. These are the methods that render the bidding pages and review listings. The other methods are restricted for 'Student' users.
 
[[File:Controller action allowed..PNG]]
 

====== Method: index ======
This method collects all the necessary information about the user, Assignment details, and Review Mapping details to render the 'review_bids/others_work', which lists the reviews that are assigned to the user. This page also includes a button to request more reviews which will add reviews up to the maximum number of reviews. To see the view rendered by this method, please go to [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#View:_others_work.html.erb "others_work"].
 
[[File:Controller_index.PNG]]
 

====== Method: show ======
This method renders the bidding page in which participants can select from a table of topics the topics they would prefer to review. It collects the necessary information about the user, Assignment, Topics, and current Review Bids to render this page. It is used in conjunction with [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_set_priority set_priority] where the users bidding information is stored, and the page updated. To see the bidding page that this function renders, visit the view [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#View:_show.html.erb "show"] 
[[File:Controller_show.PNG]]

 

====== Method: set_priority ======
In the view file [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#view:_show.html.erb "show"] there exists a line the calls this method when the page is updated or accessed. This method stores the user's bidding information in the schema database. This allows the model to access this information when collecting all the bidding information, as well as allowing users to refresh or return to this page with their selections staying in the bid table.
[[File:Controller_set_priority.PNG]]

 

====== Method: assign_bidding ======
This method utilizes methods in the model, as well as the method [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_run_bidding_algorithm "run_bidding_algorithm"] to collect the bidding information of all students of a particular Assignment, send it to the webservice algorithm, store the matched topics that are returned, and pass these matched topics to the model to be assigned. After this method is run, the students should have assigned reviews and bidding should be turned off.
[[File:Controller_assign_bidding.PNG]]
 

====== Method: run_bidding_algorithm ======
This method connects to the webservice using RestClient API, which is part of the 'require's at the top of the Controller. It will return a false if the connection fails for ay reason such as connection is not established or the algorithm has an internal error, otherwise it returns the matched topics from the algorithm as a 'json'. 
[[File:Controller_requirements.PNG]] 
[[File:Controller_run_bidding_algorithm.PNG]]
 

=== ''Model '' ===
The model contains 5 methods that are utilized by the controller to save bidding information per user or all users, retrieve bidding information per user or for all users, and assign topics to review to participants.
 

====== Method: get_bidding_data ======
This method is called by the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"] controller method. It receives the Assignment ID and a list of reviewers, which are student IDs. It creates a Hash to store bidding data into and iterates through all of the reviewers, calling [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_reviewer_bidding_data "reviewer_bidding_data"] to get each students bidding data and store it in the Hash. It returns a Hash with all the bidding data needed for the Webservice Algorithm to run.
[[File:Model_get_bidding_data.PNG]]
 

====== Method: assign_review_topics ======
This method is called by the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"] controller method. Given the Assignment ID, list of reviewers, and the matched topics returned by the webservice algorithm, it first clears any previously assigned reviews for that Assignment to prevent multiple assigns, then iterates through each reviewer and assigns them to the topics that the algorithm selected for them.
[[File:Model_assign_review_topics.PNG]]
 

====== Method: assign_topic_to_reviewer ======
This method is called by the method [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_review_topics "assign_review_topics"] and assigns a single topic to a single user to review given an Assignment ID.
[[File:Model_assign_topic_to_reviewer.PNG]]
 

====== Method: reviewer_self_topic ======
This method returns the topic id of the reviewer that is passed in. This method is utilized to retrieve the reviewers' Topic ID, which is needed for the webservice algorithm to prevent users from being assigned to review their own topic. This is also prevented in views, but this is an additional safety precaution. 
[[File:Model_reviewer_self_topic.PNG]]
 

====== Method: reviewer_bidding_data ======
This method is called by [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_get_bidding_data "get_bidding_data"]. It creates a hash to store all the reviewer's bidding information. It iterates through all the bids for a reviewer given the Assignment ID and it returns a Hash of the bidding data.
 
[[File:Model_reviewer_bidding_data.PNG]]
 

=== ''Views'' ===
There are two main views for this project. They are for bidding and for showing assigned reviews after the bidding algorithm has been run.
====== View: others_work.html.erb ======
This file renders the view where students see a list of the reviews they've been assigned, with appropriate links and related messages. It includes a "Request Additional Review" button that will allow students to view more reviews up to the max number of reviews allowed.
[[File:Views_others_work_pt1.PNG]] 
[[File:Views_others_work_pt2.PNG]] 
[[File:Views_others_work_pt3.PNG]] 
[[File:Views_others_work_pt4.PNG]] 
 
 
View rendered by above: 
[[File:Views_others_work_rendered.PNG]]

====== View: show.html.erb ======
This view file renders the Review Bidding page where participants can select which topics to review. It renders partials which are included below, and includes coloring effects from helper methods indicating likelihood of receiving that particular topic to review. 
[[File:Views_show_pt1.PNG]] 
[[File:Views_show_pt2.PNG]] 
 
 
Page rendered by above: (This rendering is an instructor view, the student view is the same without the "Edit Topics" column.) 
[[File:Views_show_rendered.PNG|900px]]
 

=== ''Webservice'' ===
===== ''app'' =====

The entry point to the Web service is the app.py file where we have the POST Method match_topics.
This receives the input in a json format from Expertiza. This json will have a list of all topics, list of all students and the topics each student selected and also the maximum number of review topics per student. Also, it has the timestamp information of the bidding time to give preference to early bidders. The input json is passed to the JsonParser class to create a Dictionary.

 
[[File:tab.jpg]]

 

===== ''json parser'' =====
The json parser converts the json into a Dictionary. The parser also creates a preferences dictionary for the Student to store their topic preferences and
one for Topics to store list of topics which have been selected by students and also the priority for the selection using the Timestamps.
The json parser does one more task of finding Popular topics. This is needed because topics that are in demand should be preserved for students who bid on it and to avoid assigning it to students who did not bid on anything. High demand topics are assigned a lower priority for such students so that there are other topics they can be assigned.

[[File:json1.png]]

 

[[File:json2.png]]

 

[[File:json3.png]]

 

===== ''Class Topic'' =====
 
The Topic class is the one that does the proposing to the students during the runtime.
There are 2 cases. In case 1 when the particular topic is asked by more students than the limit of number of students the one topic can be assigned to. In this case, it uses the priorities it already has in the dictionary to make sure the topic proposes only to those students who chose it and those who bid earlier.
In Case 2, There are less students who asked for the topic. Hence, the topic can propose to all the students who asked for it and still be under the limit. In the end of this step. All topics make their proposals to be assigned to some students.

[[File:json11.png]]

 

===== ''Class Student'' =====
 
Once the Topics propose to students, the Student has to accept some of the proposals.
First, each student sorts the proposals based on the students preferences. Now, the student accepts all the proposals they can upto the number of slots remaining for them. The max slots for the student is determined by the max_proposals_accepted field in the json. Once the acceptance is complete, the Topics are all updated so that the topics can adjust their remaining slots accordingly.

[[File:json12.png]]

===== ''Topic Matcher'' =====

Next, The Topics Matcher class is called which is the main class that runs the Algorithm.
Before running the algorithm, it creates a list of Student and Topic objects by using instances of Student and Topic class. Each student and Topic now has its own class so that it can be used to make proposals for assignment of topics.
 

[[File:json4.png]]

 
The get_student_topic_matches() method kicks off the Algorithm.
Each topic proposes to students based on the topic priorities as covered in the Topics section. It uses a max_topic_assignment_limit (say 4 for example) to make sure the same topic is not assigned to too many students.
Once the proposals are made, we iterate over students so that they can accept proposals as long as they have slots remaining to be assigned to.
 
The round is now complete. To determine if we need a next round, we ask the topics if they are done proposing using is_topics_done_proposing() method. As long as each topic has not reached its assignment limit and there are still students to propose to, the proposing continues for the next round.
Once every topic is done, it breaks out of the loop and we now have the list of students with equal number of assigned topics ( 3-4 for example) and this list is returned back to Expertiza as the Stable matching to do the assignment.

[[File:json5.png]]

===== ''Test Input'' =====

Sample Test case: 
Case: There are a few very popular topics and the students who bid for them and bid early should receive it.
Popular Topics: 3969, 3971, 3972

<code>{
"tid": [
3969,3970,3971,3972,3973,3974,3975,3976,3977
],
"users": {
"36239": {
"tid": [
3970, 3972
],
"otid": 3977,
"priority": [
2,1
],
"time": [
"Thu, 12 Nov 2020 12:01:06 EST -05:00","Thu, 12 Nov 2020 12:01:07 EST -05:00"
]
},
"36240": {
"tid": [],
"otid": 3976,
"priority": [],
"time": []
},
"36241": {
"tid": [
3969,3971,3972
],
"otid": 3975,
"priority": [
1,3,2
],
"time": [
"Thu, 12 Nov 2020 12:00:22 EST -05:00","Thu, 12 Nov 2020 12:00:25 EST -05:00","Thu, 12 Nov 2020 12:00:27 EST -05:00"
]
},
"36242": {
"tid": [
3969,3971,3973
],
"otid": 3974,
"priority": [
3,2,1
],
"time": [
"Wed, 11 Nov 2020 12:15:43 EST -05:00","Thu, 12 Nov 2020 11:59:40 EST -05:00","Thu, 12 Nov 2020 13:07:53 EST -05:00"
]
},
"36243": {
"tid": [
3971,3969,3970,3976
],
"otid": 3972,
"priority": [
4,3,2,1,5
],
"time": [
"Wed, 11 Nov 2020 11:34:50 EST -05:00","Wed, 11 Nov 2020 12:30:16 EST -05:00","Wed, 11 Nov 2020 12:30:19 EST -05:00","Thu, 12 Nov 2020 13:02:02 EST -05:00"
]
}
},
"max_accepted_proposals": 3
} </code>

 

===== ''Output'' =====

Since there is competition for these topics, the algorithm recognizes this and makes sure to assign them only to the students who bid for it.
Other students who picked other topics or did not pick anything are assigned different topics.

<code> {
"36239": [
3970,
3972,
3975
],
"36240": [
3973,
3974,
3972
],
"36241": [
3969,
3971,
3972
],
"36242": [
3969,
3971,
3973
],
"36243": [
3969,
3970,
3971
]
}</code>

==Test Plan==
This section outlines the test plan for this project including testing scenarios/edge cases, the manual testing plan, and the automatic/RSpec testing plan.

=== ''Testing Scenarios'' ===
This section outlines the various bidding scenarios that could occur. These scenarios should be tested in either '''Manual Testing''' or '''Automatic Testing'''.

==== Basic Bidding Scenario ====
*Reviewers bid on multiple projects at different times
** Bidding algorithm should give priority to reviewers that bid first and should assign based on reviewers ranking

==== Edge Cases ====
* All reviewers bid on the same project
** If different time stamps: bidding algorithm should give priority to reviewers that bid first
** If same time stamps: bidding algorithm should assign reviews randomly
* None of the reviews bid on any projects
** Bidding algorithm should assign reviews randomly
* Reviewer bids on their own project
** Bidding algorithm should not allow a reviewer to bid on their own project and should have validation to prevent assigning a reviewer their own project
* None of the reviewers bid on a specific project
** Bidding algorithm should still assign reviewers to this project

=== ''Manual Testing'' ===
Manual testing should be preformed on an Expertiza server and webserver platform 

==== Manual Testing Plan for Expertiza Server ====
* test that an instructor can allow review bidding for an assignment
* test that server routes correctly when review bidding is allowed
** test that task box links to bidding page before algorithm is run
** test that assignment->others work links to bidding page before algorithm is run
** test that task box links to review page with bidding information after algorithm is run
** test that assignment->others work links to review page with bidding information after algorithm is run
* test that the bidding UI is implemented as expected
* test that a participant can bid on projects to review
** test that a participant can move projects into the bidding column
** test that a participant can reorder projects in the bidding column
** test that a participant's bidding preferences save
** test that a participant can't bid for their own topic
* test that run algorithm button calls webserver with correct information and returns review assignments for each user

==== Directions for Manual Testing of Expertiza Server ====
1. Go to [http://152.7.99.70:8081/ testing server] 
2. Log in as username: '''instructor6''' password: '''password''' 
3. Go to '''Manage''' -> '''Assignments''' -> '''E2085 Manual Testing''' -> '''edit''' (pencil symbol)
:# In the '''General''' tab make sure '''Has teams?''' and '''Has topics?''' are checked
:# In the '''Topics''' tab check '''Allow review to choose which topic to review?''' then click '''save'''
:# In the '''Topics''' tab select '''Bidding''' then click '''save'''
4. Go to '''Assignments'''
:# '''E2085 Manual Testing review''' task should be listed in the main assignments box
::# This UI is based on '''instructor6''' being a '''participant''' assigned to topic '''Topic4'''
::# The UI blocks user from bidding on their own project
::# Check that user can bid on topics and reorder bids
::# Check that users topic preferences save on page refresh
::# Impersonate other participants and repeat bidding UI tests ('''student7601''', '''student7602''', '''student7603''', '''student7604''', '''student7605''', '''student7606''', '''student7607''', '''student7608''', '''student7609'')
5. Go to '''Manage''' -> '''Assignments''' -> '''E2085 Manual Testing''' -> '''edit''' (pencil symbol)
:# In the '''Topics''' tab click '''Run Review Algorithm''' button
6. Go to '''Assignments'''
:# '''E2085 Manual Testing review''' task should be listed in the main assignments box
::# '''E2085 Manual Testing review''' should now link to the reviewing page
::# The number of required reviews should already be displayed
::# A '''Request Another Review''' button should be present unless reviews listed is equal to the number of reviews allowed
::# '''Begin''' a review and test '''save''' and '''submit'''

==== Manual Testing Demo Video for Expertiza Server ====
Video demonstration of manual testing: [https://www.screencast.com/t/0rSg1zDW here]

=== ''Automatic/RSpec Testing '' ===
RSpec tests will need to be written to related controller(s) and model(s). 

==== RSpec Testing Plan ====
These RSpec tests:
* test basic functionality of review bids controller
* test basic functionality of review bid model

==== Directions for RSpec Testing ====
:# [https://docs.google.com/document/d/1tXmwju6R7KQbvycku-bdXxa6rXSUN4BMyvjY3ROmMSw/edit Set up Expertiza] on personal device using [https://github.com/uahamedncsu/expertiza forked repo]
:# Run the following commands:

# review bids controller:
bundle exec rspec spec/controllers/review_bids_controller_spec.rb

# review bid model:
bundle exec rspec spec/models/review_bid_spec.rb

==== RSpec Testing Demo Video ====
Video demonstration of RSpec tests passing: [https://www.screencast.com/t/uSWZHSX8s here]
 

==Team==
Ummu Kolusum Yasmin Ahamed Adam (uahamed) 
Ryan Grainger (rpgraing) 
Luis Delossantos (lgdeloss) 
Colleen "Bria" Engen (ceengen) 
'''Mentor:''' Saurabh Shingte (svshingt)

 

==Resources==
''Previous Implementations'': [[http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1986._Allow_reviewers_to_bid_on_what_to_review 1]], [[https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Spring_2020_-_E2021._Allow_reviewers_to_bid_on_what_to_review 2]] 
''Bidding Interface Implementation Pull Request'': [[https://github.com/expertiza/expertiza/pull/778 3]]

=== ''Relevant Links'' ===
''Github Repository for Expertiza'': https://github.com/uahamedncsu/expertiza/tree/beta 
''Github Repository for Web service'': https://github.com/uahamedncsu/IntelligentAssignment 
''Pull Request'': https://github.com/expertiza/expertiza/pull/1822 
''Expertiza Server'': http://152.7.99.70:8081/ 
''Web Service Server'': https://app-csc517.herokuapp.com/match_topics 
''Video Demo'': https://www.screencast.com/t/GJNOcfa9Ln

CSC/ECE 517 Fall 2020 - E2085. Allow reviewers to bid on what to review

2021-11-28T20:31:57Z

Aluthra: /* Controller */

==Project Overview==
=== ''Background'' ===
Assigning reviews to users is a complicated process. Currently, reviews are still assigned using a first-come-first-served basis. However, reviews could be bid on like how topics are bid on. This would involve matching multiple students to review a submission up to the maximum reviewers for a submission.The manual selection process can be replaced with an automated bidding approach for this purpose.

=== ''Objectives'' ===
# Implement top trading cycles on a web service.
# Add front end code to allow bidding on topics and call the appropriate web service from the lottery controller.
# Add working test cases to test the new changes.
# Code should be inclined to the Beta branch
# Link to run bidding assignment should be on the page associated with the assignment actions.
# Code such as getter methods should be added to the model and not the controller.
#The number of reviews created by default needs to be checked. One way of achieving this is creating minimum reviews and then allowing users to request more reviews if required.

=== ''Previous Work'' ===
There have been two previous implementations:
* Fall 2019 [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1986._Allow_reviewers_to_bid_on_what_to_review here]
* Spring 2020 [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Spring_2020_-_E2021._Allow_reviewers_to_bid_on_what_to_review here]

=== ''Relevant Links'' ===
''Github Repository for Expertiza'': https://github.com/uahamedncsu/expertiza/tree/beta 
''Github Repository for Web service'': https://github.com/uahamedncsu/IntelligentAssignment 
''Pull Request'': https://github.com/expertiza/expertiza/pull/1822 
''Expertiza Server'': http://152.7.99.70:8081/ 
''Web Service Server'': https://app-csc517.herokuapp.com/match_topics 
''Video Demo'': https://www.screencast.com/t/GJNOcfa9Ln 

=== ''Files Modified'' ===
This is a list of all the files added or changed in completing this project. 
 
''Changed'': 
*config/routes.rb 

*config/webservices.yml

*app/controllers/student_review_controller.rb 

*app/views/assignments/edit/_topics.html.erb 

*app/views/student_task/list.html.erb 

*app/views/student_task/view.html.erb 

*spec/factories/factories.rb 

 
''Added'': 
*app/controllers/review_bids_controller.rb 

*app/helpers/review_bids_helper.rb 

*app/models/review_bid.rb 

*app/views/review_bids/_all_actions.html.erb 

*app/views/review_bids/_table_header.html.erb 

*app/views/review_bids/_table_line.html.erb 

*app/views/review_bids/_topic_names.html.erb 

*app/views/review_bids/others_work.html.erb 

*app/views/review_bids/show.html.erb 

*db/migrate/20201107203204_create_review_bids.rb 

*spec/controllers/review_bids_controller_spec.rb 

*spec/models/review_bid_spec.rb 

==Design==

=== ''User Interface'' ===

Currently, the users are offered the entire list of topics to review and are asked to select a topic to begin a review. Topics that cannot be reviewed or have already been reviewed the maximum number of times are greyed out and cannot be selected. There is also a check box in which the student can select "I don't care which topic I review" and will be randomly assigned a topic from the list. The current User Interface (UI) for this is shown below.
 
 
[[File:current_review_UI.PNG]]

 

As discussed in the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Project_Overview Project Overview] section, a bidding system similar to the bidding system used in topic selection is preferable to the current review selection system. The UI for this process will mirror the UI for the topic bidding system. The new UI will allow users to drag a topic from a list of all the topics on the left, to their priority list on the right. The ordering of the right list indicates the priority of the users preference. The colors indicate the likelihood of being assigned that topic. By utilizing the same UI, the consistency of the user experience and the DRY-ness of the code are increased. An example of the topic selection bidding UI is shown below.
 
 
[[File:expected_review_UI.PNG]]

 

Currently, an instructor has the option while creating or editing an assignment to allow a reviewers to select the topics they want to review. This is done by checking '''Has teams?''' and '''Has topics?''' in the '''General''' tab of an assignment then checking '''Allow reviewer to choose which topic to review?''' in the '''Topics''' tab. This interface can be used for allowing review bidding either by editing the current check box or adding an additional check box for '''Allow reviewer to bid on topics to review?'''.
 
 
[[File:Instructor_ui_for_review_bidding.png|1000px]]

=== ''Analyzing the problem'' ===

Currently, the student can either pick a topic or is randomly assigned a topic, from among the ones that are currently eligible to be reviewed. In this scenario, it is possible that some good topics are already picked by a lot of students and if someone is late to do the selecting, they miss out on it since it is grayed out.
 
To solve the above problem of not getting the topics they like, the participants must be assigned topics based on a list of topics they pre-select.
This is where an algorithm will come in handy. For picking a suitable algorithm, we should first identify the type of the problem.
 
Type of problem 
Each student should be able to bid on multiple assignments of their choice and the same assignment can be assigned to multiple participants for review. 
This requires a Many to Many mapping.

[[File:many2.png]]

The algorithm of choice for this type of stable allocations is Gale-Shapely algorithm. 

In the real world, there are cases where we face this problem:
<ul><li> Student choosing their future universities [https://www.jstor.org/stable/3132114?seq=1&cid=pdf-reference#references_tab_content '''School Choice Problem'''] (One-to-Many)</li>
<li> Men and women seeking each other to create families [https://en.wikipedia.org/wiki/Stable_marriage_problem '''Stable Marriage Problem'''] (One-to-One)</li>
<li> Internet services that need to be connected with users with the smallest amount of time </li></ul>

===''Algorithm Implementation for matching students and topics''===
The Gale-Shapely Algorithm has steps to make sure it is a Stable marriage/allocation.
An unstable allocation is when the participant does not like the topic assigned to them.
But it can happen that the participant 1 might like the review topic assigned to participant 2 and participant 2 likes the topic assigned to participant 1.
This is the unstable case. To make it stable, the topics need to be exchanged between these 2 participants so that both of them get the topic they wanted.

[[File:Snippy2.png]]
Source: [https://www.journalajrcos.com/index.php/AJRCOS/article/view/24749/46281 Stable Marriage Algorithm]

===''Assignment of priority ''===
When a student places a bid, the timestamp is recorded. Priority will be given to the earlier bidder. If the student had chosen only very few bids, then this person will be given a higher priority. Any topics that were not selected by the participant are added to bottom of their preference list in a random preference order to create a situation where the number of students and number of topics is equal. 

===''Algorithm Steps ''===
<li> For the first round, the Algorithm will assign topics to each student based on the Priority and each topic is assigned to a limited number of students based on the Limit. It picks one topic and begins assigning students to it. (Note: The Algorithm makes sure the participant is not in a group that worked on this particular topic.)
<li>If the one topic is assigned to enough participants already, then it is removed from the queue and the same process is repeated for the next topic.
<li> As this goes on, all students are matched even if all matches are not stable at this point.
<li> The Algorithm terminates when all Topics that have not reached the maximum assignment level have tried to accept every student.

 
 
The time complexity of this algorithm is O(n^2) 

===''Simulation ''===

[[File:gale-shapeley-1.png]]
[[File:gale-shapeley-2.png]]

In the simulation above, we have equal number of Students (S1, S2, S3) and Review topics to assign (R1, R2, R3). The students preference of topics is to left of student. The Review topics does not have any preferences. To have preferences, the timestamps are used to determine which student bid on each topic first to create such a Preference list for topics.

# Phase 1: Both Students S1 and S2 have selected R2 as their highest preference. Since there is a conflict, The preference of R2 is checked which shows that S2 had bid earlier than S1. Thus, the topic R2 is assigned to student S2.
# Phase 2: Now, S1 and S3 both want to review R1 but R1 was first bid on by S1 as shown in the preference. Hence, R1 is assigned to S1.
# Phase 3: The only topic remaining in this case is R3 which is also on the preference list of Student S3. Therefore S3 is assigned the topic R3.
#By the end of the 3 phases, All students are assigned to a Review. The same process needs to be repeated till the required number of reviews are assigned to each student. In each iteration, if a review is assigned to max number of students already, the review is removed from consideration.

==Implementation==
This section details the implementation of the code and will highlight each component of the design in the code and its intended purpose, as well as any modifications or testing. As the implementation progresses this section will be updated.

=== ''Overview'' ===

This project was split into 2 objectives, the Expertiza MVC implementation and the webservice algorithm. The Expertiza MVC follows the MVC standard such that the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Controller controller] handles directing the users to the correct views, as well as handling the communication with the webservice. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Model model] performs all necessary computations and data handling like the creation and storing of bids on reviews and assigning reviews to students. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Views views] render the corresponding webpage based on the actions of the user. The [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Webservice webservice] contains the algorithm outlined in the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Algorithm_Implementation_for_matching_students_and_topics Algorithm Implementation] section of Design. Our Expertiza controller sends this webservice bidding information and the algorithm returns the matched topics to be assigned.

 

=== ''Controller'' ===
The controller contains 6 methods that perform all the necessary functions. The first method, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_action_allowed? "action_allowed?"] performs authentication checks The second, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_index "index"], is used to redirect the user to a list of assigned reviews after the algorithm has been run. The third, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_show "show"], is used in conjunction with fourth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_set_priority "set_priority"], to update and store the bids when users make changes in the bidding page. The fifth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"], utilizes the sixth, [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_run_bidding_algorithm "run_bidding_algorithm"], to connect to the webservice, send and receive the correct bidding information for the algorithm.
 

====== Method: action_allowed? ======
This method performs authentication by allowing only certain types of users to access controller methods. The first portion of this method allows users 'Instructor', 'Teaching Assistant', 'Administrator', 'Super-Administrator', and 'Student' to access the methods 'show', 'set_priority', and 'index'. These are the methods that render the bidding pages and review listings. The other methods are restricted for 'Student' users.
 
[[File:Controller action allowed..PNG]]
 

====== Method: index ======
This method collects all the necessary information about the user, Assignment details, and Review Mapping details to render the 'review_bids/others_work', which lists the reviews that are assigned to the user. This page also includes a button to request more reviews which will add reviews up to the maximum number of reviews. To see the view rendered by this method, please go to [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#View:_others_work.html.erb "others_work"].

[[File:Controller_index.PNG]]
 

====== Method: show ======
This method renders the bidding page in which participants can select from a table of topics the topics they would prefer to review. It collects the necessary information about the user, Assignment, Topics, and current Review Bids to render this page. It is used in conjunction with [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_set_priority set_priority] where the users bidding information is stored, and the page updated. To see the bidding page that this function renders, visit the view [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#View:_show.html.erb "show"] 
[[File:Controller_show.PNG]]

 

====== Method: set_priority ======
In the view file [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#view:_show.html.erb "show"] there exists a line the calls this method when the page is updated or accessed. This method stores the user's bidding information in the schema database. This allows the model to access this information when collecting all the bidding information, as well as allowing users to refresh or return to this page with their selections staying in the bid table.
[[File:Controller_set_priority.PNG]]

 

====== Method: assign_bidding ======
This method utilizes methods in the model, as well as the method [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_run_bidding_algorithm "run_bidding_algorithm"] to collect the bidding information of all students of a particular Assignment, send it to the webservice algorithm, store the matched topics that are returned, and pass these matched topics to the model to be assigned. After this method is run, the students should have assigned reviews and bidding should be turned off.
[[File:Controller_assign_bidding.PNG]]
 

====== Method: run_bidding_algorithm ======
This method connects to the webservice using RestClient API, which is part of the 'require's at the top of the Controller. It will return a false if the connection fails for ay reason such as connection is not established or the algorithm has an internal error, otherwise it returns the matched topics from the algorithm as a 'json'. 
[[File:Controller_requirements.PNG]] 
[[File:Controller_run_bidding_algorithm.PNG]]
 

=== ''Model '' ===
The model contains 5 methods that are utilized by the controller to save bidding information per user or all users, retrieve bidding information per user or for all users, and assign topics to review to participants.
 

====== Method: get_bidding_data ======
This method is called by the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"] controller method. It receives the Assignment ID and a list of reviewers, which are student IDs. It creates a Hash to store bidding data into and iterates through all of the reviewers, calling [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_reviewer_bidding_data "reviewer_bidding_data"] to get each students bidding data and store it in the Hash. It returns a Hash with all the bidding data needed for the Webservice Algorithm to run.
[[File:Model_get_bidding_data.PNG]]
 

====== Method: assign_review_topics ======
This method is called by the [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_bidding "assign_bidding"] controller method. Given the Assignment ID, list of reviewers, and the matched topics returned by the webservice algorithm, it first clears any previously assigned reviews for that Assignment to prevent multiple assigns, then iterates through each reviewer and assigns them to the topics that the algorithm selected for them.
[[File:Model_assign_review_topics.PNG]]
 

====== Method: assign_topic_to_reviewer ======
This method is called by the method [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_assign_review_topics "assign_review_topics"] and assigns a single topic to a single user to review given an Assignment ID.
[[File:Model_assign_topic_to_reviewer.PNG]]
 

====== Method: reviewer_self_topic ======
This method returns the topic id of the reviewer that is passed in. This method is utilized to retrieve the reviewers' Topic ID, which is needed for the webservice algorithm to prevent users from being assigned to review their own topic. This is also prevented in views, but this is an additional safety precaution. 
[[File:Model_reviewer_self_topic.PNG]]
 

====== Method: reviewer_bidding_data ======
This method is called by [https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Method:_get_bidding_data "get_bidding_data"]. It creates a hash to store all the reviewer's bidding information. It iterates through all the bids for a reviewer given the Assignment ID and it returns a Hash of the bidding data.
 
[[File:Model_reviewer_bidding_data.PNG]]
 

=== ''Views'' ===
There are two main views for this project. They are for bidding and for showing assigned reviews after the bidding algorithm has been run.
====== View: others_work.html.erb ======
This file renders the view where students see a list of the reviews they've been assigned, with appropriate links and related messages. It includes a "Request Additional Review" button that will allow students to view more reviews up to the max number of reviews allowed.
[[File:Views_others_work_pt1.PNG]] 
[[File:Views_others_work_pt2.PNG]] 
[[File:Views_others_work_pt3.PNG]] 
[[File:Views_others_work_pt4.PNG]] 
 
 
View rendered by above: 
[[File:Views_others_work_rendered.PNG]]

====== View: show.html.erb ======
This view file renders the Review Bidding page where participants can select which topics to review. It renders partials which are included below, and includes coloring effects from helper methods indicating likelihood of receiving that particular topic to review. 
[[File:Views_show_pt1.PNG]] 
[[File:Views_show_pt2.PNG]] 
 
 
Page rendered by above: (This rendering is an instructor view, the student view is the same without the "Edit Topics" column.) 
[[File:Views_show_rendered.PNG|900px]]
 

=== ''Webservice'' ===
===== ''app'' =====

The entry point to the Web service is the app.py file where we have the POST Method match_topics.
This receives the input in a json format from Expertiza. This json will have a list of all topics, list of all students and the topics each student selected and also the maximum number of review topics per student. Also, it has the timestamp information of the bidding time to give preference to early bidders. The input json is passed to the JsonParser class to create a Dictionary.

 
[[File:tab.jpg]]

 

===== ''json parser'' =====
The json parser converts the json into a Dictionary. The parser also creates a preferences dictionary for the Student to store their topic preferences and
one for Topics to store list of topics which have been selected by students and also the priority for the selection using the Timestamps.
The json parser does one more task of finding Popular topics. This is needed because topics that are in demand should be preserved for students who bid on it and to avoid assigning it to students who did not bid on anything. High demand topics are assigned a lower priority for such students so that there are other topics they can be assigned.

[[File:json1.png]]

 

[[File:json2.png]]

 

[[File:json3.png]]

 

===== ''Class Topic'' =====
 
The Topic class is the one that does the proposing to the students during the runtime.
There are 2 cases. In case 1 when the particular topic is asked by more students than the limit of number of students the one topic can be assigned to. In this case, it uses the priorities it already has in the dictionary to make sure the topic proposes only to those students who chose it and those who bid earlier.
In Case 2, There are less students who asked for the topic. Hence, the topic can propose to all the students who asked for it and still be under the limit. In the end of this step. All topics make their proposals to be assigned to some students.

[[File:json11.png]]

 

===== ''Class Student'' =====
 
Once the Topics propose to students, the Student has to accept some of the proposals.
First, each student sorts the proposals based on the students preferences. Now, the student accepts all the proposals they can upto the number of slots remaining for them. The max slots for the student is determined by the max_proposals_accepted field in the json. Once the acceptance is complete, the Topics are all updated so that the topics can adjust their remaining slots accordingly.

[[File:json12.png]]

===== ''Topic Matcher'' =====

Next, The Topics Matcher class is called which is the main class that runs the Algorithm.
Before running the algorithm, it creates a list of Student and Topic objects by using instances of Student and Topic class. Each student and Topic now has its own class so that it can be used to make proposals for assignment of topics.
 

[[File:json4.png]]

 
The get_student_topic_matches() method kicks off the Algorithm.
Each topic proposes to students based on the topic priorities as covered in the Topics section. It uses a max_topic_assignment_limit (say 4 for example) to make sure the same topic is not assigned to too many students.
Once the proposals are made, we iterate over students so that they can accept proposals as long as they have slots remaining to be assigned to.
 
The round is now complete. To determine if we need a next round, we ask the topics if they are done proposing using is_topics_done_proposing() method. As long as each topic has not reached its assignment limit and there are still students to propose to, the proposing continues for the next round.
Once every topic is done, it breaks out of the loop and we now have the list of students with equal number of assigned topics ( 3-4 for example) and this list is returned back to Expertiza as the Stable matching to do the assignment.

[[File:json5.png]]

===== ''Test Input'' =====

Sample Test case: 
Case: There are a few very popular topics and the students who bid for them and bid early should receive it.
Popular Topics: 3969, 3971, 3972

<code>{
"tid": [
3969,3970,3971,3972,3973,3974,3975,3976,3977
],
"users": {
"36239": {
"tid": [
3970, 3972
],
"otid": 3977,
"priority": [
2,1
],
"time": [
"Thu, 12 Nov 2020 12:01:06 EST -05:00","Thu, 12 Nov 2020 12:01:07 EST -05:00"
]
},
"36240": {
"tid": [],
"otid": 3976,
"priority": [],
"time": []
},
"36241": {
"tid": [
3969,3971,3972
],
"otid": 3975,
"priority": [
1,3,2
],
"time": [
"Thu, 12 Nov 2020 12:00:22 EST -05:00","Thu, 12 Nov 2020 12:00:25 EST -05:00","Thu, 12 Nov 2020 12:00:27 EST -05:00"
]
},
"36242": {
"tid": [
3969,3971,3973
],
"otid": 3974,
"priority": [
3,2,1
],
"time": [
"Wed, 11 Nov 2020 12:15:43 EST -05:00","Thu, 12 Nov 2020 11:59:40 EST -05:00","Thu, 12 Nov 2020 13:07:53 EST -05:00"
]
},
"36243": {
"tid": [
3971,3969,3970,3976
],
"otid": 3972,
"priority": [
4,3,2,1,5
],
"time": [
"Wed, 11 Nov 2020 11:34:50 EST -05:00","Wed, 11 Nov 2020 12:30:16 EST -05:00","Wed, 11 Nov 2020 12:30:19 EST -05:00","Thu, 12 Nov 2020 13:02:02 EST -05:00"
]
}
},
"max_accepted_proposals": 3
} </code>

 

===== ''Output'' =====

Since there is competition for these topics, the algorithm recognizes this and makes sure to assign them only to the students who bid for it.
Other students who picked other topics or did not pick anything are assigned different topics.

<code> {
"36239": [
3970,
3972,
3975
],
"36240": [
3973,
3974,
3972
],
"36241": [
3969,
3971,
3972
],
"36242": [
3969,
3971,
3973
],
"36243": [
3969,
3970,
3971
]
}</code>

==Test Plan==
This section outlines the test plan for this project including testing scenarios/edge cases, the manual testing plan, and the automatic/RSpec testing plan.

=== ''Testing Scenarios'' ===
This section outlines the various bidding scenarios that could occur. These scenarios should be tested in either '''Manual Testing''' or '''Automatic Testing'''.

==== Basic Bidding Scenario ====
*Reviewers bid on multiple projects at different times
** Bidding algorithm should give priority to reviewers that bid first and should assign based on reviewers ranking

==== Edge Cases ====
* All reviewers bid on the same project
** If different time stamps: bidding algorithm should give priority to reviewers that bid first
** If same time stamps: bidding algorithm should assign reviews randomly
* None of the reviews bid on any projects
** Bidding algorithm should assign reviews randomly
* Reviewer bids on their own project
** Bidding algorithm should not allow a reviewer to bid on their own project and should have validation to prevent assigning a reviewer their own project
* None of the reviewers bid on a specific project
** Bidding algorithm should still assign reviewers to this project

=== ''Manual Testing'' ===
Manual testing should be preformed on an Expertiza server and webserver platform 

==== Manual Testing Plan for Expertiza Server ====
* test that an instructor can allow review bidding for an assignment
* test that server routes correctly when review bidding is allowed
** test that task box links to bidding page before algorithm is run
** test that assignment->others work links to bidding page before algorithm is run
** test that task box links to review page with bidding information after algorithm is run
** test that assignment->others work links to review page with bidding information after algorithm is run
* test that the bidding UI is implemented as expected
* test that a participant can bid on projects to review
** test that a participant can move projects into the bidding column
** test that a participant can reorder projects in the bidding column
** test that a participant's bidding preferences save
** test that a participant can't bid for their own topic
* test that run algorithm button calls webserver with correct information and returns review assignments for each user

==== Directions for Manual Testing of Expertiza Server ====
1. Go to [http://152.7.99.70:8081/ testing server] 
2. Log in as username: '''instructor6''' password: '''password''' 
3. Go to '''Manage''' -> '''Assignments''' -> '''E2085 Manual Testing''' -> '''edit''' (pencil symbol)
:# In the '''General''' tab make sure '''Has teams?''' and '''Has topics?''' are checked
:# In the '''Topics''' tab check '''Allow review to choose which topic to review?''' then click '''save'''
:# In the '''Topics''' tab select '''Bidding''' then click '''save'''
4. Go to '''Assignments'''
:# '''E2085 Manual Testing review''' task should be listed in the main assignments box
::# This UI is based on '''instructor6''' being a '''participant''' assigned to topic '''Topic4'''
::# The UI blocks user from bidding on their own project
::# Check that user can bid on topics and reorder bids
::# Check that users topic preferences save on page refresh
::# Impersonate other participants and repeat bidding UI tests ('''student7601''', '''student7602''', '''student7603''', '''student7604''', '''student7605''', '''student7606''', '''student7607''', '''student7608''', '''student7609'')
5. Go to '''Manage''' -> '''Assignments''' -> '''E2085 Manual Testing''' -> '''edit''' (pencil symbol)
:# In the '''Topics''' tab click '''Run Review Algorithm''' button
6. Go to '''Assignments'''
:# '''E2085 Manual Testing review''' task should be listed in the main assignments box
::# '''E2085 Manual Testing review''' should now link to the reviewing page
::# The number of required reviews should already be displayed
::# A '''Request Another Review''' button should be present unless reviews listed is equal to the number of reviews allowed
::# '''Begin''' a review and test '''save''' and '''submit'''

==== Manual Testing Demo Video for Expertiza Server ====
Video demonstration of manual testing: [https://www.screencast.com/t/0rSg1zDW here]

=== ''Automatic/RSpec Testing '' ===
RSpec tests will need to be written to related controller(s) and model(s). 

==== RSpec Testing Plan ====
These RSpec tests:
* test basic functionality of review bids controller
* test basic functionality of review bid model

==== Directions for RSpec Testing ====
:# [https://docs.google.com/document/d/1tXmwju6R7KQbvycku-bdXxa6rXSUN4BMyvjY3ROmMSw/edit Set up Expertiza] on personal device using [https://github.com/uahamedncsu/expertiza forked repo]
:# Run the following commands:

# review bids controller:
bundle exec rspec spec/controllers/review_bids_controller_spec.rb

# review bid model:
bundle exec rspec spec/models/review_bid_spec.rb

==== RSpec Testing Demo Video ====
Video demonstration of RSpec tests passing: [https://www.screencast.com/t/uSWZHSX8s here]
 

==Team==
Ummu Kolusum Yasmin Ahamed Adam (uahamed) 
Ryan Grainger (rpgraing) 
Luis Delossantos (lgdeloss) 
Colleen "Bria" Engen (ceengen) 
'''Mentor:''' Saurabh Shingte (svshingt)

 

==Resources==
''Previous Implementations'': [[http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1986._Allow_reviewers_to_bid_on_what_to_review 1]], [[https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Spring_2020_-_E2021._Allow_reviewers_to_bid_on_what_to_review 2]] 
''Bidding Interface Implementation Pull Request'': [[https://github.com/expertiza/expertiza/pull/778 3]]

=== ''Relevant Links'' ===
''Github Repository for Expertiza'': https://github.com/uahamedncsu/expertiza/tree/beta 
''Github Repository for Web service'': https://github.com/uahamedncsu/IntelligentAssignment 
''Pull Request'': https://github.com/expertiza/expertiza/pull/1822 
''Expertiza Server'': http://152.7.99.70:8081/ 
''Web Service Server'': https://app-csc517.herokuapp.com/match_topics 
''Video Demo'': https://www.screencast.com/t/GJNOcfa9Ln

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-04T01:02:08Z

Aluthra: /* Previous Implementation VS New Implementation (may change as project continues) */

==Topic Overview==
===Statement of Problem===
Assigning reviews to users is a complicated process. Currently, reviews are still assigned using a first-come-first-served basis. However, reviews could be bid on like how topics are bid on. This would involve matching multiple students to review a submission up to the maximum reviewers for a submission. A similar feature is present in bidding for topics.

The goal of this topic is to allow both reviews and topics to be bid on by students following a bidding algorithm, and for both of these features to come from the same root functionality. This will involve refactoring the current bidding functionality of expertiza, as well as incorporating new code to allow this functionality to also be utilized for reviews.

==Prior Work==
Before starting, it is vital to note the substantial amount of prior work done on this task. The prior work done in Fall of 2020 (linked here [[https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Design]]) achieved much of the desired functionality for this task, and contains a comprehensive explanation of the underlying design principles behind the algorithm implemented. However, the implementation was rejected due to DRY violations. Thus, the primary goal of this project topic will be to rebuild and refactor this prior work to make it consistent with DRY principles and able to be merged into the expertiza environment.

A brief review will be given of the underlying design for the bidding algorithm. For a more exhaustive explanation, please see the aforementioned link to the prior year's work.

===Goals of this Topic===

# Update the prior work for this topic to be in line with the current state of the expertiza beta branch.
# DRY out prior implementation's controller methods by creating a single controller which handles bidding for both topics and reviews.
# DRY out prior implementation's views by implementing a view from scratch (rather than a code copy of topic bidding view).
# Create exhaustive tests for newly DRY-ed methods, including edge cases.
# Retain all functionality from prior work.

==Feature==
In the current state of implementation the reviews are still assigned using a first-come-first-served basis. Current state of the feature implementation contains bad code smell and needs improvement to resolve merge conflict and follow DRY principle in order to finalize the implementation. With the current state in mind, the new implementation will enable the capability that reviews could be bid on like how topics are bid on. Behind the scene, the process would involve matching multiple students to review a submission up to the maximum reviewers for a submission.

==Planned Changes==

===General Design Goals===

* Update the prior work for this topic to be in line with the current state of the expertiza beta branch.
* DRY out prior implementation's controller methods by creating a single controller which handles bidding for both topics and reviews.
* DRY out prior implementation's views by implementing a view from scratch (rather than a code copy of topic bidding view).
* Create exhaustive tests for newly DRY-ed methods, including edge cases.
* Retain all functionality from prior work.

===Specific Design Goals (may be expanded as project continues)===

* Inspect implementation structure to refactor out duplicated code
* Review ''def self.reviewer_self_topic'' in ''review_bid'' for refacotring to address DRY violation
* Modify ''app/views/review_bids/_all_actions.html.erb'' as fitted to integrate reafactored controller modules
* Update tests to test the refactored DRYed-out methods

===Previous Implementation VS New Implementation (may change as project continues)===
The before diagram describes the current implementation of the review bidding system. Currently, the available topics are pulled from different controllers, depending on if the user is bidding for topics or topics to review. The new implementation involves created a single controller that will populate both the topic and review bidding features.

[[File:Before-diagram.png]] [[File:After-diagram.png]]

==Github==
The Github corresponding to this task is publicly available here: https://github.com/WeiRui-Wang/expertiza

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: WeiRui Wang, Geoff Garrido, and Ayush Luthra. Our project mentor was Luis Delossantos (lgdeloss@ncsu.edu)

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-04T01:01:57Z

Aluthra: /* Before vs After Diagrams (may change as project continues) */

==Topic Overview==
===Statement of Problem===
Assigning reviews to users is a complicated process. Currently, reviews are still assigned using a first-come-first-served basis. However, reviews could be bid on like how topics are bid on. This would involve matching multiple students to review a submission up to the maximum reviewers for a submission. A similar feature is present in bidding for topics.

The goal of this topic is to allow both reviews and topics to be bid on by students following a bidding algorithm, and for both of these features to come from the same root functionality. This will involve refactoring the current bidding functionality of expertiza, as well as incorporating new code to allow this functionality to also be utilized for reviews.

==Prior Work==
Before starting, it is vital to note the substantial amount of prior work done on this task. The prior work done in Fall of 2020 (linked here [[https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Design]]) achieved much of the desired functionality for this task, and contains a comprehensive explanation of the underlying design principles behind the algorithm implemented. However, the implementation was rejected due to DRY violations. Thus, the primary goal of this project topic will be to rebuild and refactor this prior work to make it consistent with DRY principles and able to be merged into the expertiza environment.

A brief review will be given of the underlying design for the bidding algorithm. For a more exhaustive explanation, please see the aforementioned link to the prior year's work.

===Goals of this Topic===

# Update the prior work for this topic to be in line with the current state of the expertiza beta branch.
# DRY out prior implementation's controller methods by creating a single controller which handles bidding for both topics and reviews.
# DRY out prior implementation's views by implementing a view from scratch (rather than a code copy of topic bidding view).
# Create exhaustive tests for newly DRY-ed methods, including edge cases.
# Retain all functionality from prior work.

==Feature==
In the current state of implementation the reviews are still assigned using a first-come-first-served basis. Current state of the feature implementation contains bad code smell and needs improvement to resolve merge conflict and follow DRY principle in order to finalize the implementation. With the current state in mind, the new implementation will enable the capability that reviews could be bid on like how topics are bid on. Behind the scene, the process would involve matching multiple students to review a submission up to the maximum reviewers for a submission.

==Planned Changes==

===General Design Goals===

* Update the prior work for this topic to be in line with the current state of the expertiza beta branch.
* DRY out prior implementation's controller methods by creating a single controller which handles bidding for both topics and reviews.
* DRY out prior implementation's views by implementing a view from scratch (rather than a code copy of topic bidding view).
* Create exhaustive tests for newly DRY-ed methods, including edge cases.
* Retain all functionality from prior work.

===Specific Design Goals (may be expanded as project continues)===

* Inspect implementation structure to refactor out duplicated code
* Review ''def self.reviewer_self_topic'' in ''review_bid'' for refacotring to address DRY violation
* Modify ''app/views/review_bids/_all_actions.html.erb'' as fitted to integrate reafactored controller modules
* Update tests to test the refactored DRYed-out methods

===Previous Implementation VS New Implementation (may change as project continues)===
The before diagram describes the current implementation of the review bidding system. Currently, the available topics are pulled from different controllers, depending on if the user is bidding for topics or topics to review. The new implementation involves created a single controller that will populate both the topic and review bidding features.
[[File:Before-diagram.png]] [[File:After-diagram.png]]

==Github==
The Github corresponding to this task is publicly available here: https://github.com/WeiRui-Wang/expertiza

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: WeiRui Wang, Geoff Garrido, and Ayush Luthra. Our project mentor was Luis Delossantos (lgdeloss@ncsu.edu)

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-04T00:58:31Z

Aluthra: /* Before vs After Diagrams (may change as project continues) */

==Topic Overview==
===Statement of Problem===
Assigning reviews to users is a complicated process. Currently, reviews are still assigned using a first-come-first-served basis. However, reviews could be bid on like how topics are bid on. This would involve matching multiple students to review a submission up to the maximum reviewers for a submission. A similar feature is present in bidding for topics.

The goal of this topic is to allow both reviews and topics to be bid on by students following a bidding algorithm, and for both of these features to come from the same root functionality. This will involve refactoring the current bidding functionality of expertiza, as well as incorporating new code to allow this functionality to also be utilized for reviews.

==Prior Work==
Before starting, it is vital to note the substantial amount of prior work done on this task. The prior work done in Fall of 2020 (linked here [[https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Design]]) achieved much of the desired functionality for this task, and contains a comprehensive explanation of the underlying design principles behind the algorithm implemented. However, the implementation was rejected due to DRY violations. Thus, the primary goal of this project topic will be to rebuild and refactor this prior work to make it consistent with DRY principles and able to be merged into the expertiza environment.

A brief review will be given of the underlying design for the bidding algorithm. For a more exhaustive explanation, please see the aforementioned link to the prior year's work.

===Goals of this Topic===

# Update the prior work for this topic to be in line with the current state of the expertiza beta branch.
# DRY out prior implementation's controller methods by creating a single controller which handles bidding for both topics and reviews.
# DRY out prior implementation's views by implementing a view from scratch (rather than a code copy of topic bidding view).
# Create exhaustive tests for newly DRY-ed methods, including edge cases.
# Retain all functionality from prior work.

==Feature==
In the current state of implementation the reviews are still assigned using a first-come-first-served basis. Current state of the feature implementation contains bad code smell and needs improvement to resolve merge conflict and follow DRY principle in order to finalize the implementation. With the current state in mind, the new implementation will enable the capability that reviews could be bid on like how topics are bid on. Behind the scene, the process would involve matching multiple students to review a submission up to the maximum reviewers for a submission.

==Planned Changes==

===General Design Goals===

* Update the prior work for this topic to be in line with the current state of the expertiza beta branch.
* DRY out prior implementation's controller methods by creating a single controller which handles bidding for both topics and reviews.
* DRY out prior implementation's views by implementing a view from scratch (rather than a code copy of topic bidding view).
* Create exhaustive tests for newly DRY-ed methods, including edge cases.
* Retain all functionality from prior work.

===Specific Design Goals (may be expanded as project continues)===

* Inspect implementation structure to refactor out duplicated code
* Review ''def self.reviewer_self_topic'' in ''review_bid'' for refacotring to address DRY violation
* Modify ''app/views/review_bids/_all_actions.html.erb'' as fitted to integrate reafactored controller modules
* Update tests to test the refactored DRYed-out methods

===Before vs After Diagrams (may change as project continues)===
[[File:Before-diagram.png]] [[File:After-diagram.png]]

==Github==
The Github corresponding to this task is publicly available here: https://github.com/WeiRui-Wang/expertiza

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: WeiRui Wang, Geoff Garrido, and Ayush Luthra. Our project mentor was Luis Delossantos (lgdeloss@ncsu.edu)

File:After-diagram.png

2021-11-04T00:58:02Z

Aluthra:

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-04T00:57:14Z

Aluthra: /* Before vs After Diagrams (may change as project continues) */

==Topic Overview==
===Statement of Problem===
Assigning reviews to users is a complicated process. Currently, reviews are still assigned using a first-come-first-served basis. However, reviews could be bid on like how topics are bid on. This would involve matching multiple students to review a submission up to the maximum reviewers for a submission. A similar feature is present in bidding for topics.

The goal of this topic is to allow both reviews and topics to be bid on by students following a bidding algorithm, and for both of these features to come from the same root functionality. This will involve refactoring the current bidding functionality of expertiza, as well as incorporating new code to allow this functionality to also be utilized for reviews.

==Prior Work==
Before starting, it is vital to note the substantial amount of prior work done on this task. The prior work done in Fall of 2020 (linked here [[https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Design]]) achieved much of the desired functionality for this task, and contains a comprehensive explanation of the underlying design principles behind the algorithm implemented. However, the implementation was rejected due to DRY violations. Thus, the primary goal of this project topic will be to rebuild and refactor this prior work to make it consistent with DRY principles and able to be merged into the expertiza environment.

A brief review will be given of the underlying design for the bidding algorithm. For a more exhaustive explanation, please see the aforementioned link to the prior year's work.

===Goals of this Topic===

# Update the prior work for this topic to be in line with the current state of the expertiza beta branch.
# DRY out prior implementation's controller methods by creating a single controller which handles bidding for both topics and reviews.
# DRY out prior implementation's views by implementing a view from scratch (rather than a code copy of topic bidding view).
# Create exhaustive tests for newly DRY-ed methods, including edge cases.
# Retain all functionality from prior work.

==Feature==
In the current state of implementation the reviews are still assigned using a first-come-first-served basis. Current state of the feature implementation contains bad code smell and needs improvement to resolve merge conflict and follow DRY principle in order to finalize the implementation. With the current state in mind, the new implementation will enable the capability that reviews could be bid on like how topics are bid on. Behind the scene, the process would involve matching multiple students to review a submission up to the maximum reviewers for a submission.

==Planned Changes==

===General Design Goals===

* Update the prior work for this topic to be in line with the current state of the expertiza beta branch.
* DRY out prior implementation's controller methods by creating a single controller which handles bidding for both topics and reviews.
* DRY out prior implementation's views by implementing a view from scratch (rather than a code copy of topic bidding view).
* Create exhaustive tests for newly DRY-ed methods, including edge cases.
* Retain all functionality from prior work.

===Specific Design Goals (may be expanded as project continues)===

* Inspect implementation structure to refactor out duplicated code
* Review ''def self.reviewer_self_topic'' in ''review_bid'' for refacotring to address DRY violation
* Modify ''app/views/review_bids/_all_actions.html.erb'' as fitted to integrate reafactored controller modules
* Update tests to test the refactored DRYed-out methods

===Before vs After Diagrams (may change as project continues)===
[[File:Before-diagram.png]]

==Github==
The Github corresponding to this task is publicly available here: https://github.com/WeiRui-Wang/expertiza

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: WeiRui Wang, Geoff Garrido, and Ayush Luthra. Our project mentor was Luis Delossantos (lgdeloss@ncsu.edu)

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-04T00:56:48Z

Aluthra: /* Before vs After Diagrams (may change as project continues) */

==Topic Overview==
===Statement of Problem===
Assigning reviews to users is a complicated process. Currently, reviews are still assigned using a first-come-first-served basis. However, reviews could be bid on like how topics are bid on. This would involve matching multiple students to review a submission up to the maximum reviewers for a submission. A similar feature is present in bidding for topics.

The goal of this topic is to allow both reviews and topics to be bid on by students following a bidding algorithm, and for both of these features to come from the same root functionality. This will involve refactoring the current bidding functionality of expertiza, as well as incorporating new code to allow this functionality to also be utilized for reviews.

==Prior Work==
Before starting, it is vital to note the substantial amount of prior work done on this task. The prior work done in Fall of 2020 (linked here [[https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Design]]) achieved much of the desired functionality for this task, and contains a comprehensive explanation of the underlying design principles behind the algorithm implemented. However, the implementation was rejected due to DRY violations. Thus, the primary goal of this project topic will be to rebuild and refactor this prior work to make it consistent with DRY principles and able to be merged into the expertiza environment.

A brief review will be given of the underlying design for the bidding algorithm. For a more exhaustive explanation, please see the aforementioned link to the prior year's work.

===Goals of this Topic===

# Update the prior work for this topic to be in line with the current state of the expertiza beta branch.
# DRY out prior implementation's controller methods by creating a single controller which handles bidding for both topics and reviews.
# DRY out prior implementation's views by implementing a view from scratch (rather than a code copy of topic bidding view).
# Create exhaustive tests for newly DRY-ed methods, including edge cases.
# Retain all functionality from prior work.

==Feature==
In the current state of implementation the reviews are still assigned using a first-come-first-served basis. Current state of the feature implementation contains bad code smell and needs improvement to resolve merge conflict and follow DRY principle in order to finalize the implementation. With the current state in mind, the new implementation will enable the capability that reviews could be bid on like how topics are bid on. Behind the scene, the process would involve matching multiple students to review a submission up to the maximum reviewers for a submission.

==Planned Changes==

===General Design Goals===

* Update the prior work for this topic to be in line with the current state of the expertiza beta branch.
* DRY out prior implementation's controller methods by creating a single controller which handles bidding for both topics and reviews.
* DRY out prior implementation's views by implementing a view from scratch (rather than a code copy of topic bidding view).
* Create exhaustive tests for newly DRY-ed methods, including edge cases.
* Retain all functionality from prior work.

===Specific Design Goals (may be expanded as project continues)===

* Inspect implementation structure to refactor out duplicated code
* Review ''def self.reviewer_self_topic'' in ''review_bid'' for refacotring to address DRY violation
* Modify ''app/views/review_bids/_all_actions.html.erb'' as fitted to integrate reafactored controller modules
* Update tests to test the refactored DRYed-out methods

===Before vs After Diagrams (may change as project continues)===
[[File:Before-diagram.jpg]]

==Github==
The Github corresponding to this task is publicly available here: https://github.com/WeiRui-Wang/expertiza

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: WeiRui Wang, Geoff Garrido, and Ayush Luthra. Our project mentor was Luis Delossantos (lgdeloss@ncsu.edu)

File:Before-diagram.png

2021-11-04T00:55:39Z

Aluthra:

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-04T00:54:32Z

Aluthra: /* Planned Changes */

==Topic Overview==
===Statement of Problem===
Assigning reviews to users is a complicated process. Currently, reviews are still assigned using a first-come-first-served basis. However, reviews could be bid on like how topics are bid on. This would involve matching multiple students to review a submission up to the maximum reviewers for a submission. A similar feature is present in bidding for topics.

The goal of this topic is to allow both reviews and topics to be bid on by students following a bidding algorithm, and for both of these features to come from the same root functionality. This will involve refactoring the current bidding functionality of expertiza, as well as incorporating new code to allow this functionality to also be utilized for reviews.

==Prior Work==
Before starting, it is vital to note the substantial amount of prior work done on this task. The prior work done in Fall of 2020 (linked here [[https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2020_-_E2085._Allow_reviewers_to_bid_on_what_to_review#Design]]) achieved much of the desired functionality for this task, and contains a comprehensive explanation of the underlying design principles behind the algorithm implemented. However, the implementation was rejected due to DRY violations. Thus, the primary goal of this project topic will be to rebuild and refactor this prior work to make it consistent with DRY principles and able to be merged into the expertiza environment.

A brief review will be given of the underlying design for the bidding algorithm. For a more exhaustive explanation, please see the aforementioned link to the prior year's work.

===Goals of this Topic===

# Update the prior work for this topic to be in line with the current state of the expertiza beta branch.
# DRY out prior implementation's controller methods by creating a single controller which handles bidding for both topics and reviews.
# DRY out prior implementation's views by implementing a view from scratch (rather than a code copy of topic bidding view).
# Create exhaustive tests for newly DRY-ed methods, including edge cases.
# Retain all functionality from prior work.

==Feature==
In the current state of implementation the reviews are still assigned using a first-come-first-served basis. Current state of the feature implementation contains bad code smell and needs improvement to resolve merge conflict and follow DRY principle in order to finalize the implementation. With the current state in mind, the new implementation will enable the capability that reviews could be bid on like how topics are bid on. Behind the scene, the process would involve matching multiple students to review a submission up to the maximum reviewers for a submission.

==Planned Changes==

===General Design Goals===

* Update the prior work for this topic to be in line with the current state of the expertiza beta branch.
* DRY out prior implementation's controller methods by creating a single controller which handles bidding for both topics and reviews.
* DRY out prior implementation's views by implementing a view from scratch (rather than a code copy of topic bidding view).
* Create exhaustive tests for newly DRY-ed methods, including edge cases.
* Retain all functionality from prior work.

===Specific Design Goals (may be expanded as project continues)===

* Inspect implementation structure to refactor out duplicated code
* Review ''def self.reviewer_self_topic'' in ''review_bid'' for refacotring to address DRY violation
* Modify ''app/views/review_bids/_all_actions.html.erb'' as fitted to integrate reafactored controller modules
* Update tests to test the refactored DRYed-out methods

===Before vs After Diagrams (may change as project continues)===

==Github==
The Github corresponding to this task is publicly available here: https://github.com/WeiRui-Wang/expertiza

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: WeiRui Wang, Geoff Garrido, and Ayush Luthra. Our project mentor was Luis Delossantos (lgdeloss@ncsu.edu)

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-04T00:38:56Z

Aluthra: /* Diagrams */

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-04T00:38:35Z

Aluthra: /* Planned Changes */

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-03T19:21:20Z

Aluthra: /* Statement of Problem */

CSC/ECE 517 Fall 2021 - E2151. Allow reviewers to bid on what to review

2021-11-03T19:18:37Z

Aluthra: /* Contributors */

CSC/ECE 517 Fall 2021 - E2146. Introduce a Student View for instructors

2021-10-25T04:04:30Z

Aluthra: Adding mentor name to contributors section.

==Problem Statement==
Currently, Expertiza is designed such that an instructor can view all the pages a student can, and is given access to a view very similar to that of a student. However, the difference between the navigational bar shown to an instructor and that which is shown to a student means that an instructor cannot perfectly emulate the view that a student has of Expertiza.

This makes activities such as demonstrating how to navigate around the website and access certain parts of assignments more difficult. While an instructor can choose to impersonate a student, this is cumbersome if they simply wish to view things as a student would from within their instructor account.

Additionally, certain features of the navigational bar are redundant. The main feature in question is the ''Assignments'' navigational link, which navigates to the same location as ''Manage... -> Assignments''. Having such redundancy is confusing.

To summarize the problem, there is no easy way for an instructor to have the same view as a student at a moment's notice due to a difference between an instructor's and a student's navigational bars.

==Goal of this Topic==
The goal of this topic has two primary components:
*Remove the redundant "Assignments" link in the current navigational bar
*Add a new "Student View" link to the navigational bar. When clicked, the navigational bar changes to that of a student for the remainder of the session, with the addition of an "Instructor View" link. When that link is clicked, the navigational bar changes back to that of an instructor.

By making these changes, the unnecessary redundancy in the navigational bar is removed, and an instructor has an easy ability to demonstrate or visualize expertiza from the perspective of a student.

==Specific Statement of Intent==

The following is a precise, mechanical description of the intended changes.

When an instructor logs in, they should see a navigational bar with the following components:
''Home'' | ''Manage...'' | ''Survey Deployments'' | ''Course Evaluation'' | ''Profile'' | ''Contact Us'' | ''Student View''

These are the same navigational components that an instructor can currently view, with the removal of ''Assignments'' and the addition of ''Student View''.

If an instructor clicks on ''Student View'', they are redirected to the home page and their navigational bar changes to the following:

''Home'' | ''Assignments'' | ''Pending Surveys'' | ''Profile'' | ''Contact Us'' | ''Anonymized view'' | ''Instructor View''

These are the same navigational links that a student can view, with the addition of ''Instructor View''.

This navigational bar change should persist throughout the session, until the instructor chooses to change their view.

==Design==
To implement the feature, we first start with the design in our mind for the control flow as shown below.

[[File:UMLFLowDiagram_OSS1.png]]

==Implementation==

To implement these changes, our theoretical framework was as follows:
1. Add a link to the navigational bar which directs to a new controller and its controller method.
2. Write a controller method which flips the value of a session flag. This session flag determines which navigational bar you see at any time.
3. Whenever the navigational bar is loaded, this flag is checked. If in the "student view" state, it renders the student view version of the instructor's navigational bar. if not, it renders the instructor's normal navigational bar (both with the addition of the link to switch to the other state).

To make this change, the following files were affected:
*controllers/student_view_controller.rb: a controller with a single unique method, "flip_view", which flips the [:flip_user] session variable when called.
*views/shared/_navigation.html.erb: the view which handles rendering the navigation bar. Has been edited to check the status of [:flip_user] and render a view accordingly.
*config/role_instructor.yml: determines various aspects about those assigned the role of instructor. Only change made is the addition of the student_view and instructor_view Menu::Node objects, which are rendered in the navigation bar.
*config/routes.rb: the routes for expertiza. Modified to allow a route to the newly made "flip_view" method.
*spec/controllers/student_view_controller_spec.rb: A series of unit tests for the student view controller, ensuring proper functionality and permissions.

Each major change and/or will now be explained in detail.

===student_view_controller.rb===
This controller contains only two methods, flip_view and _action_allowed? The later is a method common to all controllers, which determines who has access to the method. This is defined for the student_view_controller such that only those with instructor privileges may use it.

The flip_view function is shown below.

def flip_view
if(session[:flip_user] == nil || session[:flip_user] == false) then
session[:flip_user] = true
redirect_to '/'
elsif(session[:flip_user] == true) then
session[:flip_user] = false
redirect_to '/'
end
end

This relatively simple method's purpose is to flip the parity of a session variable named :flip_user between true and false, before then redirecting back to the home page. It also sets this variable to true if uninitialized, as would be the case at the beginning of a session. The route to this method is specified in the routes file as "student_view/flip_view".

===_navigation.html.erb===
The purpose of this variable becomes clear when we investigate _navigation.html.erb. This file is the first step in rendering the navigation menu at the top. While this process does involve other files, the determination of which menu items to render is handled here, which is the point at which student view interacts. The relevant portion of code is given below:
...
<% if session[:menu] %>

<% if !session[:flip_user] || session[:flip_user] == nil %>
<%= render :partial => 'menu_items/suckerfish',
:locals => { items: session[:menu].get_menu(0) - [26, 59], level: 0 } %>

<% elsif session[:user].role.name == 'Instructor' %>
<% session[:menu] = Role.student.cache[:menu] %>
<%= render :partial => 'menu_items/suckerfish',
:locals => { items: session[:menu].get_menu(0), level: 0 } %>
<% session[:menu] = Role.instructor.cache[:menu] %>
<%= render :partial => 'menu_items/suckerfish',
:locals => { items: session[:menu].get_menu(0) & [59], level: 0 } %>
...

The second line is where the [:flip_user] session variable becomes relevant. If this variable is either false or nil (set to false or uninitialized), then the standard instructor view is rendered (the 26 and 59 subtracted from "items" are removing menu items exclusive to the Student View, Assignments and Instructor View, respectively). If this condition fails (and the user is an instructor), it implies the instructor has set that variable to true. In that case, the student view menu items are found (the cache for the student menu items is pulled from Role.student.cache[:menu]) and then rendered, with the addition of the "Instructor View" button (indicated by the & [59]).

Thus, by clicking the view button, the re-rendered page gains a newly updated navigation bar, depending on the state of session[:flip_user], which will continue to persist for the entire expertiza session.

===_role_instructor.yml===
It is typically said that one should not edit config files lightly, and so we made sure to be careful with our changes to role_instructor.yml. However, changes were necessary, as this file is what supplies the data called in _navigation.html.erb by Role.instructor.cache[:menu]. It took a while to determine this, but the navigation menu items displayed at the top of the Expertiza view are defined under the :menu: items of this file. Each listed Menu::Node item is rendered within the navigational menu, with associated label, name, and url.

Because there was no way to know what other files or controllers may rely on this file, the only change we made was the addition of two new menu nodes: student_view and instructor_view, both with a url leading to the route for student_view_controller's flip_user method. Each was added to all three databases (development, testing, and deployment), and each was assigned a corresponding id (58 for student_view, and 59 for instructor_view). The added code can be seen below:

...
58: &58 !ruby/object:Menu::Node
content_page_id:
controller_action_id: 58
id: 58
label: Student View
name: student_view
parent:
parent_id:
site_controller_id: 58
url: "/student_view/flip_view"
59: &59 !ruby/object:Menu::Node
content_page_id:
controller_action_id: 59
id: 59
label: Instructor View
name: instructor_view
parent:
parent_id:
site_controller_id: 59
url: "/student_view/flip_view"
...

Thus, these two nodes are now loaded when the navigational bar is loaded, allowing them to appear. However, 59 is removed from the normal instructor view, and is added back in with the student view.

It's worthy to note that, currently, the Assignments menu item is manually removed from the navigation bar in _navigation.html.erb (the subtracted 26 when rendering the instructor view). While it would presumably be more clean to remove the menu node object all together, due to an abundance of caution, we decided it best not to remove any data from this config file, as again, its contents may be used by other controllers and methods we are not aware of. Future development may be able to determine if this caution is unnecessary, and make this change.

This concludes all major changes and additions. All other modifications were minor changes already referenced, and so will not be discussed in detail.

==Test Plan==
In order to test the functionality of this change, a spec file was created to enforce unit tests on the student view controller, in a file titled "student_view_controller_spec.rb". The goals of this testing file were as follows:

*only instructors are able to switch between student and instructor view
*the flip user flag is worked as expected
*navigation is occurring as expected

We believe that, so long as the above functionality is working correctly, the controller is adequately tested. Testing that only instructors can access the method ensures that it is not accessed by inappropriate users such as students, matching similar functionality for features like student impersonation. Checking that the flip user flag works when navigating between pages and when calling the method ensures that the changes to the view will occur successfully, and that both student and instructor views can be toggled between. Finally, ensuring that navigation works as expected ensures that the method is being called correctly and is having no issues regarding redirection back to the homepage.

A reasonable criticism would be that only using a unit test for the controller would not test to ensure that a user could actually interact with the controller through a visible link in the view. Doing so would require a feature test, presumably involving something like Capybara ensuring that the link is visible and accessible, and that clicking it changes the view as expected. In fact, as was recently taught in Dr. Gehringer's Object Oriented Design class, this is arguably the only sort of test necessary. The internals of how the flip user flag functions and such can be considered internal functions, while we only care about an incoming command (switching views should lead to a measurable change in the navigation bar) and an outgoing command (clicking the switch view button sends a message to switch the views).

Unfortunately, for a reason we were unable to determine, the simplified view Capybara renders would not render the newly updated navigation bar. Various options and variations were tried to correct this issue, but no change had the desired effect (including ensuring we made our changes to the testing data for "role_instructor.yml"). This issue seems to only affect Capybara, as the link is visible and clickable on both the development build and a deployed Heroku build. For the time being, we were thus unable to create a feature test for this topic, although we strongly believe the feature to be working as intended. Should anyone work on this project in the future, this would be a great area of improvement.

Thus, for the moment, asserting the value of the flip_user variable is a stand-in for testing the incoming command (the :flip_user variable is treated as a public side-effect), and the access testing is a stand-in for the outgoing command (the controller must be accessible and able to make its changes).

===Actions taken before testing===
Before any testing occurs, an instructor and student are built using factories. Then, before each test, the current user is set to be the created instructor. This ensures we have the proper permissions to use the method. The created student is used for testing that we can only access this method as an instructor, not a student. The code for this test context is given below.

describe StudentViewController do
let(:instructor) { build(:instructor, id: 2) }
let(:student1) { build(:student, id: 30, name: :Amanda)}
# student view is only accessible to instructors, so we will make one
before(:each) do
stub_current_user(instructor, instructor.role.name, instructor.role)
end

===Access Tests===

The first component tested is to ensure that access is properly handled. First, we simply check if we are allowed to access the student_view_controller when logged in as an instructor. We then change our login to that of a student, and check that we are no longer able to take actions with the controller. The code that accomplishes this is given below.

describe "check access" do
it "instructor should be able to call flip user" do
expect(controller.action_allowed?).to be true
end
it "student should not be able to call flip user" do
stub_current_user(student1, student1.role.name, student1.role)
expect(controller.action_allowed?).to be false
end
end

===Student View Flag Tests===
To ensure that requesting the navigational view be flipped is working as expected, we first test that the flag is nil by default (defaulting to the instructor view), that calling "flip_view" once changes the flag to true, and that calling it again changes the flag to false. Assuming that the changes to the navigational view are working as intended (a different set of links is rendered depending upon the status of the :flip_user variable), this adequately tests that the primary functionality of the Student/Instructor View link is working. The test code is given below.

describe "the student view flag working as expected" do
it 'flag should not be initialized by default' do
expect(session[:flip_user]).to eq nil
end
it 'flag should be set to true after selecting student view' do
post :flip_view
expect(session[:flip_user]).to eq true
end
it 'flag should become false after using student view twice' do
post :flip_view
post :flip_view
expect(session[:flip_user]).to eq false
end
end

===Navigational Tests===
These last tests simply determine that, after flipping from one view to another, the user is redirected back to the home page. This redirection is done to avoid the edge cases involving a user being on a page that should not be accessible to the view they currently have selected (for example, being on an instructor-only page after changing to the student view). The code to test this is given below.

describe "navigation should work as intended" do
it 'should be redirected to homepage after entering student view' do
post :flip_view
expect(response).to redirect_to("/")
end
it 'should be redirected to homepage after exiting student view' do
post :flip_view
post :flip_view
expect(response).to redirect_to("/")
end
end
end

This concludes the testing for this feature. So long as all the above tests pass, we can be confident the controller method is working as intended, and thus that the student/instructor view is working properly.

==Feature Demo==
In place of a feature test, we here present a demo of the functionality working as intended. To summarize the contents, a user logs in as an instructor, clicks "Student View" in the navigational bar, and has their bar be changed to that of a student. They then click a newly added "Instructor View" link, and are returned to the navigational bar of an instructor.
[https://drive.google.com/file/d/1tArc7o5_mSfy8R4Yt3J9wMBDdcHUQtN_ Navi Student View for Instructor Feature Demo]

==Conclusions and Future Work==
We believe this feature to be working in its entirety and hope it will be a helpful tool to improve the teaching options of instructors.

If future changes were to be made to this feature, the following suggestions may be applicable:
*A means of performing feature tests. Due to an unknown error in capybara's simplified testing view, any tests specifically looking for the Instructor View button seemed to fail. This error has not occurred on any fully rendered version of the application, nor on the Heroku-hosted server. It would be benefitial to implement tests to ensure the link appears correctly for users, but it's not clear how to do so with this issue. Finding a way to feature test would be preferrable.
*As previously noted, if the Menu::Node objects listed in role_instructor.yml are exclusively used for rendering the navigational bar, the Assignments node object should be removed entirely, and the code where 26 is removed from the items rendered in _navigation.html.erb changed to reflect this removal.

==Github==
The Github corresponding to this task is publicly available here: https://github.com/Aeront39/expertiza

==Contributors==
This feature was created as part of Dr. Edward Gehringer's "CSC/ECE 517: Object-Oriented Design and Development" class, Fall 2021. The contributors were: WeiRui Wang, Geoff Garrido, and Ayush Luthra. Our project mentor was Yi Qiu (yqiu9@ncsu.edu)