Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-30T04:02:50Z

Svsakham: /* Links */

=== Team ===
*Akash Sarda, aksarda
*Sairam Sakhamuri, svsakham
*Sidhant Arora, sarora22
*Sai Harsha Nadendla, snadend2

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

The export functionality is however is already quite refactored to suit the strategy pattern, which is not changed by the previous group as well.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Design Plan ==

The design plan is to use Strategy patter to implement import functionality, so that all the import requests present in different models are routed through the ImportFileController. Right now since the import functionality is implemented in various model methods this leads to many if and else statements to check the type of model. So, we intent to generalize the import functionality by placing common code in the ImportFileController. But each model will still have its own import method. With this approach the redundancy is reduced by moving common code to ImportFileController and code will become DRY.

Other helper methods such as ImportFileHelper and ImportTopicHelper that are used to perform import functionality will also be removed, which keeps import functionality consistent. We will be using method overloading and overriding for the methods in ImportFileController to eliminate unnecessary if and else blocks.

To summarize our plan of changes:

* Redirect all import calls through ImportFileController
* Refactor ImportFileController by removing the redundant code and making code generic.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.

[[Image: DesignImport.PNG]]
== Implementation Details ==
=== Models ===

*ImportFileController is changed and has been made more generalized, many case statements are removed. The ImprortFileController is made so small and understandable. The import functionality is moved into the corresponding models. Previously code was present in different modules which was confusing now the functionality was all in one place.
*Note:- We have not removed the previous implementation of the import as some of the models are yet to be moved to new import framework and removing the previous code will break backward compatibility. The following code changes are made:

*Previous Implmentation
<pre>
def import_from_hash(session, params)
if params[:model] == "AssignmentTeam" or params[:model] == "CourseTeam"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
if params[:model] == "AssignmentTeam"
teamtype = AssignmentTeam
else
teamtype = CourseTeam
end
options = eval(params[:options])
options[:has_teamname] = params[:has_teamname]
Team.import(row_hash, params[:id], options, teamtype)
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "ReviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
ReviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "MetareviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
MetareviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'SignUpTopic' || params[:model] == 'SignUpSheet'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
if params[:optional_count] == '0'
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '1'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '2'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '3'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5], params[:select6]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
end
end
errors = []
begin
@header_integrated_body.each do |row_hash|
session[:assignment_id] = params[:id]
Object.const_get(params[:model]).import(row_hash, session, params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'AssignmentParticipant' || params[:model] == 'CourseParticipant'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header], contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
if params[:model] == 'AssignmentParticipant'
@header_integrated_body.each do |row_hash|
AssignmentParticipant.import(row_hash, session, params[:id])
end
elsif params[:model] == 'CourseParticipant'
@header_integrated_body.each do |row_hash|
CourseParticipant.import(row_hash, session, params[:id])
end
end
rescue
errors << $ERROR_INFO
end
else # params[:model] = "User"
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
@header_integrated_body.each do |row_hash|
User.import(row_hash, nil, session)
end
rescue StandardError
errors << $ERROR_INFO
end
end
</pre>
*Current Implementation
<pre>
def import
errors = import_from_hash(session, params)
err_msg = "The following errors were encountered during import.Other records may have been added. A second submission will not duplicate these records."
errors.each do |error|
err_msg = err_msg + error.to_s
end
err_msg += </ul>
if errors.empty?
ExpertizaLogger.info LoggerMessage.new(controller_name, session[:user].name, "The file has been successfully imported.", request)
undo_link("The file has been successfully imported.")
else
ExpertizaLogger.error LoggerMessage.new(controller_name, session[:user].name, err_msg, request)
flash[:error] = err_msg
end
redirect_to session[:return_to]
end
</pre>

*Assignment Participant is changed to use the newly implemented #import functionality. Changes are as shown below:

<pre>
def self.import(row_hash, session, id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?
raise ImportError, "The assignment with id #{id} was not found." if Assignment.find(id).nil?
unless AssignmentParticipant.exists?(user_id: user.id, parent_id: id)
new_part = AssignmentParticipant.new(user_id: user.id, parent_id: id)
new_part.set_handle
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

*Assignment Team is also changed to use the newly implemented #import functionality.

<pre>
def self.import(row_hash, session = nil, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The assignment with the id \"" + id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?" if Assignment.find_by(id: id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Course_team is is also changed to use the newly implemented #import functionality.
<pre>
def self.import(row_hash, session, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The course with the id \"" + id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?" if Course.find(id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Review response map's #import functionality is also updated to use the newly implemented #import functionality.
<pre>

def self.import(row_hash, _session, assignment_id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
reviewee_user_name = row_hash[:reviewee].to_s
reviewee_user = User.find_by(name: reviewee_user_name)
raise ArgumentError, "Cannot find reviewee user." unless reviewee_user
@@ -57,7 +58,7 @@ def self.import(row_hash, _session, assignment_id)
team_node = TeamNode.create(parent_id: assignment_id, node_object_id: reviewee_team.id)
TeamUserNode.create(parent_id: team_node.id, node_object_id: t_user.id)
end
row_hash[:reviewers].split.each do |reviewer|
reviewer_user_name = reviewer.to_s
reviewer_user = User.find_by(name: reviewer_user_name)
raise ArgumentError, "Cannot find reviewer user." unless reviewer_user
@@ -71,6 +72,20 @@ def self.import(row_hash, _session, assignment_id)
end
end

</pre>

*Refactored the Course_participant.rb to use newly added import functionality.
<pre>
def self.import(row_hash, session, id)
byebug
raise ArgumentError, "The record does not have enough items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?

user = User.find_by(name: row_hash[:name])
if user.nil?
raise ArgumentError, "The record containing #{row_hash[:name]} does not have enough items." if row_hash.length < 4
attributes = ImportFileHelper.define_attributes(row_hash)
user = ImportFileHelper.create_new_user(attributes, session)
end

course = Course.find_by(id)
raise ImportError, "The course with id " + id.to_s + " was not found." if course.nil?
unless CourseParticipant.exists?(user_id: user.id, parent_id: id)
CourseParticipant.create(user_id: user.id, parent_id: id)
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

With these new changes to #import functionality. The following modules review_response_map.rb, course_team.rb,assignment_team.rb,assignment_participant.rb,course_participant.rb have the "self.import" which will use the import functionality in the ImportFileController and they are made as concise as possible. Each module has specific checks while importing the file, so we can't have same code for all the module.

== Test Plan ==
=== Manual Testing ===
* Login with expertiza credentials.
* Navigate to Course Participant page by clicking on Manage => Courses => Add Participant.
* Click on Import course participant link and and browse for the file to import. Select the correct options for the type of file to be imported from Import Course Participant page. Please make sure the CSV file is in correct format as expected by the Import functionality. For course participant the csv should contain username | fullname | e-mail | password. Also make sure the users that are going to be imported are already present in expertiza database.
* Click in import participant button and the selected users should be imported.

=== Automated Tests ===
The import method has been implemented in a bunch of models which have been listed above. After preliminary analysis, we assume that the import functionality in a few of the models might have to be amended. These models are:
*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''review_response_map'''

Scenarios:
1. when assignment found and assignment participant does not exist, creates a new user and participant
2. when assignment cannot be found, creates a new user then raises an ImportError
3. when the assignment team does not have the required fields, raises ArgumentError
4. check what import/export actions are allowed for admin, instructor, TA, student
5. when course found and course participant does not exist, creates a new user and participant
6. when the course team does not have the required fields, raises ArgumentError

Newly added rspec test case to test the #import functionality in review_response_map_spec.rb

<pre>
it '#import' do
expect {ReviewResponseMap.import({reviewee: "name"}, nil, 1)}.to raise_error(ArgumentError, "Record does not contain required items.")
row_hash = {reviewee: "name", reviewers: "name1"}
#row_hash = {reviewee: "name", reviewers: ["name1"]}
session = nil
assignment_id = 1
# when reviewee user = nil
allow(User).to receive(:find_by).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Cannot find reviewee user.")
# when reviewee user exists but reviewee user is not a participant in this assignment
allow(User).to receive(:find_by).with(name: "name").and_return(student)
allow(AssignmentParticipant).to receive(:find_by).with(user_id: 1, parent_id: 1).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Reviewee user is not a participant in this assignment.")
end
</pre>

Newly added rspec test case to test the #import functionality in course_team_spec.rb

<pre>
describe ".import" do
let(:row) do
{teammembers: 'none'}
end
context "when a course team does not exist with id" do
it "raises ImportError" do
course_id = 1
allow(Course).to receive(:find).with(course_id).and_return(nil)
error_message = "The course with the id \"" + course_id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?"
expect { CourseTeam.import(row, nil, course_id, nil) }.
to raise_error(ImportError, error_message)
end
end

context "when the course team does not have the required fields" do
it "raises ArgumentError" do
expect { CourseTeam.import([], nil, 1, nil) }.
to raise_error(ArgumentError)
end
end

context "when a course team with the same id already exists" do
it "gets imported through Team.import" do
course_id = 1
options = []
allow(Course).to receive(:find).with(course_id).and_return(course)
expect(Team).to receive(:import_helper).with(row, course_id, options, instance_of(CourseTeam))
CourseTeam.import(row, nil, course_id, options)
end
end
end
</pre>

Newly added rspec test case to test the #import functionality in course_participant_spec.rb

<pre>
describe "CourseParticipant" do
let(:course) { build(:course, id: 1, name: 'ECE517') }
let(:assignment) { build(:assignment, id: 1, name: 'no assignment', participants: [participant], teams: [team]) }
let(:assignment_participant) {build(:participant, id: 1)}
'''describe "#copy" do
#before(:each) do
# byebug
# assignment = build(:assignment)
# course_participant = build(:course_participant)
# @assignment_participant = build(:participant)
#end
it "create a copy of participant" do
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(@assignment.id)).to be_an_instance_of(AssignmentParticipant)
end
it "returns nil if copy exist" do
allow(AssignmentParticipant).to receive(:where).and_return(AssignmentParticipant)
allow(AssignmentParticipant).to receive(:first).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(assignment.id)).to be_nil
end
end'''
</pre>

Test cases for Assignment_team.rb

<pre>
describe ".import" do
context "when an assignment team does not already exist with the same id" do
it "cannot be imported" do
assignment_id = 1
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(nil)
error_message = "The assignment with the id \"" + assignment_id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?"
expect { AssignmentTeam.import([], assignment_id, []) }.
to raise_error(ImportError, error_message)
end
end

context "when an assignment team with the same id already exists" do
it "gets imported through Team.import" do
row = []
assignment_id = 1
options = []
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(assignment)
allow(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
expect(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
AssignmentTeam.import(row, assignment_id, options)
end
end
end
</pre>

Added test for the assignment_participant_team.rb
<pre>
describe ".import" do
context 'when record is empty' do
it 'raises an ArgumentError' do
expect { AssignmentParticipant.import({}, nil, nil, nil) }.to raise_error(ArgumentError, 'No user id has been specified.')
end
end

context 'when no user is found by offered username' do
context 'when the record has less than 4 items' do
it 'raises an ArgumentError' do
row = {name: 'no one', fullname: 'no one', email: 'no_one@email.com'}
expect(ImportFileHelper).not_to receive(:create_new_user)
expect { AssignmentParticipant.import(row, nil, nil, nil) }.to raise_error('The record containing no one does not have enough items.')
end
end

context 'when new user needs to be created' do
let(:row) do
{name: 'no one', fullname: 'no one', email: 'name@email.com', role:'user_role_name', parent: 'user_parent_name'}
end
let(:attributes) do
{role_id: 1, name: 'no one', fullname: 'no one', email: 'name@email.com', email_on_submission: 'name@email.com',
email_on_review: 'name@email.com', email_on_review_of_review: 'name@email.com'}
end
let(:test_user) do
{name: 'abc', email: 'abcbbc@gmail.com'}
end
it 'create the user and number of mails sent should be 1' do
ActionMailer::Base.deliveries.clear
allow(ImportFileHelper).to receive(:define_attributes).with(row).and_return(attributes)
allow(ImportFileHelper).to receive(:create_new_user) do
test_user = User.new(name: 'abc', fullname: 'abc bbc', email: 'abcbbc@gmail.com')
test_user.id = 123
test_user.save!
test_user
end
#allow(ImportFileHelper).to receive(:create_new_user).with(attributes, {}).and_return()
allow(Assignment).to receive(:find).with(1).and_return(assignment)
allow(User).to receive(:exists?).with(name: 'no one').and_return(false)
allow(participant).to receive(:set_handle).and_return('handle')
allow(AssignmentParticipant).to receive(:exists?).and_return(false)
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(AssignmentParticipant).to receive(:set_handle)
expect{(AssignmentParticipant.import(row, nil, {}, 1))}.to change { ActionMailer::Base.deliveries.count }.by(1)
end
end
</pre>

== Future Scope ==
There are still some models that have import function but are not using the new framework like:-
* team
* metareview_response_map
* sign_up_sheet
* sign_up_topic
* user
Future work will include migrating these models to use new import framework and writing test cases for these scenarios. Also the previous implementation can then be safely removed.

== Links ==

*Pull Request: https://github.com/expertiza/expertiza/pull/2153
*Video: https://drive.google.com/file/d/19zPSOhh2AzHR-EF8v9wiwWrtGwtsUJL0/view?usp=sharing

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-30T04:02:36Z

Svsakham:

=== Team ===
*Akash Sarda, aksarda
*Sairam Sakhamuri, svsakham
*Sidhant Arora, sarora22
*Sai Harsha Nadendla, snadend2

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

The export functionality is however is already quite refactored to suit the strategy pattern, which is not changed by the previous group as well.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Design Plan ==

The design plan is to use Strategy patter to implement import functionality, so that all the import requests present in different models are routed through the ImportFileController. Right now since the import functionality is implemented in various model methods this leads to many if and else statements to check the type of model. So, we intent to generalize the import functionality by placing common code in the ImportFileController. But each model will still have its own import method. With this approach the redundancy is reduced by moving common code to ImportFileController and code will become DRY.

Other helper methods such as ImportFileHelper and ImportTopicHelper that are used to perform import functionality will also be removed, which keeps import functionality consistent. We will be using method overloading and overriding for the methods in ImportFileController to eliminate unnecessary if and else blocks.

To summarize our plan of changes:

* Redirect all import calls through ImportFileController
* Refactor ImportFileController by removing the redundant code and making code generic.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.

[[Image: DesignImport.PNG]]
== Implementation Details ==
=== Models ===

*ImportFileController is changed and has been made more generalized, many case statements are removed. The ImprortFileController is made so small and understandable. The import functionality is moved into the corresponding models. Previously code was present in different modules which was confusing now the functionality was all in one place.
*Note:- We have not removed the previous implementation of the import as some of the models are yet to be moved to new import framework and removing the previous code will break backward compatibility. The following code changes are made:

*Previous Implmentation
<pre>
def import_from_hash(session, params)
if params[:model] == "AssignmentTeam" or params[:model] == "CourseTeam"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
if params[:model] == "AssignmentTeam"
teamtype = AssignmentTeam
else
teamtype = CourseTeam
end
options = eval(params[:options])
options[:has_teamname] = params[:has_teamname]
Team.import(row_hash, params[:id], options, teamtype)
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "ReviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
ReviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "MetareviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
MetareviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'SignUpTopic' || params[:model] == 'SignUpSheet'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
if params[:optional_count] == '0'
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '1'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '2'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '3'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5], params[:select6]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
end
end
errors = []
begin
@header_integrated_body.each do |row_hash|
session[:assignment_id] = params[:id]
Object.const_get(params[:model]).import(row_hash, session, params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'AssignmentParticipant' || params[:model] == 'CourseParticipant'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header], contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
if params[:model] == 'AssignmentParticipant'
@header_integrated_body.each do |row_hash|
AssignmentParticipant.import(row_hash, session, params[:id])
end
elsif params[:model] == 'CourseParticipant'
@header_integrated_body.each do |row_hash|
CourseParticipant.import(row_hash, session, params[:id])
end
end
rescue
errors << $ERROR_INFO
end
else # params[:model] = "User"
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
@header_integrated_body.each do |row_hash|
User.import(row_hash, nil, session)
end
rescue StandardError
errors << $ERROR_INFO
end
end
</pre>
*Current Implementation
<pre>
def import
errors = import_from_hash(session, params)
err_msg = "The following errors were encountered during import.Other records may have been added. A second submission will not duplicate these records."
errors.each do |error|
err_msg = err_msg + error.to_s
end
err_msg += </ul>
if errors.empty?
ExpertizaLogger.info LoggerMessage.new(controller_name, session[:user].name, "The file has been successfully imported.", request)
undo_link("The file has been successfully imported.")
else
ExpertizaLogger.error LoggerMessage.new(controller_name, session[:user].name, err_msg, request)
flash[:error] = err_msg
end
redirect_to session[:return_to]
end
</pre>

*Assignment Participant is changed to use the newly implemented #import functionality. Changes are as shown below:

<pre>
def self.import(row_hash, session, id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?
raise ImportError, "The assignment with id #{id} was not found." if Assignment.find(id).nil?
unless AssignmentParticipant.exists?(user_id: user.id, parent_id: id)
new_part = AssignmentParticipant.new(user_id: user.id, parent_id: id)
new_part.set_handle
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

*Assignment Team is also changed to use the newly implemented #import functionality.

<pre>
def self.import(row_hash, session = nil, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The assignment with the id \"" + id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?" if Assignment.find_by(id: id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Course_team is is also changed to use the newly implemented #import functionality.
<pre>
def self.import(row_hash, session, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The course with the id \"" + id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?" if Course.find(id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Review response map's #import functionality is also updated to use the newly implemented #import functionality.
<pre>

def self.import(row_hash, _session, assignment_id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
reviewee_user_name = row_hash[:reviewee].to_s
reviewee_user = User.find_by(name: reviewee_user_name)
raise ArgumentError, "Cannot find reviewee user." unless reviewee_user
@@ -57,7 +58,7 @@ def self.import(row_hash, _session, assignment_id)
team_node = TeamNode.create(parent_id: assignment_id, node_object_id: reviewee_team.id)
TeamUserNode.create(parent_id: team_node.id, node_object_id: t_user.id)
end
row_hash[:reviewers].split.each do |reviewer|
reviewer_user_name = reviewer.to_s
reviewer_user = User.find_by(name: reviewer_user_name)
raise ArgumentError, "Cannot find reviewer user." unless reviewer_user
@@ -71,6 +72,20 @@ def self.import(row_hash, _session, assignment_id)
end
end

</pre>

*Refactored the Course_participant.rb to use newly added import functionality.
<pre>
def self.import(row_hash, session, id)
byebug
raise ArgumentError, "The record does not have enough items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?

user = User.find_by(name: row_hash[:name])
if user.nil?
raise ArgumentError, "The record containing #{row_hash[:name]} does not have enough items." if row_hash.length < 4
attributes = ImportFileHelper.define_attributes(row_hash)
user = ImportFileHelper.create_new_user(attributes, session)
end

course = Course.find_by(id)
raise ImportError, "The course with id " + id.to_s + " was not found." if course.nil?
unless CourseParticipant.exists?(user_id: user.id, parent_id: id)
CourseParticipant.create(user_id: user.id, parent_id: id)
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

With these new changes to #import functionality. The following modules review_response_map.rb, course_team.rb,assignment_team.rb,assignment_participant.rb,course_participant.rb have the "self.import" which will use the import functionality in the ImportFileController and they are made as concise as possible. Each module has specific checks while importing the file, so we can't have same code for all the module.

== Test Plan ==
=== Manual Testing ===
* Login with expertiza credentials.
* Navigate to Course Participant page by clicking on Manage => Courses => Add Participant.
* Click on Import course participant link and and browse for the file to import. Select the correct options for the type of file to be imported from Import Course Participant page. Please make sure the CSV file is in correct format as expected by the Import functionality. For course participant the csv should contain username | fullname | e-mail | password. Also make sure the users that are going to be imported are already present in expertiza database.
* Click in import participant button and the selected users should be imported.

=== Automated Tests ===
The import method has been implemented in a bunch of models which have been listed above. After preliminary analysis, we assume that the import functionality in a few of the models might have to be amended. These models are:
*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''review_response_map'''

Scenarios:
1. when assignment found and assignment participant does not exist, creates a new user and participant
2. when assignment cannot be found, creates a new user then raises an ImportError
3. when the assignment team does not have the required fields, raises ArgumentError
4. check what import/export actions are allowed for admin, instructor, TA, student
5. when course found and course participant does not exist, creates a new user and participant
6. when the course team does not have the required fields, raises ArgumentError

Newly added rspec test case to test the #import functionality in review_response_map_spec.rb

<pre>
it '#import' do
expect {ReviewResponseMap.import({reviewee: "name"}, nil, 1)}.to raise_error(ArgumentError, "Record does not contain required items.")
row_hash = {reviewee: "name", reviewers: "name1"}
#row_hash = {reviewee: "name", reviewers: ["name1"]}
session = nil
assignment_id = 1
# when reviewee user = nil
allow(User).to receive(:find_by).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Cannot find reviewee user.")
# when reviewee user exists but reviewee user is not a participant in this assignment
allow(User).to receive(:find_by).with(name: "name").and_return(student)
allow(AssignmentParticipant).to receive(:find_by).with(user_id: 1, parent_id: 1).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Reviewee user is not a participant in this assignment.")
end
</pre>

Newly added rspec test case to test the #import functionality in course_team_spec.rb

<pre>
describe ".import" do
let(:row) do
{teammembers: 'none'}
end
context "when a course team does not exist with id" do
it "raises ImportError" do
course_id = 1
allow(Course).to receive(:find).with(course_id).and_return(nil)
error_message = "The course with the id \"" + course_id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?"
expect { CourseTeam.import(row, nil, course_id, nil) }.
to raise_error(ImportError, error_message)
end
end

context "when the course team does not have the required fields" do
it "raises ArgumentError" do
expect { CourseTeam.import([], nil, 1, nil) }.
to raise_error(ArgumentError)
end
end

context "when a course team with the same id already exists" do
it "gets imported through Team.import" do
course_id = 1
options = []
allow(Course).to receive(:find).with(course_id).and_return(course)
expect(Team).to receive(:import_helper).with(row, course_id, options, instance_of(CourseTeam))
CourseTeam.import(row, nil, course_id, options)
end
end
end
</pre>

Newly added rspec test case to test the #import functionality in course_participant_spec.rb

<pre>
describe "CourseParticipant" do
let(:course) { build(:course, id: 1, name: 'ECE517') }
let(:assignment) { build(:assignment, id: 1, name: 'no assignment', participants: [participant], teams: [team]) }
let(:assignment_participant) {build(:participant, id: 1)}
'''describe "#copy" do
#before(:each) do
# byebug
# assignment = build(:assignment)
# course_participant = build(:course_participant)
# @assignment_participant = build(:participant)
#end
it "create a copy of participant" do
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(@assignment.id)).to be_an_instance_of(AssignmentParticipant)
end
it "returns nil if copy exist" do
allow(AssignmentParticipant).to receive(:where).and_return(AssignmentParticipant)
allow(AssignmentParticipant).to receive(:first).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(assignment.id)).to be_nil
end
end'''
</pre>

Test cases for Assignment_team.rb

<pre>
describe ".import" do
context "when an assignment team does not already exist with the same id" do
it "cannot be imported" do
assignment_id = 1
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(nil)
error_message = "The assignment with the id \"" + assignment_id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?"
expect { AssignmentTeam.import([], assignment_id, []) }.
to raise_error(ImportError, error_message)
end
end

context "when an assignment team with the same id already exists" do
it "gets imported through Team.import" do
row = []
assignment_id = 1
options = []
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(assignment)
allow(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
expect(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
AssignmentTeam.import(row, assignment_id, options)
end
end
end
</pre>

Added test for the assignment_participant_team.rb
<pre>
describe ".import" do
context 'when record is empty' do
it 'raises an ArgumentError' do
expect { AssignmentParticipant.import({}, nil, nil, nil) }.to raise_error(ArgumentError, 'No user id has been specified.')
end
end

context 'when no user is found by offered username' do
context 'when the record has less than 4 items' do
it 'raises an ArgumentError' do
row = {name: 'no one', fullname: 'no one', email: 'no_one@email.com'}
expect(ImportFileHelper).not_to receive(:create_new_user)
expect { AssignmentParticipant.import(row, nil, nil, nil) }.to raise_error('The record containing no one does not have enough items.')
end
end

context 'when new user needs to be created' do
let(:row) do
{name: 'no one', fullname: 'no one', email: 'name@email.com', role:'user_role_name', parent: 'user_parent_name'}
end
let(:attributes) do
{role_id: 1, name: 'no one', fullname: 'no one', email: 'name@email.com', email_on_submission: 'name@email.com',
email_on_review: 'name@email.com', email_on_review_of_review: 'name@email.com'}
end
let(:test_user) do
{name: 'abc', email: 'abcbbc@gmail.com'}
end
it 'create the user and number of mails sent should be 1' do
ActionMailer::Base.deliveries.clear
allow(ImportFileHelper).to receive(:define_attributes).with(row).and_return(attributes)
allow(ImportFileHelper).to receive(:create_new_user) do
test_user = User.new(name: 'abc', fullname: 'abc bbc', email: 'abcbbc@gmail.com')
test_user.id = 123
test_user.save!
test_user
end
#allow(ImportFileHelper).to receive(:create_new_user).with(attributes, {}).and_return()
allow(Assignment).to receive(:find).with(1).and_return(assignment)
allow(User).to receive(:exists?).with(name: 'no one').and_return(false)
allow(participant).to receive(:set_handle).and_return('handle')
allow(AssignmentParticipant).to receive(:exists?).and_return(false)
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(AssignmentParticipant).to receive(:set_handle)
expect{(AssignmentParticipant.import(row, nil, {}, 1))}.to change { ActionMailer::Base.deliveries.count }.by(1)
end
end
</pre>

== Future Scope ==
There are still some models that have import function but are not using the new framework like:-
* team
* metareview_response_map
* sign_up_sheet
* sign_up_topic
* user
Future work will include migrating these models to use new import framework and writing test cases for these scenarios. Also the previous implementation can then be safely removed.

== Links ==

*'''Pull Request: https://github.com/expertiza/expertiza/pull/2153
*'''Video: https://drive.google.com/file/d/19zPSOhh2AzHR-EF8v9wiwWrtGwtsUJL0/view?usp=sharing

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-30T03:58:24Z

Svsakham:

=== Team ===
*Akash Sarda, aksarda
*Sairam Sakhamuri, svsakham
*Sidhant Arora, sarora22
*Sai Harsha Nadendla, snadend2

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

The export functionality is however is already quite refactored to suit the strategy pattern, which is not changed by the previous group as well.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Design Plan ==

The design plan is to use Strategy patter to implement import functionality, so that all the import requests present in different models are routed through the ImportFileController. Right now since the import functionality is implemented in various model methods this leads to many if and else statements to check the type of model. So, we intent to generalize the import functionality by placing common code in the ImportFileController. But each model will still have its own import method. With this approach the redundancy is reduced by moving common code to ImportFileController and code will become DRY.

Other helper methods such as ImportFileHelper and ImportTopicHelper that are used to perform import functionality will also be removed, which keeps import functionality consistent. We will be using method overloading and overriding for the methods in ImportFileController to eliminate unnecessary if and else blocks.

To summarize our plan of changes:

* Redirect all import calls through ImportFileController
* Refactor ImportFileController by removing the redundant code and making code generic.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.

[[Image: DesignImport.PNG]]
== Implementation Details ==
=== Models ===

*ImportFileController is changed and has been made more generalized, many case statements are removed. The ImprortFileController is made so small and understandable. The import functionality is moved into the corresponding models. Previously code was present in different modules which was confusing now the functionality was all in one place.
*Note:- We have not removed the previous implementation of the import as some of the models are yet to be moved to new import framework and removing the previous code will break backward compatibility. The following code changes are made:

*Previous Implmentation
<pre>
def import_from_hash(session, params)
if params[:model] == "AssignmentTeam" or params[:model] == "CourseTeam"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
if params[:model] == "AssignmentTeam"
teamtype = AssignmentTeam
else
teamtype = CourseTeam
end
options = eval(params[:options])
options[:has_teamname] = params[:has_teamname]
Team.import(row_hash, params[:id], options, teamtype)
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "ReviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
ReviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "MetareviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
MetareviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'SignUpTopic' || params[:model] == 'SignUpSheet'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
if params[:optional_count] == '0'
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '1'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '2'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '3'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5], params[:select6]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
end
end
errors = []
begin
@header_integrated_body.each do |row_hash|
session[:assignment_id] = params[:id]
Object.const_get(params[:model]).import(row_hash, session, params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'AssignmentParticipant' || params[:model] == 'CourseParticipant'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header], contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
if params[:model] == 'AssignmentParticipant'
@header_integrated_body.each do |row_hash|
AssignmentParticipant.import(row_hash, session, params[:id])
end
elsif params[:model] == 'CourseParticipant'
@header_integrated_body.each do |row_hash|
CourseParticipant.import(row_hash, session, params[:id])
end
end
rescue
errors << $ERROR_INFO
end
else # params[:model] = "User"
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
@header_integrated_body.each do |row_hash|
User.import(row_hash, nil, session)
end
rescue StandardError
errors << $ERROR_INFO
end
end
</pre>
*Current Implementation
<pre>
def import
errors = import_from_hash(session, params)
err_msg = "The following errors were encountered during import.Other records may have been added. A second submission will not duplicate these records."
errors.each do |error|
err_msg = err_msg + error.to_s
end
err_msg += </ul>
if errors.empty?
ExpertizaLogger.info LoggerMessage.new(controller_name, session[:user].name, "The file has been successfully imported.", request)
undo_link("The file has been successfully imported.")
else
ExpertizaLogger.error LoggerMessage.new(controller_name, session[:user].name, err_msg, request)
flash[:error] = err_msg
end
redirect_to session[:return_to]
end
</pre>

*Assignment Participant is changed to use the newly implemented #import functionality. Changes are as shown below:

<pre>
def self.import(row_hash, session, id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?
raise ImportError, "The assignment with id #{id} was not found." if Assignment.find(id).nil?
unless AssignmentParticipant.exists?(user_id: user.id, parent_id: id)
new_part = AssignmentParticipant.new(user_id: user.id, parent_id: id)
new_part.set_handle
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

*Assignment Team is also changed to use the newly implemented #import functionality.

<pre>
def self.import(row_hash, session = nil, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The assignment with the id \"" + id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?" if Assignment.find_by(id: id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Course_team is is also changed to use the newly implemented #import functionality.
<pre>
def self.import(row_hash, session, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The course with the id \"" + id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?" if Course.find(id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Review response map's #import functionality is also updated to use the newly implemented #import functionality.
<pre>

def self.import(row_hash, _session, assignment_id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
reviewee_user_name = row_hash[:reviewee].to_s
reviewee_user = User.find_by(name: reviewee_user_name)
raise ArgumentError, "Cannot find reviewee user." unless reviewee_user
@@ -57,7 +58,7 @@ def self.import(row_hash, _session, assignment_id)
team_node = TeamNode.create(parent_id: assignment_id, node_object_id: reviewee_team.id)
TeamUserNode.create(parent_id: team_node.id, node_object_id: t_user.id)
end
row_hash[:reviewers].split.each do |reviewer|
reviewer_user_name = reviewer.to_s
reviewer_user = User.find_by(name: reviewer_user_name)
raise ArgumentError, "Cannot find reviewer user." unless reviewer_user
@@ -71,6 +72,20 @@ def self.import(row_hash, _session, assignment_id)
end
end

</pre>

*Refactored the Course_participant.rb to use newly added import functionality.
<pre>
def self.import(row_hash, session, id)
byebug
raise ArgumentError, "The record does not have enough items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?

user = User.find_by(name: row_hash[:name])
if user.nil?
raise ArgumentError, "The record containing #{row_hash[:name]} does not have enough items." if row_hash.length < 4
attributes = ImportFileHelper.define_attributes(row_hash)
user = ImportFileHelper.create_new_user(attributes, session)
end

course = Course.find_by(id)
raise ImportError, "The course with id " + id.to_s + " was not found." if course.nil?
unless CourseParticipant.exists?(user_id: user.id, parent_id: id)
CourseParticipant.create(user_id: user.id, parent_id: id)
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

With these new changes to #import functionality. The following modules review_response_map.rb, course_team.rb,assignment_team.rb,assignment_participant.rb,course_participant.rb have the "self.import" which will use the import functionality in the ImportFileController and they are made as concise as possible. Each module has specific checks while importing the file, so we can't have same code for all the module.

== Test Plan ==
=== Manual Testing ===
* Login with expertiza credentials.
* Navigate to Course Participant page by clicking on Manage => Courses => Add Participant.
* Click on Import course participant link and and browse for the file to import. Select the correct options for the type of file to be imported from Import Course Participant page. Please make sure the CSV file is in correct format as expected by the Import functionality. For course participant the csv should contain username | fullname | e-mail | password. Also make sure the users that are going to be imported are already present in expertiza database.
* Click in import participant button and the selected users should be imported.

=== Automated Tests ===
The import method has been implemented in a bunch of models which have been listed above. After preliminary analysis, we assume that the import functionality in a few of the models might have to be amended. These models are:
*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''review_response_map'''

Scenarios:
1. when assignment found and assignment participant does not exist, creates a new user and participant
2. when assignment cannot be found, creates a new user then raises an ImportError
3. when the assignment team does not have the required fields, raises ArgumentError
4. check what import/export actions are allowed for admin, instructor, TA, student
5. when course found and course participant does not exist, creates a new user and participant
6. when the course team does not have the required fields, raises ArgumentError

Newly added rspec test case to test the #import functionality in review_response_map_spec.rb

<pre>
it '#import' do
expect {ReviewResponseMap.import({reviewee: "name"}, nil, 1)}.to raise_error(ArgumentError, "Record does not contain required items.")
row_hash = {reviewee: "name", reviewers: "name1"}
#row_hash = {reviewee: "name", reviewers: ["name1"]}
session = nil
assignment_id = 1
# when reviewee user = nil
allow(User).to receive(:find_by).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Cannot find reviewee user.")
# when reviewee user exists but reviewee user is not a participant in this assignment
allow(User).to receive(:find_by).with(name: "name").and_return(student)
allow(AssignmentParticipant).to receive(:find_by).with(user_id: 1, parent_id: 1).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Reviewee user is not a participant in this assignment.")
end
</pre>

Newly added rspec test case to test the #import functionality in course_team_spec.rb

<pre>
describe ".import" do
let(:row) do
{teammembers: 'none'}
end
context "when a course team does not exist with id" do
it "raises ImportError" do
course_id = 1
allow(Course).to receive(:find).with(course_id).and_return(nil)
error_message = "The course with the id \"" + course_id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?"
expect { CourseTeam.import(row, nil, course_id, nil) }.
to raise_error(ImportError, error_message)
end
end

context "when the course team does not have the required fields" do
it "raises ArgumentError" do
expect { CourseTeam.import([], nil, 1, nil) }.
to raise_error(ArgumentError)
end
end

context "when a course team with the same id already exists" do
it "gets imported through Team.import" do
course_id = 1
options = []
allow(Course).to receive(:find).with(course_id).and_return(course)
expect(Team).to receive(:import_helper).with(row, course_id, options, instance_of(CourseTeam))
CourseTeam.import(row, nil, course_id, options)
end
end
end
</pre>

Newly added rspec test case to test the #import functionality in course_participant_spec.rb

<pre>
describe "CourseParticipant" do
let(:course) { build(:course, id: 1, name: 'ECE517') }
let(:assignment) { build(:assignment, id: 1, name: 'no assignment', participants: [participant], teams: [team]) }
let(:assignment_participant) {build(:participant, id: 1)}
'''describe "#copy" do
#before(:each) do
# byebug
# assignment = build(:assignment)
# course_participant = build(:course_participant)
# @assignment_participant = build(:participant)
#end
it "create a copy of participant" do
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(@assignment.id)).to be_an_instance_of(AssignmentParticipant)
end
it "returns nil if copy exist" do
allow(AssignmentParticipant).to receive(:where).and_return(AssignmentParticipant)
allow(AssignmentParticipant).to receive(:first).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(assignment.id)).to be_nil
end
end'''
</pre>

Test cases for Assignment_team.rb

<pre>
describe ".import" do
context "when an assignment team does not already exist with the same id" do
it "cannot be imported" do
assignment_id = 1
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(nil)
error_message = "The assignment with the id \"" + assignment_id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?"
expect { AssignmentTeam.import([], assignment_id, []) }.
to raise_error(ImportError, error_message)
end
end

context "when an assignment team with the same id already exists" do
it "gets imported through Team.import" do
row = []
assignment_id = 1
options = []
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(assignment)
allow(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
expect(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
AssignmentTeam.import(row, assignment_id, options)
end
end
end
</pre>

Added test for the assignment_participant_team.rb
<pre>
describe ".import" do
context 'when record is empty' do
it 'raises an ArgumentError' do
expect { AssignmentParticipant.import({}, nil, nil, nil) }.to raise_error(ArgumentError, 'No user id has been specified.')
end
end

context 'when no user is found by offered username' do
context 'when the record has less than 4 items' do
it 'raises an ArgumentError' do
row = {name: 'no one', fullname: 'no one', email: 'no_one@email.com'}
expect(ImportFileHelper).not_to receive(:create_new_user)
expect { AssignmentParticipant.import(row, nil, nil, nil) }.to raise_error('The record containing no one does not have enough items.')
end
end

context 'when new user needs to be created' do
let(:row) do
{name: 'no one', fullname: 'no one', email: 'name@email.com', role:'user_role_name', parent: 'user_parent_name'}
end
let(:attributes) do
{role_id: 1, name: 'no one', fullname: 'no one', email: 'name@email.com', email_on_submission: 'name@email.com',
email_on_review: 'name@email.com', email_on_review_of_review: 'name@email.com'}
end
let(:test_user) do
{name: 'abc', email: 'abcbbc@gmail.com'}
end
it 'create the user and number of mails sent should be 1' do
ActionMailer::Base.deliveries.clear
allow(ImportFileHelper).to receive(:define_attributes).with(row).and_return(attributes)
allow(ImportFileHelper).to receive(:create_new_user) do
test_user = User.new(name: 'abc', fullname: 'abc bbc', email: 'abcbbc@gmail.com')
test_user.id = 123
test_user.save!
test_user
end
#allow(ImportFileHelper).to receive(:create_new_user).with(attributes, {}).and_return()
allow(Assignment).to receive(:find).with(1).and_return(assignment)
allow(User).to receive(:exists?).with(name: 'no one').and_return(false)
allow(participant).to receive(:set_handle).and_return('handle')
allow(AssignmentParticipant).to receive(:exists?).and_return(false)
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(AssignmentParticipant).to receive(:set_handle)
expect{(AssignmentParticipant.import(row, nil, {}, 1))}.to change { ActionMailer::Base.deliveries.count }.by(1)
end
end
</pre>

== Future Scope ==
There are still some models that have import function but are not using the new framework like:-
* team
* metareview_response_map
* sign_up_sheet
* sign_up_topic
* user
Future work will include migrating these models to use new import framework and writing test cases for these scenarios. Also the previous implementation can then be safely removed.

== Links ==

*'''Pull Request: https://github.com/expertiza/expertiza/pull/2153
*'''Video: https://drive.google.com/file/d/19zPSOhh2AzHR-EF8v9wiwWrtGwtsUJL0/view?usp=sharing

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-30T03:45:53Z

Svsakham:

=== Team ===
*Akash Sarda, aksarda
*Sairam Sakhamuri, svsakham
*Sidhant Arora, sarora22
*Sai Harsha Nadendla, snadend2

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

The export functionality is however is already quite refactored to suit the strategy pattern, which is not changed by the previous group as well.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Design Plan ==

The design plan is to use Strategy patter to implement import functionality, so that all the import requests present in different models are routed through the ImportFileController. Right now since the import functionality is implemented in various model methods this leads to many if and else statements to check the type of model. So, we intent to generalize the import functionality by placing common code in the ImportFileController. But each model will still have its own import method. With this approach the redundancy is reduced by moving common code to ImportFileController and code will become DRY.

Other helper methods such as ImportFileHelper and ImportTopicHelper that are used to perform import functionality will also be removed, which keeps import functionality consistent. We will be using method overloading and overriding for the methods in ImportFileController to eliminate unnecessary if and else blocks.

To summarize our plan of changes:

* Redirect all import calls through ImportFileController
* Refactor ImportFileController by removing the redundant code and making code generic.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.

[[Image: DesignImport.PNG]]
== Implementation Details ==
=== Models ===

*ImportFileController is changed and has been made more generalized, many case statements are removed. The ImprortFileController is made so small and understandable. The import functionality is moved into the corresponding models. Previously code was present in different modules which was confusing now the functionality was all in one place. The following code changes are follows:

*Previous Implmentation
<pre>
def import_from_hash(session, params)
if params[:model] == "AssignmentTeam" or params[:model] == "CourseTeam"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
if params[:model] == "AssignmentTeam"
teamtype = AssignmentTeam
else
teamtype = CourseTeam
end
options = eval(params[:options])
options[:has_teamname] = params[:has_teamname]
Team.import(row_hash, params[:id], options, teamtype)
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "ReviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
ReviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "MetareviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
MetareviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'SignUpTopic' || params[:model] == 'SignUpSheet'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
if params[:optional_count] == '0'
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '1'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '2'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '3'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5], params[:select6]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
end
end
errors = []
begin
@header_integrated_body.each do |row_hash|
session[:assignment_id] = params[:id]
Object.const_get(params[:model]).import(row_hash, session, params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'AssignmentParticipant' || params[:model] == 'CourseParticipant'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header], contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
if params[:model] == 'AssignmentParticipant'
@header_integrated_body.each do |row_hash|
AssignmentParticipant.import(row_hash, session, params[:id])
end
elsif params[:model] == 'CourseParticipant'
@header_integrated_body.each do |row_hash|
CourseParticipant.import(row_hash, session, params[:id])
end
end
rescue
errors << $ERROR_INFO
end
else # params[:model] = "User"
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
@header_integrated_body.each do |row_hash|
User.import(row_hash, nil, session)
end
rescue StandardError
errors << $ERROR_INFO
end
end
</pre>
*Current Implementation
<pre>
def import
errors = import_from_hash(session, params)
err_msg = "The following errors were encountered during import.Other records may have been added. A second submission will not duplicate these records."
errors.each do |error|
err_msg = err_msg + error.to_s
end
err_msg += </ul>
if errors.empty?
ExpertizaLogger.info LoggerMessage.new(controller_name, session[:user].name, "The file has been successfully imported.", request)
undo_link("The file has been successfully imported.")
else
ExpertizaLogger.error LoggerMessage.new(controller_name, session[:user].name, err_msg, request)
flash[:error] = err_msg
end
redirect_to session[:return_to]
end
</pre>

*Assignment Participant is changed to use the newly implemented #import functionality. Changes are as shown below:

<pre>
def self.import(row_hash, session, id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?
raise ImportError, "The assignment with id #{id} was not found." if Assignment.find(id).nil?
unless AssignmentParticipant.exists?(user_id: user.id, parent_id: id)
new_part = AssignmentParticipant.new(user_id: user.id, parent_id: id)
new_part.set_handle
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

*Assignment Team is also changed to use the newly implemented #import functionality.

<pre>
def self.import(row_hash, session = nil, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The assignment with the id \"" + id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?" if Assignment.find_by(id: id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Course_team is is also changed to use the newly implemented #import functionality.
<pre>
def self.import(row_hash, session, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The course with the id \"" + id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?" if Course.find(id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Review response map's #import functionality is also updated to use the newly implemented #import functionality.
<pre>

def self.import(row_hash, _session, assignment_id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
reviewee_user_name = row_hash[:reviewee].to_s
reviewee_user = User.find_by(name: reviewee_user_name)
raise ArgumentError, "Cannot find reviewee user." unless reviewee_user
@@ -57,7 +58,7 @@ def self.import(row_hash, _session, assignment_id)
team_node = TeamNode.create(parent_id: assignment_id, node_object_id: reviewee_team.id)
TeamUserNode.create(parent_id: team_node.id, node_object_id: t_user.id)
end
row_hash[:reviewers].split.each do |reviewer|
reviewer_user_name = reviewer.to_s
reviewer_user = User.find_by(name: reviewer_user_name)
raise ArgumentError, "Cannot find reviewer user." unless reviewer_user
@@ -71,6 +72,20 @@ def self.import(row_hash, _session, assignment_id)
end
end

</pre>

*Refactored the Course_participant.rb to use newly added import functionality.
<pre>
def self.import(row_hash, session, id)
byebug
raise ArgumentError, "The record does not have enough items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?

user = User.find_by(name: row_hash[:name])
if user.nil?
raise ArgumentError, "The record containing #{row_hash[:name]} does not have enough items." if row_hash.length < 4
attributes = ImportFileHelper.define_attributes(row_hash)
user = ImportFileHelper.create_new_user(attributes, session)
end

course = Course.find_by(id)
raise ImportError, "The course with id " + id.to_s + " was not found." if course.nil?
unless CourseParticipant.exists?(user_id: user.id, parent_id: id)
CourseParticipant.create(user_id: user.id, parent_id: id)
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

With these new changes to #import functionality. The following modules review_response_map.rb, course_team.rb,assignment_team.rb,assignment_participant.rb,course_participant.rb have the "self.import" which will use the import functionality in the ImportFileController and they are made as concise as possible. Each module has specific checks while importing the file, so we can't have same code for all the module.

== Test Plan ==
=== Manual Testing ===
* Login with expertiza credentials.
* Navigate to Course Participant page by clicking on Manage => Courses => Add Participant.
* Click on Import course participant link and and browse for the file to import. Select the correct options for the type of file to be imported from Import Course Participant page. Please make sure the CSV file is in correct format as expected by the Import functionality. For course participant the csv should contain username | fullname | e-mail | password. Also make sure the users that are going to be imported are already present in expertiza database.
* Click in import participant button and the selected users should be imported.

=== Automated Tests ===
The import method has been implemented in a bunch of models which have been listed above. After preliminary analysis, we assume that the import functionality in a few of the models might have to be amended. These models are:
*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''review_response_map'''

Scenarios:
1. when assignment found and assignment participant does not exist, creates a new user and participant
2. when assignment cannot be found, creates a new user then raises an ImportError
3. when the assignment team does not have the required fields, raises ArgumentError
4. check what import/export actions are allowed for admin, instructor, TA, student
5. when course found and course participant does not exist, creates a new user and participant
6. when the course team does not have the required fields, raises ArgumentError

Newly added rspec test case to test the #import functionality in review_response_map_spec.rb

<pre>
it '#import' do
expect {ReviewResponseMap.import({reviewee: "name"}, nil, 1)}.to raise_error(ArgumentError, "Record does not contain required items.")
row_hash = {reviewee: "name", reviewers: "name1"}
#row_hash = {reviewee: "name", reviewers: ["name1"]}
session = nil
assignment_id = 1
# when reviewee user = nil
allow(User).to receive(:find_by).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Cannot find reviewee user.")
# when reviewee user exists but reviewee user is not a participant in this assignment
allow(User).to receive(:find_by).with(name: "name").and_return(student)
allow(AssignmentParticipant).to receive(:find_by).with(user_id: 1, parent_id: 1).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Reviewee user is not a participant in this assignment.")
end
</pre>

Newly added rspec test case to test the #import functionality in course_team_spec.rb

<pre>
describe ".import" do
let(:row) do
{teammembers: 'none'}
end
context "when a course team does not exist with id" do
it "raises ImportError" do
course_id = 1
allow(Course).to receive(:find).with(course_id).and_return(nil)
error_message = "The course with the id \"" + course_id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?"
expect { CourseTeam.import(row, nil, course_id, nil) }.
to raise_error(ImportError, error_message)
end
end

context "when the course team does not have the required fields" do
it "raises ArgumentError" do
expect { CourseTeam.import([], nil, 1, nil) }.
to raise_error(ArgumentError)
end
end

context "when a course team with the same id already exists" do
it "gets imported through Team.import" do
course_id = 1
options = []
allow(Course).to receive(:find).with(course_id).and_return(course)
expect(Team).to receive(:import_helper).with(row, course_id, options, instance_of(CourseTeam))
CourseTeam.import(row, nil, course_id, options)
end
end
end
</pre>

Newly added rspec test case to test the #import functionality in course_participant_spec.rb

<pre>
describe "CourseParticipant" do
let(:course) { build(:course, id: 1, name: 'ECE517') }
let(:assignment) { build(:assignment, id: 1, name: 'no assignment', participants: [participant], teams: [team]) }
let(:assignment_participant) {build(:participant, id: 1)}
'''describe "#copy" do
#before(:each) do
# byebug
# assignment = build(:assignment)
# course_participant = build(:course_participant)
# @assignment_participant = build(:participant)
#end
it "create a copy of participant" do
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(@assignment.id)).to be_an_instance_of(AssignmentParticipant)
end
it "returns nil if copy exist" do
allow(AssignmentParticipant).to receive(:where).and_return(AssignmentParticipant)
allow(AssignmentParticipant).to receive(:first).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(assignment.id)).to be_nil
end
end'''
</pre>

Test cases for Assignment_team.rb

<pre>
describe ".import" do
context "when an assignment team does not already exist with the same id" do
it "cannot be imported" do
assignment_id = 1
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(nil)
error_message = "The assignment with the id \"" + assignment_id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?"
expect { AssignmentTeam.import([], assignment_id, []) }.
to raise_error(ImportError, error_message)
end
end

context "when an assignment team with the same id already exists" do
it "gets imported through Team.import" do
row = []
assignment_id = 1
options = []
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(assignment)
allow(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
expect(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
AssignmentTeam.import(row, assignment_id, options)
end
end
end
</pre>

Added test for the assignment_participant_team.rb
<pre>
describe ".import" do
context 'when record is empty' do
it 'raises an ArgumentError' do
expect { AssignmentParticipant.import({}, nil, nil, nil) }.to raise_error(ArgumentError, 'No user id has been specified.')
end
end

context 'when no user is found by offered username' do
context 'when the record has less than 4 items' do
it 'raises an ArgumentError' do
row = {name: 'no one', fullname: 'no one', email: 'no_one@email.com'}
expect(ImportFileHelper).not_to receive(:create_new_user)
expect { AssignmentParticipant.import(row, nil, nil, nil) }.to raise_error('The record containing no one does not have enough items.')
end
end

context 'when new user needs to be created' do
let(:row) do
{name: 'no one', fullname: 'no one', email: 'name@email.com', role:'user_role_name', parent: 'user_parent_name'}
end
let(:attributes) do
{role_id: 1, name: 'no one', fullname: 'no one', email: 'name@email.com', email_on_submission: 'name@email.com',
email_on_review: 'name@email.com', email_on_review_of_review: 'name@email.com'}
end
let(:test_user) do
{name: 'abc', email: 'abcbbc@gmail.com'}
end
it 'create the user and number of mails sent should be 1' do
ActionMailer::Base.deliveries.clear
allow(ImportFileHelper).to receive(:define_attributes).with(row).and_return(attributes)
allow(ImportFileHelper).to receive(:create_new_user) do
test_user = User.new(name: 'abc', fullname: 'abc bbc', email: 'abcbbc@gmail.com')
test_user.id = 123
test_user.save!
test_user
end
#allow(ImportFileHelper).to receive(:create_new_user).with(attributes, {}).and_return()
allow(Assignment).to receive(:find).with(1).and_return(assignment)
allow(User).to receive(:exists?).with(name: 'no one').and_return(false)
allow(participant).to receive(:set_handle).and_return('handle')
allow(AssignmentParticipant).to receive(:exists?).and_return(false)
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(AssignmentParticipant).to receive(:set_handle)
expect{(AssignmentParticipant.import(row, nil, {}, 1))}.to change { ActionMailer::Base.deliveries.count }.by(1)
end
end
</pre>

== Links ==

*'''Pull Request: https://github.com/expertiza/expertiza/pull/2153
*'''Video: https://drive.google.com/file/d/19zPSOhh2AzHR-EF8v9wiwWrtGwtsUJL0/view?usp=sharing

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-30T03:44:33Z

Svsakham:

=== Team ===
*Akash Sarda, aksarda
*Sairam Sakhamuri, svsakham
*Sidhant Arora, sarora22
*Sai Harsha Nadendla, snadend2

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

The export functionality is however is already quite refactored to suit the strategy pattern, which is not changed by the previous group as well.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Design Plan ==

The design plan is to use Strategy patter to implement import functionality, so that all the import requests present in different models are routed through the ImportFileController. Right now since the import functionality is implemented in various model methods this leads to many if and else statements to check the type of model. So, we intent to generalize the import functionality by placing common code in the ImportFileController. But each model will still have its own import method. With this approach the redundancy is reduced by moving common code to ImportFileController and code will become DRY.

Other helper methods such as ImportFileHelper and ImportTopicHelper that are used to perform import functionality will also be removed, which keeps import functionality consistent. We will be using method overloading and overriding for the methods in ImportFileController to eliminate unnecessary if and else blocks.

To summarize our plan of changes:

* Redirect all import calls through ImportFileController
* Refactor ImportFileController by removing the redundant code and making code generic.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.

[[Image: DesignImport.PNG]]
== Implementation Details ==
=== Models ===

*ImportFileController is changed and has been made more generalized, many case statements are removed. The ImprortFileController is made so small and understandable. The import functionality is moved into the corresponding models. Previously code was present in different modules which was confusing now the functionality was all in one place. The following code changes are follows:

*Previous Implmentation
<pre>
def import_from_hash(session, params)
if params[:model] == "AssignmentTeam" or params[:model] == "CourseTeam"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
if params[:model] == "AssignmentTeam"
teamtype = AssignmentTeam
else
teamtype = CourseTeam
end
options = eval(params[:options])
options[:has_teamname] = params[:has_teamname]
Team.import(row_hash, params[:id], options, teamtype)
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "ReviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
ReviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "MetareviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
MetareviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'SignUpTopic' || params[:model] == 'SignUpSheet'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
if params[:optional_count] == '0'
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '1'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '2'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '3'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5], params[:select6]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
end
end
errors = []
begin
@header_integrated_body.each do |row_hash|
session[:assignment_id] = params[:id]
Object.const_get(params[:model]).import(row_hash, session, params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'AssignmentParticipant' || params[:model] == 'CourseParticipant'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header], contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
if params[:model] == 'AssignmentParticipant'
@header_integrated_body.each do |row_hash|
AssignmentParticipant.import(row_hash, session, params[:id])
end
elsif params[:model] == 'CourseParticipant'
@header_integrated_body.each do |row_hash|
CourseParticipant.import(row_hash, session, params[:id])
end
end
rescue
errors << $ERROR_INFO
end
else # params[:model] = "User"
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
@header_integrated_body.each do |row_hash|
User.import(row_hash, nil, session)
end
rescue StandardError
errors << $ERROR_INFO
end
end
</pre>
*Current Implementation
<pre>
def import
errors = import_from_hash(session, params)
err_msg = "The following errors were encountered during import.Other records may have been added. A second submission will not duplicate these records."
errors.each do |error|
err_msg = err_msg + error.to_s
end
err_msg += </ul>
if errors.empty?
ExpertizaLogger.info LoggerMessage.new(controller_name, session[:user].name, "The file has been successfully imported.", request)
undo_link("The file has been successfully imported.")
else
ExpertizaLogger.error LoggerMessage.new(controller_name, session[:user].name, err_msg, request)
flash[:error] = err_msg
end
redirect_to session[:return_to]
end
</pre>

*Assignment Participant is changed to use the newly implemented #import functionality. Changes are as shown below:

<pre>
def self.import(row_hash, session, id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?
raise ImportError, "The assignment with id #{id} was not found." if Assignment.find(id).nil?
unless AssignmentParticipant.exists?(user_id: user.id, parent_id: id)
new_part = AssignmentParticipant.new(user_id: user.id, parent_id: id)
new_part.set_handle
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

*Assignment Team is also changed to use the newly implemented #import functionality.

<pre>
def self.import(row_hash, session = nil, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The assignment with the id \"" + id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?" if Assignment.find_by(id: id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Course_team is is also changed to use the newly implemented #import functionality.
<pre>
def self.import(row_hash, session, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The course with the id \"" + id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?" if Course.find(id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Review response map's #import functionality is also updated to use the newly implemented #import functionality.
<pre>

def self.import(row_hash, _session, assignment_id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
reviewee_user_name = row_hash[:reviewee].to_s
reviewee_user = User.find_by(name: reviewee_user_name)
raise ArgumentError, "Cannot find reviewee user." unless reviewee_user
@@ -57,7 +58,7 @@ def self.import(row_hash, _session, assignment_id)
team_node = TeamNode.create(parent_id: assignment_id, node_object_id: reviewee_team.id)
TeamUserNode.create(parent_id: team_node.id, node_object_id: t_user.id)
end
row_hash[:reviewers].split.each do |reviewer|
reviewer_user_name = reviewer.to_s
reviewer_user = User.find_by(name: reviewer_user_name)
raise ArgumentError, "Cannot find reviewer user." unless reviewer_user
@@ -71,6 +72,20 @@ def self.import(row_hash, _session, assignment_id)
end
end

</pre>

*Refactored the Course_participant.rb to use newly added import functionality.
<pre>
def self.import(row_hash, session, id)
byebug
raise ArgumentError, "The record does not have enough items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?

user = User.find_by(name: row_hash[:name])
if user.nil?
raise ArgumentError, "The record containing #{row_hash[:name]} does not have enough items." if row_hash.length < 4
attributes = ImportFileHelper.define_attributes(row_hash)
user = ImportFileHelper.create_new_user(attributes, session)
end

course = Course.find_by(id)
raise ImportError, "The course with id " + id.to_s + " was not found." if course.nil?
unless CourseParticipant.exists?(user_id: user.id, parent_id: id)
CourseParticipant.create(user_id: user.id, parent_id: id)
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

With these new changes to #import functionality. The following modules review_response_map.rb, course_team.rb,assignment_team.rb,assignment_participant.rb,course_participant.rb have the "self.import" which will use the import functionality in the ImportFileController and they are made as concise as possible. Each module has specific checks while importing the file, so we can't have same code for all the module.

== Test Plan ==
=== Manual Testing ===
* Login with expertiza credentials.
* Navigate to Course Participant page by clicking on Manage => Courses => Add Participant.
* Click on Import course participant link and and browse for the file to import. Select the correct options for the type of file to be imported from Import Course Participant page. Please make sure the CSV file is in correct format as expected by the Import functionality. For course participant the csv should contain username | fullname | e-mail | password. Also make sure the users that are going to be imported are already present in expertiza database.
* Click in import participant button and the selected users should be imported.

=== Automated Tests ===
The import method has been implemented in a bunch of models which have been listed above. After preliminary analysis, we assume that the import functionality in a few of the models might have to be amended. These models are:
*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''review_response_map'''

Scenarios:
1. when assignment found and assignment participant does not exist, creates a new user and participant
2. when assignment cannot be found, creates a new user then raises an ImportError
3. when the assignment team does not have the required fields, raises ArgumentError
4. check what import/export actions are allowed for admin, instructor, TA, student
5. when course found and course participant does not exist, creates a new user and participant
6. when the course team does not have the required fields, raises ArgumentError

Newly added rspec test case to test the #import functionality in review_response_map_spec.rb

<pre>
it '#import' do
expect {ReviewResponseMap.import({reviewee: "name"}, nil, 1)}.to raise_error(ArgumentError, "Record does not contain required items.")
row_hash = {reviewee: "name", reviewers: "name1"}
#row_hash = {reviewee: "name", reviewers: ["name1"]}
session = nil
assignment_id = 1
# when reviewee user = nil
allow(User).to receive(:find_by).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Cannot find reviewee user.")
# when reviewee user exists but reviewee user is not a participant in this assignment
allow(User).to receive(:find_by).with(name: "name").and_return(student)
allow(AssignmentParticipant).to receive(:find_by).with(user_id: 1, parent_id: 1).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Reviewee user is not a participant in this assignment.")
end
</pre>

Newly added rspec test case to test the #import functionality in course_team_spec.rb

<pre>
describe ".import" do
let(:row) do
{teammembers: 'none'}
end
context "when a course team does not exist with id" do
it "raises ImportError" do
course_id = 1
allow(Course).to receive(:find).with(course_id).and_return(nil)
error_message = "The course with the id \"" + course_id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?"
expect { CourseTeam.import(row, nil, course_id, nil) }.
to raise_error(ImportError, error_message)
end
end

context "when the course team does not have the required fields" do
it "raises ArgumentError" do
expect { CourseTeam.import([], nil, 1, nil) }.
to raise_error(ArgumentError)
end
end

context "when a course team with the same id already exists" do
it "gets imported through Team.import" do
course_id = 1
options = []
allow(Course).to receive(:find).with(course_id).and_return(course)
expect(Team).to receive(:import_helper).with(row, course_id, options, instance_of(CourseTeam))
CourseTeam.import(row, nil, course_id, options)
end
end
end
</pre>

Newly added rspec test case to test the #import functionality in course_participant_spec.rb

<pre>
describe "CourseParticipant" do
let(:course) { build(:course, id: 1, name: 'ECE517') }
let(:assignment) { build(:assignment, id: 1, name: 'no assignment', participants: [participant], teams: [team]) }
let(:assignment_participant) {build(:participant, id: 1)}
'''describe "#copy" do
#before(:each) do
# byebug
# assignment = build(:assignment)
# course_participant = build(:course_participant)
# @assignment_participant = build(:participant)
#end
it "create a copy of participant" do
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(@assignment.id)).to be_an_instance_of(AssignmentParticipant)
end
it "returns nil if copy exist" do
allow(AssignmentParticipant).to receive(:where).and_return(AssignmentParticipant)
allow(AssignmentParticipant).to receive(:first).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(assignment.id)).to be_nil
end
end'''
</pre>

Test cases for Assignment_team.rb

<pre>
describe ".import" do
context "when an assignment team does not already exist with the same id" do
it "cannot be imported" do
assignment_id = 1
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(nil)
error_message = "The assignment with the id \"" + assignment_id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?"
expect { AssignmentTeam.import([], assignment_id, []) }.
to raise_error(ImportError, error_message)
end
end

context "when an assignment team with the same id already exists" do
it "gets imported through Team.import" do
row = []
assignment_id = 1
options = []
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(assignment)
allow(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
expect(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
AssignmentTeam.import(row, assignment_id, options)
end
end
end
</pre>

Added test for the assignment_participant_team.rb
<pre>
describe ".import" do
context 'when record is empty' do
it 'raises an ArgumentError' do
expect { AssignmentParticipant.import({}, nil, nil, nil) }.to raise_error(ArgumentError, 'No user id has been specified.')
end
end

context 'when no user is found by offered username' do
context 'when the record has less than 4 items' do
it 'raises an ArgumentError' do
row = {name: 'no one', fullname: 'no one', email: 'no_one@email.com'}
expect(ImportFileHelper).not_to receive(:create_new_user)
expect { AssignmentParticipant.import(row, nil, nil, nil) }.to raise_error('The record containing no one does not have enough items.')
end
end

context 'when new user needs to be created' do
let(:row) do
{name: 'no one', fullname: 'no one', email: 'name@email.com', role:'user_role_name', parent: 'user_parent_name'}
end
let(:attributes) do
{role_id: 1, name: 'no one', fullname: 'no one', email: 'name@email.com', email_on_submission: 'name@email.com',
email_on_review: 'name@email.com', email_on_review_of_review: 'name@email.com'}
end
let(:test_user) do
{name: 'abc', email: 'abcbbc@gmail.com'}
end
it 'create the user and number of mails sent should be 1' do
ActionMailer::Base.deliveries.clear
allow(ImportFileHelper).to receive(:define_attributes).with(row).and_return(attributes)
allow(ImportFileHelper).to receive(:create_new_user) do
test_user = User.new(name: 'abc', fullname: 'abc bbc', email: 'abcbbc@gmail.com')
test_user.id = 123
test_user.save!
test_user
end
#allow(ImportFileHelper).to receive(:create_new_user).with(attributes, {}).and_return()
allow(Assignment).to receive(:find).with(1).and_return(assignment)
allow(User).to receive(:exists?).with(name: 'no one').and_return(false)
allow(participant).to receive(:set_handle).and_return('handle')
allow(AssignmentParticipant).to receive(:exists?).and_return(false)
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(AssignmentParticipant).to receive(:set_handle)
expect{(AssignmentParticipant.import(row, nil, {}, 1))}.to change { ActionMailer::Base.deliveries.count }.by(1)
end
end
</pre>

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-30T03:44:06Z

Svsakham: /* Links */

=== Team ===
*Akash Sarda, aksarda
*Sairam Sakhamuri, svsakham
*Sidhant Arora, sarora22
*Sai Harsha Nadendla, snadend2

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

The export functionality is however is already quite refactored to suit the strategy pattern, which is not changed by the previous group as well.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Design Plan ==

The design plan is to use Strategy patter to implement import functionality, so that all the import requests present in different models are routed through the ImportFileController. Right now since the import functionality is implemented in various model methods this leads to many if and else statements to check the type of model. So, we intent to generalize the import functionality by placing common code in the ImportFileController. But each model will still have its own import method. With this approach the redundancy is reduced by moving common code to ImportFileController and code will become DRY.

Other helper methods such as ImportFileHelper and ImportTopicHelper that are used to perform import functionality will also be removed, which keeps import functionality consistent. We will be using method overloading and overriding for the methods in ImportFileController to eliminate unnecessary if and else blocks.

To summarize our plan of changes:

* Redirect all import calls through ImportFileController
* Refactor ImportFileController by removing the redundant code and making code generic.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.

[[Image: DesignImport.PNG]]
== Implementation Details ==
=== Models ===

*ImportFileController is changed and has been made more generalized, many case statements are removed. The ImprortFileController is made so small and understandable. The import functionality is moved into the corresponding models. Previously code was present in different modules which was confusing now the functionality was all in one place. The following code changes are follows:

*Previous Implmentation
<pre>
def import_from_hash(session, params)
if params[:model] == "AssignmentTeam" or params[:model] == "CourseTeam"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
if params[:model] == "AssignmentTeam"
teamtype = AssignmentTeam
else
teamtype = CourseTeam
end
options = eval(params[:options])
options[:has_teamname] = params[:has_teamname]
Team.import(row_hash, params[:id], options, teamtype)
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "ReviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
ReviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "MetareviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
MetareviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'SignUpTopic' || params[:model] == 'SignUpSheet'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
if params[:optional_count] == '0'
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '1'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '2'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '3'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5], params[:select6]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
end
end
errors = []
begin
@header_integrated_body.each do |row_hash|
session[:assignment_id] = params[:id]
Object.const_get(params[:model]).import(row_hash, session, params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'AssignmentParticipant' || params[:model] == 'CourseParticipant'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header], contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
if params[:model] == 'AssignmentParticipant'
@header_integrated_body.each do |row_hash|
AssignmentParticipant.import(row_hash, session, params[:id])
end
elsif params[:model] == 'CourseParticipant'
@header_integrated_body.each do |row_hash|
CourseParticipant.import(row_hash, session, params[:id])
end
end
rescue
errors << $ERROR_INFO
end
else # params[:model] = "User"
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
@header_integrated_body.each do |row_hash|
User.import(row_hash, nil, session)
end
rescue StandardError
errors << $ERROR_INFO
end
end
</pre>
*Current Implementation
<pre>
def import
errors = import_from_hash(session, params)
err_msg = "The following errors were encountered during import.Other records may have been added. A second submission will not duplicate these records."
errors.each do |error|
err_msg = err_msg + error.to_s
end
err_msg += </ul>
if errors.empty?
ExpertizaLogger.info LoggerMessage.new(controller_name, session[:user].name, "The file has been successfully imported.", request)
undo_link("The file has been successfully imported.")
else
ExpertizaLogger.error LoggerMessage.new(controller_name, session[:user].name, err_msg, request)
flash[:error] = err_msg
end
redirect_to session[:return_to]
end
</pre>

*Assignment Participant is changed to use the newly implemented #import functionality. Changes are as shown below:

<pre>
def self.import(row_hash, session, id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?
raise ImportError, "The assignment with id #{id} was not found." if Assignment.find(id).nil?
unless AssignmentParticipant.exists?(user_id: user.id, parent_id: id)
new_part = AssignmentParticipant.new(user_id: user.id, parent_id: id)
new_part.set_handle
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

*Assignment Team is also changed to use the newly implemented #import functionality.

<pre>
def self.import(row_hash, session = nil, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The assignment with the id \"" + id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?" if Assignment.find_by(id: id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Course_team is is also changed to use the newly implemented #import functionality.
<pre>
def self.import(row_hash, session, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The course with the id \"" + id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?" if Course.find(id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Review response map's #import functionality is also updated to use the newly implemented #import functionality.
<pre>

def self.import(row_hash, _session, assignment_id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
reviewee_user_name = row_hash[:reviewee].to_s
reviewee_user = User.find_by(name: reviewee_user_name)
raise ArgumentError, "Cannot find reviewee user." unless reviewee_user
@@ -57,7 +58,7 @@ def self.import(row_hash, _session, assignment_id)
team_node = TeamNode.create(parent_id: assignment_id, node_object_id: reviewee_team.id)
TeamUserNode.create(parent_id: team_node.id, node_object_id: t_user.id)
end
row_hash[:reviewers].split.each do |reviewer|
reviewer_user_name = reviewer.to_s
reviewer_user = User.find_by(name: reviewer_user_name)
raise ArgumentError, "Cannot find reviewer user." unless reviewer_user
@@ -71,6 +72,20 @@ def self.import(row_hash, _session, assignment_id)
end
end

</pre>

*Refactored the Course_participant.rb to use newly added import functionality.
<pre>
def self.import(row_hash, session, id)
byebug
raise ArgumentError, "The record does not have enough items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?

user = User.find_by(name: row_hash[:name])
if user.nil?
raise ArgumentError, "The record containing #{row_hash[:name]} does not have enough items." if row_hash.length < 4
attributes = ImportFileHelper.define_attributes(row_hash)
user = ImportFileHelper.create_new_user(attributes, session)
end

course = Course.find_by(id)
raise ImportError, "The course with id " + id.to_s + " was not found." if course.nil?
unless CourseParticipant.exists?(user_id: user.id, parent_id: id)
CourseParticipant.create(user_id: user.id, parent_id: id)
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

With these new changes to #import functionality. The following modules review_response_map.rb, course_team.rb,assignment_team.rb,assignment_participant.rb,course_participant.rb have the "self.import" which will use the import functionality in the ImportFileController and they are made as concise as possible. Each module has specific checks while importing the file, so we can't have same code for all the module.

== Test Plan ==
The import method has been implemented in a bunch of models which have been listed above. After preliminary analysis, we assume that the import functionality in a few of the models might have to be amended. These models are:
*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''review_response_map'''

Scenarios:
1. when assignment found and assignment participant does not exist, creates a new user and participant
2. when assignment cannot be found, creates a new user then raises an ImportError
3. when the assignment team does not have the required fields, raises ArgumentError
4. check what import/export actions are allowed for admin, instructor, TA, student
5. when course found and course participant does not exist, creates a new user and participant
6. when the course team does not have the required fields, raises ArgumentError

Newly added rspec test case to test the #import functionality in review_response_map_spec.rb

<pre>
it '#import' do
expect {ReviewResponseMap.import({reviewee: "name"}, nil, 1)}.to raise_error(ArgumentError, "Record does not contain required items.")
row_hash = {reviewee: "name", reviewers: "name1"}
#row_hash = {reviewee: "name", reviewers: ["name1"]}
session = nil
assignment_id = 1
# when reviewee user = nil
allow(User).to receive(:find_by).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Cannot find reviewee user.")
# when reviewee user exists but reviewee user is not a participant in this assignment
allow(User).to receive(:find_by).with(name: "name").and_return(student)
allow(AssignmentParticipant).to receive(:find_by).with(user_id: 1, parent_id: 1).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Reviewee user is not a participant in this assignment.")
end
</pre>

Newly added rspec test case to test the #import functionality in course_team_spec.rb

<pre>
describe ".import" do
let(:row) do
{teammembers: 'none'}
end
context "when a course team does not exist with id" do
it "raises ImportError" do
course_id = 1
allow(Course).to receive(:find).with(course_id).and_return(nil)
error_message = "The course with the id \"" + course_id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?"
expect { CourseTeam.import(row, nil, course_id, nil) }.
to raise_error(ImportError, error_message)
end
end

context "when the course team does not have the required fields" do
it "raises ArgumentError" do
expect { CourseTeam.import([], nil, 1, nil) }.
to raise_error(ArgumentError)
end
end

context "when a course team with the same id already exists" do
it "gets imported through Team.import" do
course_id = 1
options = []
allow(Course).to receive(:find).with(course_id).and_return(course)
expect(Team).to receive(:import_helper).with(row, course_id, options, instance_of(CourseTeam))
CourseTeam.import(row, nil, course_id, options)
end
end
end
</pre>

Newly added rspec test case to test the #import functionality in course_participant_spec.rb

<pre>
describe "CourseParticipant" do
let(:course) { build(:course, id: 1, name: 'ECE517') }
let(:assignment) { build(:assignment, id: 1, name: 'no assignment', participants: [participant], teams: [team]) }
let(:assignment_participant) {build(:participant, id: 1)}
'''describe "#copy" do
#before(:each) do
# byebug
# assignment = build(:assignment)
# course_participant = build(:course_participant)
# @assignment_participant = build(:participant)
#end
it "create a copy of participant" do
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(@assignment.id)).to be_an_instance_of(AssignmentParticipant)
end
it "returns nil if copy exist" do
allow(AssignmentParticipant).to receive(:where).and_return(AssignmentParticipant)
allow(AssignmentParticipant).to receive(:first).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(assignment.id)).to be_nil
end
end'''
</pre>

Test cases for Assignment_team.rb

<pre>
describe ".import" do
context "when an assignment team does not already exist with the same id" do
it "cannot be imported" do
assignment_id = 1
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(nil)
error_message = "The assignment with the id \"" + assignment_id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?"
expect { AssignmentTeam.import([], assignment_id, []) }.
to raise_error(ImportError, error_message)
end
end

context "when an assignment team with the same id already exists" do
it "gets imported through Team.import" do
row = []
assignment_id = 1
options = []
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(assignment)
allow(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
expect(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
AssignmentTeam.import(row, assignment_id, options)
end
end
end
</pre>

Added test for the assignment_participant_team.rb
<pre>
describe ".import" do
context 'when record is empty' do
it 'raises an ArgumentError' do
expect { AssignmentParticipant.import({}, nil, nil, nil) }.to raise_error(ArgumentError, 'No user id has been specified.')
end
end

context 'when no user is found by offered username' do
context 'when the record has less than 4 items' do
it 'raises an ArgumentError' do
row = {name: 'no one', fullname: 'no one', email: 'no_one@email.com'}
expect(ImportFileHelper).not_to receive(:create_new_user)
expect { AssignmentParticipant.import(row, nil, nil, nil) }.to raise_error('The record containing no one does not have enough items.')
end
end

context 'when new user needs to be created' do
let(:row) do
{name: 'no one', fullname: 'no one', email: 'name@email.com', role:'user_role_name', parent: 'user_parent_name'}
end
let(:attributes) do
{role_id: 1, name: 'no one', fullname: 'no one', email: 'name@email.com', email_on_submission: 'name@email.com',
email_on_review: 'name@email.com', email_on_review_of_review: 'name@email.com'}
end
let(:test_user) do
{name: 'abc', email: 'abcbbc@gmail.com'}
end
it 'create the user and number of mails sent should be 1' do
ActionMailer::Base.deliveries.clear
allow(ImportFileHelper).to receive(:define_attributes).with(row).and_return(attributes)
allow(ImportFileHelper).to receive(:create_new_user) do
test_user = User.new(name: 'abc', fullname: 'abc bbc', email: 'abcbbc@gmail.com')
test_user.id = 123
test_user.save!
test_user
end
#allow(ImportFileHelper).to receive(:create_new_user).with(attributes, {}).and_return()
allow(Assignment).to receive(:find).with(1).and_return(assignment)
allow(User).to receive(:exists?).with(name: 'no one').and_return(false)
allow(participant).to receive(:set_handle).and_return('handle')
allow(AssignmentParticipant).to receive(:exists?).and_return(false)
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(AssignmentParticipant).to receive(:set_handle)
expect{(AssignmentParticipant.import(row, nil, {}, 1))}.to change { ActionMailer::Base.deliveries.count }.by(1)
end
end
</pre>

== Links ==

*'''Pull Request: https://github.com/expertiza/expertiza/pull/2153
*'''Video: https://drive.google.com/file/d/19zPSOhh2AzHR-EF8v9wiwWrtGwtsUJL0/view?usp=sharing

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-30T03:43:49Z

Svsakham:

=== Team ===
*Akash Sarda, aksarda
*Sairam Sakhamuri, svsakham
*Sidhant Arora, sarora22
*Sai Harsha Nadendla, snadend2

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

The export functionality is however is already quite refactored to suit the strategy pattern, which is not changed by the previous group as well.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Design Plan ==

The design plan is to use Strategy patter to implement import functionality, so that all the import requests present in different models are routed through the ImportFileController. Right now since the import functionality is implemented in various model methods this leads to many if and else statements to check the type of model. So, we intent to generalize the import functionality by placing common code in the ImportFileController. But each model will still have its own import method. With this approach the redundancy is reduced by moving common code to ImportFileController and code will become DRY.

Other helper methods such as ImportFileHelper and ImportTopicHelper that are used to perform import functionality will also be removed, which keeps import functionality consistent. We will be using method overloading and overriding for the methods in ImportFileController to eliminate unnecessary if and else blocks.

To summarize our plan of changes:

* Redirect all import calls through ImportFileController
* Refactor ImportFileController by removing the redundant code and making code generic.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.

[[Image: DesignImport.PNG]]
== Implementation Details ==
=== Models ===

*ImportFileController is changed and has been made more generalized, many case statements are removed. The ImprortFileController is made so small and understandable. The import functionality is moved into the corresponding models. Previously code was present in different modules which was confusing now the functionality was all in one place. The following code changes are follows:

*Previous Implmentation
<pre>
def import_from_hash(session, params)
if params[:model] == "AssignmentTeam" or params[:model] == "CourseTeam"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
if params[:model] == "AssignmentTeam"
teamtype = AssignmentTeam
else
teamtype = CourseTeam
end
options = eval(params[:options])
options[:has_teamname] = params[:has_teamname]
Team.import(row_hash, params[:id], options, teamtype)
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "ReviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
ReviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "MetareviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
MetareviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'SignUpTopic' || params[:model] == 'SignUpSheet'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
if params[:optional_count] == '0'
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '1'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '2'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '3'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5], params[:select6]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
end
end
errors = []
begin
@header_integrated_body.each do |row_hash|
session[:assignment_id] = params[:id]
Object.const_get(params[:model]).import(row_hash, session, params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'AssignmentParticipant' || params[:model] == 'CourseParticipant'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header], contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
if params[:model] == 'AssignmentParticipant'
@header_integrated_body.each do |row_hash|
AssignmentParticipant.import(row_hash, session, params[:id])
end
elsif params[:model] == 'CourseParticipant'
@header_integrated_body.each do |row_hash|
CourseParticipant.import(row_hash, session, params[:id])
end
end
rescue
errors << $ERROR_INFO
end
else # params[:model] = "User"
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
@header_integrated_body.each do |row_hash|
User.import(row_hash, nil, session)
end
rescue StandardError
errors << $ERROR_INFO
end
end
</pre>
*Current Implementation
<pre>
def import
errors = import_from_hash(session, params)
err_msg = "The following errors were encountered during import.Other records may have been added. A second submission will not duplicate these records."
errors.each do |error|
err_msg = err_msg + error.to_s
end
err_msg += </ul>
if errors.empty?
ExpertizaLogger.info LoggerMessage.new(controller_name, session[:user].name, "The file has been successfully imported.", request)
undo_link("The file has been successfully imported.")
else
ExpertizaLogger.error LoggerMessage.new(controller_name, session[:user].name, err_msg, request)
flash[:error] = err_msg
end
redirect_to session[:return_to]
end
</pre>

*Assignment Participant is changed to use the newly implemented #import functionality. Changes are as shown below:

<pre>
def self.import(row_hash, session, id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?
raise ImportError, "The assignment with id #{id} was not found." if Assignment.find(id).nil?
unless AssignmentParticipant.exists?(user_id: user.id, parent_id: id)
new_part = AssignmentParticipant.new(user_id: user.id, parent_id: id)
new_part.set_handle
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

*Assignment Team is also changed to use the newly implemented #import functionality.

<pre>
def self.import(row_hash, session = nil, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The assignment with the id \"" + id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?" if Assignment.find_by(id: id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Course_team is is also changed to use the newly implemented #import functionality.
<pre>
def self.import(row_hash, session, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The course with the id \"" + id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?" if Course.find(id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Review response map's #import functionality is also updated to use the newly implemented #import functionality.
<pre>

def self.import(row_hash, _session, assignment_id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
reviewee_user_name = row_hash[:reviewee].to_s
reviewee_user = User.find_by(name: reviewee_user_name)
raise ArgumentError, "Cannot find reviewee user." unless reviewee_user
@@ -57,7 +58,7 @@ def self.import(row_hash, _session, assignment_id)
team_node = TeamNode.create(parent_id: assignment_id, node_object_id: reviewee_team.id)
TeamUserNode.create(parent_id: team_node.id, node_object_id: t_user.id)
end
row_hash[:reviewers].split.each do |reviewer|
reviewer_user_name = reviewer.to_s
reviewer_user = User.find_by(name: reviewer_user_name)
raise ArgumentError, "Cannot find reviewer user." unless reviewer_user
@@ -71,6 +72,20 @@ def self.import(row_hash, _session, assignment_id)
end
end

</pre>

*Refactored the Course_participant.rb to use newly added import functionality.
<pre>
def self.import(row_hash, session, id)
byebug
raise ArgumentError, "The record does not have enough items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?

user = User.find_by(name: row_hash[:name])
if user.nil?
raise ArgumentError, "The record containing #{row_hash[:name]} does not have enough items." if row_hash.length < 4
attributes = ImportFileHelper.define_attributes(row_hash)
user = ImportFileHelper.create_new_user(attributes, session)
end

course = Course.find_by(id)
raise ImportError, "The course with id " + id.to_s + " was not found." if course.nil?
unless CourseParticipant.exists?(user_id: user.id, parent_id: id)
CourseParticipant.create(user_id: user.id, parent_id: id)
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

With these new changes to #import functionality. The following modules review_response_map.rb, course_team.rb,assignment_team.rb,assignment_participant.rb,course_participant.rb have the "self.import" which will use the import functionality in the ImportFileController and they are made as concise as possible. Each module has specific checks while importing the file, so we can't have same code for all the module.

== Test Plan ==
The import method has been implemented in a bunch of models which have been listed above. After preliminary analysis, we assume that the import functionality in a few of the models might have to be amended. These models are:
*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''review_response_map'''

Scenarios:
1. when assignment found and assignment participant does not exist, creates a new user and participant
2. when assignment cannot be found, creates a new user then raises an ImportError
3. when the assignment team does not have the required fields, raises ArgumentError
4. check what import/export actions are allowed for admin, instructor, TA, student
5. when course found and course participant does not exist, creates a new user and participant
6. when the course team does not have the required fields, raises ArgumentError

Newly added rspec test case to test the #import functionality in review_response_map_spec.rb

<pre>
it '#import' do
expect {ReviewResponseMap.import({reviewee: "name"}, nil, 1)}.to raise_error(ArgumentError, "Record does not contain required items.")
row_hash = {reviewee: "name", reviewers: "name1"}
#row_hash = {reviewee: "name", reviewers: ["name1"]}
session = nil
assignment_id = 1
# when reviewee user = nil
allow(User).to receive(:find_by).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Cannot find reviewee user.")
# when reviewee user exists but reviewee user is not a participant in this assignment
allow(User).to receive(:find_by).with(name: "name").and_return(student)
allow(AssignmentParticipant).to receive(:find_by).with(user_id: 1, parent_id: 1).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Reviewee user is not a participant in this assignment.")
end
</pre>

Newly added rspec test case to test the #import functionality in course_team_spec.rb

<pre>
describe ".import" do
let(:row) do
{teammembers: 'none'}
end
context "when a course team does not exist with id" do
it "raises ImportError" do
course_id = 1
allow(Course).to receive(:find).with(course_id).and_return(nil)
error_message = "The course with the id \"" + course_id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?"
expect { CourseTeam.import(row, nil, course_id, nil) }.
to raise_error(ImportError, error_message)
end
end

context "when the course team does not have the required fields" do
it "raises ArgumentError" do
expect { CourseTeam.import([], nil, 1, nil) }.
to raise_error(ArgumentError)
end
end

context "when a course team with the same id already exists" do
it "gets imported through Team.import" do
course_id = 1
options = []
allow(Course).to receive(:find).with(course_id).and_return(course)
expect(Team).to receive(:import_helper).with(row, course_id, options, instance_of(CourseTeam))
CourseTeam.import(row, nil, course_id, options)
end
end
end
</pre>

Newly added rspec test case to test the #import functionality in course_participant_spec.rb

<pre>
describe "CourseParticipant" do
let(:course) { build(:course, id: 1, name: 'ECE517') }
let(:assignment) { build(:assignment, id: 1, name: 'no assignment', participants: [participant], teams: [team]) }
let(:assignment_participant) {build(:participant, id: 1)}
'''describe "#copy" do
#before(:each) do
# byebug
# assignment = build(:assignment)
# course_participant = build(:course_participant)
# @assignment_participant = build(:participant)
#end
it "create a copy of participant" do
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(@assignment.id)).to be_an_instance_of(AssignmentParticipant)
end
it "returns nil if copy exist" do
allow(AssignmentParticipant).to receive(:where).and_return(AssignmentParticipant)
allow(AssignmentParticipant).to receive(:first).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(assignment.id)).to be_nil
end
end'''
</pre>

Test cases for Assignment_team.rb

<pre>
describe ".import" do
context "when an assignment team does not already exist with the same id" do
it "cannot be imported" do
assignment_id = 1
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(nil)
error_message = "The assignment with the id \"" + assignment_id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?"
expect { AssignmentTeam.import([], assignment_id, []) }.
to raise_error(ImportError, error_message)
end
end

context "when an assignment team with the same id already exists" do
it "gets imported through Team.import" do
row = []
assignment_id = 1
options = []
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(assignment)
allow(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
expect(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
AssignmentTeam.import(row, assignment_id, options)
end
end
end
</pre>

Added test for the assignment_participant_team.rb
<pre>
describe ".import" do
context 'when record is empty' do
it 'raises an ArgumentError' do
expect { AssignmentParticipant.import({}, nil, nil, nil) }.to raise_error(ArgumentError, 'No user id has been specified.')
end
end

context 'when no user is found by offered username' do
context 'when the record has less than 4 items' do
it 'raises an ArgumentError' do
row = {name: 'no one', fullname: 'no one', email: 'no_one@email.com'}
expect(ImportFileHelper).not_to receive(:create_new_user)
expect { AssignmentParticipant.import(row, nil, nil, nil) }.to raise_error('The record containing no one does not have enough items.')
end
end

context 'when new user needs to be created' do
let(:row) do
{name: 'no one', fullname: 'no one', email: 'name@email.com', role:'user_role_name', parent: 'user_parent_name'}
end
let(:attributes) do
{role_id: 1, name: 'no one', fullname: 'no one', email: 'name@email.com', email_on_submission: 'name@email.com',
email_on_review: 'name@email.com', email_on_review_of_review: 'name@email.com'}
end
let(:test_user) do
{name: 'abc', email: 'abcbbc@gmail.com'}
end
it 'create the user and number of mails sent should be 1' do
ActionMailer::Base.deliveries.clear
allow(ImportFileHelper).to receive(:define_attributes).with(row).and_return(attributes)
allow(ImportFileHelper).to receive(:create_new_user) do
test_user = User.new(name: 'abc', fullname: 'abc bbc', email: 'abcbbc@gmail.com')
test_user.id = 123
test_user.save!
test_user
end
#allow(ImportFileHelper).to receive(:create_new_user).with(attributes, {}).and_return()
allow(Assignment).to receive(:find).with(1).and_return(assignment)
allow(User).to receive(:exists?).with(name: 'no one').and_return(false)
allow(participant).to receive(:set_handle).and_return('handle')
allow(AssignmentParticipant).to receive(:exists?).and_return(false)
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(AssignmentParticipant).to receive(:set_handle)
expect{(AssignmentParticipant.import(row, nil, {}, 1))}.to change { ActionMailer::Base.deliveries.count }.by(1)
end
end
</pre>

== Links ==

*Pull Request: https://github.com/expertiza/expertiza/pull/2153
*Video: https://drive.google.com/file/d/19zPSOhh2AzHR-EF8v9wiwWrtGwtsUJL0/view?usp=sharing

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T23:21:29Z

Svsakham: /* What has been Implemented */

=== Team ===
*Akash Sarda, aksarda
*Sairam Sakhamuri, svsakham
*Sidhant Arora, sarora22
*Sai Harsha Nadendla, snadend2

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

The export functionality is however is already quite refactored to suit the strategy pattern, which is not changed by the previous group as well.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Design Plan ==

The design plan is to use Strategy patter to implement import functionality, so that all the import requests present in different models are routed through the ImportFileController. Right now since the import functionality is implemented in various model methods this leads to many if and else statements to check the type of model. So, we intent to generalize the import functionality by placing common code in the ImportFileController. But each model will still have its own import method. With this approach the redundancy is reduced by moving common code to ImportFileController and code will become DRY.

Other helper methods such as ImportFileHelper and ImportTopicHelper that are used to perform import functionality will also be removed, which keeps import functionality consistent. We will be using method overloading and overriding for the methods in ImportFileController to eliminate unnecessary if and else blocks.

To summarize our plan of changes:

* Redirect all import calls through ImportFileController
* Refactor ImportFileController by removing the redundant code and making code generic.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.

[[Image: DesignImport.PNG]]
== Implementation Details ==
=== Models ===

*ImportFileController is changed and has been made more generalized, many case statements are removed. The ImprortFileController is made so small and understandable. The import functionality is moved into the corresponding models. Previously code was present in different modules which was confusing now the functionality was all in one place. The following code changes are follows:

*Previous Implmentation
<pre>
def import_from_hash(session, params)
if params[:model] == "AssignmentTeam" or params[:model] == "CourseTeam"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
if params[:model] == "AssignmentTeam"
teamtype = AssignmentTeam
else
teamtype = CourseTeam
end
options = eval(params[:options])
options[:has_teamname] = params[:has_teamname]
Team.import(row_hash, params[:id], options, teamtype)
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "ReviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
ReviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "MetareviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
MetareviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'SignUpTopic' || params[:model] == 'SignUpSheet'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
if params[:optional_count] == '0'
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '1'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '2'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '3'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5], params[:select6]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
end
end
errors = []
begin
@header_integrated_body.each do |row_hash|
session[:assignment_id] = params[:id]
Object.const_get(params[:model]).import(row_hash, session, params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'AssignmentParticipant' || params[:model] == 'CourseParticipant'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header], contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
if params[:model] == 'AssignmentParticipant'
@header_integrated_body.each do |row_hash|
AssignmentParticipant.import(row_hash, session, params[:id])
end
elsif params[:model] == 'CourseParticipant'
@header_integrated_body.each do |row_hash|
CourseParticipant.import(row_hash, session, params[:id])
end
end
rescue
errors << $ERROR_INFO
end
else # params[:model] = "User"
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
@header_integrated_body.each do |row_hash|
User.import(row_hash, nil, session)
end
rescue StandardError
errors << $ERROR_INFO
end
end
</pre>
*Current Implementation
<pre>
def import
errors = import_from_hash(session, params)
err_msg = "The following errors were encountered during import.Other records may have been added. A second submission will not duplicate these records."
errors.each do |error|
err_msg = err_msg + error.to_s
end
err_msg += </ul>
if errors.empty?
ExpertizaLogger.info LoggerMessage.new(controller_name, session[:user].name, "The file has been successfully imported.", request)
undo_link("The file has been successfully imported.")
else
ExpertizaLogger.error LoggerMessage.new(controller_name, session[:user].name, err_msg, request)
flash[:error] = err_msg
end
redirect_to session[:return_to]
end
</pre>

*Assignment Participant is changed to use the newly implemented #import functionality. Changes are as shown below:

<pre>
def self.import(row_hash, session, id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?
raise ImportError, "The assignment with id #{id} was not found." if Assignment.find(id).nil?
unless AssignmentParticipant.exists?(user_id: user.id, parent_id: id)
new_part = AssignmentParticipant.new(user_id: user.id, parent_id: id)
new_part.set_handle
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

*Assignment Team is also changed to use the newly implemented #import functionality.

<pre>
def self.import(row_hash, session = nil, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The assignment with the id \"" + id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?" if Assignment.find_by(id: id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Course_team is is also changed to use the newly implemented #import functionality.
<pre>
def self.import(row_hash, session, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The course with the id \"" + id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?" if Course.find(id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Review response map's #import functionality is also updated to use the newly implemented #import functionality.
<pre>

def self.import(row_hash, _session, assignment_id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
reviewee_user_name = row_hash[:reviewee].to_s
reviewee_user = User.find_by(name: reviewee_user_name)
raise ArgumentError, "Cannot find reviewee user." unless reviewee_user
@@ -57,7 +58,7 @@ def self.import(row_hash, _session, assignment_id)
team_node = TeamNode.create(parent_id: assignment_id, node_object_id: reviewee_team.id)
TeamUserNode.create(parent_id: team_node.id, node_object_id: t_user.id)
end
row_hash[:reviewers].split.each do |reviewer|
reviewer_user_name = reviewer.to_s
reviewer_user = User.find_by(name: reviewer_user_name)
raise ArgumentError, "Cannot find reviewer user." unless reviewer_user
@@ -71,6 +72,20 @@ def self.import(row_hash, _session, assignment_id)
end
end

</pre>

*Refactored the Course_participant.rb to use newly added import functionality.
<pre>
def self.import(row_hash, session, id)
byebug
raise ArgumentError, "The record does not have enough items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?

user = User.find_by(name: row_hash[:name])
if user.nil?
raise ArgumentError, "The record containing #{row_hash[:name]} does not have enough items." if row_hash.length < 4
attributes = ImportFileHelper.define_attributes(row_hash)
user = ImportFileHelper.create_new_user(attributes, session)
end

course = Course.find_by(id)
raise ImportError, "The course with id " + id.to_s + " was not found." if course.nil?
unless CourseParticipant.exists?(user_id: user.id, parent_id: id)
CourseParticipant.create(user_id: user.id, parent_id: id)
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

With these new changes to #import functionality. The following modules review_response_map.rb, course_team.rb,assignment_team.rb,assignment_participant.rb,course_participant.rb have the "self.import" which will use the import functionality in the ImportFileController and they are made as concise as possible. Each module has specific checks while importing the file, so we can't have same code for all the module.

== Test Plan ==
The import method has been implemented in a bunch of models which have been listed above. After preliminary analysis, we assume that the import functionality in a few of the models might have to be amended. These models are:
*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''review_response_map'''

Scenarios:
1. when assignment found and assignment participant does not exist, creates a new user and participant
2. when assignment cannot be found, creates a new user then raises an ImportError
3. when the assignment team does not have the required fields, raises ArgumentError
4. check what import/export actions are allowed for admin, instructor, TA, student
5. when course found and course participant does not exist, creates a new user and participant
6. when the course team does not have the required fields, raises ArgumentError

Newly added rspec test case to test the #import functionality in review_response_map_spec.rb

<pre>
it '#import' do
expect {ReviewResponseMap.import({reviewee: "name"}, nil, 1)}.to raise_error(ArgumentError, "Record does not contain required items.")
row_hash = {reviewee: "name", reviewers: "name1"}
#row_hash = {reviewee: "name", reviewers: ["name1"]}
session = nil
assignment_id = 1
# when reviewee user = nil
allow(User).to receive(:find_by).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Cannot find reviewee user.")
# when reviewee user exists but reviewee user is not a participant in this assignment
allow(User).to receive(:find_by).with(name: "name").and_return(student)
allow(AssignmentParticipant).to receive(:find_by).with(user_id: 1, parent_id: 1).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Reviewee user is not a participant in this assignment.")
end
</pre>

Newly added rspec test case to test the #import functionality in course_team_spec.rb

<pre>
describe ".import" do
let(:row) do
{teammembers: 'none'}
end
context "when a course team does not exist with id" do
it "raises ImportError" do
course_id = 1
allow(Course).to receive(:find).with(course_id).and_return(nil)
error_message = "The course with the id \"" + course_id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?"
expect { CourseTeam.import(row, nil, course_id, nil) }.
to raise_error(ImportError, error_message)
end
end

context "when the course team does not have the required fields" do
it "raises ArgumentError" do
expect { CourseTeam.import([], nil, 1, nil) }.
to raise_error(ArgumentError)
end
end

context "when a course team with the same id already exists" do
it "gets imported through Team.import" do
course_id = 1
options = []
allow(Course).to receive(:find).with(course_id).and_return(course)
expect(Team).to receive(:import_helper).with(row, course_id, options, instance_of(CourseTeam))
CourseTeam.import(row, nil, course_id, options)
end
end
end
</pre>

Newly added rspec test case to test the #import functionality in course_participant_spec.rb

<pre>
describe "CourseParticipant" do
let(:course) { build(:course, id: 1, name: 'ECE517') }
let(:assignment) { build(:assignment, id: 1, name: 'no assignment', participants: [participant], teams: [team]) }
let(:assignment_participant) {build(:participant, id: 1)}
'''describe "#copy" do
#before(:each) do
# byebug
# assignment = build(:assignment)
# course_participant = build(:course_participant)
# @assignment_participant = build(:participant)
#end
it "create a copy of participant" do
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(@assignment.id)).to be_an_instance_of(AssignmentParticipant)
end
it "returns nil if copy exist" do
allow(AssignmentParticipant).to receive(:where).and_return(AssignmentParticipant)
allow(AssignmentParticipant).to receive(:first).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(assignment.id)).to be_nil
end
end'''
</pre>

Test cases for Assignment_team.rb

<pre>
describe ".import" do
context "when an assignment team does not already exist with the same id" do
it "cannot be imported" do
assignment_id = 1
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(nil)
error_message = "The assignment with the id \"" + assignment_id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?"
expect { AssignmentTeam.import([], assignment_id, []) }.
to raise_error(ImportError, error_message)
end
end

context "when an assignment team with the same id already exists" do
it "gets imported through Team.import" do
row = []
assignment_id = 1
options = []
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(assignment)
allow(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
expect(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
AssignmentTeam.import(row, assignment_id, options)
end
end
end
</pre>

Added test for the assignment_participant_team.rb
<pre>
describe ".import" do
context 'when record is empty' do
it 'raises an ArgumentError' do
expect { AssignmentParticipant.import({}, nil, nil, nil) }.to raise_error(ArgumentError, 'No user id has been specified.')
end
end

context 'when no user is found by offered username' do
context 'when the record has less than 4 items' do
it 'raises an ArgumentError' do
row = {name: 'no one', fullname: 'no one', email: 'no_one@email.com'}
expect(ImportFileHelper).not_to receive(:create_new_user)
expect { AssignmentParticipant.import(row, nil, nil, nil) }.to raise_error('The record containing no one does not have enough items.')
end
end

context 'when new user needs to be created' do
let(:row) do
{name: 'no one', fullname: 'no one', email: 'name@email.com', role:'user_role_name', parent: 'user_parent_name'}
end
let(:attributes) do
{role_id: 1, name: 'no one', fullname: 'no one', email: 'name@email.com', email_on_submission: 'name@email.com',
email_on_review: 'name@email.com', email_on_review_of_review: 'name@email.com'}
end
let(:test_user) do
{name: 'abc', email: 'abcbbc@gmail.com'}
end
it 'create the user and number of mails sent should be 1' do
ActionMailer::Base.deliveries.clear
allow(ImportFileHelper).to receive(:define_attributes).with(row).and_return(attributes)
allow(ImportFileHelper).to receive(:create_new_user) do
test_user = User.new(name: 'abc', fullname: 'abc bbc', email: 'abcbbc@gmail.com')
test_user.id = 123
test_user.save!
test_user
end
#allow(ImportFileHelper).to receive(:create_new_user).with(attributes, {}).and_return()
allow(Assignment).to receive(:find).with(1).and_return(assignment)
allow(User).to receive(:exists?).with(name: 'no one').and_return(false)
allow(participant).to receive(:set_handle).and_return('handle')
allow(AssignmentParticipant).to receive(:exists?).and_return(false)
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(AssignmentParticipant).to receive(:set_handle)
expect{(AssignmentParticipant.import(row, nil, {}, 1))}.to change { ActionMailer::Base.deliveries.count }.by(1)
end
end
</pre>

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T23:21:03Z

Svsakham: /* What has been Implemented */

=== Team ===
*Akash Sarda, aksarda
*Sairam Sakhamuri, svsakham
*Sidhant Arora, sarora22
*Sai Harsha Nadendla, snadend2

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

The export functionality is however is already quite refactored to suit the strategy pattern, which is not changed by the previous group as well.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Design Plan ==

The design plan is to use Strategy patter to implement import functionality, so that all the import requests present in different models are routed through the ImportFileController. Right now since the import functionality is implemented in various model methods this leads to many if and else statements to check the type of model. So, we intent to generalize the import functionality by placing common code in the ImportFileController. But each model will still have its own import method. With this approach the redundancy is reduced by moving common code to ImportFileController and code will become DRY.

Other helper methods such as ImportFileHelper and ImportTopicHelper that are used to perform import functionality will also be removed, which keeps import functionality consistent. We will be using method overloading and overriding for the methods in ImportFileController to eliminate unnecessary if and else blocks.

To summarize our plan of changes:

* Redirect all import calls through ImportFileController
* Refactor ImportFileController by removing the redundant code and making code generic.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.

[[Image: DesignImport.PNG]]
== What has been Implemented==
=== Models ===

*ImportFileController is changed and has been made more generalized, many case statements are removed. The ImprortFileController is made so small and understandable. The import functionality is moved into the corresponding models. Previously code was present in different modules which was confusing now the functionality was all in one place. The following code changes are follows:

*Previous Implmentation
<pre>
def import_from_hash(session, params)
if params[:model] == "AssignmentTeam" or params[:model] == "CourseTeam"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
if params[:model] == "AssignmentTeam"
teamtype = AssignmentTeam
else
teamtype = CourseTeam
end
options = eval(params[:options])
options[:has_teamname] = params[:has_teamname]
Team.import(row_hash, params[:id], options, teamtype)
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "ReviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
ReviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "MetareviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
MetareviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'SignUpTopic' || params[:model] == 'SignUpSheet'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
if params[:optional_count] == '0'
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '1'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '2'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '3'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5], params[:select6]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
end
end
errors = []
begin
@header_integrated_body.each do |row_hash|
session[:assignment_id] = params[:id]
Object.const_get(params[:model]).import(row_hash, session, params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'AssignmentParticipant' || params[:model] == 'CourseParticipant'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header], contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
if params[:model] == 'AssignmentParticipant'
@header_integrated_body.each do |row_hash|
AssignmentParticipant.import(row_hash, session, params[:id])
end
elsif params[:model] == 'CourseParticipant'
@header_integrated_body.each do |row_hash|
CourseParticipant.import(row_hash, session, params[:id])
end
end
rescue
errors << $ERROR_INFO
end
else # params[:model] = "User"
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
@header_integrated_body.each do |row_hash|
User.import(row_hash, nil, session)
end
rescue StandardError
errors << $ERROR_INFO
end
end
</pre>
*Current Implementation
<pre>
def import
errors = import_from_hash(session, params)
err_msg = "The following errors were encountered during import.Other records may have been added. A second submission will not duplicate these records."
errors.each do |error|
err_msg = err_msg + error.to_s
end
err_msg += </ul>
if errors.empty?
ExpertizaLogger.info LoggerMessage.new(controller_name, session[:user].name, "The file has been successfully imported.", request)
undo_link("The file has been successfully imported.")
else
ExpertizaLogger.error LoggerMessage.new(controller_name, session[:user].name, err_msg, request)
flash[:error] = err_msg
end
redirect_to session[:return_to]
end
</pre>

*Assignment Participant is changed to use the newly implemented #import functionality. Changes are as shown below:

<pre>
def self.import(row_hash, session, id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?
raise ImportError, "The assignment with id #{id} was not found." if Assignment.find(id).nil?
unless AssignmentParticipant.exists?(user_id: user.id, parent_id: id)
new_part = AssignmentParticipant.new(user_id: user.id, parent_id: id)
new_part.set_handle
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

*Assignment Team is also changed to use the newly implemented #import functionality.

<pre>
def self.import(row_hash, session = nil, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The assignment with the id \"" + id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?" if Assignment.find_by(id: id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Course_team is is also changed to use the newly implemented #import functionality.
<pre>
def self.import(row_hash, session, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The course with the id \"" + id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?" if Course.find(id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Review response map's #import functionality is also updated to use the newly implemented #import functionality.
<pre>

def self.import(row_hash, _session, assignment_id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
reviewee_user_name = row_hash[:reviewee].to_s
reviewee_user = User.find_by(name: reviewee_user_name)
raise ArgumentError, "Cannot find reviewee user." unless reviewee_user
@@ -57,7 +58,7 @@ def self.import(row_hash, _session, assignment_id)
team_node = TeamNode.create(parent_id: assignment_id, node_object_id: reviewee_team.id)
TeamUserNode.create(parent_id: team_node.id, node_object_id: t_user.id)
end
row_hash[:reviewers].split.each do |reviewer|
reviewer_user_name = reviewer.to_s
reviewer_user = User.find_by(name: reviewer_user_name)
raise ArgumentError, "Cannot find reviewer user." unless reviewer_user
@@ -71,6 +72,20 @@ def self.import(row_hash, _session, assignment_id)
end
end

</pre>

*Refactored the Course_participant.rb to use newly added import functionality.
<pre>
def self.import(row_hash, session, id)
byebug
raise ArgumentError, "The record does not have enough items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?

user = User.find_by(name: row_hash[:name])
if user.nil?
raise ArgumentError, "The record containing #{row_hash[:name]} does not have enough items." if row_hash.length < 4
attributes = ImportFileHelper.define_attributes(row_hash)
user = ImportFileHelper.create_new_user(attributes, session)
end

course = Course.find_by(id)
raise ImportError, "The course with id " + id.to_s + " was not found." if course.nil?
unless CourseParticipant.exists?(user_id: user.id, parent_id: id)
CourseParticipant.create(user_id: user.id, parent_id: id)
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

With these new changes to #import functionality. The following modules review_response_map.rb, course_team.rb,assignment_team.rb,assignment_participant.rb,course_participant.rb have the "self.import" which will use the import functionality in the ImportFileController and they are made as concise as possible. Each module has specific checks while importing the file, so we can't have same code for all the module.

== Test Plan ==
The import method has been implemented in a bunch of models which have been listed above. After preliminary analysis, we assume that the import functionality in a few of the models might have to be amended. These models are:
*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''review_response_map'''

Scenarios:
1. when assignment found and assignment participant does not exist, creates a new user and participant
2. when assignment cannot be found, creates a new user then raises an ImportError
3. when the assignment team does not have the required fields, raises ArgumentError
4. check what import/export actions are allowed for admin, instructor, TA, student
5. when course found and course participant does not exist, creates a new user and participant
6. when the course team does not have the required fields, raises ArgumentError

Newly added rspec test case to test the #import functionality in review_response_map_spec.rb

<pre>
it '#import' do
expect {ReviewResponseMap.import({reviewee: "name"}, nil, 1)}.to raise_error(ArgumentError, "Record does not contain required items.")
row_hash = {reviewee: "name", reviewers: "name1"}
#row_hash = {reviewee: "name", reviewers: ["name1"]}
session = nil
assignment_id = 1
# when reviewee user = nil
allow(User).to receive(:find_by).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Cannot find reviewee user.")
# when reviewee user exists but reviewee user is not a participant in this assignment
allow(User).to receive(:find_by).with(name: "name").and_return(student)
allow(AssignmentParticipant).to receive(:find_by).with(user_id: 1, parent_id: 1).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Reviewee user is not a participant in this assignment.")
end
</pre>

Newly added rspec test case to test the #import functionality in course_team_spec.rb

<pre>
describe ".import" do
let(:row) do
{teammembers: 'none'}
end
context "when a course team does not exist with id" do
it "raises ImportError" do
course_id = 1
allow(Course).to receive(:find).with(course_id).and_return(nil)
error_message = "The course with the id \"" + course_id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?"
expect { CourseTeam.import(row, nil, course_id, nil) }.
to raise_error(ImportError, error_message)
end
end

context "when the course team does not have the required fields" do
it "raises ArgumentError" do
expect { CourseTeam.import([], nil, 1, nil) }.
to raise_error(ArgumentError)
end
end

context "when a course team with the same id already exists" do
it "gets imported through Team.import" do
course_id = 1
options = []
allow(Course).to receive(:find).with(course_id).and_return(course)
expect(Team).to receive(:import_helper).with(row, course_id, options, instance_of(CourseTeam))
CourseTeam.import(row, nil, course_id, options)
end
end
end
</pre>

Newly added rspec test case to test the #import functionality in course_participant_spec.rb

<pre>
describe "CourseParticipant" do
let(:course) { build(:course, id: 1, name: 'ECE517') }
let(:assignment) { build(:assignment, id: 1, name: 'no assignment', participants: [participant], teams: [team]) }
let(:assignment_participant) {build(:participant, id: 1)}
'''describe "#copy" do
#before(:each) do
# byebug
# assignment = build(:assignment)
# course_participant = build(:course_participant)
# @assignment_participant = build(:participant)
#end
it "create a copy of participant" do
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(@assignment.id)).to be_an_instance_of(AssignmentParticipant)
end
it "returns nil if copy exist" do
allow(AssignmentParticipant).to receive(:where).and_return(AssignmentParticipant)
allow(AssignmentParticipant).to receive(:first).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(assignment.id)).to be_nil
end
end'''
</pre>

Test cases for Assignment_team.rb

<pre>
describe ".import" do
context "when an assignment team does not already exist with the same id" do
it "cannot be imported" do
assignment_id = 1
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(nil)
error_message = "The assignment with the id \"" + assignment_id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?"
expect { AssignmentTeam.import([], assignment_id, []) }.
to raise_error(ImportError, error_message)
end
end

context "when an assignment team with the same id already exists" do
it "gets imported through Team.import" do
row = []
assignment_id = 1
options = []
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(assignment)
allow(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
expect(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
AssignmentTeam.import(row, assignment_id, options)
end
end
end
</pre>

Added test for the assignment_participant_team.rb
<pre>
describe ".import" do
context 'when record is empty' do
it 'raises an ArgumentError' do
expect { AssignmentParticipant.import({}, nil, nil, nil) }.to raise_error(ArgumentError, 'No user id has been specified.')
end
end

context 'when no user is found by offered username' do
context 'when the record has less than 4 items' do
it 'raises an ArgumentError' do
row = {name: 'no one', fullname: 'no one', email: 'no_one@email.com'}
expect(ImportFileHelper).not_to receive(:create_new_user)
expect { AssignmentParticipant.import(row, nil, nil, nil) }.to raise_error('The record containing no one does not have enough items.')
end
end

context 'when new user needs to be created' do
let(:row) do
{name: 'no one', fullname: 'no one', email: 'name@email.com', role:'user_role_name', parent: 'user_parent_name'}
end
let(:attributes) do
{role_id: 1, name: 'no one', fullname: 'no one', email: 'name@email.com', email_on_submission: 'name@email.com',
email_on_review: 'name@email.com', email_on_review_of_review: 'name@email.com'}
end
let(:test_user) do
{name: 'abc', email: 'abcbbc@gmail.com'}
end
it 'create the user and number of mails sent should be 1' do
ActionMailer::Base.deliveries.clear
allow(ImportFileHelper).to receive(:define_attributes).with(row).and_return(attributes)
allow(ImportFileHelper).to receive(:create_new_user) do
test_user = User.new(name: 'abc', fullname: 'abc bbc', email: 'abcbbc@gmail.com')
test_user.id = 123
test_user.save!
test_user
end
#allow(ImportFileHelper).to receive(:create_new_user).with(attributes, {}).and_return()
allow(Assignment).to receive(:find).with(1).and_return(assignment)
allow(User).to receive(:exists?).with(name: 'no one').and_return(false)
allow(participant).to receive(:set_handle).and_return('handle')
allow(AssignmentParticipant).to receive(:exists?).and_return(false)
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(AssignmentParticipant).to receive(:set_handle)
expect{(AssignmentParticipant.import(row, nil, {}, 1))}.to change { ActionMailer::Base.deliveries.count }.by(1)
end
end
</pre>

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T23:20:28Z

Svsakham: /* What has been Implemented */

=== Team ===
*Akash Sarda, aksarda
*Sairam Sakhamuri, svsakham
*Sidhant Arora, sarora22
*Sai Harsha Nadendla, snadend2

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

The export functionality is however is already quite refactored to suit the strategy pattern, which is not changed by the previous group as well.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Design Plan ==

The design plan is to use Strategy patter to implement import functionality, so that all the import requests present in different models are routed through the ImportFileController. Right now since the import functionality is implemented in various model methods this leads to many if and else statements to check the type of model. So, we intent to generalize the import functionality by placing common code in the ImportFileController. But each model will still have its own import method. With this approach the redundancy is reduced by moving common code to ImportFileController and code will become DRY.

Other helper methods such as ImportFileHelper and ImportTopicHelper that are used to perform import functionality will also be removed, which keeps import functionality consistent. We will be using method overloading and overriding for the methods in ImportFileController to eliminate unnecessary if and else blocks.

To summarize our plan of changes:

* Redirect all import calls through ImportFileController
* Refactor ImportFileController by removing the redundant code and making code generic.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.

[[Image: DesignImport.PNG]]
== What has been Implemented==
=== Models ===

*ImportFileController is changed and has been made more generalized, many case statements are removed. The ImprortFileController is made so small and understandable. The import functionality is moved into the corresponding models. Previously code was present in different modules which was confusing now the functionality was all in one place. The following code changes are follows:

*Previous Implmentation
<pre>
def import_from_hash(session, params)
if params[:model] == "AssignmentTeam" or params[:model] == "CourseTeam"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
if params[:model] == "AssignmentTeam"
teamtype = AssignmentTeam
else
teamtype = CourseTeam
end
options = eval(params[:options])
options[:has_teamname] = params[:has_teamname]
Team.import(row_hash, params[:id], options, teamtype)
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "ReviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
ReviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "MetareviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
MetareviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'SignUpTopic' || params[:model] == 'SignUpSheet'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
if params[:optional_count] == '0'
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '1'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '2'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '3'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5], params[:select6]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
end
end
errors = []
begin
@header_integrated_body.each do |row_hash|
session[:assignment_id] = params[:id]
Object.const_get(params[:model]).import(row_hash, session, params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'AssignmentParticipant' || params[:model] == 'CourseParticipant'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header], contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
if params[:model] == 'AssignmentParticipant'
@header_integrated_body.each do |row_hash|
AssignmentParticipant.import(row_hash, session, params[:id])
end
elsif params[:model] == 'CourseParticipant'
@header_integrated_body.each do |row_hash|
CourseParticipant.import(row_hash, session, params[:id])
end
end
rescue
errors << $ERROR_INFO
end
else # params[:model] = "User"
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
@header_integrated_body.each do |row_hash|
User.import(row_hash, nil, session)
end
rescue StandardError
errors << $ERROR_INFO
end
end
</pre>
*Current Implementation
<pre>
def import
errors = import_from_hash(session, params)
err_msg = "The following errors were encountered during import.Other records may have been added. A second submission will not duplicate these records."
errors.each do |error|
err_msg = err_msg + error.to_s
end
err_msg += </ul>
if errors.empty?
ExpertizaLogger.info LoggerMessage.new(controller_name, session[:user].name, "The file has been successfully imported.", request)
undo_link("The file has been successfully imported.")
else
ExpertizaLogger.error LoggerMessage.new(controller_name, session[:user].name, err_msg, request)
flash[:error] = err_msg
end
redirect_to session[:return_to]
end
</pre>

*Assignment Participant is changed to use the newly implemented #import functionality. Changes are as shown below:

<pre>
def self.import(row_hash, session, id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?
raise ImportError, "The assignment with id #{id} was not found." if Assignment.find(id).nil?
unless AssignmentParticipant.exists?(user_id: user.id, parent_id: id)
new_part = AssignmentParticipant.new(user_id: user.id, parent_id: id)
new_part.set_handle
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

*Assignment Team is also changed to use the newly implemented #import functionality.

<pre>
def self.import(row_hash, session = nil, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The assignment with the id \"" + id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?" if Assignment.find_by(id: id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Course_team is is also changed to use the newly implemented #import functionality.
<pre>
def self.import(row_hash, session, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The course with the id \"" + id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?" if Course.find(id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Review response map's #import functionality is also updated to use the newly implemented #import functionality.
<pre>

def self.import(row_hash, _session, assignment_id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
reviewee_user_name = row_hash[:reviewee].to_s
reviewee_user = User.find_by(name: reviewee_user_name)
raise ArgumentError, "Cannot find reviewee user." unless reviewee_user
@@ -57,7 +58,7 @@ def self.import(row_hash, _session, assignment_id)
team_node = TeamNode.create(parent_id: assignment_id, node_object_id: reviewee_team.id)
TeamUserNode.create(parent_id: team_node.id, node_object_id: t_user.id)
end
row_hash[:reviewers].split.each do |reviewer|
reviewer_user_name = reviewer.to_s
reviewer_user = User.find_by(name: reviewer_user_name)
raise ArgumentError, "Cannot find reviewer user." unless reviewer_user
@@ -71,6 +72,20 @@ def self.import(row_hash, _session, assignment_id)
end
end

</pre>

*Refactored the Course_participant.rb to use newly added import functionality.
<pre>
def self.import(row_hash, session, id)
byebug
raise ArgumentError, "The record does not have enough items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?

user = User.find_by(name: row_hash[:name])
if user.nil?
raise ArgumentError, "The record containing #{row_hash[:name]} does not have enough items." if row_hash.length < 4
attributes = ImportFileHelper.define_attributes(row_hash)
user = ImportFileHelper.create_new_user(attributes, session)
end

course = Course.find_by(id)
raise ImportError, "The course with id " + id.to_s + " was not found." if course.nil?
unless CourseParticipant.exists?(user_id: user.id, parent_id: id)
CourseParticipant.create(user_id: user.id, parent_id: id)
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

With these new changes to #import functionality. The following modules review_response_map.rb, course_team.rb,assignment_team.rb,assignment_participant.rb have the "self.import" which will use the import functionality in the ImportFileController and they are made as concise as possible. Each module has specific checks while importing the file, so we can't have same code for all the module.

== Test Plan ==
The import method has been implemented in a bunch of models which have been listed above. After preliminary analysis, we assume that the import functionality in a few of the models might have to be amended. These models are:
*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''review_response_map'''

Scenarios:
1. when assignment found and assignment participant does not exist, creates a new user and participant
2. when assignment cannot be found, creates a new user then raises an ImportError
3. when the assignment team does not have the required fields, raises ArgumentError
4. check what import/export actions are allowed for admin, instructor, TA, student
5. when course found and course participant does not exist, creates a new user and participant
6. when the course team does not have the required fields, raises ArgumentError

Newly added rspec test case to test the #import functionality in review_response_map_spec.rb

<pre>
it '#import' do
expect {ReviewResponseMap.import({reviewee: "name"}, nil, 1)}.to raise_error(ArgumentError, "Record does not contain required items.")
row_hash = {reviewee: "name", reviewers: "name1"}
#row_hash = {reviewee: "name", reviewers: ["name1"]}
session = nil
assignment_id = 1
# when reviewee user = nil
allow(User).to receive(:find_by).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Cannot find reviewee user.")
# when reviewee user exists but reviewee user is not a participant in this assignment
allow(User).to receive(:find_by).with(name: "name").and_return(student)
allow(AssignmentParticipant).to receive(:find_by).with(user_id: 1, parent_id: 1).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Reviewee user is not a participant in this assignment.")
end
</pre>

Newly added rspec test case to test the #import functionality in course_team_spec.rb

<pre>
describe ".import" do
let(:row) do
{teammembers: 'none'}
end
context "when a course team does not exist with id" do
it "raises ImportError" do
course_id = 1
allow(Course).to receive(:find).with(course_id).and_return(nil)
error_message = "The course with the id \"" + course_id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?"
expect { CourseTeam.import(row, nil, course_id, nil) }.
to raise_error(ImportError, error_message)
end
end

context "when the course team does not have the required fields" do
it "raises ArgumentError" do
expect { CourseTeam.import([], nil, 1, nil) }.
to raise_error(ArgumentError)
end
end

context "when a course team with the same id already exists" do
it "gets imported through Team.import" do
course_id = 1
options = []
allow(Course).to receive(:find).with(course_id).and_return(course)
expect(Team).to receive(:import_helper).with(row, course_id, options, instance_of(CourseTeam))
CourseTeam.import(row, nil, course_id, options)
end
end
end
</pre>

Newly added rspec test case to test the #import functionality in course_participant_spec.rb

<pre>
describe "CourseParticipant" do
let(:course) { build(:course, id: 1, name: 'ECE517') }
let(:assignment) { build(:assignment, id: 1, name: 'no assignment', participants: [participant], teams: [team]) }
let(:assignment_participant) {build(:participant, id: 1)}
'''describe "#copy" do
#before(:each) do
# byebug
# assignment = build(:assignment)
# course_participant = build(:course_participant)
# @assignment_participant = build(:participant)
#end
it "create a copy of participant" do
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(@assignment.id)).to be_an_instance_of(AssignmentParticipant)
end
it "returns nil if copy exist" do
allow(AssignmentParticipant).to receive(:where).and_return(AssignmentParticipant)
allow(AssignmentParticipant).to receive(:first).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(assignment.id)).to be_nil
end
end'''
</pre>

Test cases for Assignment_team.rb

<pre>
describe ".import" do
context "when an assignment team does not already exist with the same id" do
it "cannot be imported" do
assignment_id = 1
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(nil)
error_message = "The assignment with the id \"" + assignment_id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?"
expect { AssignmentTeam.import([], assignment_id, []) }.
to raise_error(ImportError, error_message)
end
end

context "when an assignment team with the same id already exists" do
it "gets imported through Team.import" do
row = []
assignment_id = 1
options = []
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(assignment)
allow(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
expect(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
AssignmentTeam.import(row, assignment_id, options)
end
end
end
</pre>

Added test for the assignment_participant_team.rb
<pre>
describe ".import" do
context 'when record is empty' do
it 'raises an ArgumentError' do
expect { AssignmentParticipant.import({}, nil, nil, nil) }.to raise_error(ArgumentError, 'No user id has been specified.')
end
end

context 'when no user is found by offered username' do
context 'when the record has less than 4 items' do
it 'raises an ArgumentError' do
row = {name: 'no one', fullname: 'no one', email: 'no_one@email.com'}
expect(ImportFileHelper).not_to receive(:create_new_user)
expect { AssignmentParticipant.import(row, nil, nil, nil) }.to raise_error('The record containing no one does not have enough items.')
end
end

context 'when new user needs to be created' do
let(:row) do
{name: 'no one', fullname: 'no one', email: 'name@email.com', role:'user_role_name', parent: 'user_parent_name'}
end
let(:attributes) do
{role_id: 1, name: 'no one', fullname: 'no one', email: 'name@email.com', email_on_submission: 'name@email.com',
email_on_review: 'name@email.com', email_on_review_of_review: 'name@email.com'}
end
let(:test_user) do
{name: 'abc', email: 'abcbbc@gmail.com'}
end
it 'create the user and number of mails sent should be 1' do
ActionMailer::Base.deliveries.clear
allow(ImportFileHelper).to receive(:define_attributes).with(row).and_return(attributes)
allow(ImportFileHelper).to receive(:create_new_user) do
test_user = User.new(name: 'abc', fullname: 'abc bbc', email: 'abcbbc@gmail.com')
test_user.id = 123
test_user.save!
test_user
end
#allow(ImportFileHelper).to receive(:create_new_user).with(attributes, {}).and_return()
allow(Assignment).to receive(:find).with(1).and_return(assignment)
allow(User).to receive(:exists?).with(name: 'no one').and_return(false)
allow(participant).to receive(:set_handle).and_return('handle')
allow(AssignmentParticipant).to receive(:exists?).and_return(false)
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(AssignmentParticipant).to receive(:set_handle)
expect{(AssignmentParticipant.import(row, nil, {}, 1))}.to change { ActionMailer::Base.deliveries.count }.by(1)
end
end
</pre>

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T23:17:10Z

Svsakham: /* Test Plan */

=== Team ===
*Akash Sarda, aksarda
*Sairam Sakhamuri, svsakham
*Sidhant Arora, sarora22
*Sai Harsha Nadendla, snadend2

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

The export functionality is however is already quite refactored to suit the strategy pattern, which is not changed by the previous group as well.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Design Plan ==

The design plan is to use Strategy patter to implement import functionality, so that all the import requests present in different models are routed through the ImportFileController. Right now since the import functionality is implemented in various model methods this leads to many if and else statements to check the type of model. So, we intent to generalize the import functionality by placing common code in the ImportFileController. But each model will still have its own import method. With this approach the redundancy is reduced by moving common code to ImportFileController and code will become DRY.

Other helper methods such as ImportFileHelper and ImportTopicHelper that are used to perform import functionality will also be removed, which keeps import functionality consistent. We will be using method overloading and overriding for the methods in ImportFileController to eliminate unnecessary if and else blocks.

To summarize our plan of changes:

* Redirect all import calls through ImportFileController
* Refactor ImportFileController by removing the redundant code and making code generic.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.

[[Image: DesignImport.PNG]]
== What has been Implemented==
=== Models ===

*ImportFileController is changed and has been made more generalized, many case statements are removed. The ImprortFileController is made so small and understandable. The import functionality is moved into the corresponding models. Previously code was present in different modules which was confusing now the functionality was all in one place. The following code changes are follows:

*Previous Implmentation
<pre>
def import_from_hash(session, params)
if params[:model] == "AssignmentTeam" or params[:model] == "CourseTeam"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
if params[:model] == "AssignmentTeam"
teamtype = AssignmentTeam
else
teamtype = CourseTeam
end
options = eval(params[:options])
options[:has_teamname] = params[:has_teamname]
Team.import(row_hash, params[:id], options, teamtype)
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "ReviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
ReviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "MetareviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
MetareviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'SignUpTopic' || params[:model] == 'SignUpSheet'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
if params[:optional_count] == '0'
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '1'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '2'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '3'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5], params[:select6]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
end
end
errors = []
begin
@header_integrated_body.each do |row_hash|
session[:assignment_id] = params[:id]
Object.const_get(params[:model]).import(row_hash, session, params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'AssignmentParticipant' || params[:model] == 'CourseParticipant'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header], contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
if params[:model] == 'AssignmentParticipant'
@header_integrated_body.each do |row_hash|
AssignmentParticipant.import(row_hash, session, params[:id])
end
elsif params[:model] == 'CourseParticipant'
@header_integrated_body.each do |row_hash|
CourseParticipant.import(row_hash, session, params[:id])
end
end
rescue
errors << $ERROR_INFO
end
else # params[:model] = "User"
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
@header_integrated_body.each do |row_hash|
User.import(row_hash, nil, session)
end
rescue StandardError
errors << $ERROR_INFO
end
end
</pre>
*Current Implementation
<pre>
def import
errors = import_from_hash(session, params)
err_msg = "The following errors were encountered during import.Other records may have been added. A second submission will not duplicate these records."
errors.each do |error|
err_msg = err_msg + error.to_s
end
err_msg += </ul>
if errors.empty?
ExpertizaLogger.info LoggerMessage.new(controller_name, session[:user].name, "The file has been successfully imported.", request)
undo_link("The file has been successfully imported.")
else
ExpertizaLogger.error LoggerMessage.new(controller_name, session[:user].name, err_msg, request)
flash[:error] = err_msg
end
redirect_to session[:return_to]
end
</pre>

*Assignment Participant is changed to use the newly implemented #import functionality. Changes are as shown below:

<pre>
def self.import(row_hash, session, id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?
raise ImportError, "The assignment with id #{id} was not found." if Assignment.find(id).nil?
unless AssignmentParticipant.exists?(user_id: user.id, parent_id: id)
new_part = AssignmentParticipant.new(user_id: user.id, parent_id: id)
new_part.set_handle
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

*Assignment Team is also changed to use the newly implemented #import functionality.

<pre>
def self.import(row_hash, session = nil, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The assignment with the id \"" + id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?" if Assignment.find_by(id: id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Course_team is is also changed to use the newly implemented #import functionality.
<pre>
def self.import(row_hash, session, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The course with the id \"" + id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?" if Course.find(id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Review response map's #import functionality is also updated to use the newly implemented #import functionality.
<pre>

def self.import(row_hash, _session, assignment_id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
reviewee_user_name = row_hash[:reviewee].to_s
reviewee_user = User.find_by(name: reviewee_user_name)
raise ArgumentError, "Cannot find reviewee user." unless reviewee_user
@@ -57,7 +58,7 @@ def self.import(row_hash, _session, assignment_id)
team_node = TeamNode.create(parent_id: assignment_id, node_object_id: reviewee_team.id)
TeamUserNode.create(parent_id: team_node.id, node_object_id: t_user.id)
end
row_hash[:reviewers].split.each do |reviewer|
reviewer_user_name = reviewer.to_s
reviewer_user = User.find_by(name: reviewer_user_name)
raise ArgumentError, "Cannot find reviewer user." unless reviewer_user
@@ -71,6 +72,20 @@ def self.import(row_hash, _session, assignment_id)
end
end

</pre>

With these new changes to #import functionality. The following modules review_response_map.rb, course_team.rb,assignment_team.rb,assignment_participant.rb have the "self.import" which will use the import functionality in the ImportFileController and they are made as concise as possible. Each module has specific checks while importing the file, so we can't have same code for all the module.

== Test Plan ==
The import method has been implemented in a bunch of models which have been listed above. After preliminary analysis, we assume that the import functionality in a few of the models might have to be amended. These models are:
*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''review_response_map'''

Scenarios:
1. when assignment found and assignment participant does not exist, creates a new user and participant
2. when assignment cannot be found, creates a new user then raises an ImportError
3. when the assignment team does not have the required fields, raises ArgumentError
4. check what import/export actions are allowed for admin, instructor, TA, student
5. when course found and course participant does not exist, creates a new user and participant
6. when the course team does not have the required fields, raises ArgumentError

Newly added rspec test case to test the #import functionality in review_response_map_spec.rb

<pre>
it '#import' do
expect {ReviewResponseMap.import({reviewee: "name"}, nil, 1)}.to raise_error(ArgumentError, "Record does not contain required items.")
row_hash = {reviewee: "name", reviewers: "name1"}
#row_hash = {reviewee: "name", reviewers: ["name1"]}
session = nil
assignment_id = 1
# when reviewee user = nil
allow(User).to receive(:find_by).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Cannot find reviewee user.")
# when reviewee user exists but reviewee user is not a participant in this assignment
allow(User).to receive(:find_by).with(name: "name").and_return(student)
allow(AssignmentParticipant).to receive(:find_by).with(user_id: 1, parent_id: 1).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Reviewee user is not a participant in this assignment.")
end
</pre>

Newly added rspec test case to test the #import functionality in course_team_spec.rb

<pre>
describe ".import" do
let(:row) do
{teammembers: 'none'}
end
context "when a course team does not exist with id" do
it "raises ImportError" do
course_id = 1
allow(Course).to receive(:find).with(course_id).and_return(nil)
error_message = "The course with the id \"" + course_id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?"
expect { CourseTeam.import(row, nil, course_id, nil) }.
to raise_error(ImportError, error_message)
end
end

context "when the course team does not have the required fields" do
it "raises ArgumentError" do
expect { CourseTeam.import([], nil, 1, nil) }.
to raise_error(ArgumentError)
end
end

context "when a course team with the same id already exists" do
it "gets imported through Team.import" do
course_id = 1
options = []
allow(Course).to receive(:find).with(course_id).and_return(course)
expect(Team).to receive(:import_helper).with(row, course_id, options, instance_of(CourseTeam))
CourseTeam.import(row, nil, course_id, options)
end
end
end
</pre>

Newly added rspec test case to test the #import functionality in course_participant_spec.rb

<pre>
describe "CourseParticipant" do
let(:course) { build(:course, id: 1, name: 'ECE517') }
let(:assignment) { build(:assignment, id: 1, name: 'no assignment', participants: [participant], teams: [team]) }
let(:assignment_participant) {build(:participant, id: 1)}
'''describe "#copy" do
#before(:each) do
# byebug
# assignment = build(:assignment)
# course_participant = build(:course_participant)
# @assignment_participant = build(:participant)
#end
it "create a copy of participant" do
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(@assignment.id)).to be_an_instance_of(AssignmentParticipant)
end
it "returns nil if copy exist" do
allow(AssignmentParticipant).to receive(:where).and_return(AssignmentParticipant)
allow(AssignmentParticipant).to receive(:first).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(assignment.id)).to be_nil
end
end'''
</pre>

Test cases for Assignment_team.rb

<pre>
describe ".import" do
context "when an assignment team does not already exist with the same id" do
it "cannot be imported" do
assignment_id = 1
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(nil)
error_message = "The assignment with the id \"" + assignment_id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?"
expect { AssignmentTeam.import([], assignment_id, []) }.
to raise_error(ImportError, error_message)
end
end

context "when an assignment team with the same id already exists" do
it "gets imported through Team.import" do
row = []
assignment_id = 1
options = []
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(assignment)
allow(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
expect(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
AssignmentTeam.import(row, assignment_id, options)
end
end
end
</pre>

Added test for the assignment_participant_team.rb
<pre>
describe ".import" do
context 'when record is empty' do
it 'raises an ArgumentError' do
expect { AssignmentParticipant.import({}, nil, nil, nil) }.to raise_error(ArgumentError, 'No user id has been specified.')
end
end

context 'when no user is found by offered username' do
context 'when the record has less than 4 items' do
it 'raises an ArgumentError' do
row = {name: 'no one', fullname: 'no one', email: 'no_one@email.com'}
expect(ImportFileHelper).not_to receive(:create_new_user)
expect { AssignmentParticipant.import(row, nil, nil, nil) }.to raise_error('The record containing no one does not have enough items.')
end
end

context 'when new user needs to be created' do
let(:row) do
{name: 'no one', fullname: 'no one', email: 'name@email.com', role:'user_role_name', parent: 'user_parent_name'}
end
let(:attributes) do
{role_id: 1, name: 'no one', fullname: 'no one', email: 'name@email.com', email_on_submission: 'name@email.com',
email_on_review: 'name@email.com', email_on_review_of_review: 'name@email.com'}
end
let(:test_user) do
{name: 'abc', email: 'abcbbc@gmail.com'}
end
it 'create the user and number of mails sent should be 1' do
ActionMailer::Base.deliveries.clear
allow(ImportFileHelper).to receive(:define_attributes).with(row).and_return(attributes)
allow(ImportFileHelper).to receive(:create_new_user) do
test_user = User.new(name: 'abc', fullname: 'abc bbc', email: 'abcbbc@gmail.com')
test_user.id = 123
test_user.save!
test_user
end
#allow(ImportFileHelper).to receive(:create_new_user).with(attributes, {}).and_return()
allow(Assignment).to receive(:find).with(1).and_return(assignment)
allow(User).to receive(:exists?).with(name: 'no one').and_return(false)
allow(participant).to receive(:set_handle).and_return('handle')
allow(AssignmentParticipant).to receive(:exists?).and_return(false)
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(AssignmentParticipant).to receive(:set_handle)
expect{(AssignmentParticipant.import(row, nil, {}, 1))}.to change { ActionMailer::Base.deliveries.count }.by(1)
end
end
</pre>

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T23:16:46Z

Svsakham: /* Test Plan */

=== Team ===
*Akash Sarda, aksarda
*Sairam Sakhamuri, svsakham
*Sidhant Arora, sarora22
*Sai Harsha Nadendla, snadend2

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

The export functionality is however is already quite refactored to suit the strategy pattern, which is not changed by the previous group as well.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Design Plan ==

The design plan is to use Strategy patter to implement import functionality, so that all the import requests present in different models are routed through the ImportFileController. Right now since the import functionality is implemented in various model methods this leads to many if and else statements to check the type of model. So, we intent to generalize the import functionality by placing common code in the ImportFileController. But each model will still have its own import method. With this approach the redundancy is reduced by moving common code to ImportFileController and code will become DRY.

Other helper methods such as ImportFileHelper and ImportTopicHelper that are used to perform import functionality will also be removed, which keeps import functionality consistent. We will be using method overloading and overriding for the methods in ImportFileController to eliminate unnecessary if and else blocks.

To summarize our plan of changes:

* Redirect all import calls through ImportFileController
* Refactor ImportFileController by removing the redundant code and making code generic.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.

[[Image: DesignImport.PNG]]
== What has been Implemented==
=== Models ===

*ImportFileController is changed and has been made more generalized, many case statements are removed. The ImprortFileController is made so small and understandable. The import functionality is moved into the corresponding models. Previously code was present in different modules which was confusing now the functionality was all in one place. The following code changes are follows:

*Previous Implmentation
<pre>
def import_from_hash(session, params)
if params[:model] == "AssignmentTeam" or params[:model] == "CourseTeam"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
if params[:model] == "AssignmentTeam"
teamtype = AssignmentTeam
else
teamtype = CourseTeam
end
options = eval(params[:options])
options[:has_teamname] = params[:has_teamname]
Team.import(row_hash, params[:id], options, teamtype)
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "ReviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
ReviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "MetareviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
MetareviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'SignUpTopic' || params[:model] == 'SignUpSheet'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
if params[:optional_count] == '0'
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '1'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '2'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '3'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5], params[:select6]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
end
end
errors = []
begin
@header_integrated_body.each do |row_hash|
session[:assignment_id] = params[:id]
Object.const_get(params[:model]).import(row_hash, session, params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'AssignmentParticipant' || params[:model] == 'CourseParticipant'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header], contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
if params[:model] == 'AssignmentParticipant'
@header_integrated_body.each do |row_hash|
AssignmentParticipant.import(row_hash, session, params[:id])
end
elsif params[:model] == 'CourseParticipant'
@header_integrated_body.each do |row_hash|
CourseParticipant.import(row_hash, session, params[:id])
end
end
rescue
errors << $ERROR_INFO
end
else # params[:model] = "User"
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
@header_integrated_body.each do |row_hash|
User.import(row_hash, nil, session)
end
rescue StandardError
errors << $ERROR_INFO
end
end
</pre>
*Current Implementation
<pre>
def import
errors = import_from_hash(session, params)
err_msg = "The following errors were encountered during import.Other records may have been added. A second submission will not duplicate these records."
errors.each do |error|
err_msg = err_msg + error.to_s
end
err_msg += </ul>
if errors.empty?
ExpertizaLogger.info LoggerMessage.new(controller_name, session[:user].name, "The file has been successfully imported.", request)
undo_link("The file has been successfully imported.")
else
ExpertizaLogger.error LoggerMessage.new(controller_name, session[:user].name, err_msg, request)
flash[:error] = err_msg
end
redirect_to session[:return_to]
end
</pre>

*Assignment Participant is changed to use the newly implemented #import functionality. Changes are as shown below:

<pre>
def self.import(row_hash, session, id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?
raise ImportError, "The assignment with id #{id} was not found." if Assignment.find(id).nil?
unless AssignmentParticipant.exists?(user_id: user.id, parent_id: id)
new_part = AssignmentParticipant.new(user_id: user.id, parent_id: id)
new_part.set_handle
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

*Assignment Team is also changed to use the newly implemented #import functionality.

<pre>
def self.import(row_hash, session = nil, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The assignment with the id \"" + id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?" if Assignment.find_by(id: id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Course_team is is also changed to use the newly implemented #import functionality.
<pre>
def self.import(row_hash, session, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The course with the id \"" + id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?" if Course.find(id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Review response map's #import functionality is also updated to use the newly implemented #import functionality.
<pre>

def self.import(row_hash, _session, assignment_id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
reviewee_user_name = row_hash[:reviewee].to_s
reviewee_user = User.find_by(name: reviewee_user_name)
raise ArgumentError, "Cannot find reviewee user." unless reviewee_user
@@ -57,7 +58,7 @@ def self.import(row_hash, _session, assignment_id)
team_node = TeamNode.create(parent_id: assignment_id, node_object_id: reviewee_team.id)
TeamUserNode.create(parent_id: team_node.id, node_object_id: t_user.id)
end
row_hash[:reviewers].split.each do |reviewer|
reviewer_user_name = reviewer.to_s
reviewer_user = User.find_by(name: reviewer_user_name)
raise ArgumentError, "Cannot find reviewer user." unless reviewer_user
@@ -71,6 +72,20 @@ def self.import(row_hash, _session, assignment_id)
end
end

</pre>

With these new changes to #import functionality. The following modules review_response_map.rb, course_team.rb,assignment_team.rb,assignment_participant.rb have the "self.import" which will use the import functionality in the ImportFileController and they are made as concise as possible. Each module has specific checks while importing the file, so we can't have same code for all the module.

== Test Plan ==
The import method has been implemented in a bunch of models which have been listed above. After preliminary analysis, we assume that the import functionality in a few of the models might have to be amended. These models are:
*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''review_response_map'''
*'''sign_up_topic'''
*'''user'''
Scenarios:
1. when assignment found and assignment participant does not exist, creates a new user and participant
2. when assignment cannot be found, creates a new user then raises an ImportError
3. when the assignment team does not have the required fields, raises ArgumentError
4. check what import/export actions are allowed for admin, instructor, TA, student
5. when course found and course participant does not exist, creates a new user and participant
6. when the course team does not have the required fields, raises ArgumentError

Newly added rspec test case to test the #import functionality in review_response_map_spec.rb

<pre>
it '#import' do
expect {ReviewResponseMap.import({reviewee: "name"}, nil, 1)}.to raise_error(ArgumentError, "Record does not contain required items.")
row_hash = {reviewee: "name", reviewers: "name1"}
#row_hash = {reviewee: "name", reviewers: ["name1"]}
session = nil
assignment_id = 1
# when reviewee user = nil
allow(User).to receive(:find_by).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Cannot find reviewee user.")
# when reviewee user exists but reviewee user is not a participant in this assignment
allow(User).to receive(:find_by).with(name: "name").and_return(student)
allow(AssignmentParticipant).to receive(:find_by).with(user_id: 1, parent_id: 1).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Reviewee user is not a participant in this assignment.")
end
</pre>

Newly added rspec test case to test the #import functionality in course_team_spec.rb

<pre>
describe ".import" do
let(:row) do
{teammembers: 'none'}
end
context "when a course team does not exist with id" do
it "raises ImportError" do
course_id = 1
allow(Course).to receive(:find).with(course_id).and_return(nil)
error_message = "The course with the id \"" + course_id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?"
expect { CourseTeam.import(row, nil, course_id, nil) }.
to raise_error(ImportError, error_message)
end
end

context "when the course team does not have the required fields" do
it "raises ArgumentError" do
expect { CourseTeam.import([], nil, 1, nil) }.
to raise_error(ArgumentError)
end
end

context "when a course team with the same id already exists" do
it "gets imported through Team.import" do
course_id = 1
options = []
allow(Course).to receive(:find).with(course_id).and_return(course)
expect(Team).to receive(:import_helper).with(row, course_id, options, instance_of(CourseTeam))
CourseTeam.import(row, nil, course_id, options)
end
end
end
</pre>

Newly added rspec test case to test the #import functionality in course_participant_spec.rb

<pre>
describe "CourseParticipant" do
let(:course) { build(:course, id: 1, name: 'ECE517') }
let(:assignment) { build(:assignment, id: 1, name: 'no assignment', participants: [participant], teams: [team]) }
let(:assignment_participant) {build(:participant, id: 1)}
'''describe "#copy" do
#before(:each) do
# byebug
# assignment = build(:assignment)
# course_participant = build(:course_participant)
# @assignment_participant = build(:participant)
#end
it "create a copy of participant" do
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(@assignment.id)).to be_an_instance_of(AssignmentParticipant)
end
it "returns nil if copy exist" do
allow(AssignmentParticipant).to receive(:where).and_return(AssignmentParticipant)
allow(AssignmentParticipant).to receive(:first).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(assignment.id)).to be_nil
end
end'''
</pre>

Test cases for Assignment_team.rb

<pre>
describe ".import" do
context "when an assignment team does not already exist with the same id" do
it "cannot be imported" do
assignment_id = 1
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(nil)
error_message = "The assignment with the id \"" + assignment_id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?"
expect { AssignmentTeam.import([], assignment_id, []) }.
to raise_error(ImportError, error_message)
end
end

context "when an assignment team with the same id already exists" do
it "gets imported through Team.import" do
row = []
assignment_id = 1
options = []
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(assignment)
allow(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
expect(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
AssignmentTeam.import(row, assignment_id, options)
end
end
end
</pre>

Added test for the assignment_participant_team.rb
<pre>
describe ".import" do
context 'when record is empty' do
it 'raises an ArgumentError' do
expect { AssignmentParticipant.import({}, nil, nil, nil) }.to raise_error(ArgumentError, 'No user id has been specified.')
end
end

context 'when no user is found by offered username' do
context 'when the record has less than 4 items' do
it 'raises an ArgumentError' do
row = {name: 'no one', fullname: 'no one', email: 'no_one@email.com'}
expect(ImportFileHelper).not_to receive(:create_new_user)
expect { AssignmentParticipant.import(row, nil, nil, nil) }.to raise_error('The record containing no one does not have enough items.')
end
end

context 'when new user needs to be created' do
let(:row) do
{name: 'no one', fullname: 'no one', email: 'name@email.com', role:'user_role_name', parent: 'user_parent_name'}
end
let(:attributes) do
{role_id: 1, name: 'no one', fullname: 'no one', email: 'name@email.com', email_on_submission: 'name@email.com',
email_on_review: 'name@email.com', email_on_review_of_review: 'name@email.com'}
end
let(:test_user) do
{name: 'abc', email: 'abcbbc@gmail.com'}
end
it 'create the user and number of mails sent should be 1' do
ActionMailer::Base.deliveries.clear
allow(ImportFileHelper).to receive(:define_attributes).with(row).and_return(attributes)
allow(ImportFileHelper).to receive(:create_new_user) do
test_user = User.new(name: 'abc', fullname: 'abc bbc', email: 'abcbbc@gmail.com')
test_user.id = 123
test_user.save!
test_user
end
#allow(ImportFileHelper).to receive(:create_new_user).with(attributes, {}).and_return()
allow(Assignment).to receive(:find).with(1).and_return(assignment)
allow(User).to receive(:exists?).with(name: 'no one').and_return(false)
allow(participant).to receive(:set_handle).and_return('handle')
allow(AssignmentParticipant).to receive(:exists?).and_return(false)
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(AssignmentParticipant).to receive(:set_handle)
expect{(AssignmentParticipant.import(row, nil, {}, 1))}.to change { ActionMailer::Base.deliveries.count }.by(1)
end
end
</pre>

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T23:15:03Z

Svsakham: /* Test Plan */

=== Team ===
*Akash Sarda, aksarda
*Sairam Sakhamuri, svsakham
*Sidhant Arora, sarora22
*Sai Harsha Nadendla, snadend2

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

The export functionality is however is already quite refactored to suit the strategy pattern, which is not changed by the previous group as well.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Design Plan ==

The design plan is to use Strategy patter to implement import functionality, so that all the import requests present in different models are routed through the ImportFileController. Right now since the import functionality is implemented in various model methods this leads to many if and else statements to check the type of model. So, we intent to generalize the import functionality by placing common code in the ImportFileController. But each model will still have its own import method. With this approach the redundancy is reduced by moving common code to ImportFileController and code will become DRY.

Other helper methods such as ImportFileHelper and ImportTopicHelper that are used to perform import functionality will also be removed, which keeps import functionality consistent. We will be using method overloading and overriding for the methods in ImportFileController to eliminate unnecessary if and else blocks.

To summarize our plan of changes:

* Redirect all import calls through ImportFileController
* Refactor ImportFileController by removing the redundant code and making code generic.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.

[[Image: DesignImport.PNG]]
== What has been Implemented==
=== Models ===

*ImportFileController is changed and has been made more generalized, many case statements are removed. The ImprortFileController is made so small and understandable. The import functionality is moved into the corresponding models. Previously code was present in different modules which was confusing now the functionality was all in one place. The following code changes are follows:

*Previous Implmentation
<pre>
def import_from_hash(session, params)
if params[:model] == "AssignmentTeam" or params[:model] == "CourseTeam"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
if params[:model] == "AssignmentTeam"
teamtype = AssignmentTeam
else
teamtype = CourseTeam
end
options = eval(params[:options])
options[:has_teamname] = params[:has_teamname]
Team.import(row_hash, params[:id], options, teamtype)
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "ReviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
ReviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "MetareviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
MetareviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'SignUpTopic' || params[:model] == 'SignUpSheet'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
if params[:optional_count] == '0'
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '1'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '2'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '3'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5], params[:select6]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
end
end
errors = []
begin
@header_integrated_body.each do |row_hash|
session[:assignment_id] = params[:id]
Object.const_get(params[:model]).import(row_hash, session, params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'AssignmentParticipant' || params[:model] == 'CourseParticipant'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header], contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
if params[:model] == 'AssignmentParticipant'
@header_integrated_body.each do |row_hash|
AssignmentParticipant.import(row_hash, session, params[:id])
end
elsif params[:model] == 'CourseParticipant'
@header_integrated_body.each do |row_hash|
CourseParticipant.import(row_hash, session, params[:id])
end
end
rescue
errors << $ERROR_INFO
end
else # params[:model] = "User"
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
@header_integrated_body.each do |row_hash|
User.import(row_hash, nil, session)
end
rescue StandardError
errors << $ERROR_INFO
end
end
</pre>
*Current Implementation
<pre>
def import
errors = import_from_hash(session, params)
err_msg = "The following errors were encountered during import.Other records may have been added. A second submission will not duplicate these records."
errors.each do |error|
err_msg = err_msg + error.to_s
end
err_msg += </ul>
if errors.empty?
ExpertizaLogger.info LoggerMessage.new(controller_name, session[:user].name, "The file has been successfully imported.", request)
undo_link("The file has been successfully imported.")
else
ExpertizaLogger.error LoggerMessage.new(controller_name, session[:user].name, err_msg, request)
flash[:error] = err_msg
end
redirect_to session[:return_to]
end
</pre>

*Assignment Participant is changed to use the newly implemented #import functionality. Changes are as shown below:

<pre>
def self.import(row_hash, session, id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?
raise ImportError, "The assignment with id #{id} was not found." if Assignment.find(id).nil?
unless AssignmentParticipant.exists?(user_id: user.id, parent_id: id)
new_part = AssignmentParticipant.new(user_id: user.id, parent_id: id)
new_part.set_handle
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

*Assignment Team is also changed to use the newly implemented #import functionality.

<pre>
def self.import(row_hash, session = nil, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The assignment with the id \"" + id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?" if Assignment.find_by(id: id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Course_team is is also changed to use the newly implemented #import functionality.
<pre>
def self.import(row_hash, session, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The course with the id \"" + id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?" if Course.find(id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Review response map's #import functionality is also updated to use the newly implemented #import functionality.
<pre>

def self.import(row_hash, _session, assignment_id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
reviewee_user_name = row_hash[:reviewee].to_s
reviewee_user = User.find_by(name: reviewee_user_name)
raise ArgumentError, "Cannot find reviewee user." unless reviewee_user
@@ -57,7 +58,7 @@ def self.import(row_hash, _session, assignment_id)
team_node = TeamNode.create(parent_id: assignment_id, node_object_id: reviewee_team.id)
TeamUserNode.create(parent_id: team_node.id, node_object_id: t_user.id)
end
row_hash[:reviewers].split.each do |reviewer|
reviewer_user_name = reviewer.to_s
reviewer_user = User.find_by(name: reviewer_user_name)
raise ArgumentError, "Cannot find reviewer user." unless reviewer_user
@@ -71,6 +72,20 @@ def self.import(row_hash, _session, assignment_id)
end
end

</pre>

With these new changes to #import functionality. The following modules review_response_map.rb, course_team.rb,assignment_team.rb,assignment_participant.rb have the "self.import" which will use the import functionality in the ImportFileController and they are made as concise as possible. Each module has specific checks while importing the file, so we can't have same code for all the module.

== Test Plan ==
The import method has been implemented in a bunch of models which have been listed above. After preliminary analysis, we assume that the import functionality in a few of the models might have to be amended. These models are:
*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''review_response_map'''
*'''sign_up_topic'''
*'''user'''
Scenarios:
1. when assignment found and assignment participant does not exist, creates a new user and participant
2. when assignment cannot be found, creates a new user then raises an ImportError
3. when the assignment team does not have the required fields, raises ArgumentError
4. check what import/export actions are allowed for admin, instructor, TA, student
5. when course found and course participant does not exist, creates a new user and participant
6. when the course team does not have the required fields, raises ArgumentError

Newly added rspec test case to test the #import functionality in review_response_map_spec.rb

<pre>
it '#import' do
expect {ReviewResponseMap.import({reviewee: "name"}, nil, 1)}.to raise_error(ArgumentError, "Record does not contain required items.")
row_hash = {reviewee: "name", reviewers: "name1"}
#row_hash = {reviewee: "name", reviewers: ["name1"]}
session = nil
assignment_id = 1
# when reviewee user = nil
allow(User).to receive(:find_by).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Cannot find reviewee user.")
# when reviewee user exists but reviewee user is not a participant in this assignment
allow(User).to receive(:find_by).with(name: "name").and_return(student)
allow(AssignmentParticipant).to receive(:find_by).with(user_id: 1, parent_id: 1).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Reviewee user is not a participant in this assignment.")
end
</pre>

Newly added rspec test case to test the #import functionality in course_team_spec.rb

<pre>
describe ".import" do
let(:row) do
{teammembers: 'none'}
end
context "when a course team does not exist with id" do
it "raises ImportError" do
course_id = 1
allow(Course).to receive(:find).with(course_id).and_return(nil)
error_message = "The course with the id \"" + course_id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?"
expect { CourseTeam.import(row, nil, course_id, nil) }.
to raise_error(ImportError, error_message)
end
end

context "when the course team does not have the required fields" do
it "raises ArgumentError" do
expect { CourseTeam.import([], nil, 1, nil) }.
to raise_error(ArgumentError)
end
end

context "when a course team with the same id already exists" do
it "gets imported through Team.import" do
course_id = 1
options = []
allow(Course).to receive(:find).with(course_id).and_return(course)
expect(Team).to receive(:import_helper).with(row, course_id, options, instance_of(CourseTeam))
CourseTeam.import(row, nil, course_id, options)
end
end
end
</pre>

Newly added rspec test case to test the #import functionality in course_participant_spec.rb

<pre>
describe "CourseParticipant" do
let(:course) { build(:course, id: 1, name: 'ECE517') }
let(:assignment) { build(:assignment, id: 1, name: 'no assignment', participants: [participant], teams: [team]) }
let(:assignment_participant) {build(:participant, id: 1)}
'''describe "#copy" do
#before(:each) do
# byebug
# assignment = build(:assignment)
# course_participant = build(:course_participant)
# @assignment_participant = build(:participant)
#end
it "create a copy of participant" do
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(@assignment.id)).to be_an_instance_of(AssignmentParticipant)
end
it "returns nil if copy exist" do
allow(AssignmentParticipant).to receive(:where).and_return(AssignmentParticipant)
allow(AssignmentParticipant).to receive(:first).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(assignment.id)).to be_nil
end
end'''
</pre>

Test cases for Assignment_team.rb

<pre>
describe ".import" do
context "when an assignment team does not already exist with the same id" do
it "cannot be imported" do
assignment_id = 1
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(nil)
error_message = "The assignment with the id \"" + assignment_id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?"
expect { AssignmentTeam.import([], assignment_id, []) }.
to raise_error(ImportError, error_message)
end
end

context "when an assignment team with the same id already exists" do
it "gets imported through Team.import" do
row = []
assignment_id = 1
options = []
allow(Assignment).to receive(:find_by).with(id: assignment_id).and_return(assignment)
allow(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
expect(Team).to receive(:import).with(row, assignment_id, options, instance_of(AssignmentTeam))
AssignmentTeam.import(row, assignment_id, options)
end
end
end
</pre>

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T23:09:30Z

Svsakham: /* Test Plan */

=== Team ===
*Akash Sarda, aksarda
*Sairam Sakhamuri, svsakham
*Sidhant Arora, sarora22
*Sai Harsha Nadendla, snadend2

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

The export functionality is however is already quite refactored to suit the strategy pattern, which is not changed by the previous group as well.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Design Plan ==

The design plan is to use Strategy patter to implement import functionality, so that all the import requests present in different models are routed through the ImportFileController. Right now since the import functionality is implemented in various model methods this leads to many if and else statements to check the type of model. So, we intent to generalize the import functionality by placing common code in the ImportFileController. But each model will still have its own import method. With this approach the redundancy is reduced by moving common code to ImportFileController and code will become DRY.

Other helper methods such as ImportFileHelper and ImportTopicHelper that are used to perform import functionality will also be removed, which keeps import functionality consistent. We will be using method overloading and overriding for the methods in ImportFileController to eliminate unnecessary if and else blocks.

To summarize our plan of changes:

* Redirect all import calls through ImportFileController
* Refactor ImportFileController by removing the redundant code and making code generic.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.

[[Image: DesignImport.PNG]]
== What has been Implemented==
=== Models ===

*ImportFileController is changed and has been made more generalized, many case statements are removed. The ImprortFileController is made so small and understandable. The import functionality is moved into the corresponding models. Previously code was present in different modules which was confusing now the functionality was all in one place. The following code changes are follows:

*Previous Implmentation
<pre>
def import_from_hash(session, params)
if params[:model] == "AssignmentTeam" or params[:model] == "CourseTeam"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
if params[:model] == "AssignmentTeam"
teamtype = AssignmentTeam
else
teamtype = CourseTeam
end
options = eval(params[:options])
options[:has_teamname] = params[:has_teamname]
Team.import(row_hash, params[:id], options, teamtype)
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "ReviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
ReviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "MetareviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
MetareviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'SignUpTopic' || params[:model] == 'SignUpSheet'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
if params[:optional_count] == '0'
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '1'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '2'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '3'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5], params[:select6]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
end
end
errors = []
begin
@header_integrated_body.each do |row_hash|
session[:assignment_id] = params[:id]
Object.const_get(params[:model]).import(row_hash, session, params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'AssignmentParticipant' || params[:model] == 'CourseParticipant'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header], contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
if params[:model] == 'AssignmentParticipant'
@header_integrated_body.each do |row_hash|
AssignmentParticipant.import(row_hash, session, params[:id])
end
elsif params[:model] == 'CourseParticipant'
@header_integrated_body.each do |row_hash|
CourseParticipant.import(row_hash, session, params[:id])
end
end
rescue
errors << $ERROR_INFO
end
else # params[:model] = "User"
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
@header_integrated_body.each do |row_hash|
User.import(row_hash, nil, session)
end
rescue StandardError
errors << $ERROR_INFO
end
end
</pre>
*Current Implementation
<pre>
def import
errors = import_from_hash(session, params)
err_msg = "The following errors were encountered during import.Other records may have been added. A second submission will not duplicate these records."
errors.each do |error|
err_msg = err_msg + error.to_s
end
err_msg += </ul>
if errors.empty?
ExpertizaLogger.info LoggerMessage.new(controller_name, session[:user].name, "The file has been successfully imported.", request)
undo_link("The file has been successfully imported.")
else
ExpertizaLogger.error LoggerMessage.new(controller_name, session[:user].name, err_msg, request)
flash[:error] = err_msg
end
redirect_to session[:return_to]
end
</pre>

*Assignment Participant is changed to use the newly implemented #import functionality. Changes are as shown below:

<pre>
def self.import(row_hash, session, id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?
raise ImportError, "The assignment with id #{id} was not found." if Assignment.find(id).nil?
unless AssignmentParticipant.exists?(user_id: user.id, parent_id: id)
new_part = AssignmentParticipant.new(user_id: user.id, parent_id: id)
new_part.set_handle
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

*Assignment Team is also changed to use the newly implemented #import functionality.

<pre>
def self.import(row_hash, session = nil, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The assignment with the id \"" + id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?" if Assignment.find_by(id: id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Course_team is is also changed to use the newly implemented #import functionality.
<pre>
def self.import(row_hash, session, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The course with the id \"" + id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?" if Course.find(id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Review response map's #import functionality is also updated to use the newly implemented #import functionality.
<pre>

def self.import(row_hash, _session, assignment_id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
reviewee_user_name = row_hash[:reviewee].to_s
reviewee_user = User.find_by(name: reviewee_user_name)
raise ArgumentError, "Cannot find reviewee user." unless reviewee_user
@@ -57,7 +58,7 @@ def self.import(row_hash, _session, assignment_id)
team_node = TeamNode.create(parent_id: assignment_id, node_object_id: reviewee_team.id)
TeamUserNode.create(parent_id: team_node.id, node_object_id: t_user.id)
end
row_hash[:reviewers].split.each do |reviewer|
reviewer_user_name = reviewer.to_s
reviewer_user = User.find_by(name: reviewer_user_name)
raise ArgumentError, "Cannot find reviewer user." unless reviewer_user
@@ -71,6 +72,20 @@ def self.import(row_hash, _session, assignment_id)
end
end

</pre>

With these new changes to #import functionality. The following modules review_response_map.rb, course_team.rb,assignment_team.rb,assignment_participant.rb have the "self.import" which will use the import functionality in the ImportFileController and they are made as concise as possible. Each module has specific checks while importing the file, so we can't have same code for all the module.

== Test Plan ==
The import method has been implemented in a bunch of models which have been listed above. After preliminary analysis, we assume that the import functionality in a few of the models might have to be amended. These models are:
*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''review_response_map'''
*'''sign_up_topic'''
*'''user'''
Scenarios:
1. when assignment found and assignment participant does not exist, creates a new user and participant
2. when assignment cannot be found, creates a new user then raises an ImportError
3. when the assignment team does not have the required fields, raises ArgumentError
4. check what import/export actions are allowed for admin, instructor, TA, student
5. when course found and course participant does not exist, creates a new user and participant
6. when the course team does not have the required fields, raises ArgumentError

Newly added rspec test case to test the #import functionality in review_response_map_spec.rb

<pre>
it '#import' do
expect {ReviewResponseMap.import({reviewee: "name"}, nil, 1)}.to raise_error(ArgumentError, "Record does not contain required items.")
row_hash = {reviewee: "name", reviewers: "name1"}
#row_hash = {reviewee: "name", reviewers: ["name1"]}
session = nil
assignment_id = 1
# when reviewee user = nil
allow(User).to receive(:find_by).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Cannot find reviewee user.")
# when reviewee user exists but reviewee user is not a participant in this assignment
allow(User).to receive(:find_by).with(name: "name").and_return(student)
allow(AssignmentParticipant).to receive(:find_by).with(user_id: 1, parent_id: 1).and_return(nil)
expect { ReviewResponseMap.import(row_hash, session, 1) }.to raise_error(ArgumentError, "Reviewee user is not a participant in this assignment.")
end
</pre>

Newly added rspec test case to test the #import functionality in course_team_spec.rb

<pre>
describe ".import" do
let(:row) do
{teammembers: 'none'}
end
context "when a course team does not exist with id" do
it "raises ImportError" do
course_id = 1
allow(Course).to receive(:find).with(course_id).and_return(nil)
error_message = "The course with the id \"" + course_id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?"
expect { CourseTeam.import(row, nil, course_id, nil) }.
to raise_error(ImportError, error_message)
end
end

context "when the course team does not have the required fields" do
it "raises ArgumentError" do
expect { CourseTeam.import([], nil, 1, nil) }.
to raise_error(ArgumentError)
end
end

context "when a course team with the same id already exists" do
it "gets imported through Team.import" do
course_id = 1
options = []
allow(Course).to receive(:find).with(course_id).and_return(course)
expect(Team).to receive(:import_helper).with(row, course_id, options, instance_of(CourseTeam))
CourseTeam.import(row, nil, course_id, options)
end
end
end
</pre>

Newly added rspec test case to test the #import functionality in course_participant_spec.rb

<pre>
describe "CourseParticipant" do
let(:course) { build(:course, id: 1, name: 'ECE517') }
let(:assignment) { build(:assignment, id: 1, name: 'no assignment', participants: [participant], teams: [team]) }
let(:assignment_participant) {build(:participant, id: 1)}
'''describe "#copy" do
#before(:each) do
# byebug
# assignment = build(:assignment)
# course_participant = build(:course_participant)
# @assignment_participant = build(:participant)
#end
it "create a copy of participant" do
allow(AssignmentParticipant).to receive(:create).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(@assignment.id)).to be_an_instance_of(AssignmentParticipant)
end
it "returns nil if copy exist" do
allow(AssignmentParticipant).to receive(:where).and_return(AssignmentParticipant)
allow(AssignmentParticipant).to receive(:first).and_return(participant)
allow(assignment_participant).to receive(:set_handle).and_return(true)
expect(course_participant.copy(assignment.id)).to be_nil
end
end'''
</pre>

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T23:03:49Z

Svsakham: /* Test Plan */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T22:54:23Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T22:00:08Z

Svsakham: /* What has been Implemented */

=== Team ===
*Akash Sarda, aksarda
*Sairam Sakhamuri, svsakham
*Sidhant Arora, sarora22
*Sai Harsha Nadendla, snadend2

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

The export functionality is however is already quite refactored to suit the strategy pattern, which is not changed by the previous group as well.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Design Plan ==

The design plan is to use Strategy patter to implement import functionality, so that all the import requests present in different models are routed through the ImportFileController. Right now since the import functionality is implemented in various model methods this leads to many if and else statements to check the type of model. So, we intent to generalize the import functionality by placing common code in the ImportFileController. But each model will still have its own import method. With this approach the redundancy is reduced by moving common code to ImportFileController and code will become DRY.

Other helper methods such as ImportFileHelper and ImportTopicHelper that are used to perform import functionality will also be removed, which keeps import functionality consistent. We will be using method overloading and overriding for the methods in ImportFileController to eliminate unnecessary if and else blocks.

To summarize our plan of changes:

* Redirect all import calls through ImportFileController
* Refactor ImportFileController by removing the redundant code and making code generic.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.

[[Image: DesignImport.PNG]]
== What has been Implemented==
=== Models ===

*ImportFileController is changed and has been made more generalized, many case statements are removed. The ImprortFileController is made so small and understandable. The following code changes are follows:

*Previous Implmentation
<pre>
def import_from_hash(session, params)
if params[:model] == "AssignmentTeam" or params[:model] == "CourseTeam"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
if params[:model] == "AssignmentTeam"
teamtype = AssignmentTeam
else
teamtype = CourseTeam
end
options = eval(params[:options])
options[:has_teamname] = params[:has_teamname]
Team.import(row_hash, params[:id], options, teamtype)
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "ReviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
ReviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == "MetareviewResponseMap"
contents_hash = eval(params[:contents_hash])
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
errors = []
begin
@header_integrated_body.each do |row_hash|
MetareviewResponseMap.import(row_hash,session,params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'SignUpTopic' || params[:model] == 'SignUpSheet'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
if params[:optional_count] == '0'
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '1'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '2'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
elsif params[:optional_count] == '3'
new_header = [params[:select1], params[:select2], params[:select3], params[:select4], params[:select5], params[:select6]]
@header_integrated_body = hash_rows_with_headers(new_header,contents_hash[:body])
end
end
errors = []
begin
@header_integrated_body.each do |row_hash|
session[:assignment_id] = params[:id]
Object.const_get(params[:model]).import(row_hash, session, params[:id])
end
rescue
errors << $ERROR_INFO
end
elsif params[:model] == 'AssignmentParticipant' || params[:model] == 'CourseParticipant'
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header], contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3], params[:select4]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
if params[:model] == 'AssignmentParticipant'
@header_integrated_body.each do |row_hash|
AssignmentParticipant.import(row_hash, session, params[:id])
end
elsif params[:model] == 'CourseParticipant'
@header_integrated_body.each do |row_hash|
CourseParticipant.import(row_hash, session, params[:id])
end
end
rescue
errors << $ERROR_INFO
end
else # params[:model] = "User"
contents_hash = eval(params[:contents_hash])
if params[:has_header] == 'true'
@header_integrated_body = hash_rows_with_headers(contents_hash[:header],contents_hash[:body])
else
new_header = [params[:select1], params[:select2], params[:select3]]
@header_integrated_body = hash_rows_with_headers(new_header, contents_hash[:body])
end
errors = []
begin
@header_integrated_body.each do |row_hash|
User.import(row_hash, nil, session)
end
rescue StandardError
errors << $ERROR_INFO
end
end
</pre>
*Current Implementation
<pre>
def import
errors = import_from_hash(session, params)
err_msg = "The following errors were encountered during import.Other records may have been added. A second submission will not duplicate these records."
errors.each do |error|
err_msg = err_msg + error.to_s
end
err_msg += </ul>
if errors.empty?
ExpertizaLogger.info LoggerMessage.new(controller_name, session[:user].name, "The file has been successfully imported.", request)
undo_link("The file has been successfully imported.")
else
ExpertizaLogger.error LoggerMessage.new(controller_name, session[:user].name, err_msg, request)
flash[:error] = err_msg
end
redirect_to session[:return_to]
end
</pre>

*Assignment Participant is changed to use the newly implemented #import functionality. Changes are as shown below:

<pre>
def self.import(row_hash, session, id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
user = User.find_by(name: row_hash[:name])
user = User.import(row_hash, session, nil) if user.nil?
raise ImportError, "The assignment with id #{id} was not found." if Assignment.find(id).nil?
unless AssignmentParticipant.exists?(user_id: user.id, parent_id: id)
new_part = AssignmentParticipant.new(user_id: user.id, parent_id: id)
new_part.set_handle
end
end

def self.required_import_fields
{"name" => "Name",
"fullname" => "Full Name",
"email" => "Email"}
end

def self.optional_import_fields(id=nil)
{}
end

def self.import_options
{}
end
</pre>

*Assignment Team is also changed to use the newly implemented #import functionality.

<pre>
def self.import(row_hash, session = nil, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The assignment with the id \"" + id.to_s + "\" was not found. <a href='/assignment/new'>Create</a> this assignment?" if Assignment.find_by(id: id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Course_team is is also changed to use the newly implemented #import functionality.
<pre>
def self.import(row_hash, session, id, options)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
raise ImportError, "The course with the id \"" + id.to_s + "\" was not found. <a href='/course/new'>Create</a> this course?" if Course.find(id).nil?
Team.import_helper(row_hash, id, options, prototype)
end

def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

*Review response map's #import functionality is also updated to use the newly implemented #import functionality.
<pre>

def self.import(row_hash, _session, assignment_id)
raise ArgumentError, "Record does not contain required items." if row_hash.length < self.required_import_fields.length
reviewee_user_name = row_hash[:reviewee].to_s
reviewee_user = User.find_by(name: reviewee_user_name)
raise ArgumentError, "Cannot find reviewee user." unless reviewee_user
@@ -57,7 +58,7 @@ def self.import(row_hash, _session, assignment_id)
team_node = TeamNode.create(parent_id: assignment_id, node_object_id: reviewee_team.id)
TeamUserNode.create(parent_id: team_node.id, node_object_id: t_user.id)
end
row_hash[:reviewers].split.each do |reviewer|
reviewer_user_name = reviewer.to_s
reviewer_user = User.find_by(name: reviewer_user_name)
raise ArgumentError, "Cannot find reviewer user." unless reviewer_user
@@ -71,6 +72,20 @@ def self.import(row_hash, _session, assignment_id)
end
end

</pre>

*Questionnaire.rb has a #import method that is now routed through the ImportFileController and the actual act of importing works. We removed all functionality related to importing from the QuestionnaireController and QuestionnaireHelper.

*We removed the ImportFileHelper and ImportTopicsHelper and moved that functionality into the corresponding models. Having the code segmented made things confusing since none of the functionality was all in one place.

*We removed import functionality from '''question.rb''' because there is no way to import a question out of context from a questionnaire. Importing a questionnaire, means importing questions to fill that questionnaire.

*SignUpSheet no longer has an import method. That functionality was never used and does not actually currently work in the production version of Expertiza. After discussing with our mentor, we were instructed to removed the import link on the front-end, the import method, and all related tests.

*The previous way of determining required import fields were to populate the @expected_fields variable which was incredibly hard to find in the code. We have eliminated the need for that variable and have removed all instances of it.

*We have made things as generic as possible in the .html.erb files so that you can be in any model and the code works on the front end seamlessly. This is done by adding these three methods to each model that has an import method:

<pre>
def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

The above content is specific to the course_team.rb but the three method names are consistent to all the models and are called on the front end.

*The models that have a "self.import" method that does not branch out into other controllers beside ImportFileController will be looked at to make sure they are as concise as possible. None of them can be all the same because they all need to have checks specific to what they need to import. We can, however, make sure that similar models, like assignment_team and course_team, have similar imports. That is what we have done for assignment_team/course_team and assignment_participant/course_participant.

Now, these are the final models/places where a user may import a file into Expertiza:

- assignment_participant

- assignment_team

- course_participant

- course_team

- review_response_map

- metareview_response_map

- sign_up_topic

- user

- questionnaire

*We have updated some text on the front end related to questionnaire. The import link did not match the capitalization format in the rest Expertiza.

=== Code Climate ===
'''Note: Code Climate was not running on Expertiza's beta branch for the last four days of the project. We have done the best we could to removed extraneous lines and white spaces.'''

There were many code climate issues in reference to our project. We have managed to fix 46 issues as a byproduct of refactoring the code. Here is a list of them with their frequency put in parenthesis':

* Method get_questions_from_csv has a Cognitive Complexity of 62 (exceeds 5 allowed). Consider refactoring.

* Method get_questions_from_csv has 41 lines of code (exceeds 25 allowed). Consider refactoring.

* File import_file_controller.rb has 278 lines of code (exceeds 250 allowed). Consider refactoring.

* Avoid deeply nested control flow statements. (3)

* Similar blocks of code found in 2 locations. Consider refactoring. (2)

* Unescaped parameter value

* Useless assignment to variable - a.

* Cyclomatic complexity for get_questions_from_csv is too high. [18/6]

* Assignment Branch Condition size for get_questions_from_csv is too high. [43.3/15]

* Block has too many lines. [36/25]

* Avoid more than 3 levels of block nesting. (3)

* Perceived complexity for get_questions_from_csv is too high. [16/7]

* Align elsif with if.

* Space missing after comma. (13)

* Line is too long. [172/160]

* Method has too many lines. [108/60]

* Use the return of the conditional for variable assignment and comparison.

* Move @optional_count = 0 out of the conditional. (2)

* Move contents_hash = eval(params[:contents_hash]) out of the conditional. (6)

* Convert if nested inside else to elsif.

* Don't use parentheses around the condition of an if. (3)

== Test Plan ==
The import method has been implemented in a bunch of models which have been listed above. After preliminary analysis, we assume that the import functionality in a few of the models might have to be amended. These models are:
*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''review_response_map'''
*'''sign_up_topic'''
*'''user'''
We will update the test plan for those models in which the import code is amended to fit into the new framework. If we amend the import code in any other model, we will also update the tests in their respective .spec files to ensure 100% coverage. We also plan to create a spec file for the '''import_file_controller'''.

Scenarios:
1. when assignment found and assignment participant does not exist, creates a new user and participant
2. when assignment cannot be found, creates a new user then raises an ImportError
3. when the assignment team does not have the required fields, raises ArgumentError
4. check what import/export actions are allowed for admin, instructor, TA, student
5. when course found and course participant does not exist, creates a new user and participant
6. when the course team does not have the required fields, raises ArgumentError

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:52:32Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:49:22Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:36:51Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:34:47Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:33:51Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:32:19Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:31:59Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:31:38Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:31:27Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:30:43Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:30:27Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:29:54Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:29:15Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:28:51Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:27:46Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:26:34Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:25:50Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:25:17Z

Svsakham: /* What has been Implemented */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:24:12Z

Svsakham: /* Implementation Details */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:23:33Z

Svsakham: /* What has been done */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-29T21:07:19Z

Svsakham:

CSC/ECE 517 Fall 2021 - E2162. Further refactoring and improvement of review mapping helper

2021-11-03T16:03:55Z

Svsakham:

==Introduction==
This page gives a description of the changes made for the ''review_mapping_helper.rb'' of Expertiza based OSS project.

===Expertiza Background ===
Expertiza is a web application where students can submit and peer-review learning objects (articles, codes, websites, etc). Instructors add and grade the assignments submitted by students to Expertiza. Students can be assigned in teams based on their selection of the topics. It has functionalities such as peer reviews in which students can provide feedback on other's work which helps peer in better developing the project. It is supported by the National Science Foundation.

=== Problem Statement ===

The review_mapping_helper.rb has multiple functions with a wrong naming convention, they are classic examples of bad function names. These functions are to be refactored accordingly. Few of the variables names used in the method must also be refactored to make it more relevant and understandable. In addition, some function's code is to be optimized to ensure that it follows DRY principle. Tests must be written for the functions which are modified. Also, comments must be added to all functions of the file. The create_report_table_header function contains HTML code which should be ideally placed in the view as partials, which will allow to easily reuse the code in Rails application.

==Implementation==

===Flowchart===

The following process is carried out to complete the project-

[[File:Steps for project.png]]

===Refactoring===

====Refactoring method names====
All the method names that are refactored are mentioned below-

'''Before:'''

get_each_round_review_and_feedback_response_map_for_feedback_report

'''After:'''

get_each_review_and_feedback_response_map ''[Since the method and file is already about feedback report, we do not need to put feedback_report in the method name]''

'''Before:'''

get_certain_round_review_and_feedback_response_map_for_feedback_report

'''After:'''

get_certain_review_and_feedback_response_map ''[Since the method and file is already about feedback report, we do not need to put feedback_report in the method name]''

'''Before:'''

get_team_name_color_in_review_report

'''After:'''

get_team_colour ''[The method suggests that team color is defined in the review report, so we do not need to put in_review_report in the method name]''

'''Before:'''

get_each_round_score_awarded_for_review_report

'''After:'''

get_awarded_review_score ''[The method gets the awarded score for review, so we do not need to put for_review_report in the method name]''

'''Before:'''

get_min_max_avg_value_for_review_report

'''After:'''

get_review_metrics ''[Since minimum, maximum and average value is the part of metrics in review report, we can replace it all by metrics]''

====Refactoring method====

The following method was not used or called in any of the files in the project. Due to that reason, the entire function was commented out.

<pre>
def get_current_round(reviewer_id)
user_id = Participant.find(reviewer_id).user.id
topic_id = SignedUpTeam.topic_id(@assignment.id, user_id)
@assignment.number_of_current_round(topic_id)
@assignment.num_review_rounds if @assignment.get_current_stage(topic_id) == "Finished" || @assignment.get_current_stage(topic_id) == "metareview"
end
</pre>

===Comments added to various functions===
There were several methods which didn't have any comments or the comments weren't meaningful. In all those cases, comments have been added or changed which are mentioned as follows:

<pre>
# gets the review score awarded based on each round of the review
def get_each_round_score_awarded_for_review_report(reviewer_id, team_id)
</pre>

<pre>
# gets minimum, maximum and average value for all the reviews
def get_min_max_avg_value_for_review_report(round, team_id)
</pre>

<pre>
# sorts the reviewers by the average volume of reviews in each round, in descending order
def sort_reviewer_by_review_volume_desc
</pre>

<pre>
# displays all the average scores in round 1, 2 and 3
def display_volume_metric(overall_avg_vol, avg_vol_in_round_1, avg_vol_in_round_2, avg_vol_in_round_3)
</pre>

<pre>
# moves data of reviews in each round from a current round
def initialize_chart_elements(reviewer)
</pre>

<pre>
# The data of all the reviews is displayed in the form of a bar chart
def display_volume_metric_chart(reviewer)
</pre>

<pre>
# For assignments with 1 team member, the following method returns user's fullname else it returns "team name" that a particular reviewee belongs to.
def get_team_reviewed_link_name(max_team_size, response, reviewee_id)
</pre>

<pre>
# if the current stage is "submission" or "review", function returns the current round number otherwise
# if the current stage is "Finished" or "metareview", function returns the number of rounds of review completed
def get_current_round_for_review_report(reviewer_id)
</pre>

<pre>
# gets the response map data such as reviewer id, reviewd object id and type for the review report
def get_data_for_review_report(reviewed_object_id, reviewer_id, type)
</pre>

<pre>
# gets the team name's color according to review and assignment submission status
def get_team_name_color_in_review_report(response_map)
</pre>

<pre>
# checks if a review was submitted in every round and gives the total responses count
def response_for_each_round?(response_map)
</pre>

<pre>
# returns hyperlink of the assignment that has been submitted on the due date
def submitted_hyperlink(round, response_map, assignment_created, assignment_due_dates)
</pre>

===Function placed in the view as partials===

The following method has been removed from the '''review_mapping_helper.rb''' and added to the views as partials:

<pre>
def create_report_table_header(headers = {})
table_header = "<div class = 'reviewreport'>\
<table width='100% cellspacing='0' cellpadding='2' border='0' class='table table-striped'>\
<tr bgcolor='#CCCCCC'>"
headers.each do |header, percentage|
table_header += if percentage
"<th width = #{percentage}>\
#{header.humanize}\
</th>"
else
"<th>\
#{header.humanize}\
</th>"
end
end
table_header += "</tr>"
table_header.html_safe
end
</pre>

The following is the code available in views in the file '''_report_table_header.html.erb''':

<pre>
<% table_header = "<div class = 'reviewreport'><table width='100% cellspacing='0' cellpadding='2' border='0' class='table table-striped'><tr bgcolor='#CCCCCC'>" %>
<% headers.each do |header, percentage| %>
<% table_header += if percentage %>
<% "<th width = #{percentage}> #{header.humanize} </th>" %>
<% else %>
<%"<th> #{header.humanize} </th>" %>
<% end %>
<% end %>
<% table_header += "</tr>"%>
<% return table_header.html_safe %>
</pre>

==Testing==

===Manual Testing===

''To be carried out''

===RSpec Testing===

RSpec testing was carried out for the methods that had been refactored. Given below are the tests which are written for the functions that are modified:

<pre>
describe 'My Test Cases' do
before(:each) do
create(:instructor)
create(:role_of_student)
login_as("instructor6")
visit '/tree_display/list'
click_link 'View reports'
expect(page).to have_current_path('/reports/response_report')
click_link 'View'
end

it "can display review metrics", js: true do
expect(page).to have_content('Metrics')
end

it "can display review grades of each round", js: true do
expect(page).to have_content('Score awarded')
end

it "can display review summary", js: true do
expect(page).to have_content('Reviews done')
end

it "can display review summary", js: true do
expect(page).to have_content('Reviewer')
end

it "can display review summary", js: true do
expect(page).to have_content('Team reviewed')
end
end
</pre>

==Design Pattern==

A design pattern is a general repeatable solution to a commonly occurring problem in software design. It is a description or template for how to solve a problem that can be used in many different situations.

During the process of refactoring methods as well as method names,'''Strategy Pattern''' was used in the implementation. The Strategy pattern is most useful when you want to provide multiple ways of processing a request, without hard-coding knowledge about those different methods into the object that handles the request.

==References==

1) https://www.rubyguides.com/2015/12/ruby-refactoring/

2) http://rspec.info/documentation/3.8/rspec-core/

3) http://wiki.expertiza.ncsu.edu/index.php/Main_Page

==Team Members==

Ashish Kumar Jayantilal Jain

Aishwarya Tirumala

Devang Upadhyay

CSC/ECE 517 Fall 2021 - E2162. Further refactoring and improvement of review mapping helper

2021-11-03T16:02:30Z

Svsakham:

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-03T04:48:16Z

Svsakham:

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-03T04:44:30Z

Svsakham: /* Design Plan */

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Design Plan ==

The design plan is to use Strategy patter to implement import functionality, so that all the import requests present in different models are routed through the ImportFileController. Right now since the import functionality is implemented in various model methods this leads to many if and else statements to check the type of model. So, we intent to generalize the import functionality by placing common code in the ImportFileController. But each model will still have its own import method. With this approach the redundancy is reduced by moving common code to ImportFileController and code will become DRY.

Other helper methods such as ImportFileHelper and ImportTopicHelper that are used to perform import functionality will also be removed, which keeps import functionality consistent. We will be using method overloading and overriding for the methods in ImportFileController to eliminate unnecessary if and else blocks.

To summarize our plan of changes:

* Redirect all import calls through ImportFileController
* Refactor ImportFileController by removing the redundant code and making code generic.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.

[[Image: DesignImport.PNG]]

== Test Plan ==

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-03T04:43:47Z

Svsakham: /* Design Plan */

File:DesignImport.PNG

2021-11-03T04:43:14Z

Svsakham:

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-03T04:17:34Z

Svsakham:

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-03T04:16:14Z

Svsakham: /* Proposed Import Changes */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-03T03:44:29Z

Svsakham:

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

The current import functionality is done through the ImportFileController and there are different import class methods implemented in different models that are performing import function.

In the SignUpTopic and User models use helper classes and the attributes are taken from a hash and new ActiveRecord object is created.

=== Controllers ===

*'''import_file_controller''', methods:
** Methods used to process files:
*** #get_delimiter - Sets delimiter for filetype
*** #parse_line - Processes line of the file
*** #parse_to_grid - parses the file into 2D array
*** #parse_to_hash - parses file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

=== Models ===

We have found that the below mentioned models are using the import functionality defined in ImportFileController. SignUpTopic and User are dependent on the helper methods to use import functionality.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Proposed Import Changes ==

The import functionality should be routed through the ImportFileController. Everything should still work as it does now after our changes; we are cleaning up code and may move functionality to a different location but nothing will get removed permanently. Luckily, most import functionality is routed through the ImportFileController and import class methods on models that are being imported. It makes sense for each model to have its own import method because each model knows what to expect for itself. Making things overly generic will end up contradicting the DRY principle and lead to many nested if statements.

We intend to remove other helpers and files, such as the ImportFileHelper, the ImportTopicHelper, and the QuestionnaireHelper to keep import routing consistent. We will also remove the QuestionnaireController and route that import through the ImportFileController. In addition, we will update ImportFileController methods (in particular, #hash_rows_with_headers and #import_from_hash) to make better use of polymorphism and eliminate large and unnecessary if/else blocks.

We will also insert more specific object creation specifications. Consider if a user's import contains objects that do not exist in the database. We will provide an option for the user to specify whether or not new/dependent objects should be created and account for this boolean in relevant ::import functions for the models.

To summarize our changes:

* Route all import traffic through ImportFileController and ::import calls on models.
* Refactor ImportFileController.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.
* Add ability for ImportFileController to import Question objects for Questionnaires. (Add question view to views/import_file, introduce Question logic into the controller.)

Visually this is what is happening right now (with some files missing for brevity's sake):

[[Image: badUML.png]]

What we want it to look like:

[[Image: goodUML.png]]

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-03T03:43:23Z

Svsakham: /* Existing Import Functionality */

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-03T03:23:15Z

Svsakham:

== Import and Export Functionality ==
In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that have been created or need to be created, lists of topics that users are to be able to sign up for, or instructor or peer scores that have been given for a particular assignment. Historically, all of these import and export methods were written individually, using similar code patterns.
The instructor might end up adding those in excel or having that data in excel - this functionality allows the expertiza to accept that csv file and tabulate it infront of the user. The user can then select to assign columns to that data and then import it into the database.
Eg. list of topics or feedback comments from the students can be tabular if it is say a MCQ questionnaire. The instructor will then have to either enter each entry manually through the UI or can upload this file to automate it.

== Problem Description ==
Initially the different classes (Topics, Assignments, Course Participants) had different implementations of taking in the csv file which was tightly coupled with the feature / model class. However, it was discovered that instead of defining the new method again and again, this process can be automated, by pushing common code of reading the headers and writing to the database into a single generic function. This would basically make it really easy to add this functionality to other model classes and to the new features yet to come to Expertiza.

== Existing Import Functionality ==

Existing import functionality is primarily routed through the ImportFileController and an import class method for various models.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

Questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

=== Models ===

All these models have an ::import method called by the ImportFileController. In addition, SignUpTopic and User objects rely on their helpers, which are mentioned above.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

CSC/ECE 517 Fall 2021 - E2160. Implementing and testing import export controllers

2021-11-03T02:52:17Z

Svsakham: Created page with "== Import and Export Functionality == In Expertiza, many kinds of files can be imported or exported: lists of users for whom accounts are to be created, lists of teams that ha..."