Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-26T03:55:23Z

Spriyad2:

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

*Bollineni, Rachana Sri rbollin@ncsu.edu
*Nangia, Chirrag - cnangia@ncsu.edu
*Priyadarshan, Saswat - spriyad2@ncsu.edu
*Vemuri, Pradyumna pvemuri@ncsu.edu

== Project Purpose & Description ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

== Current Project Scope ==

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438
But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

'''Below listed are our primary goals for this project:'''
* Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
* Write test cases for the new import_file_controller and the existing export_file_controller.
* Find and fix the existing bugs in the import and export process flows.

== Extended Project Scope ==

'''Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.'''

* After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

== Existing Import/Export Controller design ==

The existing import and export functionality primarily uses the '''import_file_controller''' and '''export_file_controller'''.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

*'''export_file_controller''', the list of methods in the controller are the following:
** ::find_delim_filename - Sets and returns attributes for User object from hash.
** ::exportdetails - Generates the metadata for the data that needs to be exported.
** ::export - Converts the data into CSV format files.
** ::export_advices - Export question advice data to CSV file.
** ::export_tags - Export the tags.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Find the filename and delimiter.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

== Proposed Design Change ==
* Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
* Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
* Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
* Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

'''In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):'''

[[Image: badUML.png]]

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

'''What we want it to look like: UML Diagram'''

[[Image: Ic usecase.png|600px]]

[[Image: EC usecase.png|600px]]

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference.
The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

== Actual Changes Made ==

* ''' Analyzing Export Functionality code flow ''' - We started by analyzing the code flow of export_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, Grades and Topics'''. On analyzing the code we found that all these three models are using the export_file_controller.rb to generate the respective CSVs. However, there were a few bugs that we found in the existing ''' Export Grades ''' functionality. We debugged this flow completely and fixed those bugs. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section. Apart from that, there was no ''' Export Topics ''' functionality present in the system. We created a completely new flow for exporting the topics leveraging the existing export_file_controller logic to keep the modules consistent. More information about the new features added can be found in the ''' New Features Added ''' section.

* ''' Analyzing Import Functionality code flow ''' - We analyzed the code flow of import_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, and Topics'''. On analyzing the code we found that these two models are using the export_file_controller.rb to import the respective CSVs with required data. However, we found a few bugs in the Import Topics functionality and we fixed those. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section.

Analysis of these two workflows (Export and Import) took a considerable amount of our time since there were a lot of moving pieces present in these controllers. We had to manually test these flows for different models because of the different data and customizations present.

* '''Test Cases for export_file_controller''' - After we finished the analysis and development of the required features for the export functionality, we proceeded with writing the test cases for the export_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users, Grades and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

* '''Test Cases for import_file_controller''' - After we finished the analysis and development of the required features for the import functionality, we proceeded with writing the test cases for the import_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

== New Features Added ==

*'''Export Topics''' - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to '''Assignments > Click on Edit (pencil icon) > Click on Topics menu'''. Once you click the Export Topics button, you will be taken to a customizations page where you can select what columns you want in your CSV file. You can select/deselect the options by clicking on the checkboxes provided. Once you select the desired cloumns, click on the export button to generate the CSV file. The Export Topics and Customizations pages are attached below.

[[File:Export Topics.png|800x520px]]

[[File:Customization.png|800x520px]]

*'''''Code Changes for Export Topics'''''

[[File:C1.png|800x520px]]

[[File:C2.png|800x520px]]

[[File:C3.png|800x520px]]

[[File:C4.png|800x520px]]

[[File:C5.png|800x520px]]

[[File:C6.png|800x520px]]

== Bug Fixes ==

On analyzing the code flow for import and export functionality we found that the '''Export Grades''' and '''Import Topics''' functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

*'''Fix 1: Export grades into a file successfully''' - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. After fixing that, there were some lines related to the business logic were missing here. We added those lines and it can be seen in the code diff.

[[File:Bug1.png|800x520px]]

*'''''Code Changes for Fix 1: Export grades into a file successfully'''''

[[File:Fix1 code.png|800x520px]]

*'''Fix 2: Incorrect headers in the exported file''' - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.
[[File:Incorrect grouped column.jpeg|800x520px]]

*'''''Code Changes for Fix 2: Incorrect headers in the exported file'''''

[[File:Fix2 code.png|800x520px]]

*'''Fix 3: Only export selected columns from the UI''' - Removed author feedback score from being exported by unchecking that option in the UI as seen in image below.

[[File:E3.png|800x520px]]

But what happens in the exported file?
** Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
** Problem 2: Also only the headers have changed but all values of all columns are still being exported.

[[File:E3 1.png|800x520px]]

*'''''Code Changes for Fix 3: Only export selected columns from the UI'''''

[[File:Fix3 code.png|800x520px]]

[[File:B3.jpeg|800x520px]]

*'''Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button''' - We tried importing the topics.csv and it imported all the data except the Category, Description, and Link columns. These columns were optional columns and when we selected the checkboxes for these, the data was not persisted in the database. We debugged this workflow and fixed this bug.

*'''''Code Changes for Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button'''''

[[File:Fix 4.png|800x520px]]

== Test Plan ==

Writing test cases for '''import_file_controller''' and '''export_file_controller'''. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:
*'''action_allowed''' - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''exportdetails''' - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
*'''export''' - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.

For the import_file_controller class, we identified the following methods to be tested:
*'''action_allowed''' - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''show''' - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
*'''import''' - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.

== Execution of Test Plan ==

To run the test cases for '''export_file_controller''' use the command - '''''rspec spec/controllers/export_file_controller_spec.rb'''''

To run the test cases for '''import_file_controller''' use the command - '''''rspec spec/controllers/import_file_controller_spec.rb'' ''''

We wrote the test cases for the above-mentioned methods focussing on the ''' models (Users, Topics and Grades) ''' since these were the high priority models assigned to us. Our Test Coverage so far is as follows:

*'''''export_file_controller test coverage - 54.84%'''''

[[File:Export test coverage.png|800x520px]]

*'''''import_file_controller test coverage - 52.43%'''''

[[File:Import test coverage.png|800x520px]]

The reason for low test coverage is the fact that our test cases revolve around the high-priority models mentioned above. We covered all the scenarios associated with those models only. Future work will involve writing test cases for other models as well.

We also implemented '''''fixtures''''' for testing the generated CSVs. This is a new concept that was not implemented earlier in this project and can be leveraged in the future to test the CSVs. This provides the developers ability to test the Export and Import functionalities for CSV files.

We did not attach the code difference screenshots for the test cases because it was quite big. Kindly refer to the PR to view the code changes for the test cases corresponding to '''export_file_controller''' and '''import_file_controller'''. Our Test cases are quite exhaustive, kindly refer to the PR to check the different contexts for each test case in both the controllers. For the sake of simplicity and to keep redundant stuff out we did not write each context on the Wiki page.

== '''''Link to Pull Request:''''' https://github.com/expertiza/expertiza/pull/2396 ==

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-26T03:52:01Z

Spriyad2:

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

*Bollineni, Rachana Sri rbollin@ncsu.edu
*Nangia, Chirrag - cnangia@ncsu.edu
*Priyadarshan, Saswat - spriyad2@ncsu.edu
*Vemuri, Pradyumna pvemuri@ncsu.edu

== Project Purpose & Description ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

== Current Project Scope ==

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438
But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

'''Below listed are our primary goals for this project:'''
* Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
* Write test cases for the new import_file_controller and the existing export_file_controller.
* Find and fix the existing bugs in the import and export process flows.

== Extended Project Scope ==

'''Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.'''

* After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

== Existing Import/Export Controller design ==

The existing import and export functionality primarily uses the '''import_file_controller''' and '''export_file_controller'''.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

*'''export_file_controller''', the list of methods in the controller are the following:
** ::find_delim_filename - Sets and returns attributes for User object from hash.
** ::exportdetails - Generates the metadata for the data that needs to be exported.
** ::export - Converts the data into CSV format files.
** ::export_advices - Export question advice data to CSV file.
** ::export_tags - Export the tags.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Find the filename and delimiter.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

== Proposed Design Change ==
* Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
* Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
* Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
* Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

'''In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):'''

[[Image: badUML.png]]

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

'''What we want it to look like: UML Diagram'''

[[Image: Ic usecase.png|600px]]

[[Image: EC usecase.png|600px]]

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference.
The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

== Actual Changes Made ==

* ''' Analyzing Export Functionality code flow ''' - We started by analyzing the code flow of export_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, Grades and Topics'''. On analyzing the code we found that all these three models are using the export_file_controller.rb to generate the respective CSVs. However, there were a few bugs that we found in the existing ''' Export Grades ''' functionality. We debugged this flow completely and fixed those bugs. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section. Apart from that, there was no ''' Export Topics ''' functionality present in the system. We created a completely new flow for exporting the topics leveraging the existing export_file_controller logic to keep the modules consistent. More information about the new features added can be found in the ''' New Features Added ''' section.

* ''' Analyzing Import Functionality code flow ''' - We analyzed the code flow of import_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, and Topics'''. On analyzing the code we found that these two models are using the export_file_controller.rb to import the respective CSVs with required data. However, we found a few bugs in the Import Topics functionality and we fixed those. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section.

Analysis of these two workflows (Export and Import) took a considerable amount of our time since there were a lot of moving pieces present in these controllers. We had to manually test these flows for different models because of the different data and customizations present.

* '''Test Cases for export_file_controller''' - After we finished the analysis and development of the required features for the export functionality, we proceeded with writing the test cases for the export_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users, Grades and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

* '''Test Cases for import_file_controller''' - After we finished the analysis and development of the required features for the import functionality, we proceeded with writing the test cases for the import_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

== New Features Added ==

*'''Export Topics''' - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to '''Assignments > Click on Edit (pencil icon) > Click on Topics menu'''. Once you click the Export Topics button, you will be taken to a customizations page where you can select what columns you want in your CSV file. You can select/deselect the options by clicking on the checkboxes provided. Once you select the desired cloumns, click on the export button to generate the CSV file. The Export Topics and Customizations pages are attached below.

[[File:Export Topics.png|800x520px]]

[[File:Customization.png|800x520px]]

*'''''Code Changes for Export Topics'''''

[[File:C1.png|800x520px]]

[[File:C2.png|800x520px]]

[[File:C3.png|800x520px]]

[[File:C4.png|800x520px]]

[[File:C5.png|800x520px]]

[[File:C6.png|800x520px]]

== Bug Fixes ==

On analyzing the code flow for import and export functionality we found that the '''Export Grades''' and '''Import Topics''' functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

*'''Fix 1: Export grades into a file successfully''' - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. After fixing that, there were some lines related to the business logic were missing here. We added those lines and it can be seen in the code diff.

[[File:Bug1.png|800x520px]]

*'''''Code Changes for Fix 1: Export grades into a file successfully'''''

[[File:Fix1 code.png|800x520px]]

*'''Fix 2: Incorrect headers in the exported file''' - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.
[[File:Incorrect grouped column.jpeg|800x520px]]

*'''''Code Changes for Fix 2: Incorrect headers in the exported file'''''

[[File:Fix2 code.png|800x520px]]

*'''Fix 3: Only export selected columns from the UI''' - Removed author feedback score from being exported by unchecking that option in the UI as seen in image below.

[[File:E3.png|800x520px]]

But what happens in the exported file?
** Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
** Problem 2: Also only the headers have changed but all values of all columns are still being exported.

[[File:E3 1.png|800x520px]]

*'''''Code Changes for Fix 3: Only export selected columns from the UI'''''

[[File:Fix3 code.png|800x520px]]

[[File:B3.jpeg|800x520px]]

*'''Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button''' - We tried importing the topics.csv and it imported all the data except the Category, Description, and Link columns. These columns were optional columns and when we selected the checkboxes for these, the data was not persisted in the database. We debugged this workflow and fixed this bug.

*'''''Code Changes for Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button'''''

[[File:Fix 4.png|800x520px]]

== Test Plan ==

Writing test cases for '''import_file_controller''' and '''export_file_controller'''. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:
*'''action_allowed''' - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''exportdetails''' - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
*'''export''' - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.

For the import_file_controller class, we identified the following methods to be tested:
*'''action_allowed''' - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''show''' - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
*'''import''' - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.

== Execution of Test Plan ==

To run the test cases for '''export_file_controller''' use the command - '''''rspec spec/controllers/export_file_controller_spec.rb'''''

To run the test cases for '''import_file_controller''' use the command - '''''rspec spec/controllers/import_file_controller_spec.rb'' ''''

We wrote the test cases for the above-mentioned methods focussing on the ''' models (Users, Topics and Grades) ''' since these were the high priority models assigned to us. Our Test Coverage so far is as follows:

*'''''export_file_controller test coverage - 54.84%'''''

[[File:Export test coverage.png|800x520px]]

*'''''import_file_controller test coverage - 52.43%'''''

[[File:Import test coverage.png|800x520px]]

We also implemented '''''fixtures''''' for testing the generated CSVs. This is a new concept that was not implemented earlier in this project and can be leveraged in future to test the CSVs. This provides the developers ability to test the Export and Import functionalities for CSV files.

We did not attach the code difference screenshots for the test cases because it was quite big. Kindly refer to the PR to view the code changes for the test cases corresponding to '''export_file_controller''' and '''import_file_controller'''. Our Test cases are quite exhaustive, kindly refer to the PR to check the different contexts for each test case in both the controllers. For the sake of simplicity and keep redundant stuff out we did not write each context in the Wiki page.

== '''''Link to Pull Request:''''' https://github.com/expertiza/expertiza/pull/2396 ==

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-26T03:48:04Z

Spriyad2:

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

*Bollineni, Rachana Sri rbollin@ncsu.edu
*Nangia, Chirrag - cnangia@ncsu.edu
*Priyadarshan, Saswat - spriyad2@ncsu.edu
*Vemuri, Pradyumna pvemuri@ncsu.edu

== Project Purpose & Description ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

== Current Project Scope ==

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438
But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

'''Below listed are our primary goals for this project:'''
* Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
* Write test cases for the new import_file_controller and the existing export_file_controller.
* Find and fix the existing bugs in the import and export process flows.

== Extended Project Scope ==

'''Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.'''

* After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

== Existing Import/Export Controller design ==

The existing import and export functionality primarily uses the '''import_file_controller''' and '''export_file_controller'''.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

*'''export_file_controller''', the list of methods in the controller are the following:
** ::find_delim_filename - Sets and returns attributes for User object from hash.
** ::exportdetails - Generates the metadata for the data that needs to be exported.
** ::export - Converts the data into CSV format files.
** ::export_advices - Export question advice data to CSV file.
** ::export_tags - Export the tags.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Find the filename and delimiter.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

== Proposed Design Change ==
* Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
* Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
* Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
* Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

'''In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):'''

[[Image: badUML.png]]

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

'''What we want it to look like: UML Diagram'''

[[Image: Ic usecase.png|600px]]

[[Image: EC usecase.png|600px]]

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference.
The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

== Actual Changes Made ==

* ''' Analyzing Export Functionality code flow ''' - We started by analyzing the code flow of export_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, Grades and Topics'''. On analyzing the code we found that all these three models are using the export_file_controller.rb to generate the respective CSVs. However, there were a few bugs that we found in the existing ''' Export Grades ''' functionality. We debugged this flow completely and fixed those bugs. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section. Apart from that, there was no ''' Export Topics ''' functionality present in the system. We created a completely new flow for exporting the topics leveraging the existing export_file_controller logic to keep the modules consistent. More information about the new features added can be found in the ''' New Features Added ''' section.

* ''' Analyzing Import Functionality code flow ''' - We analyzed the code flow of import_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, and Topics'''. On analyzing the code we found that these two models are using the export_file_controller.rb to import the respective CSVs with required data. However, we found a few bugs in the Import Topics functionality and we fixed those. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section.

Analysis of these two workflows (Export and Import) took a considerable amount of our time since there were a lot of moving pieces present in these controllers. We had to manually test these flows for different models because of the different data and customizations present.

* '''Test Cases for export_file_controller''' - After we finished the analysis and development of the required features for the export functionality, we proceeded with writing the test cases for the export_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users, Grades and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

* '''Test Cases for import_file_controller''' - After we finished the analysis and development of the required features for the import functionality, we proceeded with writing the test cases for the import_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

== New Features Added ==

*'''Export Topics''' - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to '''Assignments > Click on Edit (pencil icon) > Click on Topics menu'''. Once you click the Export Topics button, you will be taken to a customizations page where you can select what columns you want in your CSV file. You can select/deselect the options by clicking on the checkboxes provided. Once you select the desired cloumns, click on the export button to generate the CSV file. The Export Topics and Customizations pages are attached below.

[[File:Export Topics.png|800x520px]]

[[File:Customization.png|800x520px]]

*'''''Code Changes for Export Topics'''''

[[File:C1.png|800x520px]]

[[File:C2.png|800x520px]]

[[File:C3.png|800x520px]]

[[File:C4.png|800x520px]]

[[File:C5.png|800x520px]]

[[File:C6.png|800x520px]]

== Bug Fixes ==

On analyzing the code flow for import and export functionality we found that the '''Export Grades''' and '''Import Topics''' functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

*'''Fix 1: Export grades into a file successfully''' - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. After fixing that, there were some lines related to the business logic were missing here. We added those lines and it can be seen in the code diff.

[[File:Bug1.png|800x520px]]

*'''''Code Changes for Fix 1: Export grades into a file successfully'''''

[[File:Fix1 code.png|800x520px]]

*'''Fix 2: Incorrect headers in the exported file''' - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.
[[File:Incorrect grouped column.jpeg|800x520px]]

*'''''Code Changes for Fix 2: Incorrect headers in the exported file'''''

[[File:Fix2 code.png|800x520px]]

*'''Fix 3: Only export selected columns from the UI''' - Removed author feedback score from being exported by unchecking that option in the UI as seen in image below.

[[File:E3.png|800x520px]]

But what happens in the exported file?
** Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
** Problem 2: Also only the headers have changed but all values of all columns are still being exported.

[[File:E3 1.png|800x520px]]

*'''''Code Changes for Fix 3: Only export selected columns from the UI'''''

[[File:Fix3 code.png|800x520px]]

[[File:B3.jpeg|800x520px]]

*'''Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button''' - We tried importing the topics.csv and it imported all the data except the Category, Description, and Link columns. These columns were optional columns and when we selected the checkboxes for these, the data was not persisted in the database. We debugged this workflow and fixed this bug.

*'''''Code Changes for Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button'''''

[[File:Fix 4.png|800x520px]]

== Test Plan ==

Writing test cases for '''import_file_controller''' and '''export_file_controller'''. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:
*'''action_allowed''' - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''exportdetails''' - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
*'''export''' - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.

For the import_file_controller class, we identified the following methods to be tested:
*'''action_allowed''' - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''show''' - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
*'''import''' - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.

== Execution of Test Plan ==

To run the test cases for '''export_file_controller''' use the command - '''''rspec spec/controllers/export_file_controller_spec.rb'''''

To run the test cases for '''import_file_controller''' use the command - '''''rspec spec/controllers/import_file_controller_spec.rb'' ''''

We wrote the test cases for the above-mentioned methods focussing on the ''' models (Users, Topics and Grades) ''' since these were the high priority models assigned to us. Our Test Coverage so far is as follows:

*'''''export_file_controller test coverage - 54.84%'''''

[[File:Export test coverage.png|800x520px]]

*'''''import_file_controller test coverage - 52.43%'''''

[[File:Import test coverage.png|800x520px]]

We also implemented '''''fixtures''''' for testing the generated CSVs. This is a new concept that was not implemented earlier in this project and can be leveraged in future to test the CSVs. This provides the developers ability to test the Export and Import functionalities for CSV files.

We did not attach the code difference screenshots for the test cases because it was quite big. Kindly refer to the PR to view the code changes for the test cases corresponding to '''export_file_controller''' and '''import_file_controller'''.

== '''''Link to Pull Request:''''' https://github.com/expertiza/expertiza/pull/2396 ==

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-26T03:45:10Z

Spriyad2:

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

*Bollineni, Rachana Sri rbollin@ncsu.edu
*Nangia, Chirrag - cnangia@ncsu.edu
*Priyadarshan, Saswat - spriyad2@ncsu.edu
*Vemuri, Pradyumna pvemuri@ncsu.edu

== Project Purpose & Description ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

== Current Project Scope ==

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438
But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

'''Below listed are our primary goals for this project:'''
* Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
* Write test cases for the new import_file_controller and the existing export_file_controller.
* Find and fix the existing bugs in the import and export process flows.

== Extended Project Scope ==

'''Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.'''

* After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

== Existing Import/Export Controller design ==

The existing import and export functionality primarily uses the '''import_file_controller''' and '''export_file_controller'''.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

*'''export_file_controller''', the list of methods in the controller are the following:
** ::find_delim_filename - Sets and returns attributes for User object from hash.
** ::exportdetails - Generates the metadata for the data that needs to be exported.
** ::export - Converts the data into CSV format files.
** ::export_advices - Export question advice data to CSV file.
** ::export_tags - Export the tags.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Find the filename and delimiter.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

== Proposed Design Change ==
* Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
* Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
* Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
* Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

'''In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):'''

[[Image: badUML.png]]

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

'''What we want it to look like: UML Diagram'''

[[Image: Ic usecase.png|600px]]

[[Image: EC usecase.png|600px]]

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference.
The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

== Actual Changes Made ==

* ''' Analyzing Export Functionality code flow ''' - We started by analyzing the code flow of export_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, Grades and Topics'''. On analyzing the code we found that all these three models are using the export_file_controller.rb to generate the respective CSVs. However, there were a few bugs that we found in the existing ''' Export Grades ''' functionality. We debugged this flow completely and fixed those bugs. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section. Apart from that, there was no ''' Export Topics ''' functionality present in the system. We created a completely new flow for exporting the topics leveraging the existing export_file_controller logic to keep the modules consistent. More information about the new features added can be found in the ''' New Features Added ''' section.

* ''' Analyzing Import Functionality code flow ''' - We analyzed the code flow of import_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, and Topics'''. On analyzing the code we found that these two models are using the export_file_controller.rb to import the respective CSVs with required data. However, we found a few bugs in the Import Topics functionality and we fixed those. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section.

Analysis of these two workflows (Export and Import) took a considerable amount of our time since there were a lot of moving pieces present in these controllers. We had to manually test these flows for different models because of the different data and customizations present.

* '''Test Cases for export_file_controller''' - After we finished the analysis and development of the required features for the export functionality, we proceeded with writing the test cases for the export_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users, Grades and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

* '''Test Cases for import_file_controller''' - After we finished the analysis and development of the required features for the import functionality, we proceeded with writing the test cases for the import_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

== New Features Added ==

*'''Export Topics''' - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to '''Assignments > Click on Edit (pencil icon) > Click on Topics menu'''. Once you click the Export Topics button, you will be taken to a customizations page where you can select what columns you want in your CSV file. You can select/deselect the options by clicking on the checkboxes provided. Once you select the desired cloumns, click on the export button to generate the CSV file. The Export Topics and Customizations pages are attached below.

[[File:Export Topics.png|800x520px]]

[[File:Customization.png|800x520px]]

*'''''Code Changes for Export Topics'''''

[[File:C1.png|800x520px]]

[[File:C2.png|800x520px]]

[[File:C3.png|800x520px]]

[[File:C4.png|800x520px]]

[[File:C5.png|800x520px]]

[[File:C6.png|800x520px]]

== Bug Fixes ==

On analyzing the code flow for import and export functionality we found that the '''Export Grades''' and '''Import Topics''' functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

*'''Fix 1: Export grades into a file successfully''' - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. After fixing that, there were some lines related to the business logic were missing here. We added those lines and it can be seen in the code diff.

[[File:Bug1.png|800x520px]]

*'''''Code Changes for Fix 1: Export grades into a file successfully'''''

[[File:Fix1 code.png|800x520px]]

*'''Fix 2: Incorrect headers in the exported file''' - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.
[[File:Incorrect grouped column.jpeg|800x520px]]

*'''''Code Changes for Fix 2: Incorrect headers in the exported file'''''

[[File:Fix2 code.png|800x520px]]

*'''Fix 3: Only export selected columns from the UI''' - Removed author feedback score from being exported by unchecking that option in the UI as seen in image below.

[[File:E3.png|800x520px]]

But what happens in the exported file?
** Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
** Problem 2: Also only the headers have changed but all values of all columns are still being exported.

[[File:E3 1.png|800x520px]]

*'''''Code Changes for Fix 3: Only export selected columns from the UI'''''

[[File:Fix3 code.png|800x520px]]

[[File:B3.jpeg|800x520px]]

*'''Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button''' - We tried importing the topics.csv and it imported all the data except the Category, Description, and Link columns. These columns were optional columns and when we selected the checkboxes for these, the data was not persisted in the database. We debugged this workflow and fixed this bug.

*'''''Code Changes for Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button'''''

[[File:Fix 4.png|800x520px]]

== Test Plan ==

Writing test cases for '''import_file_controller''' and '''export_file_controller'''. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:
*'''action_allowed''' - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''exportdetails''' - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
*'''export''' - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.

For the import_file_controller class, we identified the following methods to be tested:
*'''action_allowed''' - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''show''' - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
*'''import''' - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.

== Execution of Test Plan ==

To run the test cases for '''export_file_controller''' use the command - '''''rspec spec/controllers/export_file_controller_spec.rb'''''

To run the test cases for '''import_file_controller''' use the command - '''''rspec spec/controllers/import_file_controller_spec.rb'' ''''

We wrote the test cases for the above-mentioned methods focussing on the ''' models (Users, Topics and Grades) ''' since these were the high priority models assigned to us. Our Test Coverage so far is as follows:

*'''''export_file_controller test coverage - 54.84%'''''

[[File:Export test coverage.png|800x520px]]

*'''''import_file_controller test coverage - 52.43%'''''

[[File:Import test coverage.png|800x520px]]

We also implemented '''''fixtures''''' for testing the generated CSVs. This is a new concept that was not implemented earlier in this project and can be leveraged in future to test the CSVs. This provides the developers ability to test the Export and Import functionalities for CSV files.

We did not attach the code difference screenshots for the test cases because it was quite big. Kindly refer to the PR to view the code changes for the test cases corresponding to '''export_file_controller''' and '''import_file_controller'''.

== '''''Link to Pull Request:''''' https://github.com/expertiza/expertiza/pull/2396 ==

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-26T03:44:12Z

Spriyad2:

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

*Bollineni, Rachana Sri rbollin@ncsu.edu
*Nangia, Chirrag - cnangia@ncsu.edu
*Priyadarshan, Saswat - spriyad2@ncsu.edu
*Vemuri, Pradyumna pvemuri@ncsu.edu

== Project Purpose & Description ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

== Current Project Scope ==

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438
But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

'''Below listed are our primary goals for this project:'''
* Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
* Write test cases for the new import_file_controller and the existing export_file_controller.
* Find and fix the existing bugs in the import and export process flows.

== Extended Project Scope ==

'''Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.'''

* After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

== Existing Import/Export Controller design ==

The existing import and export functionality primarily uses the '''import_file_controller''' and '''export_file_controller'''.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

*'''export_file_controller''', the list of methods in the controller are the following:
** ::find_delim_filename - Sets and returns attributes for User object from hash.
** ::exportdetails - Generates the metadata for the data that needs to be exported.
** ::export - Converts the data into CSV format files.
** ::export_advices - Export question advice data to CSV file.
** ::export_tags - Export the tags.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Find the filename and delimiter.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

== Proposed Design Change ==
* Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
* Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
* Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
* Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

'''In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):'''

[[Image: badUML.png]]

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

'''What we want it to look like: UML Diagram'''

[[Image: Ic usecase.png|600px]]

[[Image: EC usecase.png|600px]]

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference.
The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

== Actual Changes Made ==

* ''' Analyzing Export Functionality code flow ''' - We started by analyzing the code flow of export_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, Grades and Topics'''. On analyzing the code we found that all these three models are using the export_file_controller.rb to generate the respective CSVs. However, there were a few bugs that we found in the existing ''' Export Grades ''' functionality. We debugged this flow completely and fixed those bugs. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section. Apart from that, there was no ''' Export Topics ''' functionality present in the system. We created a completely new flow for exporting the topics leveraging the existing export_file_controller logic to keep the modules consistent. More information about the new features added can be found in the ''' New Features Added ''' section.

* ''' Analyzing Import Functionality code flow ''' - We analyzed the code flow of import_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, and Topics'''. On analyzing the code we found that these two models are using the export_file_controller.rb to import the respective CSVs with required data. However, we found a few bugs in the Import Topics functionality and we fixed those. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section.

Analysis of these two workflows (Export and Import) took a considerable amount of our time since there were a lot of moving pieces present in these controllers. We had to manually test these flows for different models because of the different data and customizations present.

* '''Test Cases for export_file_controller''' - After we finished the analysis and development of the required features for the export functionality, we proceeded with writing the test cases for the export_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users, Grades and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

* '''Test Cases for import_file_controller''' - After we finished the analysis and development of the required features for the import functionality, we proceeded with writing the test cases for the import_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

== New Features Added ==

*'''Export Topics''' - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to '''Assignments > Click on Edit (pencil icon) > Click on Topics menu'''. Once you click the Export Topics button, you will be taken to a customizations page where you can select what columns you want in your CSV file. You can select/deselect the options by clicking on the checkboxes provided. Once you select the desired cloumns, click on the export button to generate the CSV file. The Export Topics and Customizations pages are attached below.

[[File:Export Topics.png|800x520px]]

[[File:Customization.png|800x520px]]

*'''''Code Changes for Export Topics'''''

[[File:C1.png|800x520px]]

[[File:C2.png|800x520px]]

[[File:C3.png|800x520px]]

[[File:C4.png|800x520px]]

[[File:C5.png|800x520px]]

[[File:C6.png|800x520px]]

== Bug Fixes ==

On analyzing the code flow for import and export functionality we found that the '''Export Grades''' and '''Import Topics''' functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

*'''Fix 1: Export grades into a file successfully''' - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. After fixing that, there were some lines related to the business logic were missing here. We added those lines and it can be seen in the code diff.

[[File:Bug1.png|800x520px]]

*'''''Code Changes for Fix 1: Export grades into a file successfully'''''

[[File:Fix1 code.png|800x520px]]

*'''Fix 2: Incorrect headers in the exported file''' - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.
[[File:Incorrect grouped column.jpeg|800x520px]]

*'''''Code Changes for Fix 2: Incorrect headers in the exported file'''''

[[File:Fix2 code.png|800x520px]]

*'''Fix 3: Only export selected columns from the UI''' - Removed author feedback score from being exported by unchecking that option in the UI as seen in image below.

[[File:E3.png|800x520px]]

But what happens in the exported file?
** Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
** Problem 2: Also only the headers have changed but all values of all columns are still being exported.

[[File:E3 1.png|800x520px]]

*'''''Code Changes for Fix 3: Only export selected columns from the UI'''''

[[File:Fix3 code.png|800x520px]]

[[File:B3.jpeg|800x520px]]

*'''Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button''' - We tried importing the topics.csv and it imported all the data except the Category, Description, and Link columns. These columns were optional columns and when we selected the checkboxes for these, the data was not persisted in the database. We debugged this workflow and fixed this bug.

*'''''Code Changes for Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button'''''

[[File:Fix 4.png|800x520px]]

== Test Plan ==

Writing test cases for '''import_file_controller''' and '''export_file_controller'''. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:
*'''action_allowed''' - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''exportdetails''' - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
*'''export''' - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.
*'''export_advices''' - This method is similar to export but only exports "Question" type models along with "QuestionAdvice". We will test this method by checking if the correct data is being exported.
*'''export_tags''' - This method exports the user's answers with their respective tags and comments. We will test this method by checking if the correct data is being exported.

For the import_file_controller class, we identified the following methods to be tested:
*'''action_allowed''' - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''show''' - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
*'''import''' - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.

== Execution of Test Plan ==

To run the test cases for '''export_file_controller''' use the command - '''''rspec spec/controllers/export_file_controller_spec.rb'''''

To run the test cases for '''import_file_controller''' use the command - '''''rspec spec/controllers/import_file_controller_spec.rb'' ''''

We wrote the test cases for the above-mentioned methods focussing on the ''' models (Users, Topics and Grades) ''' since these were the high priority models assigned to us. Our Test Coverage so far is as follows:

*'''''export_file_controller test coverage - 54.84%'''''

[[File:Export test coverage.png|800x520px]]

*'''''import_file_controller test coverage - 52.43%'''''

[[File:Import test coverage.png|800x520px]]

We also implemented '''''fixtures''''' for testing the generated CSVs. This is a new concept that was not implemented earlier in this project and can be leveraged in future to test the CSVs. This provides the developers ability to test the Export and Import functionalities for CSV files.

We did not attach the code difference screenshots for the test cases because it was quite big. Kindly refer to the PR to view the code changes for the test cases corresponding to '''export_file_controller''' and '''import_file_controller'''.

== '''''Link to Pull Request:''''' https://github.com/expertiza/expertiza/pull/2396 ==

File:B3.jpeg

2022-04-26T03:42:25Z

Spriyad2:

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-25T21:40:00Z

Spriyad2:

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

*Bollineni, Rachana Sri rbollin@ncsu.edu
*Nangia, Chirrag - cnangia@ncsu.edu
*Priyadarshan, Saswat - spriyad2@ncsu.edu
*Vemuri, Pradyumna pvemuri@ncsu.edu

== Project Purpose & Description ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

== Current Project Scope ==

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438
But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

'''Below listed are our primary goals for this project:'''
* Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
* Write test cases for the new import_file_controller and the existing export_file_controller.
* Find and fix the existing bugs in the import and export process flows.

== Extended Project Scope ==

'''Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.'''

* After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

== Existing Import/Export Controller design ==

The existing import and export functionality primarily uses the '''import_file_controller''' and '''export_file_controller'''.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

*'''export_file_controller''', the list of methods in the controller are the following:
** ::find_delim_filename - Sets and returns attributes for User object from hash.
** ::exportdetails - Generates the metadata for the data that needs to be exported.
** ::export - Converts the data into CSV format files.
** ::export_advices - Export question advice data to CSV file.
** ::export_tags - Export the tags.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Find the filename and delimiter.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

== Proposed Design Change ==
* Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
* Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
* Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
* Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

'''In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):'''

[[Image: badUML.png]]

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

'''What we want it to look like: UML Diagram'''

[[Image: import_usecase.png|600px]]

[[Image: export_usecase.png|600px]]

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference.
The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

== Actual Changes Made ==

* ''' Analyzing Export Functionality code flow ''' - We started by analyzing the code flow of export_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, Grades and Topics'''. On analyzing the code we found that all these three models are using the export_file_controller.rb to generate the respective CSVs. However, there were a few bugs that we found in the existing ''' Export Grades ''' functionality. We debugged this flow completely and fixed those bugs. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section. Apart from that, there was no ''' Export Topics ''' functionality present in the system. We created a completely new flow for exporting the topics leveraging the existing export_file_controller logic to keep the modules consistent. More information about the new features added can be found in the ''' New Features Added ''' section.

* ''' Analyzing Import Functionality code flow ''' - We analyzed the code flow of import_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, and Topics'''. On analyzing the code we found that these two models are using the export_file_controller.rb to import the respective CSVs with required data. However, we found a few bugs in the Import Topics functionality and we fixed those. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section.

Analysis of these two workflows (Export and Import) took a considerable amount of our time since there were a lot of moving pieces present in these controllers. We had to manually test these flows for different models because of the different data and customizations present.

* '''Test Cases for export_file_controller''' - After we finished the analysis and development of the required features for the export functionality, we proceeded with writing the test cases for the export_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users, Grades and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

* '''Test Cases for import_file_controller''' - After we finished the analysis and development of the required features for the import functionality, we proceeded with writing the test cases for the import_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

== New Features Added ==

*'''Export Topics''' - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to '''Assignments > Click on Edit (pencil icon) > Click on Topics menu'''. Once you click the Export Topics button, you will be taken to a customizations page where you can select what columns you want in your CSV file. You can select/deselect the options by clicking on the checkboxes provided. Once you select the desired cloumns, click on the export button to generate the CSV file. The Export Topics and Customizations pages are attached below.

[[File:Export Topics.png|800x520px]]

[[File:Customization.png|800x520px]]

*'''''Code Changes for Export Topics'''''

[[File:C1.png|800x520px]]

[[File:C2.png|800x520px]]

[[File:C3.png|800x520px]]

[[File:C4.png|800x520px]]

[[File:C5.png|800x520px]]

[[File:C6.png|800x520px]]

== Bug Fixes ==

On analyzing the code flow for import and export functionality we found that the '''Export Grades''' functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

*'''Fix 1: Export grades into a file successfully''' - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. Some lines related to the business logic were missing here.

[[File:Bug1.png|800x520px]]

*'''''Code Changes for Fix 1: Export grades into a file successfully'''''

[[File:Fix1 code.png|800x520px]]

*'''Fix 2: Incorrect headers in the exported file''' - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.
[[File:Incorrect grouped column.jpeg|800x520px]]

*'''''Code Changes for Fix 2: Incorrect headers in the exported file'''''

[[File:Fix2 code.png|800x520px]]

*'''Fix 3: Only export selected columns from the UI''' - Removed author feedback score from being exported. What happens in the file?
** Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
** Problem 2: Also only the headers have changed but all values of all columns are still being exported.

[[File:E3.png|800x520px]]

[[File:E3 1.png|800x520px]]

*'''''Code Changes for Fix 3: Only export selected columns from the UI'''''

[[File:Fix3 code.png|800x520px]]

*'''Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button''' - We tried importing the topics.csv and it imported all the data except the Category, Description, and Link columns. These columns were optional columns and when we selected the checkboxes for these, the data was not persisted in the database. We debugged this workflow and fixed this bug.

*'''''Code Changes for Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button'''''

[[File:Fix 4.png|800x520px]]

== Test Plan ==

Writing test cases for '''import_file_controller''' and '''export_file_controller'''. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:
*'''action_allowed''' - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''exportdetails'' - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
*'''export''' - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.
*'''export_advices''' - This method is similar to export but only exports "Question" type models along with "QuestionAdvice". We will test this method by checking if the correct data is being exported.
*'''export_tags''' - This method exports the user's answers with their respective tags and comments. We will test this method by checking if the correct data is being exported.

For the import_file_controller class, we identified the following methods to be tested:
*'''action_allowed''' - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''show''' - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
*'''import''' - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.

== Execution of Test Plan ==

To run the test cases for '''export_file_controller''' use the command - '''''rspec spec/controllers/export_file_controller_spec.rb'''''

To run the test cases for '''import_file_controller''' use the command - '''''rspec spec/controllers/import_file_controller_spec.rb'' ''''

We wrote the test cases for the above-mentioned methods focussing on the ''' models (Users, Topics and Grades) ''' since these were the high priority models assigned to us. Our Test Coverage so far is as follows:

*'''''export_file_controller test coverage - 54.84%'''''

[[File:Export test coverage.png|800x520px]]

*'''''import_file_controller test coverage - 52.43%'''''

[[File:Import test coverage.png|800x520px]]

We also implemented '''''fixtures''''' for testing the generated CSVs. This is a new concept that was not implemented earlier in this project and can be leveraged in future to test the CSVs. This provides the developers ability to test the Export and Import functionalities for CSV files.

We did not attach the code difference screenshots for the test cases because it was quite big. Kindly refer to the PR to view the code changes for the test cases corresponding to '''export_file_controller''' and '''import_file_controller'''.

== '''''Link to Pull Request:''''' https://github.com/expertiza/expertiza/pull/2396 ==

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-25T21:39:15Z

Spriyad2:

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

*Bollineni, Rachana Sri rbollin@ncsu.edu
*Nangia, Chirrag - cnangia@ncsu.edu
*Priyadarshan, Saswat - spriyad2@ncsu.edu
*Vemuri, Pradyumna pvemuri@ncsu.edu

== Project Purpose & Description ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

== Current Project Scope ==

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438
But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

'''Below listed are our primary goals for this project:'''
* Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
* Write test cases for the new import_file_controller and the existing export_file_controller.
* Find and fix the existing bugs in the import and export process flows.

== Extended Project Scope ==

'''Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.'''

* After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

== Existing Import/Export Controller design ==

The existing import and export functionality primarily uses the '''import_file_controller''' and '''export_file_controller'''.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

*'''export_file_controller''', the list of methods in the controller are the following:
** ::find_delim_filename - Sets and returns attributes for User object from hash.
** ::exportdetails - Generates the metadata for the data that needs to be exported.
** ::export - Converts the data into CSV format files.
** ::export_advices - Export question advice data to CSV file.
** ::export_tags - Export the tags.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Find the filename and delimiter.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

== Proposed Design Change ==
* Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
* Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
* Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
* Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

'''In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):'''

[[Image: badUML.png]]

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

'''What we want it to look like: UML Diagram'''

[[Image: import_usecase.png|600px]]

[[Image: export_usecase.png|600px]]

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference.
The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

== Actual Changes Made ==

* ''' Analyzing Export Functionality code flow ''' - We started by analyzing the code flow of export_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, Grades and Topics'''. On analyzing the code we found that all these three models are using the export_file_controller.rb to generate the respective CSVs. However, there were a few bugs that we found in the existing ''' Export Grades ''' functionality. We debugged this flow completely and fixed those bugs. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section. Apart from that, there was no ''' Export Topics ''' functionality present in the system. We created a completely new flow for exporting the topics leveraging the existing export_file_controller logic to keep the modules consistent. More information about the new features added can be found in the ''' New Features Added ''' section.

* ''' Analyzing Import Functionality code flow ''' - We analyzed the code flow of import_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, and Topics'''. On analyzing the code we found that these two models are using the export_file_controller.rb to import the respective CSVs with required data. However, we found a few bugs in the Import Topics functionality and we fixed those. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section.

Analysis of these two workflows (Export and Import) took a considerable amount of our time since there were a lot of moving pieces present in these controllers. We had to manually test these flows for different models because of the different data and customizations present.

* '''Test Cases for export_file_controller''' - After we finished the analysis and development of the required features for the export functionality, we proceeded with writing the test cases for the export_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users, Grades and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

* '''Test Cases for import_file_controller''' - After we finished the analysis and development of the required features for the import functionality, we proceeded with writing the test cases for the import_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

== New Features Added ==

*'''Export Topics''' - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to '''Assignments > Click on Edit (pencil icon) > Click on Topics menu'''. Once you click the Export Topics button, you will be taken to a customizations page where you can select what columns you want in your CSV file. You can select/deselect the options by clicking on the checkboxes provided. Once you select the desired cloumns, click on the export button to generate the CSV file. The Export Topics and Customizations pages are attached below.

[[File:Export Topics.png|800x520px]]

[[File:Customization.png|800x520px]]

*'''''Code Changes for Export Topics'''''

[[File:C1.png|800x520px]]

[[File:C2.png|800x520px]]

[[File:C3.png|800x520px]]

[[File:C4.png|800x520px]]

[[File:C5.png|800x520px]]

[[File:C6.png|800x520px]]

== Bug Fixes ==

On analyzing the code flow for import and export functionality we found that the '''Export Grades''' functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

*'''Fix 1: Export grades into a file successfully''' - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. Some lines related to the business logic were missing here.

[[File:Bug1.png|800x520px]]

*'''''Code Changes for Fix 1: Export grades into a file successfully'''''

[[File:Fix1 code.png|800x520px]]

*'''Fix 2: Incorrect headers in the exported file''' - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.
[[File:Incorrect grouped column.jpeg|800x520px]]

*'''''Code Changes for Fix 2: Incorrect headers in the exported file'''''

[[File:Fix2 code.png|800x520px]]

*'''Fix 3: Only export selected columns from the UI''' - Removed author feedback score from being exported. What happens in the file?
** Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
** Problem 2: Also only the headers have changed but all values of all columns are still being exported.

[[File:E3.png|800x520px]]

[[File:E3 1.png|800x520px]]

*'''''Code Changes for Fix 3: Only export selected columns from the UI'''''

[[File:Fix3 code.png|800x520px]]

*'''Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button''' - We tried importing the topics.csv and it imported all the data except the Category, Description, and Link columns. These columns were optional columns and when we selected the checkboxes for these, the data was not persisted in the database. We debugged this workflow and fixed this bug.

*'''''Code Changes for Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button'''''

[[File:Fix 4.png|800x520px]]

== Test Plan ==

Writing test cases for '''import_file_controller''' and '''export_file_controller'''. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:
*'''action_allowed''' - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''exportdetails'' - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
*'''export''' - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.
*'''export_advices''' - This method is similar to export but only exports "Question" type models along with "QuestionAdvice". We will test this method by checking if the correct data is being exported.
*'''export_tags''' - This method exports the user's answers with their respective tags and comments. We will test this method by checking if the correct data is being exported.

For the import_file_controller class, we identified the following methods to be tested:
*'''action_allowed''' - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''show''' - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
*'''import''' - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.

== Execution of Test Plan ==

To run the test cases for '''export_file_controller''' use the command - '''rspec spec/controllers/export_file_controller_spec.rb'''

To run the test cases for '''import_file_controller''' use the command - '''rspec spec/controllers/import_file_controller_spec.rb ''''

We wrote the test cases for the above-mentioned methods focussing on the ''' models (Users, Topics and Grades) ''' since these were the high priority models assigned to us. Our Test Coverage so far is as follows:

*'''''export_file_controller test coverage - 54.84%'''''

[[File:Export test coverage.png|800x520px]]

*'''''import_file_controller test coverage - 52.43%'''''

[[File:Import test coverage.png|800x520px]]

We also implemented fixtures for testing the generated CSVs. This is a new concept that was not implemented earlier in this project and can be leveraged in future to test the CSVs. This provides the developers ability to test the Export and Import functionalities for CSV files.

We did not attach the code difference screenshots for the test cases because it was quite big. Kindly refer to the PR to view the code changes for the test cases corresponding to '''export_file_controller''' and '''import_file_controller'''.

== '''''Link to Pull Request:''''' https://github.com/expertiza/expertiza/pull/2396 ==

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-25T21:38:47Z

Spriyad2:

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

*Bollineni, Rachana Sri rbollin@ncsu.edu
*Nangia, Chirrag - cnangia@ncsu.edu
*Priyadarshan, Saswat - spriyad2@ncsu.edu
*Vemuri, Pradyumna pvemuri@ncsu.edu

== Project Purpose & Description ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

== Current Project Scope ==

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438
But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

'''Below listed are our primary goals for this project:'''
* Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
* Write test cases for the new import_file_controller and the existing export_file_controller.
* Find and fix the existing bugs in the import and export process flows.

== Extended Project Scope ==

'''Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.'''

* After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

== Existing Import/Export Controller design ==

The existing import and export functionality primarily uses the '''import_file_controller''' and '''export_file_controller'''.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

*'''export_file_controller''', the list of methods in the controller are the following:
** ::find_delim_filename - Sets and returns attributes for User object from hash.
** ::exportdetails - Generates the metadata for the data that needs to be exported.
** ::export - Converts the data into CSV format files.
** ::export_advices - Export question advice data to CSV file.
** ::export_tags - Export the tags.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Find the filename and delimiter.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

== Proposed Design Change ==
* Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
* Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
* Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
* Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

'''In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):'''

[[Image: badUML.png]]

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

'''What we want it to look like: UML Diagram'''

[[Image: import_usecase.png|600px]]

[[Image: export_usecase.png|600px]]

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference.
The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

== Actual Changes Made ==

* ''' Analyzing Export Functionality code flow ''' - We started by analyzing the code flow of export_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, Grades and Topics'''. On analyzing the code we found that all these three models are using the export_file_controller.rb to generate the respective CSVs. However, there were a few bugs that we found in the existing ''' Export Grades ''' functionality. We debugged this flow completely and fixed those bugs. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section. Apart from that, there was no ''' Export Topics ''' functionality present in the system. We created a completely new flow for exporting the topics leveraging the existing export_file_controller logic to keep the modules consistent. More information about the new features added can be found in the ''' New Features Added ''' section.

* ''' Analyzing Import Functionality code flow ''' - We analyzed the code flow of import_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, and Topics'''. On analyzing the code we found that these two models are using the export_file_controller.rb to import the respective CSVs with required data. However, we found a few bugs in the Import Topics functionality and we fixed those. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section.

Analysis of these two workflows (Export and Import) took a considerable amount of our time since there were a lot of moving pieces present in these controllers. We had to manually test these flows for different models because of the different data and customizations present.

* '''Test Cases for export_file_controller''' - After we finished the analysis and development of the required features for the export functionality, we proceeded with writing the test cases for the export_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users, Grades and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

* '''Test Cases for import_file_controller''' - After we finished the analysis and development of the required features for the import functionality, we proceeded with writing the test cases for the import_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

== New Features Added ==

*'''Export Topics''' - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to '''Assignments > Click on Edit (pencil icon) > Click on Topics menu'''. Once you click the Export Topics button, you will be taken to a customizations page where you can select what columns you want in your CSV file. You can select/deselect the options by clicking on the checkboxes provided. Once you select the desired cloumns, click on the export button to generate the CSV file. The Export Topics and Customizations pages are attached below.

[[File:Export Topics.png|800x520px]]

[[File:Customization.png|800x520px]]

*'''''Code Changes for Export Topics'''''

[[File:C1.png|800x520px]]

[[File:C2.png|800x520px]]

[[File:C3.png|800x520px]]

[[File:C4.png|800x520px]]

[[File:C5.png|800x520px]]

[[File:C6.png|800x520px]]

== Bug Fixes ==

On analyzing the code flow for import and export functionality we found that the '''Export Grades''' functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

*'''Fix 1: Export grades into a file successfully''' - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. Some lines related to the business logic were missing here.

[[File:Bug1.png|800x520px]]

*'''''Code Changes for Fix 1: Export grades into a file successfully'''''

[[File:Fix1 code.png|800x520px]]

*'''Fix 2: Incorrect headers in the exported file''' - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.
[[File:Incorrect grouped column.jpeg|800x520px]]

*'''''Code Changes for Fix 2: Incorrect headers in the exported file'''''

[[File:Fix2 code.png|800x520px]]

*'''Fix 3: Only export selected columns from the UI''' - Removed author feedback score from being exported. What happens in the file?
** Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
** Problem 2: Also only the headers have changed but all values of all columns are still being exported.

[[File:E3.png|800x520px]]

[[File:E3 1.png|800x520px]]

*'''''Code Changes for Fix 3: Only export selected columns from the UI'''''

[[File:Fix3 code.png|800x520px]]

*'''Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button''' - We tried importing the topics.csv and it imported all the data except the Category, Description, and Link columns. These columns were optional columns and when we selected the checkboxes for these, the data was not persisted in the database. We debugged this workflow and fixed this bug.

*'''''Code Changes for Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button'''''

[[File:Fix 4.png|800x520px]]

== Test Plan ==

Writing test cases for '''import_file_controller''' and '''export_file_controller'''. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:
*'''action_allowed''' - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''exportdetails'' - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
*'''export''' - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.
*'''export_advices''' - This method is similar to export but only exports "Question" type models along with "QuestionAdvice". We will test this method by checking if the correct data is being exported.
*'''export_tags''' - This method exports the user's answers with their respective tags and comments. We will test this method by checking if the correct data is being exported.

For the import_file_controller class, we identified the following methods to be tested:
*'''action_allowed''' - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''show''' - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
*'''import''' - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.

== Execution of Test Plan ==

To run the test cases for '''export_file_controller''' use the command - '''rspec spec/controllers/export_file_controller_spec.rb'''''Italic text''

To run the test cases for '''import_file_controller''' use the command - '''rspec spec/controllers/import_file_controller_spec.rb '''''Italic text''

We wrote the test cases for the above-mentioned methods focussing on the ''' models (Users, Topics and Grades) ''' since these were the high priority models assigned to us. Our Test Coverage so far is as follows:

*'''''export_file_controller test coverage - 54.84%'''''

[[File:Export test coverage.png|800x520px]]

*'''''import_file_controller test coverage - 52.43%'''''

[[File:Import test coverage.png|800x520px]]

We also implemented fixtures for testing the generated CSVs. This is a new concept that was not implemented earlier in this project and can be leveraged in future to test the CSVs. This provides the developers ability to test the Export and Import functionalities for CSV files.

We did not attach the code difference screenshots for the test cases because it was quite big. Kindly refer to the PR to view the code changes for the test cases corresponding to '''export_file_controller''' and '''import_file_controller'''.

== '''''Link to Pull Request:''''' https://github.com/expertiza/expertiza/pull/2396 ==

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-25T21:30:21Z

Spriyad2:

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

*Bollineni, Rachana Sri rbollin@ncsu.edu
*Nangia, Chirrag - cnangia@ncsu.edu
*Priyadarshan, Saswat - spriyad2@ncsu.edu
*Vemuri, Pradyumna pvemuri@ncsu.edu

== Project Purpose & Description ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

== Current Project Scope ==

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438
But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

'''Below listed are our primary goals for this project:'''
* Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
* Write test cases for the new import_file_controller and the existing export_file_controller.
* Find and fix the existing bugs in the import and export process flows.

== Extended Project Scope ==

'''Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.'''

* After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

== Existing Import/Export Controller design ==

The existing import and export functionality primarily uses the '''import_file_controller''' and '''export_file_controller'''.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

*'''export_file_controller''', the list of methods in the controller are the following:
** ::find_delim_filename - Sets and returns attributes for User object from hash.
** ::exportdetails - Generates the metadata for the data that needs to be exported.
** ::export - Converts the data into CSV format files.
** ::export_advices - Export question advice data to CSV file.
** ::export_tags - Export the tags.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Find the filename and delimiter.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

== Proposed Design Change ==
* Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
* Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
* Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
* Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

'''In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):'''

[[Image: badUML.png]]

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

'''What we want it to look like: UML Diagram'''

[[Image: import_usecase.png|600px]]

[[Image: export_usecase.png|600px]]

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference.
The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

== Actual Changes Made ==

* ''' Analyzing Export Functionality code flow ''' - We started by analyzing the code flow of export_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, Grades and Topics'''. On analyzing the code we found that all these three models are using the export_file_controller.rb to generate the respective CSVs. However, there were a few bugs that we found in the existing ''' Export Grades ''' functionality. We debugged this flow completely and fixed those bugs. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section. Apart from that, there was no ''' Export Topics ''' functionality present in the system. We created a completely new flow for exporting the topics leveraging the existing export_file_controller logic to keep the modules consistent. More information about the new features added can be found in the ''' New Features Added ''' section.

* ''' Analyzing Import Functionality code flow ''' - We analyzed the code flow of import_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, and Topics'''. On analyzing the code we found that these two models are using the export_file_controller.rb to import the respective CSVs with required data. However, we found a few bugs in the Import Topics functionality and we fixed those. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section.

Analysis of these two workflows (Export and Import) took a considerable amount of our time since there were a lot of moving pieces present in these controllers. We had to manually test these flows for different models because of the different data and customizations present.

* '''Test Cases for export_file_controller''' - After we finished the analysis and development of the required features for the export functionality, we proceeded with writing the test cases for the export_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users, Grades and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

* '''Test Cases for import_file_controller''' - After we finished the analysis and development of the required features for the import functionality, we proceeded with writing the test cases for the import_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

== New Features Added ==

*'''Export Topics''' - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to '''Assignments > Click on Edit (pencil icon) > Click on Topics menu'''. Once you click the Export Topics button, you will be taken to a customizations page where you can select what columns you want in your CSV file. You can select/deselect the options by clicking on the checkboxes provided. Once you select the desired cloumns, click on the export button to generate the CSV file. The Export Topics and Customizations pages are attached below.

[[File:Export Topics.png|800x520px]]

[[File:Customization.png|800x520px]]

*'''''Code Changes for Export Topics'''''

[[File:C1.png|800x520px]]

[[File:C2.png|800x520px]]

[[File:C3.png|800x520px]]

[[File:C4.png|800x520px]]

[[File:C5.png|800x520px]]

[[File:C6.png|800x520px]]

== Bug Fixes ==

On analyzing the code flow for import and export functionality we found that the '''Export Grades''' functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

*'''Fix 1: Export grades into a file successfully''' - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. Some lines related to the business logic were missing here.

[[File:Bug1.png|800x520px]]

*'''''Code Changes for Fix 1: Export grades into a file successfully'''''

[[File:Fix1 code.png|800x520px]]

*'''Fix 2: Incorrect headers in the exported file''' - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.
[[File:Incorrect grouped column.jpeg|800x520px]]

*'''''Code Changes for Fix 2: Incorrect headers in the exported file'''''

[[File:Fix2 code.png|800x520px]]

*'''Fix 3: Only export selected columns from the UI''' - Removed author feedback score from being exported. What happens in the file?
** Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
** Problem 2: Also only the headers have changed but all values of all columns are still being exported.

[[File:E3.png|800x520px]]

[[File:E3 1.png|800x520px]]

*'''''Code Changes for Fix 3: Only export selected columns from the UI'''''

[[File:Fix3 code.png|800x520px]]

*'''Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button''' - We tried importing the topics.csv and it imported all the data except the Category, Description, and Link columns. These columns were optional columns and when we selected the checkboxes for these, the data was not persisted in the database. We debugged this workflow and fixed this bug.

*'''''Code Changes for Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button'''''

[[File:Fix 4.png|800x520px]]

== Test Plan ==

Writing test cases for '''import_file_controller''' and '''export_file_controller'''. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:
*'''action_allowed''' - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''exportdetails'' - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
*'''export''' - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.
*'''export_advices''' - This method is similar to export but only exports "Question" type models along with "QuestionAdvice". We will test this method by checking if the correct data is being exported.
*'''export_tags''' - This method exports the user's answers with their respective tags and comments. We will test this method by checking if the correct data is being exported.

For the import_file_controller class, we identified the following methods to be tested:
*'''action_allowed''' - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''show''' - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
*'''import''' - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.

== Execution of Test Plan ==

We wrote the test cases for the above-mentioned methods focussing on the ''' models (Users, Topics and Grades) ''' since these were the high priority models assigned to us. Our Test Coverage so far is as follows:

*'''''export_file_controller test coverage - 54.84%'''''

[[File:Export test coverage.png|800x520px]]

*'''''import_file_controller test coverage - 52.43%'''''

[[File:Import test coverage.png|800x520px]]

We also implemented fixtures for testing the generated CSVs. This is a new concept that was not implemented earlier in this project and can be leveraged in future to test the CSVs. This provides the developers ability to test the Export and Import functionalities for CSV files.

We did not attach the code difference screenshots for the test cases because it was quite big. Kindly refer to the PR to view the code changes for the test cases corresponding to '''export_file_controller''' and '''import_file_controller'''.

== '''''Link to Pull Request:''''' https://github.com/expertiza/expertiza/pull/2396 ==

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-25T21:27:53Z

Spriyad2:

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

*Bollineni, Rachana Sri rbollin@ncsu.edu
*Nangia, Chirrag - cnangia@ncsu.edu
*Priyadarshan, Saswat - spriyad2@ncsu.edu
*Vemuri, Pradyumna pvemuri@ncsu.edu

== Project Purpose & Description ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

== Current Project Scope ==

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438
But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

'''Below listed are our primary goals for this project:'''
* Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
* Write test cases for the new import_file_controller and the existing export_file_controller.
* Find and fix the existing bugs in the import and export process flows.

== Extended Project Scope ==

'''Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.'''

* After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

== Existing Import/Export Controller design ==

The existing import and export functionality primarily uses the '''import_file_controller''' and '''export_file_controller'''.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

*'''export_file_controller''', the list of methods in the controller are the following:
** ::find_delim_filename - Sets and returns attributes for User object from hash.
** ::exportdetails - Generates the metadata for the data that needs to be exported.
** ::export - Converts the data into CSV format files.
** ::export_advices - Export question advice data to CSV file.
** ::export_tags - Export the tags.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Find the filename and delimiter.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

== Proposed Design Change ==
* Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
* Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
* Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
* Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

'''In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):'''

[[Image: badUML.png]]

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

'''What we want it to look like: UML Diagram'''

[[Image: import_usecase.png|600px]]

[[Image: export_usecase.png|600px]]

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference.
The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

== Actual Changes Made ==

* ''' Analyzing Export Functionality code flow ''' - We started by analyzing the code flow of export_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, Grades and Topics'''. On analyzing the code we found that all these three models are using the export_file_controller.rb to generate the respective CSVs. However, there were a few bugs that we found in the existing ''' Export Grades ''' functionality. We debugged this flow completely and fixed those bugs. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section. Apart from that, there was no ''' Export Topics ''' functionality present in the system. We created a completely new flow for exporting the topics leveraging the existing export_file_controller logic to keep the modules consistent. More information about the new features added can be found in the ''' New Features Added ''' section.

* ''' Analyzing Import Functionality code flow ''' - We analyzed the code flow of import_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, and Topics'''. On analyzing the code we found that these two models are using the export_file_controller.rb to import the respective CSVs with required data. However, we found a few bugs in the Import Topics functionality and we fixed those. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section.

Analysis of these two workflows (Export and Import) took a considerable amount of our time since there were a lot of moving pieces present in these controllers. We had to manually test these flows for different models because of the different data and customizations present.

* '''Test Cases for export_file_controller''' - After we finished the analysis and development of the required features for the export functionality, we proceeded with writing the test cases for the export_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users, Grades and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

* '''Test Cases for import_file_controller''' - After we finished the analysis and development of the required features for the import functionality, we proceeded with writing the test cases for the import_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

== New Features Added ==

*'''Export Topics''' - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to '''Assignments > Click on Edit (pencil icon) > Click on Topics menu'''. Once you click the Export Topics button, you will be taken to a customizations page where you can select what columns you want in your CSV file. You can select/deselect the options by clicking on the checkboxes provided. Once you select the desired cloumns, click on the export button to generate the CSV file. The Export Topics and Customizations pages are attached below.

[[File:Export Topics.png|800x520px]]

[[File:Customization.png|800x520px]]

*'''''Code Changes for Export Topics'''''

[[File:C1.png|800x520px]]

[[File:C2.png|800x520px]]

[[File:C3.png|800x520px]]

[[File:C4.png|800x520px]]

[[File:C5.png|800x520px]]

[[File:C6.png|800x520px]]

== Bug Fixes ==

On analyzing the code flow for import and export functionality we found that the '''Export Grades''' functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

*'''Fix 1: Export grades into a file successfully''' - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. Some lines related to the business logic were missing here.

[[File:Bug1.png|800x520px]]

*'''''Code Changes for Fix 1: Export grades into a file successfully'''''

[[File:Fix1 code.png|800x520px]]

*'''Fix 2: Incorrect headers in the exported file''' - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.
[[File:Incorrect grouped column.jpeg|800x520px]]

*'''''Code Changes for Fix 2: Incorrect headers in the exported file'''''

[[File:Fix2 code.png|800x520px]]

*'''Fix 3: Only export selected columns from the UI''' - Removed author feedback score from being exported. What happens in the file?
** Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
** Problem 2: Also only the headers have changed but all values of all columns are still being exported.

[[File:E3.png|800x520px]]

[[File:E3 1.png|800x520px]]

*'''''Code Changes for Fix 3: Only export selected columns from the UI'''''

[[File:Fix3 code.png|800x520px]]

*'''Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button''' - We tried importing the topics.csv and it imported all the data except the Category, Description, and Link columns. These columns were optional columns and when we selected the checkboxes for these, the data was not persisted in the database. We debugged this workflow and fixed this bug.

*'''''Code Changes for Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button'''''

[[File:Fix 4.png|800x520px]]

== Test Plan ==

Writing test cases for '''import_file_controller''' and '''export_file_controller'''. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:
*'''action_allowed''' - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''exportdetails'' - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
*'''export''' - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.
*'''export_advices''' - This method is similar to export but only exports "Question" type models along with "QuestionAdvice". We will test this method by checking if the correct data is being exported.
*'''export_tags''' - This method exports the user's answers with their respective tags and comments. We will test this method by checking if the correct data is being exported.

For the import_file_controller class, we identified the following methods to be tested:
*'''action_allowed''' - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''show''' - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
*'''import''' - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.

== Execution of Test Plan ==

We wrote the test cases for the above-mentioned methods focussing on the ''' models (Users, Topics and Grades) ''' since these were the high priority models assigned to us. Our Test Coverage so far is as follows:

*'''''export_file_controller test coverage - 54.84%'''''

[[File:Export test coverage.png|800x520px]]

*'''''import_file_controller test coverage - 52.43%'''''

[[File:Import test coverage.png|800x520px]]

We also implemented fixtures for testing the generated CSVs. This is a new concept that was not implemented earlier in this project and can be leveraged in future to test the CSVs. This provides the developers ability to test the Export and Import functionalities for CSV files.

== '''''Link to Pull Request:''''' https://github.com/expertiza/expertiza/pull/2396 ==

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-25T21:27:27Z

Spriyad2:

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

*Bollineni, Rachana Sri rbollin@ncsu.edu
*Nangia, Chirrag - cnangia@ncsu.edu
*Priyadarshan, Saswat - spriyad2@ncsu.edu
*Vemuri, Pradyumna pvemuri@ncsu.edu

== Project Purpose & Description ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

== Current Project Scope ==

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438
But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

'''Below listed are our primary goals for this project:'''
* Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
* Write test cases for the new import_file_controller and the existing export_file_controller.
* Find and fix the existing bugs in the import and export process flows.

== Extended Project Scope ==

'''Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.'''

* After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

== Existing Import/Export Controller design ==

The existing import and export functionality primarily uses the '''import_file_controller''' and '''export_file_controller'''.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

*'''export_file_controller''', the list of methods in the controller are the following:
** ::find_delim_filename - Sets and returns attributes for User object from hash.
** ::exportdetails - Generates the metadata for the data that needs to be exported.
** ::export - Converts the data into CSV format files.
** ::export_advices - Export question advice data to CSV file.
** ::export_tags - Export the tags.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Find the filename and delimiter.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

== Proposed Design Change ==
* Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
* Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
* Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
* Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

'''In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):'''

[[Image: badUML.png]]

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

'''What we want it to look like: UML Diagram'''

[[Image: import_usecase.png|600px]]

[[Image: export_usecase.png|600px]]

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference.
The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

== Actual Changes Made ==

* ''' Analyzing Export Functionality code flow ''' - We started by analyzing the code flow of export_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, Grades and Topics'''. On analyzing the code we found that all these three models are using the export_file_controller.rb to generate the respective CSVs. However, there were a few bugs that we found in the existing ''' Export Grades ''' functionality. We debugged this flow completely and fixed those bugs. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section. Apart from that, there was no ''' Export Topics ''' functionality present in the system. We created a completely new flow for exporting the topics leveraging the existing export_file_controller logic to keep the modules consistent. More information about the new features added can be found in the ''' New Features Added ''' section.

* ''' Analyzing Import Functionality code flow ''' - We analyzed the code flow of import_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, and Topics'''. On analyzing the code we found that these two models are using the export_file_controller.rb to import the respective CSVs with required data. However, we found a few bugs in the Import Topics functionality and we fixed those. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section.

Analysis of these two workflows (Export and Import) took a considerable amount of our time since there were a lot of moving pieces present in these controllers. We had to manually test these flows for different models because of the different data and customizations present.

* '''Test Cases for export_file_controller''' - After we finished the analysis and development of the required features for the export functionality, we proceeded with writing the test cases for the export_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users, Grades and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

* '''Test Cases for import_file_controller''' - After we finished the analysis and development of the required features for the import functionality, we proceeded with writing the test cases for the import_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

== New Features Added ==

*'''Export Topics''' - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to '''Assignments > Click on Edit (pencil icon) > Click on Topics menu'''. Once you click the Export Topics button, you will be taken to a customizations page where you can select what columns you want in your CSV file. You can select/deselect the options by clicking on the checkboxes provided. Once you select the desired cloumns, click on the export button to generate the CSV file. The Export Topics and Customizations pages are attached below.

[[File:Export Topics.png|800x520px]]

[[File:Customization.png|800x520px]]

*'''''Code Changes for Export Topics'''''

[[File:C1.png|800x520px]]

[[File:C2.png|800x520px]]

[[File:C3.png|800x520px]]

[[File:C4.png|800x520px]]

[[File:C5.png|800x520px]]

[[File:C6.png|800x520px]]

== Bug Fixes ==

On analyzing the code flow for import and export functionality we found that the '''Export Grades''' functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

*'''Fix 1: Export grades into a file successfully''' - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. Some lines related to the business logic were missing here.

[[File:Bug1.png|800x520px]]

*'''''Code Changes for Fix 1: Export grades into a file successfully'''''

[[File:Fix1 code.png|800x520px]]

*'''Fix 2: Incorrect headers in the exported file''' - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.
[[File:Incorrect grouped column.jpeg|800x520px]]

*'''''Code Changes for Fix 2: Incorrect headers in the exported file'''''

[[File:Fix2 code.png|800x520px]]

*'''Fix 3: Only export selected columns from the UI''' - Removed author feedback score from being exported. What happens in the file?
** Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
** Problem 2: Also only the headers have changed but all values of all columns are still being exported.

[[File:E3.png|800x520px]]

[[File:E3 1.png|800x520px]]

*'''''Code Changes for Fix 3: Only export selected columns from the UI'''''

[[File:Fix3 code.png|800x520px]]

*'''Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button''' - We tried importing the topics.csv and it imported all the data except the Category, Description, and Link columns. These columns were optional columns and when we selected the checkboxes for these, the data was not persisted in the database. We debugged this workflow and fixed this bug.

*'''''Code Changes for Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button'''''

[[File:Fix 4.png|800x520px]]

== Test Plan ==

Writing test cases for '''import_file_controller''' and '''export_file_controller'''. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:
*'''action_allowed''' - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''exportdetails'' - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
*'''export''' - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.
*'''export_advices''' - This method is similar to export but only exports "Question" type models along with "QuestionAdvice". We will test this method by checking if the correct data is being exported.
*'''export_tags''' - This method exports the user's answers with their respective tags and comments. We will test this method by checking if the correct data is being exported.

For the import_file_controller class, we identified the following methods to be tested:
*'''action_allowed''' - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''show''' - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
*'''import''' - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.

== Execution of Test Plan ==

We wrote the test cases for the above-mentioned methods focussing on the ''' models (Users, Topics and Grades) ''' since these were the high priority models assigned to us. Our Test Coverage so far is as follows:

*'''''export_file_controller test coverage - 54.84%'''''

[[File:Export test coverage.png|800x520px]]

*'''''import_file_controller test coverage - 52.43%'''''

[[File:Import test coverage.png|800x520px]]

We also implemented fixtures for testing the generated CSVs. This is a new concept that was not implemented earlier in this project and can be leveraged in future to test the CSVs. This provides the developers ability to test the Export and Import functionalities for CSV files.

'''''Link to Pull Request:''''' https://github.com/expertiza/expertiza/pull/2396

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-25T21:26:04Z

Spriyad2:

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

*Bollineni, Rachana Sri rbollin@ncsu.edu
*Nangia, Chirrag - cnangia@ncsu.edu
*Priyadarshan, Saswat - spriyad2@ncsu.edu
*Vemuri, Pradyumna pvemuri@ncsu.edu

== Project Purpose & Description ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

== Current Project Scope ==

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438
But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

'''Below listed are our primary goals for this project:'''
* Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
* Write test cases for the new import_file_controller and the existing export_file_controller.
* Find and fix the existing bugs in the import and export process flows.

== Extended Project Scope ==

'''Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.'''

* After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

== Existing Import/Export Controller design ==

The existing import and export functionality primarily uses the '''import_file_controller''' and '''export_file_controller'''.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

*'''export_file_controller''', the list of methods in the controller are the following:
** ::find_delim_filename - Sets and returns attributes for User object from hash.
** ::exportdetails - Generates the metadata for the data that needs to be exported.
** ::export - Converts the data into CSV format files.
** ::export_advices - Export question advice data to CSV file.
** ::export_tags - Export the tags.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Find the filename and delimiter.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

== Proposed Design Change ==
* Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
* Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
* Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
* Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

'''In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):'''

[[Image: badUML.png]]

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

'''What we want it to look like: UML Diagram'''

[[Image: import_usecase.png|600px]]

[[Image: export_usecase.png|600px]]

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference.
The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

== Actual Changes Made ==

* ''' Analyzing Export Functionality code flow ''' - We started by analyzing the code flow of export_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, Grades and Topics'''. On analyzing the code we found that all these three models are using the export_file_controller.rb to generate the respective CSVs. However, there were a few bugs that we found in the existing ''' Export Grades ''' functionality. We debugged this flow completely and fixed those bugs. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section. Apart from that, there was no ''' Export Topics ''' functionality present in the system. We created a completely new flow for exporting the topics leveraging the existing export_file_controller logic to keep the modules consistent. More information about the new features added can be found in the ''' New Features Added ''' section.

* ''' Analyzing Import Functionality code flow ''' - We analyzed the code flow of import_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, and Topics'''. On analyzing the code we found that these two models are using the export_file_controller.rb to import the respective CSVs with required data. However, we found a few bugs in the Import Topics functionality and we fixed those. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section.

Analysis of these two workflows (Export and Import) took a considerable amount of our time since there were a lot of moving pieces present in these controllers. We had to manually test these flows for different models because of the different data and customizations present.

* '''Test Cases for export_file_controller''' - After we finished the analysis and development of the required features for the export functionality, we proceeded with writing the test cases for the export_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users, Grades and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

* '''Test Cases for import_file_controller''' - After we finished the analysis and development of the required features for the import functionality, we proceeded with writing the test cases for the import_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

== New Features Added ==

*'''Export Topics''' - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to '''Assignments > Click on Edit (pencil icon) > Click on Topics menu'''. Once you click the Export Topics button, you will be taken to a customizations page where you can select what columns you want in your CSV file. You can select/deselect the options by clicking on the checkboxes provided. Once you select the desired cloumns, click on the export button to generate the CSV file. The Export Topics and Customizations pages are attached below.

[[File:Export Topics.png|800x520px]]

[[File:Customization.png|800x520px]]

*'''''Code Changes for Export Topics'''''

[[File:C1.png|800x520px]]

[[File:C2.png|800x520px]]

[[File:C3.png|800x520px]]

[[File:C4.png|800x520px]]

[[File:C5.png|800x520px]]

[[File:C6.png|800x520px]]

== Bug Fixes ==

On analyzing the code flow for import and export functionality we found that the '''Export Grades''' functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

*'''Fix 1: Export grades into a file successfully''' - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. Some lines related to the business logic were missing here.

[[File:Bug1.png|800x520px]]

*'''''Code Changes for Fix 1: Export grades into a file successfully'''''

[[File:Fix1 code.png|800x520px]]

*'''Fix 2: Incorrect headers in the exported file''' - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.
[[File:Incorrect grouped column.jpeg|800x520px]]

*'''''Code Changes for Fix 2: Incorrect headers in the exported file'''''

[[File:Fix2 code.png|800x520px]]

*'''Fix 3: Only export selected columns from the UI''' - Removed author feedback score from being exported. What happens in the file?
** Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
** Problem 2: Also only the headers have changed but all values of all columns are still being exported.

[[File:E3.png|800x520px]]

[[File:E3 1.png|800x520px]]

*'''''Code Changes for Fix 3: Only export selected columns from the UI'''''

[[File:Fix3 code.png|800x520px]]

*'''Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button''' - We tried importing the topics.csv and it imported all the data except the Category, Description, and Link columns. These columns were optional columns and when we selected the checkboxes for these, the data was not persisted in the database. We debugged this workflow and fixed this bug.

*'''''Code Changes for Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button'''''

[[File:Fix 4.png|800x520px]]

== Test Plan ==

Writing test cases for '''import_file_controller''' and '''export_file_controller'''. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:
*'''action_allowed''' - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''exportdetails'' - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
*'''export''' - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.
*'''export_advices''' - This method is similar to export but only exports "Question" type models along with "QuestionAdvice". We will test this method by checking if the correct data is being exported.
*'''export_tags''' - This method exports the user's answers with their respective tags and comments. We will test this method by checking if the correct data is being exported.

For the import_file_controller class, we identified the following methods to be tested:
*'''action_allowed''' - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''show''' - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
*'''import''' - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.

== Execution of Test Plan ==

We wrote the test cases for the above-mentioned methods focussing on the ''' models (Users, Topics and Grades) ''' since these were the high priority models assigned to us. Our Test Coverage so far is as follows:

*'''''export_file_controller test coverage - 54.84%'''''

[[File:Export test coverage.png|800x520px]]

*'''''import_file_controller test coverage - 52.43%'''''

[[File:Import test coverage.png|800x520px]]

We also implemented fixtures for testing the generated CSVs. This is a new concept that was not implemented earlier in this project and can be leveraged in future to test the CSVs. This provides the developers ability to test the Export and Import functionalities for CSV files.

File:Fixtures.png

2022-04-25T21:23:15Z

Spriyad2: Spriyad2 uploaded a new version of File:Fixtures.png

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-25T21:19:02Z

Spriyad2:

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

*Bollineni, Rachana Sri rbollin@ncsu.edu
*Nangia, Chirrag - cnangia@ncsu.edu
*Priyadarshan, Saswat - spriyad2@ncsu.edu
*Vemuri, Pradyumna pvemuri@ncsu.edu

== Project Purpose & Description ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

== Current Project Scope ==

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438
But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

'''Below listed are our primary goals for this project:'''
* Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
* Write test cases for the new import_file_controller and the existing export_file_controller.
* Find and fix the existing bugs in the import and export process flows.

== Extended Project Scope ==

'''Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.'''

* After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

== Existing Import/Export Controller design ==

The existing import and export functionality primarily uses the '''import_file_controller''' and '''export_file_controller'''.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

*'''export_file_controller''', the list of methods in the controller are the following:
** ::find_delim_filename - Sets and returns attributes for User object from hash.
** ::exportdetails - Generates the metadata for the data that needs to be exported.
** ::export - Converts the data into CSV format files.
** ::export_advices - Export question advice data to CSV file.
** ::export_tags - Export the tags.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Find the filename and delimiter.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

== Proposed Design Change ==
* Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
* Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
* Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
* Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

'''In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):'''

[[Image: badUML.png]]

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

'''What we want it to look like: UML Diagram'''

[[Image: import_usecase.png|600px]]

[[Image: export_usecase.png|600px]]

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference.
The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

== Actual Changes Made ==

* ''' Analyzing Export Functionality code flow ''' - We started by analyzing the code flow of export_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, Grades and Topics'''. On analyzing the code we found that all these three models are using the export_file_controller.rb to generate the respective CSVs. However, there were a few bugs that we found in the existing ''' Export Grades ''' functionality. We debugged this flow completely and fixed those bugs. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section. Apart from that, there was no ''' Export Topics ''' functionality present in the system. We created a completely new flow for exporting the topics leveraging the existing export_file_controller logic to keep the modules consistent. More information about the new features added can be found in the ''' New Features Added ''' section.

* ''' Analyzing Import Functionality code flow ''' - We analyzed the code flow of import_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, and Topics'''. On analyzing the code we found that these two models are using the export_file_controller.rb to import the respective CSVs with required data. However, we found a few bugs in the Import Topics functionality and we fixed those. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section.

Analysis of these two workflows (Export and Import) took a considerable amount of our time since there were a lot of moving pieces present in these controllers. We had to manually test these flows for different models because of the different data and customizations present.

* '''Test Cases for export_file_controller''' - After we finished the analysis and development of the required features for the export functionality, we proceeded with writing the test cases for the export_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users, Grades and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

* '''Test Cases for import_file_controller''' - After we finished the analysis and development of the required features for the import functionality, we proceeded with writing the test cases for the import_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these '''models (Users and Topics)''' since the methods inside these controllers are pretty big and contain a lot of if-else for different models.

== New Features Added ==

*'''Export Topics''' - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to '''Assignments > Click on Edit (pencil icon) > Click on Topics menu'''. Once you click the Export Topics button, you will be taken to a customizations page where you can select what columns you want in your CSV file. You can select/deselect the options by clicking on the checkboxes provided. Once you select the desired cloumns, click on the export button to generate the CSV file. The Export Topics and Customizations pages are attached below.

[[File:Export Topics.png|800x520px]]

[[File:Customization.png|800x520px]]

*'''''Code Changes for Export Topics'''''

[[File:C1.png|800x520px]]

[[File:C2.png|800x520px]]

[[File:C3.png|800x520px]]

[[File:C4.png|800x520px]]

[[File:C5.png|800x520px]]

[[File:C6.png|800x520px]]

== Bug Fixes ==

On analyzing the code flow for import and export functionality we found that the '''Export Grades''' functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

*'''Fix 1: Export grades into a file successfully''' - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. Some lines related to the business logic were missing here.

[[File:Bug1.png|800x520px]]

*'''''Code Changes for Fix 1: Export grades into a file successfully'''''

[[File:Fix1 code.png|800x520px]]

*'''Fix 2: Incorrect headers in the exported file''' - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.
[[File:Incorrect grouped column.jpeg|800x520px]]

*'''''Code Changes for Fix 2: Incorrect headers in the exported file'''''

[[File:Fix2 code.png|800x520px]]

*'''Fix 3: Only export selected columns from the UI''' - Removed author feedback score from being exported. What happens in the file?
** Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
** Problem 2: Also only the headers have changed but all values of all columns are still being exported.

[[File:E3.png|800x520px]]

[[File:E3 1.png|800x520px]]

*'''''Code Changes for Fix 3: Only export selected columns from the UI'''''

[[File:Fix3 code.png|800x520px]]

*'''Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button''' - We tried importing the topics.csv and it imported all the data except the Category, Description, and Link columns. These columns were optional columns and when we selected the checkboxes for these, the data was not persisted in the database. We debugged this workflow and fixed this bug.

*'''''Code Changes for Fix 4: Category, Description, and Link were not being updated in the database while importing the CSV using the Import Topics button'''''

[[File:Fix 4.png|800x520px]]

== Test Plan ==

Writing test cases for '''import_file_controller''' and '''export_file_controller'''. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:
*'''action_allowed''' - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''exportdetails'' - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
*'''export''' - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.
*'''export_advices''' - This method is similar to export but only exports "Question" type models along with "QuestionAdvice". We will test this method by checking if the correct data is being exported.
*'''export_tags''' - This method exports the user's answers with their respective tags and comments. We will test this method by checking if the correct data is being exported.

For the import_file_controller class, we identified the following methods to be tested:
*'''action_allowed''' - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''show''' - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
*'''import''' - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.

== Execution of Test Plan ==

We wrote the test cases for the above-mentioned methods focussing on the ''' models (Users, Topics and Grades) ''' since these were the high priority models assigned to us. Our Test Coverage so far is as follows:

*'''''export_file_controller test coverage - 54.84'''''

[[File:Export test coverage.png|800x520px]]

*'''''import_file_controller test coverage - 52.43'''''

[[File:Import test coverage.png|800x520px]]

File:Import test coverage.png

2022-04-25T21:16:51Z

Spriyad2:

File:Export test coverage.png

2022-04-25T21:15:32Z

Spriyad2:

File:C5.png

2022-04-25T21:03:15Z

Spriyad2:

File:C6.png

2022-04-25T21:01:57Z

Spriyad2:

File:C4.png

2022-04-25T21:00:09Z

Spriyad2:

File:C3.png

2022-04-25T20:59:06Z

Spriyad2:

File:C2.png

2022-04-25T20:57:52Z

Spriyad2:

File:C1.png

2022-04-25T20:56:05Z

Spriyad2:

File:Fix 4.png

2022-04-25T20:43:52Z

Spriyad2:

File:Bug1.png

2022-04-25T20:38:09Z

Spriyad2:

File:Fix3 code.png

2022-04-25T20:35:30Z

Spriyad2:

File:Fix2 code.png

2022-04-25T20:33:11Z

Spriyad2:

File:Fix1 code.png

2022-04-25T20:29:52Z

Spriyad2:

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-25T20:24:54Z

Spriyad2:

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

*Bollineni, Rachana Sri rbollin@ncsu.edu
*Nangia, Chirrag - cnangia@ncsu.edu
*Priyadarshan, Saswat - spriyad2@ncsu.edu
*Vemuri, Pradyumna pvemuri@ncsu.edu

== Project Purpose & Description ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

== Current Project Scope ==

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438
But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

'''Below listed are our primary goals for this project:'''
* Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
* Write test cases for the new import_file_controller and the existing export_file_controller.
* Find and fix the existing bugs in the import and export process flows.

== Extended Project Scope ==

'''Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.'''

* After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

== Existing Import/Export Controller design ==

The existing import and export functionality primarily uses the '''import_file_controller''' and '''export_file_controller'''.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

*'''export_file_controller''', the list of methods in the controller are the following:
** ::find_delim_filename - Sets and returns attributes for User object from hash.
** ::exportdetails - Generates the metadata for the data that needs to be exported.
** ::export - Converts the data into CSV format files.
** ::export_advices - Export question advice data to CSV file.
** ::export_tags - Export the tags.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Find the filename and delimiter.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

== Proposed Design Change ==
* Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
* Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
* Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
* Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

'''In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):'''

[[Image: badUML.png]]

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

'''What we want it to look like: UML Diagram'''

[[Image: import_usecase.png|600px]]

[[Image: export_usecase.png|600px]]

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference.
The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

== Actual Changes Made ==

* ''' Analyzing Export Functionality code flow ''' - We started by analyzing the code flow of export_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, Grades and Topics'''. On analyzing the code we found that all these three models are using the export_file_controller.rb to generate the respective CSVs. However, there were a few bugs that we found in the existing ''' Export Grades ''' functionality. We debugged this flow completely and fixed those bugs. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section. Apart from that, there was no ''' Export Topics ''' functionality present in the system. We created a completely new flow for exporting the topics leveraging the existing export_file_controller logic to keep the modules consistent. More information about the new features added can be found in the ''' New Features Added ''' section.

* ''' Analyzing Import Functionality code flow ''' - We analyzed the code flow of import_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, and Topics'''. On analyzing the code we found that these two models are using the export_file_controller.rb to import the respective CSVs with required data. The functionality was also working as expected and there were no bugs found for these two models.

Analysis of these two workflows (Export and Import) took a considerable amount of our time since there were a lot of moving pieces present in these controllers. We had to manually test these flows for different models because of the different data and customizations present.

* '''Test Cases for export_file_controller''' - After we finished the analysis and development of the required features, we proceeded with writing the test cases for the export_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these models since the methods inside these controllers are pretty big and contains a lot of if-else for different models.

== New Features Added ==

*'''Export Topics''' - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to '''Assignments > Click on Edit (pencil icon) > Click on Topics menu'''. Once you click the Export Topics button, you will be taken to a customizations page where you can select what columns you want in your CSV file. You can select/deselect the options by clicking on the checkboxes provided. Once you select the desired cloumns, click on the export button to generate the CSV file. The Export Topics and Customizations pages are attached below.
[[File:Export Topics.png|800x520px]]
[[File:Customization.png|800x520px]]

== Bug Fixes ==

On analyzing the code flow for import and export functionality we found that the '''Export Grades''' functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

*'''Fix 1: Export grades into a file successfully''' - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. Some lines related to the business logic were missing here.

*'''Fix 2: Incorrect headers in the exported file''' - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.
[[File:Incorrect grouped column.jpeg|800x520px]]

*'''Fix 3: Only export selected columns from the UI''' - Removed author feedback score from being exported. What happens in the file?
** Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
** Problem 2: Also only the headers have changed but all values of all columns are still being exported.

[[File:E3.png|800x520px]]

[[File:E3 1.png|800x520px]]

== Test Plan ==

Writing test cases for '''import_file_controller''' and '''export_file_controller'''. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:
*'''action_allowed''' - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''exportdetails''' - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
*'''export''' - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.
*'''export_advices''' - This method is similar to export but only exports "Question" type models along with "QuestionAdvice". We will test this method by checking if the correct data is being exported.
*'''export_tags''' - This method exports the user's answers with their respective tags and comments. We will test this method by checking if the correct data is being exported.

For the import_file_controller class, we identified the following methods to be tested:
*'''action_allowed''' - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''show''' - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
*'''import''' - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.

File:E3 1.png

2022-04-25T20:22:46Z

Spriyad2:

File:E3.png

2022-04-25T20:21:22Z

Spriyad2:

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-25T17:47:45Z

Spriyad2:

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

*Bollineni, Rachana Sri rbollin@ncsu.edu
*Nangia, Chirrag - cnangia@ncsu.edu
*Priyadarshan, Saswat - spriyad2@ncsu.edu
*Vemuri, Pradyumna pvemuri@ncsu.edu

== Project Purpose & Description ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

== Current Project Scope ==

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438
But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

'''Below listed are our primary goals for this project:'''
* Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
* Write test cases for the new import_file_controller and the existing export_file_controller.
* Find and fix the existing bugs in the import and export process flows.

== Extended Project Scope ==

'''Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.'''

* After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

== Existing Import/Export Controller design ==

The existing import and export functionality primarily uses the '''import_file_controller''' and '''export_file_controller'''.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

*'''export_file_controller''', the list of methods in the controller are the following:
** ::find_delim_filename - Sets and returns attributes for User object from hash.
** ::exportdetails - Generates the metadata for the data that needs to be exported.
** ::export - Converts the data into CSV format files.
** ::export_advices - Export question advice data to CSV file.
** ::export_tags - Export the tags.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Find the filename and delimiter.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

== Proposed Design Change ==
* Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
* Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
* Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
* Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

'''In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):'''

[[Image: badUML.png]]

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

'''What we want it to look like: UML Diagram'''

[[Image: import_usecase.png|600px]]

[[Image: export_usecase.png|600px]]

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference.
The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

== Actual Changes Made ==

* ''' Analyzing Export Functionality code flow ''' - We started by analyzing the code flow of export_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, Grades and Topics'''. On analyzing the code we found that all these three models are using the export_file_controller.rb to generate the respective CSVs. However, there were a few bugs that we found in the existing ''' Export Grades ''' functionality. We debugged this flow completely and fixed those bugs. The bugs and their corresponding fixes are mentioned below in the ''' Bug Fixes ''' section. Apart from that, there was no ''' Export Topics ''' functionality present in the system. We created a completely new flow for exporting the topics leveraging the existing export_file_controller logic to keep the modules consistent. More information about the new features added can be found in the ''' New Features Added ''' section.

* ''' Analyzing Import Functionality code flow ''' - We analyzed the code flow of import_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, and Topics'''. On analyzing the code we found that these two models are using the export_file_controller.rb to import the respective CSVs with required data. The functionality was also working as expected and there were no bugs found for these two models.

Analysis of these two workflows (Export and Import) took a considerable amount of our time since there were a lot of moving pieces present in these controllers. We had to manually test these flows for different models because of the different data and customizations present.

* '''Test Cases for export_file_controller''' - After we finished the analysis and development of the required features, we proceeded with writing the test cases for the export_file_controller.rb file. There were no test cases present for this file and we wrote all the test cases from scratch. Our test cases revolve around these models since the methods inside these controllers are pretty big and contains a lot of if-else for different models.

== New Features Added ==

*'''Export Topics''' - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to '''Assignments > Click on Edit (pencil icon) > Click on Topics menu'''. Once you click the Export Topics button, you will be taken to a customizations page where you can select what columns you want in your CSV file. You can select/deselect the options by clicking on the checkboxes provided. Once you select the desired cloumns, click on the export button to generate the CSV file. The Export Topics and Customizations pages are attached below.
[[File:Export Topics.png|800x520px]]
[[File:Customization.png|800x520px]]

== Bug Fixes ==

On analyzing the code flow for import and export functionality we found that the '''Export Grades''' functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

*'''Fix 1: Export grades into a file successfully''' - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. Some lines related to the business logic were missing here.

*'''Fix 2: Incorrect headers in the exported file''' - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.
[[File:Incorrect grouped column.jpeg|800x520px]]

*'''Fix 3: Only export selected columns from the UI''' - Removed author feedback score from being exported. What happens in the file?
** Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
** Problem 2: Also only the headers have changed but all values of all columns are still being exported.
[[File:Incorrect column data.jpeg|800x520px]]

== Test Plan ==

Writing test cases for '''import_file_controller''' and '''export_file_controller'''. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:
*'''action_allowed''' - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''exportdetails''' - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
*'''export''' - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.
*'''export_advices''' - This method is similar to export but only exports "Question" type models along with "QuestionAdvice". We will test this method by checking if the correct data is being exported.
*'''export_tags''' - This method exports the user's answers with their respective tags and comments. We will test this method by checking if the correct data is being exported.

For the import_file_controller class, we identified the following methods to be tested:
*'''action_allowed''' - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''show''' - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
*'''import''' - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-25T17:24:09Z

Spriyad2:

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

*Bollineni, Rachana Sri rbollin@ncsu.edu
*Nangia, Chirrag - cnangia@ncsu.edu
*Priyadarshan, Saswat - spriyad2@ncsu.edu
*Vemuri, Pradyumna pvemuri@ncsu.edu

== Project Purpose & Description ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

== Current Project Scope ==

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438
But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

'''Below listed are our primary goals for this project:'''
* Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
* Write test cases for the new import_file_controller and the existing export_file_controller.
* Find and fix the existing bugs in the import and export process flows.

== Extended Project Scope ==

'''Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.'''

* After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

== Existing Import/Export Controller design ==

The existing import and export functionality primarily uses the '''import_file_controller''' and '''export_file_controller'''.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

*'''export_file_controller''', the list of methods in the controller are the following:
** ::find_delim_filename - Sets and returns attributes for User object from hash.
** ::exportdetails - Generates the metadata for the data that needs to be exported.
** ::export - Converts the data into CSV format files.
** ::export_advices - Export question advice data to CSV file.
** ::export_tags - Export the tags.

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Find the filename and delimiter.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

== Proposed Design Change ==
* Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
* Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
* Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
* Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

'''In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):'''

[[Image: badUML.png]]

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

'''What we want it to look like: UML Diagram'''

[[Image: import_usecase.png|600px]]

[[Image: export_usecase.png|600px]]

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference.
The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

== Actual Changes Made ==

* '''1. Analyzing Export Functionality code flow''' - We started by analyzing the code flow of export_file_controller.rb and whether it is being used by different models to export data or not. '''The models we focussed on are Users, Grades and Topics'''. On analyzing the code we found that all these three models are using the export_file_controller.rb to generate the respective CSVs. However, there were a few bugs that we found in the existing '''Export Grades''' functionality. We debugged this flow completely and fixed those bugs. The bugs and their corresponding fixes are mentioned below in the '''Bug Fixes''' section. Apart from that there was no '''Export Topics''' functionality present in the system. We created a completely new flow for exporting the topics leveraging the existing export_file_controller logic to keep the modules consistent. More information about the new features added can be found in the '''New Features Added''' section.

== New Features Added ==

*'''Export Topics''' - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to '''Assignments > Click on Edit (pencil icon) > Click on Topics menu'''. Once you click the Export Topics button, you will be taken to a customizations page where you can select what columns you want in your CSV file. You can select/deselect the options by clicking on the checkboxes provided. Once you select the desired cloumns, click on the export button to generate the CSV file. The Export Topics and Customizations pages are attached below.
[[File:Export Topics.png|800x520px]]
[[File:Customization.png|800x520px]]

== Bug Fixes ==

On analyzing the code flow for import and export functionality we found that the '''Export Grades''' functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

*'''Fix 1: Export grades into a file successfully''' - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. Some lines related to the business logic were missing here.

*'''Fix 2: Incorrect headers in the exported file''' - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.
[[File:Incorrect grouped column.jpeg|800x520px]]

*'''Fix 3: Only export selected columns from the UI''' - Removed author feedback score from being exported. What happens in the file?
** Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
** Problem 2: Also only the headers have changed but all values of all columns are still being exported.
[[File:Incorrect column data.jpeg|800x520px]]

== Test Plan ==

Writing test cases for '''import_file_controller''' and '''export_file_controller'''. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:
*'''action_allowed''' - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''exportdetails''' - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
*'''export''' - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.
*'''export_advices''' - This method is similar to export but only exports "Question" type models along with "QuestionAdvice". We will test this method by checking if the correct data is being exported.
*'''export_tags''' - This method exports the user's answers with their respective tags and comments. We will test this method by checking if the correct data is being exported.

For the import_file_controller class, we identified the following methods to be tested:
*'''action_allowed''' - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
*'''show''' - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
*'''import''' - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.

File:Incorrect grouped column.jpeg

2022-04-25T17:07:53Z

Spriyad2:

File:Incorrect column data.jpeg

2022-04-25T17:05:52Z

Spriyad2:

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-25T16:59:25Z

Spriyad2:

File:Customization.png

2022-04-25T16:48:14Z

Spriyad2: Screenshot displaying customization page for Export Topics

Screenshot displaying customization page for Export Topics

File:Export Topics.png

2022-04-25T16:47:30Z

Spriyad2: Screenshot displaying Export Topics page

Screenshot displaying Export Topics page

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-25T16:45:25Z

Spriyad2:

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-25T16:31:17Z

Spriyad2:

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-25T16:17:05Z

Spriyad2:

CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

2022-04-06T22:58:23Z

Spriyad2: Created page with "== Team== === Mentor === * Vinay Deshmukh === Team Members === *Bollineni, Rachana Sri rbollin@ncsu.edu *Nangia, Chirrag - cnangia@ncsu.edu *Priyadarshan, Saswat - spriyad2..."

CSC/ECE 517 Spring 2022

2022-04-06T21:16:12Z

Spriyad2:

== OSS Projects ==

* [[CSC/ECE 517 Spring 2022 - E2200: Testing advice_controller]]
* [[CSC/ECE 517 Spring 2022 - E2203: Adding tests for courses_controller, eula_controller]]
* [[CSC/ECE 517 Spring 2022 - E2204: Adding tests for markup_styles_controller, lock_controller]]
* [[CSC/ECE 517 Spring 2022 - E2208: Testing for submission_records_controller, profile_controller]]
* [[CSC/ECE 517 Spring 2022 - E2218: Refactor response_controller]]
* [[CSC/ECE 517 Spring 2022 - E2217: Refactor questionnaires_controller]]
* [[CSC/ECE 517 Spring 2022 - E2216: Refactor late_policies_controller]]
* [[CSC/ECE 517 Spring 2022 - E2215: Refactor student_quizzes_controller]]
* [[CSC/ECE 517 Spring 2022 - E2214: Refactor teams_controller]]
* [[CSC/ECE 517 Spring 2022 - E2219: Improve assessment360_controller.rb]]
* [[CSC/ECE 517 Spring 2022 - E2211: Testing for summary_helper]]
* [[CSC/ECE 517 Spring 2022 - E2225: Refactor review_mapping_helper.rb]]
* [[CSC/ECE 517 Spring 2022 - E2221: Refactor submitted content controller.rb]]
* [[CSC/ECE 517 Spring 2022 - E2206: Testing for users_controller]]
* [[CSC/ECE 517 Spring 2022 - E2207: Testing for submitted_content_controller.rb]]
* [[CSC/ECE 517 Spring 2022 - E2205: Testing for participants_controller, versions_controller]]
* [[CSC/ECE 517 Spring 2022 - E2212: Testing for hamer.rb]]
* [[CSC/ECE 517 Spring 2022 - S2200: Refactoring evaluation of SQL queries (Java/JUnit)]]
* [[CSC/ECE 517 Spring 2022 - S2222: Refactor impersonate_controller]]
* [[CSC/ECE 517 Spring 2022 - E2202- Testing for badges_controller, publishing_controller]]
*[[CSC/ECE 517 Spring 2022 - E2220: Refactor reputation_web_service_controller]]
* [[CSC/ECE 517 Spring 2022 - E2224: Refactor review_mapping_controller]]
* [[CSC/ECE 517 Spring 2022 - E2209: Testing for analytic helper.rb, join team requests helper.rb]]
* [[CSC/ECE 517 Spring 2022 - E2227: SQL Injection Fix]]
* [[CSC/ECE 517 Spring 2022 - E2223. Refactor sign up sheet controller.rb]]
* [[CSC/ECE 517 Spring 2022 - E2201: Testing for assignment_questionnaire_controller.rb]]
* [[CSC/ECE 517 Spring 2022 - E2243. Refactor student_teams functionality]]

== Final Projects ==
* [[CSC/ECE 517 Spring 2022 - E2245: View for results of bidding]]
* [[CSC/ECE 517 Spring 2022 - E2241: Heatgrid fixes and improvements]]
* [[CSC/ECE 517 Spring 2022 - E2237: Grading audit trail]]
* [[CSC/ECE 517 Spring 2022 - S2201: Improving User Experience for SQLFE]]
* [[CSC/ECE 517 Spring 2022 - E2236: Fix issues related to deadlines and late policies]]
* [[CSC/ECE 517 Spring 2022 - E2231: Allow reviewers to bid on what to review]]
* [[CSC/ECE 517 Spring 2022 - E2233. Improving search facility in Expertiza]]
* [[CSC/ECE 517 Spring 2022 - E2234. Calibration submissions should be copied along with calibration assignments]]
* [[CSC/ECE 517 Spring 2022 - E2244. Support for saying your team pair-programmed]]
* [[CSC/ECE 517 Spring 2022 - E2246: Email your authors/reviewers, with complete tests]]
* [[CSC/ECE 517 Spring 2022 - E2229: Track the time students look at other submissions]]
* [[CSC/ECE 517 Spring 2022 - E2239: Further refactoring and improvement of review mapping helper.rb ]]
* [[CSC/ECE 517 Spring 2022 - E2240. Re-write waitlist functionality ]]
* [[CSC/ECE 517 Spring 2022 - E2242. Fix teammate-review view ]]
* [[CSC/ECE 517 Spring 2022 - E2235. Issues related to meta-reviewing]]
* [[CSC/ECE 517 Spring 2022 - E2230. Integrate Suggestion Detection Algorithm]]
* [[CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers]]

CSC/ECE 517 Spring 2022 - E2211: Testing for summary helper

2022-03-28T00:35:04Z

Spriyad2:

== About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an assignment/project management portal that can be used by both instructors and students for collaborative learning and feedback. It is an open-source project based on [http://rubyonrails.org/ Ruby on Rails] framework. It allows the instructors not only to create and customize new or existing assignments but also to create a list of topics the students can sign up for. Students can form teams to work on various projects and assignments. Expertiza also lets students peer-review other students' submissions, enabling them to work together to improve others' learning experiences.

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

* Saswat Priyadarshan
* Rachel Son
* Bhuwan Bhatt

== Description and Testing ==
Our project deals with writing test cases for the summary_helper.rb file. The summary helper is responsible for doing the following tasks:
* Get all answers for each question and send them to summarization WS.
* Get average scores and a summary for each question in a review by a reviewer.

=== Files Involved ===

* summary_helper.rb
* summary_helper_spec.rb

=== Running Tests ===
To successfully run summary_helper_test in local machine, please run the below rspec command.
<pre>
rspec spec/helpers/summary_helper_spec.rb
</pre>

=== Requirement ===

summary_helper class had methods which were not tested. Our job was to test all the methods so that we can verify summary_helper class is working fine.

=== Test Plan ===

After going through all the methods in the class, we figured out there were some methods that were never being called. So, we removed all the redundant methods that are not being called. Now we had only 9 methods in the class that we need to test. Nine methods in summary_helper class that needed to be tested were:

* summarize_reviews_by_reviewee
* summarize_reviews_by_reviewee_question
* get_max_score_for_question
* summarize_sentences
* get_sentences
* break_up_comments_to_sentences
* calculate_avg_score_by_criterion
* calculate_round_score
* calculate_avg_score_by_round
* get_sentences

For the testing, we have mock following items :
<pre>
let(:answer) { Answer.new(answer: 1, comments: 'This is a sentence. This is another sentence.', question_id: 1) }
let(:answer1){ Answer.new(answer: 2, comments: 'This is a sentence1. This is another sentence1.', question_id: 2)}
let(:question) {build(:question, weight:1, type:"Criterion")}
let(:avg_scores_by_criterion) { {a:2.345} }
</pre>

* get_sentences
To test this method we mocked an answer object and checked if the number of sentences is equal to the number of comments provided in the answer.
We identified two test cases for this method:
1. When the answer is nil
2. When the comment is two sentences

Code Snippet:
<pre>
describe '#get_sentences' do
context 'when the answer is nil' do
it 'returns a nil object' do
expect(@summary.get_sentences(nil)).to eq(nil)
end
end
context 'when the comment is two sentences' do
it 'returns an array of two sentences' do
sentences = @summary.get_sentences(answer)
expect(sentences.length).to be(2)
end
end
end
</pre>

* get_max_score_for_question
To test this method we mocked two different types of questions which are as follows:
1. let(:questionOne){Question.new(type:'Checkbox')}
2. let(:questionTwo) { build(:question, questionnaire: questionnaire1, weight: 1, id: 1) }
Now based on the different question types we tested two scenarios for this method:
1. When the question type is Checkbox
2. When question type is not Checkbox

Code Snippet:
<pre>
describe 'get_max_score_for_question' do
context 'When question type is Checkbox' do
let(:questionOne){Question.new(type:'Checkbox')}
it 'returns 1' do
max_score = @summary.get_max_score_for_question(questionOne)
expect(max_score).to be(1)
end
end
context 'When question type is not Checkbox' do
let(:questionnaire1) { build(:questionnaire, id: 2) }
let(:questionTwo) { build(:question, questionnaire: questionnaire1, weight: 1, id: 1) }
it 'return the max score for the provided question' do
allow(Questionnaire).to receive(:where).with(id:2).and_return(questionnaire1)
allow(questionnaire1).to receive(:first).and_return(questionnaire1)
expect(@summary.get_max_score_for_question(questionTwo)).to eql(5)
end
end
end

</pre>

* summarize_sentence
To test this method we mocked a comments array with the following values ["Hello this is first comment", "This is second comment"]. This method is making a web service call at the following Web Service:
'http://peerlogic.csc.ncsu.edu/sum/v1.0/summary/8/lsa'
We are using expect to compare the expected result with the actual result from the web service call.
Disclaimer: This web service call results in a bad gateway error while running the test case. We have already informed our mentor and the professor regarding the same.

Code Snippet:
<pre>
describe '#summarize_sentence' do
context 'successful webservice call' do
comments = ["Hello this is first comment", "This is second comment"]
summary_ws_url = WEBSERVICE_CONFIG['summary_webservice_url']
it 'return success' do
expect(@summary.summarize_sentences(comments,summary_ws_url)).not_to eql(nil)
end
end
end
</pre>

* break_up_comments_to_sentences
This methods question answer array and breaks them into comments.
In our testing we provided the question_answer array of length 2 and see if the number of comments is equal to 2.
Code Snippet:
<pre>
describe '#break_up_comments_to_sentences' do
context 'when the question_answers is not nil' do
it 'add the comment to an array to be converted as a json request' do
comments = @summary.break_up_comments_to_sentences([answer])
expect(comments.length).to be(2)
end
end
context 'when the question_answers is nil' do
it 'returns an empty array' do
comments = @summary.break_up_comments_to_sentences([])
expect(comments.length).to be(0)
end
end
end
</pre>

* calculate_avg_score_by_criterion
This test is to see when question answer was given, the output of method is correctly calculated as percentage of question score.
Code Snippet:
<pre>
describe '#calculate_avg_score_by_criterion' do
context 'when question_answers are available' do
it 'calculate percentage question_score & no float' do
expect(@summary.calculate_avg_score_by_criterion([answer,answer1], 3)).to be_within(0).of(50)
end
end
</pre>

This test is to see when question answer was nil, the output of method is correctly calculated as 0.
Code Snippet:
<pre>
context 'when question_answers are not available' do
it 'gives question scores 0.0' do
expect(@summary.calculate_avg_score_by_criterion([], 3)).to eq(0.0)
end
end
</pre>

This test is to see when q_max_score = 0, the output of method is correctly calculated.
Code Snippet:
<pre>
context 'when q_max_score = 0' do
it 'gives pure question_score' do
expect(@summary.calculate_avg_score_by_criterion([answer,answer1], 0)).to eq(3.0)
end
end
end
</pre>

* calculate_round_score
The test is to see when the criteria input is nil, the method gives right output of nil, which is 0 in this case.
Code Snippet:
<pre>
describe '#calculate_round_score' do
context 'when criteria not available' do
it 'returns 0.0 since round_score = 0.0' do
expect(@summary.calculate_round_score(avg_scores_by_criterion, nil)).to eq(0.to_f)
end
end
</pre>

The test is to see when the criteria input is not nil, the method gives the right output, which is to get a two-digit round score. The criteria input was defined as question in mock section.
Code Snippet:
<pre>
context 'when criteria not nil' do
it 'get 2 round_score ' do
expect(@summary.calculate_round_score(avg_scores_by_criterion, question)).to be_within(0.01).of(2.345)
end
end
end
</pre>

* calculate_avg_score_by_round
This test is to see the method round an average number up to Two-digit. The input of avg_scores_by_criterion was given as 2.345, and we tested here that it gives two-digit round number as 2.35.
Code Snippet:
<pre>
describe '#calculate_avg_score_by_round'do
context 'when avg_scores_by_criterion available' do
it 'gives 2 round value' do
expect(@summary.calculate_avg_score_by_round(avg_scores_by_criterion, question)).to eq(2.35)
end
end
end
</pre>

=== Test Execution ===
We divided the work among the teammates and started tackling the problems. We stubbed the data using factory and mocked the method calls which were being done internally to get the desired output from the methods that were calling other methods internally.

=== Test Coverage ===
Our test coverage is shown in [https://coveralls.io/builds/47520691/source?filename=app%2Fhelpers%2Fsummary_helper.rb coverails].

=== Conclusion ===
While we tried to test all the methods in the summary_helper.rb class, we faced some blockers. All the blockers are described below:-
* summarize_sentences method

summary_webservice_url: 'http://peerlogic.csc.ncsu.edu/sum/v1.0/summary/8/lsa'
This URL is being used in the summary_helper.rb file and we were not able to mock it. It's giving us bad gateway error while we try to hit that url.

* summarize_reviews_by_reviewee method

In this method we found out in the loop, questions[round] is not the correct way of passing the individual question since questions is an array and use 0 based indexing. Whereas in the above logic it is being treated as a hash.

* summarize_reviews_by_reviewee_question method

This method uses the instance variables from a different method(summarize_reviews_by_reviewee) which are not being passed as arguments to this method. Technically this type of initialization is bad coding practice and hence needs refratoring.

We already informed our Mentor and professor regarding these issues and they said it will be included in future work.

We tested all other remaining methods in the summary_helper.rb class. All the unit tests passed and the methods were working as expected. We tried to cover corner cases but there is room for some improvement.

CSC/ECE 517 Spring 2022 - E2211: Testing for summary helper

2022-03-28T00:34:08Z

Spriyad2:

CSC/ECE 517 Spring 2022 - E2211: Testing for summary helper

2022-03-28T00:33:10Z

Spriyad2:

== About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an assignment/project management portal that can be used by both instructors and students for collaborative learning and feedback. It is an open-source project based on [http://rubyonrails.org/ Ruby on Rails] framework. It allows the instructors not only to create and customize new or existing assignments but also to create a list of topics the students can sign up for. Students can form teams to work on various projects and assignments. Expertiza also lets students peer-review other students' submissions, enabling them to work together to improve others' learning experiences.

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

* Saswat Priyadarshan
* Rachel Son
* Bhuwan Bhatt

== Description and Testing ==
Our project deals with writing test cases for the summary_helper.rb file. The summary helper is responsible for doing the following tasks:
* Get all answers for each question and send them to summarization WS.
* Get average scores and a summary for each question in a review by a reviewer.

=== Files Involved ===

* summary_helper.rb
* summary_helper_spec.rb

=== Running Tests ===
To successfully run summary_helper_test in local machine, please run the below rspec command.
<pre>
rspec spec/helpers/summary_helper_spec.rb
</pre>

=== Requirement ===

summary_helper class had methods which were not tested. Our job was to test all the methods so that we can verify summary_helper class is working fine.

=== Test Plan ===

After going through all the methods in the class, we figured out there were some methods that were never being called. So, we removed all the redundant methods that are not being called. Now we had only 9 methods in the class that we need to test. Nine methods in summary_helper class that needed to be tested were:

* summarize_reviews_by_reviewee
* summarize_reviews_by_reviewee_question
* get_max_score_for_question
* summarize_sentences
* get_sentences
* break_up_comments_to_sentences
* calculate_avg_score_by_criterion
* calculate_round_score
* calculate_avg_score_by_round
* get_sentences

For the testing, we have mock following items :
<pre>
let(:answer) { Answer.new(answer: 1, comments: 'This is a sentence. This is another sentence.', question_id: 1) }
let(:answer1){ Answer.new(answer: 2, comments: 'This is a sentence1. This is another sentence1.', question_id: 2)}
let(:question) {build(:question, weight:1, type:"Criterion")}
let(:avg_scores_by_criterion) { {a:2.345} }
</pre>

* get_sentences
To test this method we mocked an answer object and checked if the number of sentences is equal to the number of comments provided in the answer.
We identified two test cases for this method:
1. When the answer is nil
2. When the comment is two sentences

Code Snippet:
<pre>
describe '#get_sentences' do
context 'when the answer is nil' do
it 'returns a nil object' do
expect(@summary.get_sentences(nil)).to eq(nil)
end
end
context 'when the comment is two sentences' do
it 'returns an array of two sentences' do
sentences = @summary.get_sentences(answer)
expect(sentences.length).to be(2)
end
end
end
</pre>

* get_max_score_for_question
To test this method we mocked two different types of questions which are as follows:
1. let(:questionOne){Question.new(type:'Checkbox')}
2. let(:questionTwo) { build(:question, questionnaire: questionnaire1, weight: 1, id: 1) }
Now based on the different question types we tested two scenarios for this method:
1. When the question type is Checkbox
2. When question type is not Checkbox

Code Snippet:
<pre>
describe 'get_max_score_for_question' do
context 'When question type is Checkbox' do
let(:questionOne){Question.new(type:'Checkbox')}
it 'returns 1' do
max_score = @summary.get_max_score_for_question(questionOne)
expect(max_score).to be(1)
end
end
context 'When question type is not Checkbox' do
let(:questionnaire1) { build(:questionnaire, id: 2) }
let(:questionTwo) { build(:question, questionnaire: questionnaire1, weight: 1, id: 1) }
it 'return the max score for the provided question' do
allow(Questionnaire).to receive(:where).with(id:2).and_return(questionnaire1)
allow(questionnaire1).to receive(:first).and_return(questionnaire1)
expect(@summary.get_max_score_for_question(questionTwo)).to eql(5)
end
end
end

</pre>

* summarize_sentence
To test this method we mocked a comments array with the following values ["Hello this is first comment", "This is second comment"]. This method is making a web service call at the following Web Service:
'http://peerlogic.csc.ncsu.edu/sum/v1.0/summary/8/lsa'
We are using expect to compare the expected result with the actual result from the web service call.
Disclaimer: This web service call results in a bad gateway error while running the test case. We have already informed our mentor and the professor regarding the same.

Code Snippet:
<pre>
describe '#summarize_sentence' do
context 'successful webservice call' do
comments = ["Hello this is first comment", "This is second comment"]
summary_ws_url = WEBSERVICE_CONFIG['summary_webservice_url']
it 'return success' do
expect(@summary.summarize_sentences(comments,summary_ws_url)).not_to eql(nil)
end
end
end
</pre>

* break_up_comments_to_sentences
This methods question answer array and breaks them into comments.
In our testing we provided the question_answer array of length 2 and see if the number of comments is equal to 2.
Code Snippet:
<pre>
describe '#break_up_comments_to_sentences' do
context 'when the question_answers is not nil' do
it 'add the comment to an array to be converted as a json request' do
comments = @summary.break_up_comments_to_sentences([answer])
expect(comments.length).to be(2)
end
end
context 'when the question_answers is nil' do
it 'returns an empty array' do
comments = @summary.break_up_comments_to_sentences([])
expect(comments.length).to be(0)
end
end
end
</pre>

* calculate_avg_score_by_criterion
This test is to see when question answer was given, the output of method is correctly calculated as percentage of question score.
Code Snippet:
<pre>
describe '#calculate_avg_score_by_criterion' do
context 'when question_answers are available' do
it 'calculate percentage question_score & no float' do
expect(@summary.calculate_avg_score_by_criterion([answer,answer1], 3)).to be_within(0).of(50)
end
end
</pre>

This test is to see when question answer was nil, the output of method is correctly calculated as 0.
Code Snippet:
<pre>
context 'when question_answers are not available' do
it 'gives question scores 0.0' do
expect(@summary.calculate_avg_score_by_criterion([], 3)).to eq(0.0)
end
end
</pre>

This test is to see when q_max_score = 0, the output of method is correctly calculated.
Code Snippet:
<pre>
context 'when q_max_score = 0' do
it 'gives pure question_score' do
expect(@summary.calculate_avg_score_by_criterion([answer,answer1], 0)).to eq(3.0)
end
end
end
</pre>

* calculate_round_score
The test is to see when the criteria input is nil, the method gives right output of nil, which is 0 in this case.
Code Snippet:
<pre>
describe '#calculate_round_score' do
context 'when criteria not available' do
it 'returns 0.0 since round_score = 0.0' do
expect(@summary.calculate_round_score(avg_scores_by_criterion, nil)).to eq(0.to_f)
end
end
</pre>

The test is to see when the criteria input is not nil, the method gives the right output, which is to get a two-digit round score. The criteria input was defined as question in mock section.
Code Snippet:
<pre>
context 'when criteria not nil' do
it 'get 2 round_score ' do
expect(@summary.calculate_round_score(avg_scores_by_criterion, question)).to be_within(0.01).of(2.345)
end
end
end
</pre>

* calculate_avg_score_by_round
This test is to see the method round an average number up to Two-digit. The input of avg_scores_by_criterion was given as 2.345, and we tested here that it gives two-digit round number as 2.35.
Code Snippet:
<pre>
describe '#calculate_avg_score_by_round'do
context 'when avg_scores_by_criterion available' do
it 'gives 2 round value' do
expect(@summary.calculate_avg_score_by_round(avg_scores_by_criterion, question)).to eq(2.35)
end
end
end
</pre>

=== Test Execution ===
We divided the work among the teammates and started tackling the problems. We stubbed the data using factory and mocked the method calls which were being done internally to get the desired output from the methods that were calling other methods internally.

=== Test Coverage ===
Our test coverage shows in [https://coveralls.io/builds/47520691/source?filename=app%2Fhelpers%2Fsummary_helper.rb coverails].

=== Conclusion ===
While we tried to test all the methods in the summary_helper.rb class, we faced some blockers. All the blockers are described below:-
* summarize_sentences method

summary_webservice_url: 'http://peerlogic.csc.ncsu.edu/sum/v1.0/summary/8/lsa'
This URL is being used in the summary_helper.rb file and we were not able to mock it. It's giving us bad gateway error while we try to hit that url.

* summarize_reviews_by_reviewee method

In this method we found out in the loop, questions[round] is not the correct way of passing the individual question since questions is an array and use 0 based indexing. Whereas in the above logic it is being treated as a hash.

* summarize_reviews_by_reviewee_question method

This method uses the instance variables from a different method(summarize_reviews_by_reviewee) which are not being passed as arguments to this method. Technically this type of initialization is bad coding practice and hence needs refratoring.

We already informed our Mentor and professor regarding these issues and they said it will be included in future work.

We tested all other remaining methods in the summary_helper.rb class. All the unit tests passed and the methods were working as expected. We tried to cover corner cases but I think there is room for some improvement.

CSC/ECE 517 Spring 2022 - E2211: Testing for summary helper

2022-03-28T00:28:36Z

Spriyad2:

== About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an assignment/project management portal that can be used by both instructors and students for collaborative learning and feedback. It is an open-source project based on [http://rubyonrails.org/ Ruby on Rails] framework. It allows the instructors not only to create and customize new or existing assignments but also to create a list of topics the students can sign up for. Students can form teams to work on various projects and assignments. Expertiza also lets students peer-review other students' submissions, enabling them to work together to improve others' learning experiences.

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

* Saswat Priyadarshan
* Rachel Son
* Bhuwan Bhatt

== Description and Testing ==
Our project deals with writing test cases for the summary_helper.rb file. The summary helper is responsible for doing the following tasks:
* Get all answers for each question and send them to summarization WS.
* Get average scores and a summary for each question in a review by a reviewer.

=== Files Involved ===

* summary_helper.rb
* summary_helper_spec.rb

=== Running Tests ===
To successfully run summary_helper_test in local machine, please run the below rspec command.
<pre>
rspec spec/helpers/summary_helper_spec.rb
</pre>

=== Requirement ===

summary_helper class had methods which were not tested. Our job was to test all the methods so that we can verify summary_helper class is working fine.

=== Test Plan ===

After going through all the methods in the class, we figured out there were some methods that were never being called. So, we removed all the redundant methods that are not being called. Now we had only 9 methods in the class that we need to test. Nine methods in summary_helper class that needed to be tested were:

* summarize_reviews_by_reviewee
* summarize_reviews_by_reviewee_question
* get_max_score_for_question
* summarize_sentences
* get_sentences
* break_up_comments_to_sentences
* calculate_avg_score_by_criterion
* calculate_round_score
* calculate_avg_score_by_round
* get_sentences

For the testing, we have mock following items :
<pre>
let(:answer) { Answer.new(answer: 1, comments: 'This is a sentence. This is another sentence.', question_id: 1) }
let(:answer1){ Answer.new(answer: 2, comments: 'This is a sentence1. This is another sentence1.', question_id: 2)}
let(:question) {build(:question, weight:1, type:"Criterion")}
let(:avg_scores_by_criterion) { {a:2.345} }
</pre>

* get_sentences
To test this method we mocked an answer object and checked if the number of sentences is equal to the number of comments provided in the answer.
We identified two test cases for this method:
1. When the answer is nil
2. When the comment is two sentences

Code Snippet:
<pre>
describe '#get_sentences' do
context 'when the answer is nil' do
it 'returns a nil object' do
expect(@summary.get_sentences(nil)).to eq(nil)
end
end
context 'when the comment is two sentences' do
it 'returns an array of two sentences' do
sentences = @summary.get_sentences(answer)
expect(sentences.length).to be(2)
end
end
end
</pre>

* get_max_score_for_question
To test this method we mocked two different types of questions which are as follows:
1. let(:questionOne){Question.new(type:'Checkbox')}
2. let(:questionTwo) { build(:question, questionnaire: questionnaire1, weight: 1, id: 1) }
Now based on the different question types we tested two scenarios for this method:
1. When the question type is Checkbox
2. When question type is not Checkbox

Code Snippet:
<pre>
describe 'get_max_score_for_question' do
context 'When question type is Checkbox' do
let(:questionOne){Question.new(type:'Checkbox')}
it 'returns 1' do
max_score = @summary.get_max_score_for_question(questionOne)
expect(max_score).to be(1)
end
end
context 'When question type is not Checkbox' do
let(:questionnaire1) { build(:questionnaire, id: 2) }
let(:questionTwo) { build(:question, questionnaire: questionnaire1, weight: 1, id: 1) }
it 'return the max score for the provided question' do
allow(Questionnaire).to receive(:where).with(id:2).and_return(questionnaire1)
allow(questionnaire1).to receive(:first).and_return(questionnaire1)
expect(@summary.get_max_score_for_question(questionTwo)).to eql(5)
end
end
end

</pre>

* summarize_sentence
To test this method we mocked a comments array with the following values ["Hello this is first comment", "This is second comment"]. This method is making a web service call at the following Web Service:
'http://peerlogic.csc.ncsu.edu/sum/v1.0/summary/8/lsa'
We are using expect to compare the expected result with the actual result from the web service call.
Disclaimer: This web service call results in a bad gateway error while running the test case. We have already informed our mentor and the professor regarding the same.

Code Snippet:
<pre>
describe '#summarize_sentence' do
context 'successful webservice call' do
comments = ["Hello this is first comment", "This is second comment"]
summary_ws_url = WEBSERVICE_CONFIG['summary_webservice_url']
it 'return success' do
expect(@summary.summarize_sentences(comments,summary_ws_url)).not_to eql(nil)
end
end
end
</pre>

* break_up_comments_to_sentences
This methods question answer array and breaks them into comments.
In our testing we provided the question_answer array of length 2 and see if the number of comments is equal to 2.
Code Snippet:
<pre>
describe '#break_up_comments_to_sentences' do
context 'when the question_answers is not nil' do
it 'add the comment to an array to be converted as a json request' do
comments = @summary.break_up_comments_to_sentences([answer])
expect(comments.length).to be(2)
end
end
context 'when the question_answers is nil' do
it 'returns an empty array' do
comments = @summary.break_up_comments_to_sentences([])
expect(comments.length).to be(0)
end
end
end
</pre>

* calculate_avg_score_by_criterion
This test is to see when question answer was given, the output of method is correctly calculated as percentage of question score.
Code Snippet:
<pre>
describe '#calculate_avg_score_by_criterion' do
context 'when question_answers are available' do
it 'calculate percentage question_score & no float' do
expect(@summary.calculate_avg_score_by_criterion([answer,answer1], 3)).to be_within(0).of(50)
end
end
</pre>

This test is to see when question answer was nil, the output of method is correctly calculated as 0.
Code Snippet:
<pre>
context 'when question_answers are not available' do
it 'gives question scores 0.0' do
expect(@summary.calculate_avg_score_by_criterion([], 3)).to eq(0.0)
end
end
</pre>

This test is to see when q_max_score = 0, the output of method is correctly calculated.
Code Snippet:
<pre>
context 'when q_max_score = 0' do
it 'gives pure question_score' do
expect(@summary.calculate_avg_score_by_criterion([answer,answer1], 0)).to eq(3.0)
end
end
end
</pre>

* calculate_round_score
The test is to see when the criteria input is nil, the method gives right output of nil, which is 0 in this case.
Code Snippet:
<pre>
describe '#calculate_round_score' do
context 'when criteria not available' do
it 'returns 0.0 since round_score = 0.0' do
expect(@summary.calculate_round_score(avg_scores_by_criterion, nil)).to eq(0.to_f)
end
end
</pre>

The test is to see when the criteria input is not nil, the method gives the right output, which is to get a two-digit round score. The criteria input was defined as question in mock section.
Code Snippet:
<pre>
context 'when criteria not nil' do
it 'get 2 round_score ' do
expect(@summary.calculate_round_score(avg_scores_by_criterion, question)).to be_within(0.01).of(2.345)
end
end
end
</pre>

* calculate_avg_score_by_round
This test is to see the method round an average number up to Two-digit. The input of avg_scores_by_criterion was given as 2.345, and we tested here that it gives two-digit round number as 2.35.
Code Snippet:
<pre>
describe '#calculate_avg_score_by_round'do
context 'when avg_scores_by_criterion available' do
it 'gives 2 round value' do
expect(@summary.calculate_avg_score_by_round(avg_scores_by_criterion, question)).to eq(2.35)
end
end
end
</pre>

=== Test Execution ===
We divided the work among the teammates and started tackling the problems. We stubbed the data using factory and mocked the method calls which were being done internally to get the desired output from the methods that were calling other methods internally.

=== Test Coverage ===
Our test coverage shows in [https://coveralls.io/builds/47520691/source?filename=app%2Fhelpers%2Fsummary_helper.rb coverails].

=== Conclusion ===
While we tried to test all the methods in the summary_helper.rb class, we faced some blockers. All the blockers are described below:-
* summarize_sentences method

summary_webservice_url: 'http://peerlogic.csc.ncsu.edu/sum/v1.0/summary/8/lsa'
This URL is being used in the summary_helper.rb file and we were not able to mock it. It's giving us bad gateway error while we try to hit that url.

* summarize_reviews_by_reviewee method

In this method we found out in the loop, questions[round] is not the correct way of passing the individual question since questions is an array and use 0 based indexing. Whereas in the above logic it is being treated as a hash.

* summarize_reviews_by_reviewee_question method

We were not able to write test cases for this method. This method uses the instance variables from a different method(summarize_reviews_by_reviewee) which are not being passed as arguments to this method. Hence we were not able to initialize those variables and it always gives nil value.

We tested all other remaining methods in the summary_helper.rb class. All the unit tests passed and the methods were working as expected. We tried to cover corner cases but I think there is room for some improvement.

CSC/ECE 517 Spring 2022 - E2211: Testing for summary helper

2022-03-28T00:28:17Z

Spriyad2:

== About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an assignment/project management portal that can be used by both instructors and students for collaborative learning and feedback. It is an open-source project based on [http://rubyonrails.org/ Ruby on Rails] framework. It allows the instructors not only to create and customize new or existing assignments but also to create a list of topics the students can sign up for. Students can form teams to work on various projects and assignments. Expertiza also lets students peer-review other students' submissions, enabling them to work together to improve others' learning experiences.

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

* Saswat Priyadarshan
* Rachel Son
* Bhuwan Bhatt

== Description and Testing ==
Our project deals with writing test cases for the summary_helper.rb file. The summary helper is responsible for doing the following tasks:
* Get all answers for each question and send them to summarization WS.
* Get average scores and a summary for each question in a review by a reviewer.

=== Files Involved ===

* summary_helper.rb
* summary_helper_spec.rb

=== Running Tests ===
To successfully run summary_helper_test in local machine, please run the below rspec command.
<pre>
rspec spec/helpers/summary_helper_spec.rb
</pre>

=== Requirement ===

summary_helper class had methods which were not tested. Our job was to test all the methods so that we can verify summary_helper class is working fine.

=== Test Plan ===

After going through all the methods in the class, we figured out there were some methods that were never being called. So, we removed all the redundant methods that are not being called. Now we had only 9 methods in the class that we need to test. Nine methods in summary_helper class that needed to be tested were:

* summarize_reviews_by_reviewee
* summarize_reviews_by_reviewee_question
* get_max_score_for_question
* summarize_sentences
* get_sentences
* break_up_comments_to_sentences
* calculate_avg_score_by_criterion
* calculate_round_score
* calculate_avg_score_by_round
* get_sentences

For the testing, we have mock following items :
<pre>
let(:answer) { Answer.new(answer: 1, comments: 'This is a sentence. This is another sentence.', question_id: 1) }
let(:answer1){ Answer.new(answer: 2, comments: 'This is a sentence1. This is another sentence1.', question_id: 2)}
let(:question) {build(:question, weight:1, type:"Criterion")}
let(:avg_scores_by_criterion) { {a:2.345} }
</pre>

* get_sentences
To test this method we mocked an answer object and checked if the number of sentences is equal to the number of comments provided in the answer.
We identified two test cases for this method:
1. When the answer is nil
2. When the comment is two sentences

Code Snippet:
<pre>
describe '#get_sentences' do
context 'when the answer is nil' do
it 'returns a nil object' do
expect(@summary.get_sentences(nil)).to eq(nil)
end
end
context 'when the comment is two sentences' do
it 'returns an array of two sentences' do
sentences = @summary.get_sentences(answer)
expect(sentences.length).to be(2)
end
end
end
</pre>

* get_max_score_for_question
To test this method we mocked two different types of questions which are as follows:
1. let(:questionOne){Question.new(type:'Checkbox')}
2. let(:questionTwo) { build(:question, questionnaire: questionnaire1, weight: 1, id: 1) }
Now based on the different question types we tested two scenarios for this method:
1. When the question type is Checkbox
2. When question type is not Checkbox

Code Snippet:
<pre>
describe 'get_max_score_for_question' do
context 'When question type is Checkbox' do
let(:questionOne){Question.new(type:'Checkbox')}
it 'returns 1' do
max_score = @summary.get_max_score_for_question(questionOne)
expect(max_score).to be(1)
end
end
context 'When question type is not Checkbox' do
let(:questionnaire1) { build(:questionnaire, id: 2) }
let(:questionTwo) { build(:question, questionnaire: questionnaire1, weight: 1, id: 1) }
it 'return the max score for the provided question' do
allow(Questionnaire).to receive(:where).with(id:2).and_return(questionnaire1)
allow(questionnaire1).to receive(:first).and_return(questionnaire1)
expect(@summary.get_max_score_for_question(questionTwo)).to eql(5)
end
end
end

</pre>

* summarize_sentence
To test this method we mocked a comments array with the following values ["Hello this is first comment", "This is second comment"]. This method is making a web service call at the following Web Service:
'http://peerlogic.csc.ncsu.edu/sum/v1.0/summary/8/lsa'
We are using expect to compare the expected result with the actual result from the web service call.
Disclaimer: This web service call results in a bad gateway error while running the test case. We have already informed our mentor and the professor regarding the same.

Code Snippet:
<pre>
describe '#summarize_sentence' do
context 'successful webservice call' do
comments = ["Hello this is first comment", "This is second comment"]
summary_ws_url = WEBSERVICE_CONFIG['summary_webservice_url']
it 'return success' do
expect(@summary.summarize_sentences(comments,summary_ws_url)).not_to eql(nil)
end
end
end
</pre>

* break_up_comments_to_sentences
This methods question answer array and breaks them into comments.
In our testing we provided the question_answer array of length 2 and see if the number of comments is equal to 2.
Code Snippet:
<pre>
describe '#break_up_comments_to_sentences' do
context 'when the question_answers is not nil' do
it 'add the comment to an array to be converted as a json request' do
comments = @summary.break_up_comments_to_sentences([answer])
expect(comments.length).to be(2)
end
end
context 'when the question_answers is nil' do
it 'returns an empty array' do
comments = @summary.break_up_comments_to_sentences([])
expect(comments.length).to be(0)
end
end
end
</pre>

* calculate_avg_score_by_criterion
This test is to see when question answer was given, the output of method is correctly calculated as percentage of question score.
Code Snippet:
<pre>
describe '#calculate_avg_score_by_criterion' do
context 'when question_answers are available' do
it 'calculate percentage question_score & no float' do
expect(@summary.calculate_avg_score_by_criterion([answer,answer1], 3)).to be_within(0).of(50)
end
end
</pre>

This test is to see when question answer was nil, the output of method is correctly calculated as 0.
Code Snippet:
<pre>
context 'when question_answers are not available' do
it 'gives question scores 0.0' do
expect(@summary.calculate_avg_score_by_criterion([], 3)).to eq(0.0)
end
end
</pre>

This test is to see when q_max_score = 0, the output of method is correctly calculated.
Code Snippet:
<pre>
context 'when q_max_score = 0' do
it 'gives pure question_score' do
expect(@summary.calculate_avg_score_by_criterion([answer,answer1], 0)).to eq(3.0)
end
end
end
</pre>

* calculate_round_score
The test is to see when the criteria input is nil, the method gives right output of nil, which is 0 in this case.
Code Snippet:
<pre>
describe '#calculate_round_score' do
context 'when criteria not available' do
it 'returns 0.0 since round_score = 0.0' do
expect(@summary.calculate_round_score(avg_scores_by_criterion, nil)).to eq(0.to_f)
end
end
</pre>

The test is to see when the criteria input is not nil, the method gives the right output, which is to get a two-digit round score. The criteria input was defined as question in mock section.
Code Snippet:
<pre>
context 'when criteria not nil' do
it 'get 2 round_score ' do
expect(@summary.calculate_round_score(avg_scores_by_criterion, question)).to be_within(0.01).of(2.345)
end
end
end
</pre>

* calculate_avg_score_by_round
This test is to see the method round an average number up to Two-digit. The input of avg_scores_by_criterion was given as 2.345, and we tested here that it gives two-digit round number as 2.35.
Code Snippet:
<pre>
describe '#calculate_avg_score_by_round'do
context 'when avg_scores_by_criterion available' do
it 'gives 2 round value' do
expect(@summary.calculate_avg_score_by_round(avg_scores_by_criterion, question)).to eq(2.35)
end
end
end
</pre>

=== Test Execution ===
We divided the work among the teammates and started tackling the problems. We stubbed the data using factory and mocked the method calls which were being done internally to get the desired output from the methods that were calling other methods internally.

=== Test Coverage ===
Our test coverage shows in [https://coveralls.io/builds/47520691/source?filename=app%2Fhelpers%2Fsummary_helper.rb coverails].

=== Conclusion ===
While we tried to test all the methods in the summary_helper.rb class, we faced some blockers. All the blockers are described below:-
* summarize_sentences method

summary_webservice_url: 'http://peerlogic.csc.ncsu.edu/sum/v1.0/summary/8/lsa'
This URL is being used in the summary_helper.rb file and we were not able to mock it. It's giving us bad gateway error while we try to hit that url.

* summarize_reviews_by_reviewee method

In this method we found out in the loop, questions[round] is not the correct way of passing the individual question since questions is an array and use 0 based indexing. Whereas in the above logic it is being treated as a hash.

* summarize_reviews_by_reviewee_question method

We were not able to write test cases for this method. This method uses the instance variables from a different method(summarize_reviews_by_reviewee) which are not being passed as arguments to this method. Hence we were not able to initialize those variables and it always gives nil value.

We tested all other remaining methods in the summary_helper.rb class. All the unit tests passed and the methods were working as expected. We tried to cover corner cases but I think there is room for some improvement.

CSC/ECE 517 Spring 2022 - E2211: Testing for summary helper

2022-03-28T00:25:49Z

Spriyad2:

== About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an assignment/project management portal that can be used by both instructors and students for collaborative learning and feedback. It is an open-source project based on [http://rubyonrails.org/ Ruby on Rails] framework. It allows the instructors not only to create and customize new or existing assignments but also to create a list of topics the students can sign up for. Students can form teams to work on various projects and assignments. Expertiza also lets students peer-review other students' submissions, enabling them to work together to improve others' learning experiences.

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

* Saswat Priyadarshan
* Rachel Son
* Bhuwan Bhatt

== Description and Testing ==
Our project deals with writing test cases for the summary_helper.rb file. The summary helper is responsible for doing the following tasks:
* Get all answers for each question and send them to summarization WS.
* Get average scores and a summary for each question in a review by a reviewer.

=== Files Involved ===

* summary_helper.rb
* summary_helper_spec.rb

=== Running Tests ===
To successfully run summary_helper_test in local machine, please run the below rspec command.
<pre>
rspec spec/helpers/summary_helper_spec.rb
</pre>

=== Requirement ===

summary_helper class had methods which were not tested. Our job was to test all the methods so that we can verify summary_helper class is working fine.

=== Test Plan ===

After going through all the methods in the class, we figured out there were some methods that were never being called. So, we removed all the redundant methods that are not being called. Now we had only 9 methods in the class that we need to test. Nine methods in summary_helper class that needed to be tested were:

* summarize_reviews_by_reviewee
* summarize_reviews_by_reviewee_question
* get_max_score_for_question
* summarize_sentences
* get_sentences
* break_up_comments_to_sentences
* calculate_avg_score_by_criterion
* calculate_round_score
* calculate_avg_score_by_round
* get_sentences

For the testing, we have mock following items :
<pre>
let(:answer) { Answer.new(answer: 1, comments: 'This is a sentence. This is another sentence.', question_id: 1) }
let(:answer1){ Answer.new(answer: 2, comments: 'This is a sentence1. This is another sentence1.', question_id: 2)}
let(:question) {build(:question, weight:1, type:"Criterion")}
let(:avg_scores_by_criterion) { {a:2.345} }
</pre>

* get_sentences
To test this method we mocked an answer object and checked if the number of sentences is equal to the number of comments provided in the answer.
We identified two test cases for this method:
1. When the answer is nil
2. When the comment is two sentences

Code Snippet:
<pre>
describe '#get_sentences' do
context 'when the answer is nil' do
it 'returns a nil object' do
expect(@summary.get_sentences(nil)).to eq(nil)
end
end
context 'when the comment is two sentences' do
it 'returns an array of two sentences' do
sentences = @summary.get_sentences(answer)
expect(sentences.length).to be(2)
end
end
end

</pre>

* get_max_score_for_question
To test this method we mocked two different types of questions which are as follows:
1. let(:questionOne){Question.new(type:'Checkbox')}
2. let(:questionTwo) { build(:question, questionnaire: questionnaire1, weight: 1, id: 1) }
Now based on the different question types we tested two scenarios for this method:
1. When the question type is Checkbox
2. When question type is not Checkbox

Code Snippet:
<pre>
describe 'get_max_score_for_question' do
context 'When question type is Checkbox' do
let(:questionOne){Question.new(type:'Checkbox')}
it 'returns 1' do
max_score = @summary.get_max_score_for_question(questionOne)
expect(max_score).to be(1)
end
end
context 'When question type is not Checkbox' do
let(:questionnaire1) { build(:questionnaire, id: 2) }
let(:questionTwo) { build(:question, questionnaire: questionnaire1, weight: 1, id: 1) }
it 'return the max score for the provided question' do
allow(Questionnaire).to receive(:where).with(id:2).and_return(questionnaire1)
allow(questionnaire1).to receive(:first).and_return(questionnaire1)
expect(@summary.get_max_score_for_question(questionTwo)).to eql(5)
end
end
end

</pre>

* summarize_sentence
To test this method we mocked a comments array with the following values ["Hello this is first comment", "This is second comment"]. This method is making a web service call at the following Web Service:
'http://peerlogic.csc.ncsu.edu/sum/v1.0/summary/8/lsa'
We are using expect to compare the expected result with the actual result from the web service call.
Disclaimer: This web service call results in a bad gateway error while running the test case. We have already informed our mentor and the professor regarding the same.

Code Snippet:
<pre>
describe '#summarize_sentence' do
context 'successful webservice call' do
comments = ["Hello this is first comment", "This is second comment"]
summary_ws_url = WEBSERVICE_CONFIG['summary_webservice_url']
it 'return success' do
expect(@summary.summarize_sentences(comments,summary_ws_url)).not_to eql(nil)
end
end
end
</pre>

* break_up_comments_to_sentences
This methods question answer array and breaks them into comments.
In our testing we provided the question_answer array of length 2 and see if the number of comments is equal to 2.
Code Snippet:
<pre>
describe '#break_up_comments_to_sentences' do
context 'when the question_answers is not nil' do
it 'add the comment to an array to be converted as a json request' do
comments = @summary.break_up_comments_to_sentences([answer])
expect(comments.length).to be(2)
end
end
context 'when the question_answers is nil' do
it 'returns an empty array' do
comments = @summary.break_up_comments_to_sentences([])
expect(comments.length).to be(0)
end
end
end
</pre>

* calculate_avg_score_by_criterion
This test is to see when question answer was given, the output of method is correctly calculated as percentage of question score.
Code Snippet:
<pre>
describe '#calculate_avg_score_by_criterion' do
context 'when question_answers are available' do
it 'calculate percentage question_score & no float' do
expect(@summary.calculate_avg_score_by_criterion([answer,answer1], 3)).to be_within(0).of(50)
end
end
</pre>

This test is to see when question answer was nil, the output of method is correctly calculated as 0.
Code Snippet:
<pre>
context 'when question_answers are not available' do
it 'gives question scores 0.0' do
expect(@summary.calculate_avg_score_by_criterion([], 3)).to eq(0.0)
end
end
</pre>

This test is to see when q_max_score = 0, the output of method is correctly calculated.
Code Snippet:
<pre>
context 'when q_max_score = 0' do
it 'gives pure question_score' do
expect(@summary.calculate_avg_score_by_criterion([answer,answer1], 0)).to eq(3.0)
end
end
end
</pre>

* calculate_round_score
The test is to see when the criteria input is nil, the method gives right output of nil, which is 0 in this case.
Code Snippet:
<pre>
describe '#calculate_round_score' do
context 'when criteria not available' do
it 'returns 0.0 since round_score = 0.0' do
expect(@summary.calculate_round_score(avg_scores_by_criterion, nil)).to eq(0.to_f)
end
end
</pre>

The test is to see when the criteria input is not nil, the method gives the right output, which is to get a two-digit round score. The criteria input was defined as question in mock section.
Code Snippet:
<pre>
context 'when criteria not nil' do
it 'get 2 round_score ' do
expect(@summary.calculate_round_score(avg_scores_by_criterion, question)).to be_within(0.01).of(2.345)
end
end
end
</pre>

* calculate_avg_score_by_round
This test is to see the method round an average number up to Two-digit. The input of avg_scores_by_criterion was given as 2.345, and we tested here that it gives two-digit round number as 2.35.
Code Snippet:
<pre>
describe '#calculate_avg_score_by_round'do
context 'when avg_scores_by_criterion available' do
it 'gives 2 round value' do
expect(@summary.calculate_avg_score_by_round(avg_scores_by_criterion, question)).to eq(2.35)
end
end
end
</pre>

=== Test Execution ===
We divided the work among the teammates and started tackling the problems. We stubbed the data using factory and mocked the method calls which were being done internally to get the desired output from the methods that were calling other methods internally.

=== Test Coverage ===
Our test coverage shows in [https://coveralls.io/builds/47520691/source?filename=app%2Fhelpers%2Fsummary_helper.rb coverails].

=== Conclusion ===
While we tried to test all the methods in the summary_helper.rb class, we faced some blockers. All the blockers are described below:-
* summarize_sentences method

summary_webservice_url: 'http://peerlogic.csc.ncsu.edu/sum/v1.0/summary/8/lsa'
This URL is being used in the summary_helper.rb file and we were not able to mock it. It's giving us bad gateway error while we try to hit that url.

* summarize_reviews_by_reviewee method

In this method we found out in the loop, questions[round] is not the correct way of passing the individual question since questions is an array and use 0 based indexing. Whereas in the above logic it is being treated as a hash.

* summarize_reviews_by_reviewee_question method

We were not able to write test cases for this method. This method uses the instance variables from a different method(summarize_reviews_by_reviewee) which are not being passed as arguments to this method. Hence we were not able to initialize those variables and it always gives nil value.

We tested all other remaining methods in the summary_helper.rb class. All the unit tests passed and the methods were working as expected. We tried to cover corner cases but I think there is room for some improvement.

CSC/ECE 517 Spring 2022 - E2211: Testing for summary helper

2022-03-28T00:18:17Z

Spriyad2:

== About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an assignment/project management portal that can be used by both instructors and students for collaborative learning and feedback. It is an open-source project based on [http://rubyonrails.org/ Ruby on Rails] framework. It allows the instructors not only to create and customize new or existing assignments but also to create a list of topics the students can sign up for. Students can form teams to work on various projects and assignments. Expertiza also lets students peer-review other students' submissions, enabling them to work together to improve others' learning experiences.

== Team==
=== Mentor ===

* Vinay Deshmukh

=== Team Members ===

* Saswat Priyadarshan
* Rachel Son
* Bhuwan Bhatt

== Description and Testing ==
Our project deals with writing test cases for the summary_helper.rb file. The summary helper is responsible for doing the following tasks:
* Get all answers for each question and send them to summarization WS.
* Get average scores and a summary for each question in a review by a reviewer.

=== Files Involved ===

* summary_helper.rb
* summary_helper_spec.rb

=== Running Tests ===
To successfully run summary_helper_test in local machine, please run the below rspec command.
<pre>
rspec spec/helpers/summary_helper_spec.rb
</pre>

=== Requirement ===

summary_helper class had methods which were not tested. Our job was to test all the methods so that we can verify summary_helper class is working fine.

=== Test Plan ===

After going through all the methods in the class, we figured out there were some methods that were never being called. So, we removed all the redundant methods that are not being called. Now we had only 9 methods in the class that we need to test. Nine methods in summary_helper class that needed to be tested were:

* summarize_reviews_by_reviewee
* summarize_reviews_by_reviewee_question
* get_max_score_for_question
* summarize_sentences
* get_sentences
* break_up_comments_to_sentences
* calculate_avg_score_by_criterion
* calculate_round_score
* calculate_avg_score_by_round
* get_sentences

For the testing, we have mock following items :
<pre>
let(:answer) { Answer.new(answer: 1, comments: 'This is a sentence. This is another sentence.', question_id: 1) }
let(:answer1){ Answer.new(answer: 2, comments: 'This is a sentence1. This is another sentence1.', question_id: 2)}
let(:question) {build(:question, weight:1, type:"Criterion")}
let(:avg_scores_by_criterion) { {a:2.345} }
</pre>

* get_sentences
To test this method we mocked an answer object and checked if the number of sentences is equal to the number of comments provided in the answer.
We identified two test cases for this method:
1. When the answer is nil
2. When the comment is two sentences

Code Snippet:
<pre>
describe '#get_sentences' do
context 'when the answer is nil' do
it 'returns a nil object' do
expect(@summary.get_sentences(nil)).to eq(nil)
end
end
context 'when the comment is two sentences' do
it 'returns an array of two sentences' do
sentences = @summary.get_sentences(answer)
expect(sentences.length).to be(2)
end
end
end

</pre>

* get_max_score_for_question
To test this method we mocked two different types of questions which are as follows:
1. let(:questionOne){Question.new(type:'Checkbox')}
2. let(:questionTwo) { build(:question, questionnaire: questionnaire1, weight: 1, id: 1) }
Now based on the different question types we tested two scenarios for this method:
1. When the question type is Checkbox
2. When question type is not Checkbox

Code Snippet:
<pre>
describe 'get_max_score_for_question' do
context 'When question type is Checkbox' do
let(:questionOne){Question.new(type:'Checkbox')}
it 'returns 1' do
max_score = @summary.get_max_score_for_question(questionOne)
expect(max_score).to be(1)
end
end
context 'When question type is not Checkbox' do
let(:questionnaire1) { build(:questionnaire, id: 2) }
let(:questionTwo) { build(:question, questionnaire: questionnaire1, weight: 1, id: 1) }
it 'return the max score for the provided question' do
allow(Questionnaire).to receive(:where).with(id:2).and_return(questionnaire1)
allow(questionnaire1).to receive(:first).and_return(questionnaire1)
expect(@summary.get_max_score_for_question(questionTwo)).to eql(5)
end
end
end

</pre>

* break_up_comments_to_sentences
This methods question answer array and breaks them into comments.
In our testing we provided the question_answer array of length 2 and see if the number of comments is equal to 2.
Code Snippet:
<pre>
describe '#break_up_comments_to_sentences' do
context 'when the question_answers is not nil' do
it 'add the comment to an array to be converted as a json request' do
comments = @summary.break_up_comments_to_sentences([answer])
expect(comments.length).to be(2)
end
end
context 'when the question_answers is nil' do
it 'returns an empty array' do
comments = @summary.break_up_comments_to_sentences([])
expect(comments.length).to be(0)
end
end
end
</pre>

* calculate_avg_score_by_criterion
This test is to see when question answer was given, the output of method is correctly calculated as percentage of question score.
Code Snippet:
<pre>
describe '#calculate_avg_score_by_criterion' do
context 'when question_answers are available' do
it 'calculate percentage question_score & no float' do
expect(@summary.calculate_avg_score_by_criterion([answer,answer1], 3)).to be_within(0).of(50)
end
end
</pre>

This test is to see when question answer was nil, the output of method is correctly calculated as 0.
Code Snippet:
<pre>
context 'when question_answers are not available' do
it 'gives question scores 0.0' do
expect(@summary.calculate_avg_score_by_criterion([], 3)).to eq(0.0)
end
end
</pre>

This test is to see when q_max_score = 0, the output of method is correctly calculated.
Code Snippet:
<pre>
context 'when q_max_score = 0' do
it 'gives pure question_score' do
expect(@summary.calculate_avg_score_by_criterion([answer,answer1], 0)).to eq(3.0)
end
end
end
</pre>

* calculate_round_score
The test is to see when the criteria input is nil, the method gives right output of nil, which is 0 in this case.
Code Snippet:
<pre>
describe '#calculate_round_score' do
context 'when criteria not available' do
it 'returns 0.0 since round_score = 0.0' do
expect(@summary.calculate_round_score(avg_scores_by_criterion, nil)).to eq(0.to_f)
end
end
</pre>

The test is to see when the criteria input is not nil, the method gives the right output, which is to get a two-digit round score. The criteria input was defined as question in mock section.
Code Snippet:
<pre>
context 'when criteria not nil' do
it 'get 2 round_score ' do
expect(@summary.calculate_round_score(avg_scores_by_criterion, question)).to be_within(0.01).of(2.345)
end
end
end
</pre>

* calculate_avg_score_by_round
This test is to see the method round an average number up to Two-digit. The input of avg_scores_by_criterion was given as 2.345, and we tested here that it gives two-digit round number as 2.35.
Code Snippet:
<pre>
describe '#calculate_avg_score_by_round'do
context 'when avg_scores_by_criterion available' do
it 'gives 2 round value' do
expect(@summary.calculate_avg_score_by_round(avg_scores_by_criterion, question)).to eq(2.35)
end
end
end
</pre>

=== Test Execution ===
We divided the work among the teammates and started tackling the problems. We stubbed the data using factory and mocked the method calls which were being done internally to get the desired output from the methods that were calling other methods internally.

=== Test Coverage ===
Our test coverage shows in [https://coveralls.io/builds/47520691/source?filename=app%2Fhelpers%2Fsummary_helper.rb coverails].

=== Conclusion ===
While we tried to test all the methods in the summary_helper.rb class, we faced some blockers. All the blockers are described below:-
* summarize_sentences method

summary_webservice_url: 'http://peerlogic.csc.ncsu.edu/sum/v1.0/summary/8/lsa'
This URL is being used in the summary_helper.rb file and we were not able to mock it. It's giving us bad gateway error while we try to hit that url.

* summarize_reviews_by_reviewee method

In this method we found out in the loop, questions[round] is not the correct way of passing the individual question since questions is an array and use 0 based indexing. Whereas in the above logic it is being treated as a hash.

* summarize_reviews_by_reviewee_question method

We were not able to write test cases for this method. This method uses the instance variables from a different method(summarize_reviews_by_reviewee) which are not being passed as arguments to this method. Hence we were not able to initialize those variables and it always gives nil value.

We tested all other remaining methods in the summary_helper.rb class. All the unit tests passed and the methods were working as expected. We tried to cover corner cases but I think there is room for some improvement.