CSC/ECE 517 Spring 2022 - E2238. Implementing and testing Import & Export controllers

Team

Mentor

• Vinay Deshmukh

Team Members

• Bollineni, Rachana Sri rbollin@ncsu.edu
• Nangia, Chirrag - cnangia@ncsu.edu
• Priyadarshan, Saswat - spriyad2@ncsu.edu
• Vemuri, Pradyumna pvemuri@ncsu.edu

Project Purpose & Description

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from the database and saves it as a file in the desired format. However, the same functionality is implemented multiple times for exporting and importing different things. The aim of this project is to design a generic export/import feature that can be used across the application. This will help in maintaining the codebase and keep things consistent throughout.

Current Project Scope

A generic import_file_controller has already been experimented with in Project E1923: https://github.com/expertiza/expertiza/pull/1438 But as discussed with the professor and the mentors, the PR hasn’t been merged, and it’s not known whether the code was correct or it had issues. And given the existing code has lots of if-else, it would be better to rewrite it from scratch.

Below listed are our primary goals for this project:

• Redo the code for import_file_controller.rb and export_file_controller.rb, by referring to the code in the pull request. Our plan of action is to rewrite the code than try to refactor it, as it’s not known whether the code in the PR is correct.
• Write test cases for the new import_file_controller and the existing export_file_controller.
• Find and fix the existing bugs in the import and export process flows.

Extended Project Scope

Below listed is the stretch goal for this project. We'll try to pick it only if the time permits.

• After the functionality is fixed, the next step is to leverage the import and export controllers for importing/exporting users, topics, teams, questionnaires, and grades. Initially, High Priority modules are users, topics, and grades, and then move on to the other modules.

Existing Import/Export Controller design

The existing import and export functionality primarily uses the import_file_controller and export_file_controller.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

The questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

Controllers

• import_file_controller, the list of methods in the controller are the following:
  • File processing methods:
    • #get_delimiter - Sets proper delimiter for filetype
    • #parse_line - Processes line (row) of the file
    • #parse_to_grid - Turns file into 2D array
    • #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
    • #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
  • Import methods:
    • #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
    • #import - Larger controller of import, sets error messages and displays.

• questionnaires_controller, the list of methods in the controller are the following:
  • ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

• export_file_controller, the list of methods in the controller are the following:
  • ::find_delim_filename - Sets and returns attributes for User object from hash.
  • ::exportdetails - Generates the metadata for the data that needs to be exported.
  • ::export - Converts the data into CSV format files.
  • ::export_advices - Export question advice data to CSV file.
  • ::export_tags - Export the tags.

Helpers

• import_file_helper, the list of methods in the file are the following:
  • ::create_new_user - Makes a user object in the database.

• import_topics_helper, the list of methods in the file are the following:
  • ::define_attributes - Find the filename and delimiter.

• questionnaire_helper (Appears to be deprecated/unused), the list of methods in the file are the following:
  • ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

Proposed Design Change

• Search for occurrences of “import” and “export” to find all the valid models that can be imported or exported.
• Identify the flow of import/export functionality involving those models and figure out whether they are using import_file_controller/export_file_controller or using their own logic to perform these actions.
• Once all the involved pieces are identified, we will change the code to ensure that all the involved modules make use of the new import_file_controller/export_file_controller.
• Once everything is in working condition, we will proceed with testing.

Everything should work as expected based on our analysis so far. However, there are a few models which use the import/export functionality and have very different structures. We need to create helper methods for them to make the imports and exports specific to those models leveraging the common controllers created.

In a more generic way, this is how the import functionality is working as of now (taking one example for simplicity):

Import_file_controller is used for importing the data of team and course_participant models in a CSV. Questionnaires_controller is being used to import data of the question model.

What we want it to look like: UML Diagram

A generic import_file_controller/export_file_controller should be used for importing/exporting the data of all modules i.e., team, course_participant and questions model.

Refer to the above PR (https://github.com/expertiza/expertiza/pull/1438) for reference. The major part of our effort will be targeted at rewriting this import_file_controller controller and removing the pieces which are breaking. Once we achieve that, we will create helper methods specific to each method and leverage the controllers to do the import and export in a consistent fashion across the application.

New Features Added

• Export Topics - The Export Topics Feature was not present in the application. You can navigate to Export Topics by going to Assignments > Click on Edit (pencil icon) > Click on Topics menu.

Bug Fixes

On analyzing the code flow for import and export functionality we found that the Export Grades functionality was broken. There were a few things that were creating problems in this flow. Mentioned below are the fixes that we made in this flow:

• Fix 1: Export grades into a file successfully - When exporting grades, self.export method of Assignment.rb is called. This internally calls review_grades method in Scoring.rb. But review_grades method was not being invoked because it was being called from a class method but was imported as an instance method. Then self.export method internally invokes self.export_data method. Some lines related to the business logic were missing here.

• Fix 2: Incorrect headers in the exported file - During the export, the generated CSV had some issues with the headers. A few headers were not separated into multiple columns which was not the desired result. We fixed those as well.

• Fix 3: Only export selected columns from the UI - Removed author feedback score from being exported. What happens in the file?
  • Problem 1: The meta-review feedback column is removed but the author feedback header still shows up.
  • Problem 2: Also only the headers have changed but all values of all columns are still being exported.

Test Plan

Writing test cases for import_file_controller and export_file_controller. We would be focussing on writing test cases for the controllers and the helpers that we will create specific to the models.

Now we have only 5 methods in the export_file_controller class that we need to test. Five methods in the export_file_controller class that needed to be tested were:

• action_allowed - This is a method that calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the export operation. We write a test for this method expecting that a user calling the method has required permissions.
• exportdetails - This method fetches the delimiter and the file name required to perform the export operation. It also stores additional metadata from the Assignments class in a CSV file. We will test this method by verifying whether the export process works with supported model types.
• export - This is the main method that does the actual exporting task. It gathers all the necessary data such as - assignment, course, participant id, team, questions, answers, reviews, etc, and saves the data unless the model is not supported. We will test this method by verifying whether the export process works with supported model types.
• export_advices - This method is similar to export but only exports "Question" type models along with "QuestionAdvice". We will test this method by checking if the correct data is being exported.
• export_tags - This method exports the user's answers with their respective tags and comments. We will test this method by checking if the correct data is being exported.

For the import_file_controller class, we identified the following methods to be tested:

• action_allowed - Similar to the one in export_file_controller, this is a method that in turn calls "current_user_has_ta_privileges?" to ensure that the current user has the proper privileges to perform the import operation. We write a test for this method expecting that a user calling the method has required permissions.
• show - This method stores necessary metadata required to perform import operations such as - user id, delimiters, header, and also additional information such as the assignment id and team number. We write a test to check whether the necessary variables in the method are populated.
• import - The import method is used to verify the hash header or body, and generate error messages if needed. We will be writing a test to verify if this method can throw an error if something goes wrong with the import process.