CSC517 Spring 2019/E1923 New Framework For Import/Export

Project Introduction

Team

• Veeha Khanna, vkhanna
• Randy Paluszkiewicz, rpalusz
• Drew Marshburn, rdmarshb
• Andrew Miller, asmille5

Project Purpose

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from database and save it as a file in desired format. However, same functionality is implemented multiple times for exporting and importing different things.

The aim of this project is to design a generic export/import feature. What you need to do is implement a framework which supports exporting and importing functionality based on input. There are substantial degrees of freedom in this project. You need to design the module which will take the database table names, column names from which data needs to be exported or the imported into. So that this module can be used for every export/import feature that expertiza provides.

Most imports and exports just import or export a single table. But sometimes exports involve “details.” For example, when questionnaires are imported or exported, the “advice” (guidelines on what characteristics of work merit each score) may be imported or exported along with the questionnaire. I suspect that all instances of import/export of multiple tables are called “details” in the current code. Provide a mechanism so that “details” can be imported or exported along with the main table involved.

Existing Import Functionality

Existing import functionality is primarily routed through the import_file_controller and an import class method for various models.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

Questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

Controllers

• import_file_controller
  • File processing methods:
    • #get_delimiter - Sets proper delimiter for filetype
    • #parse_line - Processes line (row) of the file
    • #parse_to_grid - Turns file into 2D array
    • #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
    • #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
  • Import methods:
    • #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
    • #import - Larger controller of import, sets error messages and displays.

• questionnaires_controller
  • ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

Helpers

• import_file_helper
  • ::define_attributes - Sets and returns attributes for User object from hash.
  • ::create_new_user - Makes a user object in the database.

• import_topics_helper
  • ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
  • ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

• questionnaire_helper (Appears to be deprecated/unused)
  • ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

Models

All models have an ::import method called by the ImportFileController. In addition, SignUpTopic and User objects rely on their helpers, which are mentioned above.

• assignment_participant
• assignment_team
• course_participant
• course_team
• team
• metareview_response_map
• question
• review_response_map
• sign_up_sheet
• sign_up_topic
• user

Proposed Import Changes

The import functionality should be routed through the ImportFileController. Everything should still work as it does now after our changes; we are cleaning up code and may move functionality to a different location but nothing will get removed permanently. Luckily, most import functionality is routed through the ImportFileController and import class methods on models that are being imported. It makes sense for each model to have its own import method because each model is know what to expect for itself. Making things overly generic will end up contradicting the DRY principle and lead to many nested if statements.

We intend to remove other helpers and files, such as the ImportFileHelper, the ImportTopicHelper, and the QuestionnaireHelper to keep import routing consistent. We will also remove the QuestionnaireController and route that import through the ImportFileController. In addition, we will update ImportFileController methods (in particular, #hash_rows_with_headers and #import_from_hash) to make better use of polymorphism and eliminate large and unnecessary if/else blocks.

We will also insert more specific object creation specifications. Consider if a user's import contains objects that do not exist in the database. We will provide an option for the user to specify whether or not new/dependent objects should be created and account for this boolean in relevant ::import functions for the models.

To summarize our changes:

• Route all import traffic through ImportFileController and ::import calls on models.
• Refactor ImportFileController.
• Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.
• Add ability for ImportFileController to import Question objects for Questionnaires. (Add question view to views/import_file, introduce Question logic into the controller.)

Visually this is what is happening right now (with some files missing for brevity's sake):

What we want it to look like:

What has been done so far

We have re-routed how the Questionnaire import works so that it now gets routed to the ImportFileController instead of the QuestionnairesController. We have asked our mentor for some test files to try importing and seeing if what we did works.

The models that have a "self.import" method work

We have also started refactoring the #show and #import_from_hash. The first model that we will refactor is SignUpTopic since it is the most complicated and messy. There is a lot of hard coded information in these files so there is only so much we can do without breaking everything. Our goal is it mimic the current functionality in a cleaner more DRY way.

In the #show method this is what we wanted to clean up:

What is happening here is that there are three optional columns that the imported file could have. To know if they are a part of that file the user checks a check box; when this happens this bit of code counts how many extra columns there are by incrementing '@optional_count' by one. This is very messy and not good coding practice.

This is how we have changed it:

File:NewShow.png

Now for loop something.

In the #import_from_hash this is what we wanted to clean up:

What is happening here is that depending on the value of '@optional_count' the appropriate number of headers will be generated when parsing the file (given a header is not present) and then #hash_row_with_headers is called. This is again very repetitive and long.

This is what we have changed it to:

File:NewHash.png

Existing Export Functionality

These are the files that contain an method with export in the name. The Export functionality, unlike the Import functionality, is different in every place it was implemented.

Controllers

• export_file_controller
  • File processing methods:
    • #find_delim_filename - Find delimiter and filename
    • #start - Assign titles to models for display
  • Export methods:
    • #exportdetails - Exports details from assignments
    • #export - Exports the options parameter for the following models: Assignment, AssignmentParticipant, AssignmentTeam, CourseParticipant, CourseTeam, MetareviewResponseMap, Question, ReviewResponseMap, User, Team
    • #export_advice - Exports question advice data to CSV file

• 106_update_controller_for_export
  • ::up - It seems to update permissions for export by for administrative assignments
  • ::down - Empty method

Helpers

• export_file_helper
  • It is empty

Models

All models have some kind of an ::export method:

• assignment
• assignment_team
• course_team
• metareview_response_map
• participant
• question
• review_response_map
• user
• team
• question_advice

Proposed Export Changes

The goal is to get all the export functionality routed through one place, the ExportFileController. The overall effect will be similar to what we hope to achomplish when updating the Import functionality.

We will work on Import before we start with Export.

Test Plan

Every file we are amending has test cases corresponding to the import/export functionalities. Since we are rearranging import/export functionality and not re-creating it tests will similarly be rearranged so they keep passing and are being represented in the correct file.

If we end up adding new functionality we will add new tests accordingly, we do not foresee a need for this yet.

Deployment

The project is deployed on an NCSU VCL server. It can be accessed at http://152.46.16.252:8080/.

A sample instructor login is:

• Username: instructor6
• Password: password