Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export

2025-12-03T07:08:46Z

Tjbrown8:

== Introduction ==
The export/import functionality is one of the most valuable tools for instructors when setting up assignments in Expertiza. Instructors often have lists of students, teams, and other data from their learning management systems. The ability to seamlessly export and import this data into Expertiza significantly reduces setup time and effort.

* '''Import Functionality Video''' — https://youtu.be/OYMAJIws0gI
* '''Export Functionality Video''' — https://youtu.be/vhzYiPaoAK8

== Objective ==
The previous version of Expertiza offered multiple export and import features—for example, exporting students, teams, rubrics, topics, and more. These features typically retrieve data from the database and save it in a specified format. However, the same logic is often duplicated across different implementations for various data types.

The goal of this project is to design and implement a generic export/import framework that can handle various types of data based on input parameters. You will create a module that accepts database table names and column names to determine what data should be exported or imported. This module should be flexible and reusable across all export/import features in Expertiza.

== Design Document ==
=== Class Diagram ===
[[File:CSC517Final_4.png|1200px]]

==== Architecture ====
The system is divided into two main layers:

* '''Backend''' — Handles import/export logic, data validation, and duplicate record management.
* '''Frontend''' — Provides user interfaces for triggering imports and exports, communicating with backend controllers.
----
==== Backend Components ====

===== FieldMapping =====
Represents the relationship between class fields and their corresponding data columns.
* '''''Attributes:'''''
** <code>class: Class</code> — The associated class for data mapping.
** <code>ordered_fields: Array[String]</code> — Field order for import/export operations.

===== DuplicateAction (Mixin) =====
Defines actions to take when duplicate records are detected.
* '''''Attributes'''''
** <code>name: String</code> — Name of the duplicate action.
* '''''Methods:'''''
** <code>on_duplicate_record()</code> — Handles logic when a duplicate record is found.

===== ImportableExportable (Mixin) =====
Provides shared functionality for classes that can be imported or exported.
* '''''Attributes:'''''
** <code>mandatory_fields: Array[String]</code> — Fields required for processing.
** <code>optional_fields: Array[String]</code> — Fields that can optionally be included.
** <code>available_duplicate_actions: Array[DuplicateAction]</code> — List of supported duplicate handling strategies.
* '''''Methods:'''''
** <code>register_detail(class, field: String)</code>
** <code>try_insert_record(DuplicateAction)</code>
** <code>is_duplicate_record()</code>
** <code>get_detail_field(record, detail_class, detail_field: String)</code>
** <code>to_hash()</code>

===== ImportExportManager =====
Acts as the central interface for managing import and export workflows.
* '''''Methods:'''''
** <code>export(class, mapping: FieldMapping, filename: String)</code> — Exports class data based on mapping.
** <code>import(class, mapping: FieldMapping, filename: String, use_header: bool)</code> — Imports data into the specified class.
** <code>default_mapping(class)</code> — Retrieves the default mapping configuration.
** <code>suggest_filename(class, mode: String)</code> — Suggests a filename based on class and mode.

===== ImportController =====
Handles import-related HTTP requests.
* '''''Endpoints:'''''
** <code>index</code> — Returns import configuration data.
** <code>import</code> — Accepts incoming data and triggers import operations.
* '''''Interactions:'''''
** Sends mandatory and optional fields, as well as available duplicate actions, to the frontend.
** Receives chosen ordered fields and duplicate action from the frontend.

===== ExportController =====
Handles export-related HTTP requests.
* '''''Endpoints:'''''
** <code>index</code> — Provides available export configurations.
** <code>export</code> — Generates export files based on selected parameters.
* '''''Interactions:'''''
** Sends mandatory and optional fields to the frontend.
** Receives chosen ordered fields for export.
----
==== Frontend Components ====

===== Import (.tsx) =====
* '''''Methods:'''''
** <code>on_import</code> — Initiates the import process.
* '''''Interactions:'''''
** Receives mandatory/optional field data and duplicate actions.
** Sends selected ordered fields and chosen duplicate handling action back to the backend.

===== Export (.tsx) =====
* '''''Methods:'''''
** <code>on_export</code> — Initiates the export process.
* '''''Interactions:'''''
** Receives mandatory/optional field data from backend.
** Sends selected ordered fields for export.
----
==== Data Flow ====

===== Import Sequence =====
# Frontend requests import configuration.
# Backend responds with mandatory/optional fields and duplicate actions.
# User selects ordered fields and a duplicate action.
# Frontend sends selections to backend for processing via <code>ImportController.import</code>.

===== Export Sequence =====
# Frontend requests export configuration.
# Backend sends available fields.
# User selects ordered fields to include.
# Frontend sends selections to backend via <code>ExportController.export</code>.
----

=== Use Case Diagram ===
[[File:CSC517FinalUseCases_2.png]]

==== Primary Actor ====

'''Instructor / Administrator'''
* Initiates both import and export operations for any class (e.g., Users, Teams, Rubrics).
* Goal is to efficiently transfer data into or out of the system without requiring separate tools for each entity.

==== Use Cases ====

===== Main Use Cases =====

;Import data for any class from a CSV file
# The actor uploads a CSV file.
# The system reads available headers or prompts the actor to select them.
# The actor confirms header order and selects a duplicate-handling option.
# The system validates data and imports records according to the selected duplicate policy.

;Export data for any class to a CSV file
# The actor selects which fields to include.
# The actor orders these fields as desired.
# The system generates a downloadable CSV file containing the chosen data.

===== Included Use Cases =====

;Choose/Auto Read Headers Included in the File
* When importing, the system attempts to automatically detect headers from the uploaded CSV.
* If headers are missing or unclear, the actor manually specifies them.

;Choose the Order of the Headers
* Applies to both import and export operations.
* The actor arranges the order of headers to ensure correct mapping between CSV columns and database fields.

;Choose Headers that will be Included in the File
* Applies primarily to export operations.
* The actor selects specific headers to include in the output CSV file, ensuring only relevant data is exported.

==== Relationships ====

* Both the '''Import''' and '''Export''' use cases include the sub-use cases for choosing headers and their order.
* The actor directly initiates the primary use cases and indirectly interacts with the included ones through system prompts.

==== System Responsibilities ====

* Provide the actor with available class fields and supported duplicate actions.
* Manage CSV validation, field mapping, and record processing.
* Generate clear feedback on import/export results (e.g., success, skipped, or error entries).
----

==== Design Notes ====
The design emphasizes modularity and extensibility because the same flow supports multiple data entities using a unified import/export framework.
* Included use cases ensure flexibility for various file structures and actor preferences.
* Error handling (e.g., missing headers, duplicate data) is encapsulated within the main import process.

== Usage ==

=== Creating CSV for Importing===
----
When creating a CSV that will be imported, there are a few guidelines that should be followed to ensure information is uploaded successfully.

;Guidelines
# CSV should be made with headers that names align with the fields in the program. The names should be in Capital case (ie Name, User id, Txt, etc). These need to align with the program version of the fields (ie. name, user_id, txt) because they will be converted programmatically.
# CSV must include all mandatory fields and necessary external fields. Optional fields may be included as well. To find these, check the import page for the item you are trying to import.
# For Headers that refer to an external class, the class name should be appended to the front of the field. (ex. If a user is trying to reference a Role by its name field, the header should be "Role name", not "name").
# For CSVs that contain Duplicate Headers for External classes (Ex. Items and QuestionAdvice), make sure the external class appears in the CSV grouped together and in sequence.

=== Importing a File ===
----
The Import Modal is a reusable modal used to upload CSV files for different models (Items, Teams, Users, Assignments, etc.).
----
; Opening the Import Modal
:* From the relevant page (e.g., Items index), click the Import button. This opens the Import <ModelName> modal.
:* As soon as the modal opens, the frontend automatically calls:
:*: GET /import/<modelClass> to fetch import metadata for that model. This metadata is what populates the lists at the top of the modal.
; Field Requirements
:* At the top of the modal you’ll see three lists:
:# Mandatory fields – columns that must be present in your import for it to succeed.
:# Optional fields – extra columns you may include if you have that data.
:# External fields – related fields that can be pulled from or linked to other records (e.g., questionnaire_name, question_advice_id, etc.)
:* These lists are read-only and meant to guide how you structure your CSV.
[[File:ImportFrontend.png|600px]]
; Selecting a CSV
:* Under CSV file:
:# Click Choose File and select your .csv file.
:# Decide whether your file’s first row is a header row:
:#* First row contains headers (switch ON)
:#** Use this if your first row has column names like txt, weight, seq.
:#** The backend will match CSV headers to field names automatically.
:#* First row contains headers (switch OFF)
:#** Use this if your file has no header row (just data).
[[File:UploadFile.png|600px]]
; Manual Column Mapping
:* If “First row contains headers” is OFF, the modal:
:** Reads the first one or two lines from your CSV.
:** For each column, shows:
:*** A dropdown to choose which field this column represents.
:*** A “First Row Value” snippet so you can see an example value from that column.
:* You must:
:*# Go through each column.
:*# Use the dropdown to select the corresponding field name (e.g., map column 1 → txt, column 2 → weight, etc.).
:*# Ensure that all mandatory fields appear in at least one column.
:** If mandatory fields are missing from your selection, the modal will block the import and show a status message.
[[File:ColumnMapping.png|600px]]
; Handling Duplicates
:* In the Duplicate handling section:
:** If the backend has configured options, you’ll see one or more radio buttons (e.g., skip, overwrite).
:** Choose how the system should behave if it encounters duplicate records during the import.
:* If no duplicate options are available, you’ll see: “No duplicate options.”
; Running the Import
:* When everything is set:
:*# Click "import"
:*# The frontend sends a POST /import/<modelClass> request with:
:*#* csv_file – the file you selected.
:*#* use_headers – true or false depending on the toggle.
:*#* ordered_fields – only included if header mode is off, listing the fields you selected for each column.
:* While the request is in progress, the modal shows a loading state.
:* When the response returns, the Status line at the bottom will show either:
:** A success message from the backend (e.g., “Import complete” or a custom message)
:** An error message if something went wrong (e.g., missing mandatory fields, invalid format)
:* You can then close the modal with cancel or by using the close icon.
[[File:ImportComplete.png|600px]]

=== Exporting a File ===
----
; Opening the Export Modal
:* Click the Export button on the page for the model you want to export (e.g., Teams).
:* When the modal opens, it automatically makes a request to:
:*: `GET /export/<modelClass>`
:* to fetch the field metadata for the selected model.
; Field Lists Displayed
:* The modal uses the metadata received from the backend to show three types of fields:
:# Mandatory fields – These must always be included in the export and cannot be unchecked.
:# Optional fields – These may be included in the export at the user’s discretion.
:# External fields – Additional related fields made available for export (if applicable).
:* Users can hover over an info icon to view the full field list in a tooltip.
; Choosing Columns to Export
:* The modal provides a checkbox list of all available fields.
:* Users may:
:# Check or uncheck optional fields.
:# View mandatory fields (locked and always selected).
:# Reorder any fields using ↑ and ↓ arrows, which determines the ordering of columns in the exported CSV.
; Column Ordering
:* Fields appear in the export file in the exact order shown in the modal.
:* The user can:
:# Move any non-mandatory field up.
:# Move any non-mandatory field down.
:* Mandatory fields are always included and their order can also be adjusted if they are part of the list returned by the backend.
; Generating the CSV
:* When satisfied with selections:
:** Click export
:* The frontend gathers:
:** The selected fields
:* It then generates a CSV where:
:** The first row contains the field names (in the chosen order)
:** All following rows contain the corresponding values
:** A downloadable `.csv` file is automatically created and saved to the user’s system.
; Status Feedback
:* The modal displays a status message during and after the export operation, for example:
:# “Generating CSV…”
:# “Export complete”
:* If no fields are selected, the modal will prevent exporting and display a warning.
[[File:ExportFile.png|600px]]

== Testing ==

=== Fixtures ===
* <code>empty.csv</code>
*: A completely empty CSV file. This file is used to ensure that if no content is included, the import service will fail gracefully.
* <code>empty_with_headers.csv</code>
*: A CSV file that only contains the headers for the User class. This file is used to ensure a file with no records is handled gracefully.
* <code>multiple_users_no_headers.csv</code>
*: A CSV file with no headers for importing records for the User class. It contains multiple User records (John Doe, Jane Doe) to import.
* <code>multiple_users_with_headers.csv</code>
*: A CSV file with headers for importing records for the User class. It contains multiple User records (John Doe, Jane Doe) to import.
* <code>questionnaire_item_with_headers.csv</code>
*: A CSV file with headers for importing records for the Item class. These Items make up a questionnaire. It contains a single Item that needs to be imported.
* <code>single_user_no_headers.csv</code>
*: A CSV file with headers for importing records for the User class. It contains a single User (John Doe) who needs to be imported.
* <code>single_user_with_headers.csv</code>
*: A CSV file with headers for importing records for the User class. It contains a single User (John Doe) who needs to be imported.
* <code>users_duplicate_records.csv</code>
*: A CSV file containing multiple duplicate records that need to be handled.
* <code>single_user_email_invalid.csv</code>
*: A CSV file that contains a user with an invalid email. That is an Internal field that needs to be validated.
* <code>single_user_role_doe_not_exist.csv</code>
*: A CSV file that contains a user with a role that does not exist. That is a field from an External Class that needs to be looked up.

=== Unit Testing ===

----
==== ImportableExportableHelper - Importing ====
----
;Create tests for each of the different importable classes
* Show a class with no headers can be imported (User)
*: This test tries to import the 'single_user_no_headers.csv' file. This file doesn't have headers, so the test passes a list of headers to the import statement. This CSV file calls for the Role Name of the user, which is set to Student. This tests whether the import statement is able to look up a role by its name. It is able to do this because the name is a lookup field for the External Class Role, which is assigned to the User Class.
* Show a class with headers can be imported (User)
*: This test tries to import the 'single_user_with_headers.csv' file. This file has headers, so the test passes nil to the header variable of the import statement. This CSV file calls for the Role ID of the user, which is set to 4 (Teaching Assistant). This tests whether the import statement is able to look up a role by its ID. It can do this even though ID is not the set lookup field, because the lookup function uses the primary key if the lookup field isn't in the file.
* Show a class with multiple records can be imported (User)
*: This test tries to import the 'multiple_users_with_headers.csv' file. This file contains multiple user records that need to be imported (John Doe and Jane Doe). This tests that multiple records successfully get imported with the right information.
* Show a class with external create classes can take duplicate headers (Questionnaire w/ multiple Advice)
*: This test tries to import the 'questionnaire_item_with_headers.csv' file that creates an external class. Not only that, but it can detect if there are multiple instances of this external class and save them individually.

;Create Tests to test Errors/Edge Cases
* Import a class with external lookup class that doesn't exist
*: Tries to import a csv with an invalid external field. The user's role doesn't exist, so looking up this field should fail and ultimately cause the import to fail.
* Import a class with an invalid field (User with ian nvalid email)
*: Tries to import a csv with an invalid internal field. The user email is validated inside the User class, so make sure that validation raises the correct error.
* Import an empty CSV (With Headers)
*: Tries to import an empty csv with no records inside. This is tested with the <code>use_header</code> option set to true.
* Import an empty CSV (Without Headers)
*: Tries to import an empty csv with nothing inside. This is tested with the <code>use_header</code> option set to false.

==== ImportExportService - Importing ====
----
(Taylor - Expand on these)
;Create tests to make sure the duplicate actions work
* Try to create with duplicates for the skip action
* Try to create with duplicates for rename action
* Try to create with duplicates for update record

;Create Tests to test Errors/Edge Cases
* Create a test to make sure duplicate action is specified
* Create a test that doesn't include the mandatory fields

==== ImportExportService - Exporting ====
----
(Taylor - Expand on these)
;Create tests for each of the different exportable classes
* Export a class with the default headers
* Export a class with only the mandatory headers
* Export a class with only some optional headers
* Export a class with a different order of headers

;Create tests to test Errors/Edge Cases
* Export a class with chosen headers that are empty

----

=== Integration Testing ===
----
(Taylor)

==== Import Controller ====
----
(Taylor)

==== Export Controller ====
----
(Taylor)

=== Frontend Testing ===
----
==== Import Modal ====
----

* Shows loading state when isLoading is true
*: Ensures that when `useAPI` reports `isLoading=true`, the modal hides all content and displays a “Loading…” message.
* Renders field summary and duplicate options from metadata
*: Confirms that metadata returned from the backend (mandatory, optional, external fields, and duplicate handling actions) are rendered correctly in the modal.
* Shows status when importing with no file selected
*: Validates that clicking “import” without choosing a CSV sets an error status message and prevents the import call.
* Shows column mapping and first-row values when header mode is off
*: After uploading a CSV and disabling the “First row contains headers” toggle, verifies that the “Column order” UI appears, column dropdowns are rendered, and the first-row preview text is displayed.
* Calls sendImport when import is valid
*: Uploads a valid CSV containing mandatory fields and clicks “import.” Checks that the `sendRequest` mock is called, proving that the modal sends a correctly prepared API request.
* Calls onHide when cancel is clicked
*: Simulates clicking the “cancel” button and confirms the modal fires the `onHide` callback to close itself.

==== Export Modal ====
----

== Future work ==
* Adding event-based entity registration.
* Extending support for new duplicate resolution policies.
* Integration with the engine demo for visualization and debugging.

=== Other Notes/Recommendations ===
* For Items, limit the string values that question_type can be using built-in validation rules as well, or else importing will allow just any string.
* For Questionnaires, break_before can't be false because of the validation rules. Maybe try this `validates :break_before, inclusion: { in: [true, false] }`

CSC/ECE 517 Fall 2025

2025-11-19T01:11:09Z

Tjbrown8: /* Final Projects */

* [[CSC/ECE 517 Fall 2025 - E2550. Response hierarchy and responses_controller back end]]
* [[CSC/ECE 517 Fall 2025 - E2551. Reimplementing SubmittedContentController]]
* [[CSC/ECE 517 Fall 2025 - E2552. ProjectTopic and SignedUpTeam]]
* [[CSC/ECE 517 Fall 2025 - E2553. Heatgrid and Grades Controller (Frontend + Backend)]]
* [[CSC/ECE 517 Fall 2025 - E2554. Front end for View Submissions Page]]
* [[CSC/ECE 517 Fall 2025 - E2555. Front end for Participants Page]]
* [[CSC/ECE 517 Fall 2025 - E2556. Front ends for Create Teams and Assign Reviewers]]
* [[CSC/ECE 517 Fall 2025 - E2557. Specialized rubrics for different topic types]]
* [[CSC/ECE 517 Fall 2025 - E2558. Tabbed view for creating and editing assignments]]

== Final Projects ==
* [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]
* [[CSC/ECE 517 Fall 2025 - E2561. Finish up SubmittedContentController]]
* [[CSC/ECE 517 Fall 2025 - E2562. Review grading dashboard]]
* [[CSC/ECE 517 Fall 2025 - E2563. Review tableau]]
* [[CSC/ECE 517 Fall 2025 - E2564. Review calibration]]
* [[CSC/ECE 517 Fall 2025 - E2565. Integration of JoinTeamRequests]]
* [[CSC/ECE 517 Fall 2025 - E2566. Finish DueDates]]
* [[CSC/ECE 517 Fall 2025 - E2567. Review rubrics varying by round]]
* [[CSC/ECE 517 Fall 2025 - E2568. Finish tabbed view for Assignments, including Topics and Calibration tabs]]

CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export

2025-11-10T06:53:44Z

Tjbrown8: