Expertiza_Wiki - User contributions [en]

File:Export-sequence.png

2026-04-28T04:12:52Z

Kdnadkar: Kdnadkar uploaded a new version of File:Export-sequence.png

File:Import-sequence.png

2026-04-28T04:12:45Z

Kdnadkar: Kdnadkar uploaded a new version of File:Import-sequence.png

File:Import-export-architecture.png

2026-04-28T04:12:22Z

Kdnadkar: Kdnadkar uploaded a new version of File:Import-export-architecture.png

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-04-28T04:12:09Z

Kdnadkar:

= Final Project Design Document: E2606 Import/Export =

== Purpose ==

The E2606 import/export work builds on the shared import/export framework introduced by [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. The final design keeps the reusable pipeline, but applies it to practical instructor-facing workflows such as users, course participants, assignment participants, teams, topics, questionnaires, and grades. The goal is to let instructors move data in and out reliably without exposing internal database fields or maintaining one-off CSV scripts for each entity.

The central design distinction is that '''importable''' and '''exportable''' are related but not identical capabilities:

* An '''importable''' class can create or update application records from uploaded CSV data. It must define required fields, optional fields, external lookup fields, request context, and duplicate handling behavior.
* An '''exportable''' class can produce CSV data from existing application records. It may expose calculated, flattened, or reporting-oriented fields that should not necessarily be accepted during import.
* Some workflows, such as grades and questionnaire packages, are intentionally specialized instead of using the generic single-model path.

== Previous Work ==

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. E2560 introduced the shared import/export framework that replaced one-off entity-specific CSV logic with reusable model metadata, service-layer CSV processing, field mapping, and duplicate-handling strategies.

E2606 extends that foundation by applying the framework to additional class types and by tightening the design around importable versus exportable classes, hidden/system-managed fields, assignment-scoped data, questionnaire packages, and grade export.

== Team ==

* Kiran Nadkarni (kdnadkar@ncsu.edu)
* Isaac Martin (iemartin@ncsu.edu)

== Problem Statement ==

The previous generic import/export framework made the system more reusable, but it still needed refinement before it could support the full set of entities expected by instructors. The main problems were:

* The system needed clearer boundaries around which classes can be imported or exported.
* User-facing field lists needed to hide internal or system-managed columns such as IDs, timestamps, and foreign keys.
* Multi-table workflows such as teams with members and questionnaires with items/advice needed special handling.
* Scoped data, such as teams, assignment participants, and course participants, needed request context so imports could follow the same rules as the application.
* Grades needed to be treated as an export-only reporting workflow rather than as a normal mutable model.

This design addresses those issues by combining a shared import/export pipeline with explicit class support, hidden fields, context-aware model behavior, and specialized services where the generic model path is not a good fit.

== Capability Matrix ==

{| class="wikitable"
! Class or workflow
! Importable
! Exportable
! Generic endpoint?
! Notes
|-
| User
| Yes
| Yes
| Yes
| Imports use role and institution external lookups. Exports omit hidden fields.
|-
| CourseParticipant
| Yes
| Yes
| Yes
| Requires course_id context. CSV intentionally contains user_name only.
|-
| Team
| Yes
| Yes
| Yes
| Requires assignment_id context. Exports dynamic participant_1 through participant_n columns.
|-
| AssignmentParticipant
| Yes
| Yes
| Yes
| Requires assignment_id context. CSV intentionally contains user_name only.
|-
| ProjectTopic
| Yes
| Yes
| Yes
| Assignment-backed topic records.
|-
| Questionnaire*
| Yes
| Yes
| Yes
| Generic model metadata exists, but template movement is handled with Item and QuestionAdvice through the package workflow.
|-
| Item*
| Yes
| Yes
| Yes
| Represents questionnaire questions/items and participates in the questionnaire package workflow.
|-
| QuestionAdvice*
| Yes
| Yes
| Yes
| Represents scoring advice associated with questionnaire items and participates in the questionnaire package workflow.
|-
| Grades
| No
| Yes
| No, specialized
| Export-only reporting workflow. No import path because grades are derived or instructor-assigned data.
|-
|}

<nowiki>*</nowiki> Questionnaire, Item, and QuestionAdvice can be treated as individual import/export classes, but the preferred template workflow handles them together as a multi-file or zip package so their relationships are preserved.

== High-Level Architecture ==

[[File:Import-export-architecture.png|thumb|center|700px|Generic import/export framework with specialized questionnaire and grades paths]]

The architecture diagram is intended to depict the generic framework, with specialized questionnaire and grades workflows shown as side paths. The reusable core is organized around two generic controllers and two service objects:

* '''ImportController''' exposes metadata and upload endpoints for supported import classes.
* '''ExportController''' exposes metadata and download endpoints for supported export classes.
* '''Import''' coordinates CSV import, model row processing, and duplicate handling.
* '''Export''' coordinates CSV generation and preserves requested field order.
* '''FieldMapping''' normalizes CSV headers and selected fields into the model's internal field names.
* '''ImportableExportableHelper''' is the model-level declaration layer for mandatory fields, optional fields, hidden fields, external fields, filters, and import row behavior.
* Questionnaire package services handle multi-file questionnaire template export/import outside the generic single-model export path.

One important improvement is explicit class support. Instead of blindly resolving any client-provided class name, the controllers resolve requests through supported import/export class lists. This makes the API safer and makes the supported surface easier to test and document.

== Importable Classes ==

Importable classes must be safe to mutate from CSV input. A class is importable only when it can define the following:

* Mandatory CSV columns
* Optional CSV columns
* Hidden or system-managed columns
* External lookup columns for related records
* Whether referenced records should be looked up, created, or rejected if missing
* Duplicate handling options
* Any request context required to make the import safe, such as assignment_id or course_id

The generic import flow is:

# ImportController receives class, csv_file, use_headers, ordered_fields, duplicate action, and optional context.
# The controller resolves the requested class from the supported import class list.
# Import delegates row processing to klass.try_import_records.
# FieldMapping normalizes selected or header-provided columns.
# ImportableExportableHelper builds model instances, resolves external fields, applies defaults, saves records, and returns duplicates.

[[File:Import-sequence.png|thumb|center|700px|Generic import request sequence]]

Special import behavior exists where a raw table-shaped import would be unsafe:

* Team import requires assignment context and creates AssignmentTeam rows with participant links.
* AssignmentParticipant import accepts usernames only and attaches existing users to an assignment.
* CourseParticipant import accepts usernames only and attaches existing users to a course.
* QuestionnairePackageImportService imports related files together because questionnaire templates are graph-shaped rather than single-table data.

== Exportable Classes ==

Exportable classes are read-only from the user's perspective. They do not need duplicate handling, and they may expose calculated or flattened fields that are useful in CSV output.

The generic export flow is:

# ExportController receives class, ordered_fields, and optional context.
# The controller resolves the requested class from the supported export class list.
# Export.perform exports the class directly. Specialized workflows such as questionnaire packages and grades use their own services/controllers.
# FieldMapping preserves the selected column order.
# The frontend receives one or more CSV payloads and downloads them.

[[File:Export-sequence.png|thumb|center|700px|Generic export request sequence]]

Some exports intentionally avoid the generic path:

* Grades export is handled by GradesController#export because it is assignment-scoped and gradebook-oriented.
* Questionnaire package export uses QuestionnairePackageExportService because it produces a zip containing multiple related CSV files.

== Hidden Fields and Field Visibility ==

A major usability issue in generic import/export systems is exposing too many database fields. End users should not need to see or map internal IDs, timestamps, tokens, or foreign keys that are managed by the application.

The design uses hidden fields to remove system-managed fields from the metadata returned to the frontend. Models still know those fields exist, but the import/export UI only receives the fields that are relevant to users.

Hidden fields take precedence over mandatory fields. If a field is marked as hidden, it is removed from the user-facing mandatory and optional field lists even if it exists in the database schema or model metadata.

Examples include:

* id
* created_at
* updated_at
* instructor_id
* institution_id
* course_id

For CSV-facing relationships, the design prefers readable fields such as course_name and user_name.

== Duplicate Handling ==

Import supports strategy-based duplicate handling. The available strategies can vary by entity, but the shared model metadata lets the frontend display the available options before import.

The main duplicate actions are:

* '''SkipRecordAction''': ignore incoming rows that conflict with existing records.
* '''UpdateExistingRecordAction''': update the existing record with incoming values.
* '''ChangeOffendingFieldAction''': modify the conflicting field to make the incoming record unique.

This keeps duplicate behavior explicit. It also allows sensitive workflows to restrict the available actions. For example, participant import is intentionally narrower than user import because it should attach existing users to an assignment or course, not silently create or rewrite user accounts.

== Specialized Workflows ==

=== Assignment Participants ===

Assignment participant import/export is intentionally narrower than a normal model export. The CSV exposes only user_name, and the request must include assignment_id.

This prevents participant import from accidentally creating or editing users. Import resolves each username against existing User records, then creates or reuses the AssignmentParticipant link for the selected assignment.

This design keeps team and participant membership tied to the correct assignment scope, matching the same conceptual rules used when adding participants through the application.

The exported filename includes the assignment name and ID, while the CSV body stays username-only so it can be imported again without extra cleanup.

=== Course Participants ===

Course participant import/export follows the same safety boundary as assignment participants. The CSV exposes only user_name, and the request must include course_id.

This prevents participant import from accidentally creating or editing users. Import resolves each username against existing User records, then creates or reuses the CourseParticipant link for the selected course.

This design keeps course roster membership tied to the correct course scope, matching the same conceptual rules used when adding participants through the application.

The exported filename includes the course name and ID, while the CSV body stays username-only so it can be imported again without extra cleanup.

=== Teams ===

Team import/export also requires assignment context. The export surface is flattened:

<pre>
name,participant_1,participant_2,participant_3,...
</pre>

The number of participant columns is based on assignment size when available, otherwise it falls back to a default count. On import, the team name identifies the team and the participant columns identify assignment participants to attach to that team.

The frontend may display this as a single participant_ids option while expanding it back to the backend's individual participant columns. This keeps the UI compact without changing the backend CSV contract.

=== Topics ===

Project topics are assignment-backed records. They can use the same generic import/export path because they are closer to single-model data than teams or questionnaire packages.

The important design point is naming clarity. The implementation uses ProjectTopic rather than the older sign-up topic naming, so the import/export documentation and endpoints should use the current class name.

=== Questionnaire Packages ===

Questionnaire templates are not just single rows in a questionnaires table. A useful questionnaire export must preserve related items and advice. Because of that, questionnaire templates use a package workflow in addition to the generic model path.

The package controller and services handle:

* Exporting questionnaires, items, and advice together
* Importing either a zip package or separate CSV files
* Downloading blank templates
* Previewing import effects before committing
* Applying duplicate actions through a package-specific allowlist

This keeps questionnaire import/export understandable and avoids forcing a multi-table template into a single CSV.

=== Grades ===

Grades are export-only. They are not imported through the generic framework because grade data is either computed from related review data or assigned by instructors in application workflows.

The export is assignment-scoped and gradebook-oriented. It uses fixed columns such as:

* username
* grade
* comment
* optional email

The View Scores page reuses the grades export endpoint so the on-screen table and downloaded CSV are based on the same username, grade, comment, and optional email data.

This also avoids confusing instructor-assigned marks with computed peer-review, teammate-review, or author-feedback aggregates.

== Questionnaire Packages ==

Some data cannot be represented well by exporting a single model alone. The main example is questionnaire template data, where a useful export may need the questionnaire, its items, and related advice.

The final design uses explicit package services for this workflow. This is clearer than generic relationship traversal because it defines exactly which files belong in the export, supports zip-based transfer, provides blank templates, and supports preview before import.

For the final wiki, the important design point is that multi-table exports are handled deliberately rather than forcing every related record into a single CSV.

== Frontend Design ==

The frontend uses reusable modals for most generic classes:

* '''ImportModal''' fetches metadata from /import/:class, previews CSV rows, lets users map columns when headers are absent, sends duplicate-action choices, and forwards context parameters such as assignment_id or course_id.
* '''ExportModal''' fetches metadata from /export/:class, lets users choose and order columns, sends the selected order, and downloads returned CSV payloads.

The import modal supports:

* CSV file selection
* Header-based import
* Manual column-to-field mapping
* Preview before confirmation
* Duplicate action selection
* Sample CSV download

Specialized pages use custom controls where the generic modal is not enough:

* Questionnaire package modals support zip export/import, separate CSV uploads, preview, and template downloads.
* Grades export is launched from the grades or assignment workflow rather than the generic export modal.
* The View Scores page reads from the grades export endpoint to display the same saved grade data that export downloads.

== Design Rationale ==

The design separates class capability from class existence. Open-ended class resolution can expose models simply because the client supplies a class name. Explicit supported import and export class lists make behavior visible, testable, and safer.

The system also avoids assuming every exportable thing should be importable. Grades are a reporting artifact, so they are export-only. Questionnaire packages are graph-shaped, so they use a package service. Assignment participants, course participants, and teams need request context, so they override the generic table-shaped behavior.

This produces a hybrid design:

* Generic where the data is model-shaped and safe to process uniformly
* Specialized where the data is contextual, graph-shaped, computed, or safety-sensitive

== Testing and Verification ==

The implementation should be verified at three levels:

* Model and helper specs for metadata, hidden fields, package export/import, and row-level import behavior
* Request specs for metadata endpoints, import endpoints, export endpoints, unsupported classes, and missing context
* Frontend tests or manual verification for modal metadata loading, preview, duplicate action selection, and generated downloads

Important cases include:

* Unsupported class names are rejected
* Missing assignment_id is rejected for assignment-scoped teams and participants
* Missing course_id is rejected for course participants
* Missing referenced records produce readable validation errors
* Duplicate rows use the selected duplicate strategy
* Questionnaire package preview reports intended actions before import
* Grade export is limited to the selected assignment and teaching-staff permissions

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-04-27T02:27:05Z

Kdnadkar:

= Final Project Design Document: E2606 Import/Export =

== Purpose ==

The E2606 import/export work builds on the shared import/export framework introduced by [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. The final design keeps the reusable pipeline, but applies it to practical instructor-facing workflows such as users, courses, assignments, teams, participants, topics, questionnaires, and grades. The goal is to let instructors move data in and out reliably without exposing internal database fields or maintaining one-off CSV scripts for each entity.

The central design distinction is that '''importable''' and '''exportable''' are related but not identical capabilities:

* An '''importable''' class can create or update application records from uploaded CSV data. It must define required fields, optional fields, external lookup fields, request context, and duplicate handling behavior.
* An '''exportable''' class can produce CSV data from existing application records. It may expose calculated, flattened, or reporting-oriented fields that should not necessarily be accepted during import.
* Some workflows, such as grades and questionnaire packages, are intentionally specialized instead of using the generic single-model path.

== Previous Work ==

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. E2560 introduced the shared import/export framework that replaced one-off entity-specific CSV logic with reusable model metadata, service-layer CSV processing, field mapping, and duplicate-handling strategies.

E2606 extends that foundation by applying the framework to additional class types and by tightening the design around importable versus exportable classes, hidden/system-managed fields, assignment-scoped data, questionnaire packages, and grade export.

== Team ==

* Kiran Nadkarni (kdnadkar@ncsu.edu)
* Isaac Martin (iemartin@ncsu.edu)

== Problem Statement ==

The previous generic import/export framework made the system more reusable, but it still needed refinement before it could support the full set of entities expected by instructors. The main problems were:

* The system needed clearer boundaries around which classes can be imported or exported.
* User-facing field lists needed to hide internal or system-managed columns such as IDs, timestamps, and foreign keys.
* Multi-table workflows such as teams with members and questionnaires with items/advice needed special handling.
* Assignment-scoped data, such as teams and assignment participants, needed request context so imports could follow the same rules as the application.
* Grades needed to be treated as an export-only reporting workflow rather than as a normal mutable model.

This design addresses those issues by combining a shared import/export pipeline with explicit class support, hidden fields, context-aware model behavior, and specialized services where the generic model path is not a good fit.

== Capability Matrix ==

{| class="wikitable"
! Class or workflow
! Importable
! Exportable
! Generic endpoint?
! Notes
|-
| User
| Yes
| Yes
| Yes
| Imports use role and institution external lookups. Exports omit hidden fields.
|-
| Course
| Yes
| Yes
| Yes
| Uses instructor_name and institution_name as CSV-facing fields instead of raw foreign keys.
|-
| Assignment
| Yes
| Yes
| Yes
| Uses instructor_name and optional course_name instead of raw IDs.
|-
| Team
| Yes
| Yes
| Yes
| Requires assignment_id context. Exports dynamic participant_1 through participant_n columns.
|-
| AssignmentParticipant
| Yes
| Yes
| Yes
| Requires assignment_id context. CSV intentionally contains user_name only.
|-
| ProjectTopic
| Yes
| Yes
| Yes
| Assignment-backed topic records.
|-
| Questionnaire*
| Yes
| Yes
| Yes
| Generic model metadata exists, but template movement is handled with Item and QuestionAdvice through the package workflow.
|-
| Item*
| Yes
| Yes
| Yes
| Represents questionnaire questions/items and participates in the questionnaire package workflow.
|-
| QuestionAdvice*
| Yes
| Yes
| Yes
| Represents scoring advice associated with questionnaire items and participates in the questionnaire package workflow.
|-
| Answer
| Yes
| Yes
| Yes
| Supported by the generic metadata model.
|-
| QuizItem
| Yes
| Yes
| Yes
| Supported by the generic metadata model.
|-
| Grades
| No
| Yes
| No, specialized
| Export-only reporting workflow. No import path because grades are derived or instructor-assigned data.
|-
|}

<nowiki>*</nowiki> Questionnaire, Item, and QuestionAdvice can be treated as individual import/export classes, but the preferred template workflow handles them together as a multi-file or zip package so their relationships are preserved.

== High-Level Architecture ==

[[File:Import-export-architecture.png|thumb|center|700px|Generic import/export framework with specialized questionnaire and grades paths]]

The architecture diagram is intended to depict the generic framework, with specialized questionnaire and grades workflows shown as side paths. The reusable core is organized around two generic controllers and two service objects:

* '''ImportController''' exposes metadata and upload endpoints for supported import classes.
* '''ExportController''' exposes metadata and download endpoints for supported export classes.
* '''Import''' coordinates CSV import, model row processing, and duplicate handling.
* '''Export''' coordinates CSV generation and preserves requested field order.
* '''FieldMapping''' normalizes CSV headers and selected fields into the model's internal field names.
* '''ImportableExportableHelper''' is the model-level declaration layer for mandatory fields, optional fields, hidden fields, external fields, filters, and import row behavior.
* Questionnaire package services handle multi-file questionnaire template export/import outside the generic single-model export path.

One important improvement is explicit class support. Instead of blindly resolving any client-provided class name, the controllers resolve requests through supported import/export class lists. This makes the API safer and makes the supported surface easier to test and document.

== Importable Classes ==

Importable classes must be safe to mutate from CSV input. A class is importable only when it can define the following:

* Mandatory CSV columns
* Optional CSV columns
* Hidden or system-managed columns
* External lookup columns for related records
* Whether referenced records should be looked up, created, or rejected if missing
* Duplicate handling options
* Any request context required to make the import safe, such as assignment_id

The generic import flow is:

# ImportController receives class, csv_file, use_headers, ordered_fields, duplicate action, and optional context.
# The controller resolves the requested class from the supported import class list.
# Import delegates row processing to klass.try_import_records.
# FieldMapping normalizes selected or header-provided columns.
# ImportableExportableHelper builds model instances, resolves external fields, applies defaults, saves records, and returns duplicates.

[[File:Import-sequence.png|thumb|center|700px|Generic import request sequence]]

Special import behavior exists where a raw table-shaped import would be unsafe:

* Team import requires assignment context and creates AssignmentTeam rows with participant links.
* AssignmentParticipant import accepts usernames only and attaches existing users to an assignment.
* Course and Assignment import use human-readable association fields, such as instructor_name and institution_name, instead of raw foreign keys.
* QuestionnairePackageImportService imports related files together because questionnaire templates are graph-shaped rather than single-table data.

== Exportable Classes ==

Exportable classes are read-only from the user's perspective. They do not need duplicate handling, and they may expose calculated or flattened fields that are useful in CSV output.

The generic export flow is:

# ExportController receives class, ordered_fields, and optional context.
# The controller resolves the requested class from the supported export class list.
# Export.perform exports the class directly. Specialized workflows such as questionnaire packages and grades use their own services/controllers.
# FieldMapping preserves the selected column order.
# The frontend receives one or more CSV payloads and downloads them.

[[File:Export-sequence.png|thumb|center|700px|Generic export request sequence]]

Some exports intentionally avoid the generic path:

* Grades export is handled by GradesController#export because it is assignment-scoped and gradebook-oriented.
* Questionnaire package export uses QuestionnairePackageExportService because it produces a zip containing multiple related CSV files.

== Hidden Fields and Field Visibility ==

A major usability issue in generic import/export systems is exposing too many database fields. End users should not need to see or map internal IDs, timestamps, tokens, or foreign keys that are managed by the application.

The design uses hidden fields to remove system-managed fields from the metadata returned to the frontend. Models still know those fields exist, but the import/export UI only receives the fields that are relevant to users.

Hidden fields take precedence over mandatory fields. If a field is marked as hidden, it is removed from the user-facing mandatory and optional field lists even if it exists in the database schema or model metadata.

Examples include:

* id
* created_at
* updated_at
* instructor_id
* institution_id
* course_id

For CSV-facing relationships, the design prefers readable fields such as instructor_name, institution_name, course_name, and user_name.

== Duplicate Handling ==

Import supports strategy-based duplicate handling. The available strategies can vary by entity, but the shared model metadata lets the frontend display the available options before import.

The main duplicate actions are:

* '''SkipRecordAction''': ignore incoming rows that conflict with existing records.
* '''UpdateExistingRecordAction''': update the existing record with incoming values.
* '''ChangeOffendingFieldAction''': modify the conflicting field to make the incoming record unique.

This keeps duplicate behavior explicit. It also allows sensitive workflows to restrict the available actions. For example, assignment participant import is intentionally narrower than user import because it should attach existing users to an assignment, not silently create or rewrite user accounts.

== Specialized Workflows ==

=== Courses and Assignments ===

Courses and assignments avoid exposing raw foreign keys in CSVs:

* Course uses instructor_name and institution_name.
* Assignment uses instructor_name and optional course_name.

Before validation, imported names are resolved into actual associations. If a referenced instructor, institution, or course cannot be found, validation fails with a readable error.

This keeps CSV files understandable to instructors while still preserving normalized database relationships internally.

=== Assignment Participants ===

Assignment participant import/export is intentionally narrower than a normal model export. The CSV exposes only user_name, and the request must include assignment_id.

This prevents participant import from accidentally creating or editing users. Import resolves each username against existing User records, then creates or reuses the AssignmentParticipant link for the selected assignment.

This design keeps team and participant membership tied to the correct assignment scope, matching the same conceptual rules used when adding participants through the application.

=== Teams ===

Team import/export also requires assignment context. The export surface is flattened:

<pre>
name,participant_1,participant_2,participant_3,...
</pre>

The number of participant columns is based on assignment size when available, otherwise it falls back to a default count. On import, the team name identifies the team and the participant columns identify assignment participants to attach to that team.

The frontend may display this as a single participant_ids option while expanding it back to the backend's individual participant columns. This keeps the UI compact without changing the backend CSV contract.

=== Topics ===

Project topics are assignment-backed records. They can use the same generic import/export path because they are closer to single-model data than teams or questionnaire packages.

The important design point is naming clarity. The implementation uses ProjectTopic rather than the older sign-up topic naming, so the import/export documentation and endpoints should use the current class name.

=== Questionnaire Packages ===

Questionnaire templates are not just single rows in a questionnaires table. A useful questionnaire export must preserve related items and advice. Because of that, questionnaire templates use a package workflow in addition to the generic model path.

The package controller and services handle:

* Exporting questionnaires, items, and advice together
* Importing either a zip package or separate CSV files
* Downloading blank templates
* Previewing import effects before committing
* Applying duplicate actions through a package-specific allowlist

This keeps questionnaire import/export understandable and avoids forcing a multi-table template into a single CSV.

=== Grades ===

Grades are export-only. They are not imported through the generic framework because grade data is either computed from related review data or assigned by instructors in application workflows.

The export is assignment-scoped and gradebook-oriented. It uses fixed columns such as:

* username
* grade
* comment
* optional email

This also avoids confusing instructor-assigned marks with computed peer-review, teammate-review, or author-feedback aggregates.

== Questionnaire Packages ==

Some data cannot be represented well by exporting a single model alone. The main example is questionnaire template data, where a useful export may need the questionnaire, its items, and related advice.

The final design uses explicit package services for this workflow. This is clearer than generic relationship traversal because it defines exactly which files belong in the export, supports zip-based transfer, provides blank templates, and supports preview before import.

For the final wiki, the important design point is that multi-table exports are handled deliberately rather than forcing every related record into a single CSV.

== Frontend Design ==

The frontend uses reusable modals for most generic classes:

* '''ImportModal''' fetches metadata from /import/:class, previews CSV rows, lets users map columns when headers are absent, sends duplicate-action choices, and forwards context parameters such as assignment_id.
* '''ExportModal''' fetches metadata from /export/:class, lets users choose and order columns, sends the selected order, and downloads returned CSV payloads.

The import modal supports:

* CSV file selection
* Header-based import
* Manual column-to-field mapping
* Preview before confirmation
* Duplicate action selection
* Sample CSV download

Specialized pages use custom controls where the generic modal is not enough:

* Questionnaire package modals support zip export/import, separate CSV uploads, preview, and template downloads.
* Grades export is launched from the grades or assignment workflow rather than the generic export modal.

== Design Rationale ==

The design separates class capability from class existence. Open-ended class resolution can expose models simply because the client supplies a class name. Explicit supported import and export class lists make behavior visible, testable, and safer.

The system also avoids assuming every exportable thing should be importable. Grades are a reporting artifact, so they are export-only. Questionnaire packages are graph-shaped, so they use a package service. Assignment participants and teams need assignment context, so they override the generic table-shaped behavior.

This produces a hybrid design:

* Generic where the data is model-shaped and safe to process uniformly
* Specialized where the data is contextual, graph-shaped, computed, or safety-sensitive

== Testing and Verification ==

The implementation should be verified at three levels:

* Model and helper specs for metadata, hidden fields, package export/import, and row-level import behavior
* Request specs for metadata endpoints, import endpoints, export endpoints, unsupported classes, and missing context
* Frontend tests or manual verification for modal metadata loading, preview, duplicate action selection, and generated downloads

Important cases include:

* Unsupported class names are rejected
* Missing assignment_id is rejected for assignment-scoped teams and participants
* Missing referenced records produce readable validation errors
* Duplicate rows use the selected duplicate strategy
* Questionnaire package preview reports intended actions before import
* Grade export is limited to the selected assignment and teaching-staff permissions

File:Import-sequence.png

2026-04-27T02:26:08Z

Kdnadkar: Kdnadkar uploaded a new version of File:Import-sequence.png

File:Export-sequence.png

2026-04-27T02:25:59Z

Kdnadkar: Kdnadkar uploaded a new version of File:Export-sequence.png

File:Import-export-architecture.png

2026-04-27T02:25:46Z

Kdnadkar: Kdnadkar uploaded a new version of File:Import-export-architecture.png

File:Export-sequence.png

2026-04-27T02:21:59Z

Kdnadkar:

File:Import-export-architecture.png

2026-04-27T02:21:48Z

Kdnadkar: Kdnadkar uploaded a new version of File:Import-export-architecture.png

File:Import-sequence.png

2026-04-27T02:21:34Z

Kdnadkar:

File:Import-export-architecture.png

2026-04-27T02:13:50Z

Kdnadkar:

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-04-14T01:17:44Z

Kdnadkar: /* Final Project Design Document */

= Final Project Design Document =

For the final project work on E2606, we will be performing the following tasks:
* More comments to the code contributions made, as many methods added currently go unexplained as highlighted in the existing E2606 feedback
* An explicit whitelist of classes available for import/export added within the Wiki and mix-in comments
* Picture / Video Examples of importing and exporting each class type will be added
* Rework or Strengthen Questionnaires import/export to simplify the process and code readability
* Standardizing successes and failures in the import pipeline
* Revisit Team import/export documentation as it currently is focused on <Code>AssignmentTeam</code> and <Code>MentoredTeam</code> and NOT <Code>CourseTeam</code>
* Creating a UML Diagram for the importable/exportable additions.
* Adding the ImportableExportable mixin to the Course and AssignmentParticipant classes.

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Importing Teams and Topics utilized the same single-model flow as Users and is covered in [[#Expansion of Existing Import/Export |Expansion of Existing Import/Export]]
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in given they would be single model exports. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "Users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has Participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

To reach the above-mentioned Import/Export features from Home, the admin can navigate to:

1. Manage -> Users

2. Manage -> Assignments -> Edit an Assignment -> Topics

3. Manage -> Assignments -> Edit an Assignment -> Etc. -> Create Teams

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system handles both exporting single models and exporting sub-models that have <code> has_many</code> or <code>belongs_to</code> relations. This can be triggered by the "export related sub-models" switch on the frontend.

The graph export section lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

= Additional Changes =

Some QoL updates were made within this project, to allow easier functional tests. This was with respect to user and team editing. When performing initial checkout, roles were not appearing within the front-end Users table, so this project addresses that by adjusting the user_serializer.rb wiring to allow role, parent, and institution to be correctly viewable.

= Members =

* Kiran Nadkarni (kdnadkar@ncsu.edu)
* Mihir Kamat (mskamat@ncsu.edu)

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-04-14T01:02:27Z

Kdnadkar: /* Final Project */

= Final Project Design Document =

For the final project work on E2606, we will be performing the following tasks:
* More comments to the code contributions made, as many methods added currently go unexplained as highlighted in the existing E2606 feedback
* An explicit whitelist of classes available for import/export added within the Wiki and mix-in comments
* Picture / Video Examples of importing and exporting each class type will be added
* Rework or Strengthen Questionnaires import/export to simplify the process and code readability
* Standardizing successes and failures in the import pipeline
* Revisit Team import/export documentation as it currently is focused on <Code>AssignmentTeam</code> and <Code>MentoredTeam</code> and NOT <Code>CourseTeam</code>
* Creating a UML Diagram for the importable/exportable additions.

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Importing Teams and Topics utilized the same single-model flow as Users and is covered in [[#Expansion of Existing Import/Export |Expansion of Existing Import/Export]]
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in given they would be single model exports. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "Users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has Participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

To reach the above-mentioned Import/Export features from Home, the admin can navigate to:

1. Manage -> Users

2. Manage -> Assignments -> Edit an Assignment -> Topics

3. Manage -> Assignments -> Edit an Assignment -> Etc. -> Create Teams

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system handles both exporting single models and exporting sub-models that have <code> has_many</code> or <code>belongs_to</code> relations. This can be triggered by the "export related sub-models" switch on the frontend.

The graph export section lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

= Additional Changes =

Some QoL updates were made within this project, to allow easier functional tests. This was with respect to user and team editing. When performing initial checkout, roles were not appearing within the front-end Users table, so this project addresses that by adjusting the user_serializer.rb wiring to allow role, parent, and institution to be correctly viewable.

= Members =

* Kiran Nadkarni (kdnadkar@ncsu.edu)
* Mihir Kamat (mskamat@ncsu.edu)

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-04-14T00:52:55Z

Kdnadkar: /* Final Project */

= Final Project =

For the final project work on E2606, we will be performing the following tasks:
* More comments to the code contributions made, as many methods added currently go unexplained as highlighted in the existing E2606 feedback
* An explicit whitelist of classes available for import/export added within the Wiki and mix-in comments
* Picture / Video Examples of importing and exporting each class type will be added
* Rework or Strengthen Questionnaires import/export to simplify the process and code readability
* Standardizing successes and failures in the import pipeline
* Revisit Team import/export documentation as it currently is focused on <Code>AssignmentTeam</code> and <Code>MentoredTeam</code> and NOT <Code>CourseTeam</code>
* Creating a UML Diagram for the importable/exportable additions.

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Importing Teams and Topics utilized the same single-model flow as Users and is covered in [[#Expansion of Existing Import/Export |Expansion of Existing Import/Export]]
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in given they would be single model exports. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "Users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has Participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

To reach the above-mentioned Import/Export features from Home, the admin can navigate to:

1. Manage -> Users

2. Manage -> Assignments -> Edit an Assignment -> Topics

3. Manage -> Assignments -> Edit an Assignment -> Etc. -> Create Teams

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system handles both exporting single models and exporting sub-models that have <code> has_many</code> or <code>belongs_to</code> relations. This can be triggered by the "export related sub-models" switch on the frontend.

The graph export section lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

= Additional Changes =

Some QoL updates were made within this project, to allow easier functional tests. This was with respect to user and team editing. When performing initial checkout, roles were not appearing within the front-end Users table, so this project addresses that by adjusting the user_serializer.rb wiring to allow role, parent, and institution to be correctly viewable.

= Members =

* Kiran Nadkarni (kdnadkar@ncsu.edu)
* Mihir Kamat (mskamat@ncsu.edu)

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-04-13T23:34:37Z

Kdnadkar:

= Final Project =

For the final project work on E2606, we will be performing the following tasks:
* More comments to the code contributions made, as many methods added currently go unexplained
* An explicit whitelist of classes available for import/export
* Picture / Video Examples of importing and exporting each class type will be added
* Rework or Strengthen Questionnaires import/export to simplify the process and code readability
* Standardizing successes and failures in the import pipeline
* Revisit Team import/export documentation as it currently is focused on <Code>AssignmentTeam</code> and <Code>MentoredTeam</code> and NOT <Code>CourseTeam</code>

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Importing Teams and Topics utilized the same single-model flow as Users and is covered in [[#Expansion of Existing Import/Export |Expansion of Existing Import/Export]]
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in given they would be single model exports. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "Users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has Participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

To reach the above-mentioned Import/Export features from Home, the admin can navigate to:

1. Manage -> Users

2. Manage -> Assignments -> Edit an Assignment -> Topics

3. Manage -> Assignments -> Edit an Assignment -> Etc. -> Create Teams

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system handles both exporting single models and exporting sub-models that have <code> has_many</code> or <code>belongs_to</code> relations. This can be triggered by the "export related sub-models" switch on the frontend.

The graph export section lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

= Additional Changes =

Some QoL updates were made within this project, to allow easier functional tests. This was with respect to user and team editing. When performing initial checkout, roles were not appearing within the front-end Users table, so this project addresses that by adjusting the user_serializer.rb wiring to allow role, parent, and institution to be correctly viewable.

= Members =

* Kiran Nadkarni (kdnadkar@ncsu.edu)
* Mihir Kamat (mskamat@ncsu.edu)

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-04-13T23:34:05Z

Kdnadkar:

= Final Project =

For the final project work on E2606, we will be performing the following tasks:
* More comments to the code contributions made, as many methods added currently go unexplained
* An explicit whitelist of classes available for import/export
* Picture / Video Examples of importing and exporting each class type will be added
* Rework or Strengthen Questionnaires import/export to simplify the process and code readability
* Standardizing successes and failures in the import pipeline
* Revisit Team import/export documentation as it currently is focused on <Code>AssignmentTeam<\code> and <Code>MentoredTeam<\code> and NOT <Code>CourseTeam<\code>

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Importing Teams and Topics utilized the same single-model flow as Users and is covered in [[#Expansion of Existing Import/Export |Expansion of Existing Import/Export]]
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in given they would be single model exports. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "Users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has Participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

To reach the above-mentioned Import/Export features from Home, the admin can navigate to:

1. Manage -> Users

2. Manage -> Assignments -> Edit an Assignment -> Topics

3. Manage -> Assignments -> Edit an Assignment -> Etc. -> Create Teams

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system handles both exporting single models and exporting sub-models that have <code> has_many</code> or <code>belongs_to</code> relations. This can be triggered by the "export related sub-models" switch on the frontend.

The graph export section lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

= Additional Changes =

Some QoL updates were made within this project, to allow easier functional tests. This was with respect to user and team editing. When performing initial checkout, roles were not appearing within the front-end Users table, so this project addresses that by adjusting the user_serializer.rb wiring to allow role, parent, and institution to be correctly viewable.

= Members =

* Kiran Nadkarni (kdnadkar@ncsu.edu)
* Mihir Kamat (mskamat@ncsu.edu)

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-04-13T23:33:44Z

Kdnadkar:

= Final Project =

For the final project work on E2606, we will be performing the following tasks:
* More comments to the code contributions made, as many methods added currently go unexplained
* An explicit whitelist of classes available for import/export
* Picture / Video Examples of importing and exporting each class type will be added
* Rework or Strengthen Questionnaires import/export to simplify the process and code readability
* Standardizing successes and failures in the import pipeline
* Revisit Team import/export documentation as it currently is focused on <Code>AssignmentTeam<\Code> and <Code>MentoredTeam<\Code> and NOT <Code>CourseTeam<\Code>

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Importing Teams and Topics utilized the same single-model flow as Users and is covered in [[#Expansion of Existing Import/Export |Expansion of Existing Import/Export]]
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in given they would be single model exports. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "Users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has Participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

To reach the above-mentioned Import/Export features from Home, the admin can navigate to:

1. Manage -> Users

2. Manage -> Assignments -> Edit an Assignment -> Topics

3. Manage -> Assignments -> Edit an Assignment -> Etc. -> Create Teams

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system handles both exporting single models and exporting sub-models that have <code> has_many</code> or <code>belongs_to</code> relations. This can be triggered by the "export related sub-models" switch on the frontend.

The graph export section lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

= Additional Changes =

Some QoL updates were made within this project, to allow easier functional tests. This was with respect to user and team editing. When performing initial checkout, roles were not appearing within the front-end Users table, so this project addresses that by adjusting the user_serializer.rb wiring to allow role, parent, and institution to be correctly viewable.

= Members =

* Kiran Nadkarni (kdnadkar@ncsu.edu)
* Mihir Kamat (mskamat@ncsu.edu)

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-04-13T23:32:27Z

Kdnadkar:

= Final Project =

For the final project work on E2606, we will be performing the following tasks:
* More comments to the code contributions made, as many methods added currently go unexplained
* An explicit whitelist of classes available for import/export
* Picture / Video Examples of importing and exporting each class type will be added
* Rework or Strengthen Questionnaires import/export to simplify the process and code readability
* Standardizing successes and failures in the import pipeline
* Revisit Team import/export documentation as it currently is focused on <Code>AssignmentTeam<Code\> and <Code>MentoredTeam<Code\> and NOT <Code>CourseTeam<Code\>

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Importing Teams and Topics utilized the same single-model flow as Users and is covered in [[#Expansion of Existing Import/Export |Expansion of Existing Import/Export]]
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in given they would be single model exports. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "Users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has Participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

To reach the above-mentioned Import/Export features from Home, the admin can navigate to:

1. Manage -> Users

2. Manage -> Assignments -> Edit an Assignment -> Topics

3. Manage -> Assignments -> Edit an Assignment -> Etc. -> Create Teams

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system handles both exporting single models and exporting sub-models that have <code> has_many</code> or <code>belongs_to</code> relations. This can be triggered by the "export related sub-models" switch on the frontend.

The graph export section lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

= Additional Changes =

Some QoL updates were made within this project, to allow easier functional tests. This was with respect to user and team editing. When performing initial checkout, roles were not appearing within the front-end Users table, so this project addresses that by adjusting the user_serializer.rb wiring to allow role, parent, and institution to be correctly viewable.

= Members =

* Kiran Nadkarni (kdnadkar@ncsu.edu)
* Mihir Kamat (mskamat@ncsu.edu)

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-04-13T23:31:17Z

Kdnadkar:

= Final Project =

For the final project work on E2606, we will be performing the following tasks:
* More comments to the code contributions made, as many methods added currently go unexplained
* An explicit whitelist of classes available for import/export
* Picture / Video Examples of importing and exporting each class type will be added
* Rework or Strengthen Questionnaires import/export to simplify the process and code readability
* Standardizing successes and failures in the import pipeline
* Revisit Team import/export documentation as it currently is focused on (AssignmentTeam) and (MentoredTeam) and NOT (CourseTeam)

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Importing Teams and Topics utilized the same single-model flow as Users and is covered in [[#Expansion of Existing Import/Export |Expansion of Existing Import/Export]]
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in given they would be single model exports. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "Users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has Participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

To reach the above-mentioned Import/Export features from Home, the admin can navigate to:

1. Manage -> Users

2. Manage -> Assignments -> Edit an Assignment -> Topics

3. Manage -> Assignments -> Edit an Assignment -> Etc. -> Create Teams

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system handles both exporting single models and exporting sub-models that have <code> has_many</code> or <code>belongs_to</code> relations. This can be triggered by the "export related sub-models" switch on the frontend.

The graph export section lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

= Additional Changes =

Some QoL updates were made within this project, to allow easier functional tests. This was with respect to user and team editing. When performing initial checkout, roles were not appearing within the front-end Users table, so this project addresses that by adjusting the user_serializer.rb wiring to allow role, parent, and institution to be correctly viewable.

= Members =

* Kiran Nadkarni (kdnadkar@ncsu.edu)
* Mihir Kamat (mskamat@ncsu.edu)

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-03-31T02:23:22Z

Kdnadkar: /* Approach */

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Importing Teams and Topics utilized the same single-model flow as Users and is covered in [[#Expansion of Existing Import/Export |Expansion of Existing Import/Export]]
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in given they would be single model exports. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "Users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has Participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

To reach the above-mentioned Import/Export features from Home, the admin can navigate to:

1. Manage -> Users

2. Manage -> Assignments -> Edit an Assignment -> Topics

3. Manage -> Assignments -> Edit an Assignment -> Etc. -> Create Teams

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system handles both exporting single models and exporting sub-models that have <code> has_many</code> or <code>belongs_to</code> relations. This can be triggered by the "export related sub-models" switch on the frontend.

The graph export section lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

= Additional Changes =

Some QoL updates were made within this project, to allow easier functional tests. This was with respect to user and team editing. When performing initial checkout, roles were not appearing within the front-end Users table, so this project addresses that by adjusting the user_serializer.rb wiring to allow role, parent, and institution to be correctly viewable.

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-03-31T02:17:57Z

Kdnadkar: /* Expansion of Existing Import/Export */

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Importing Teams and Topics is covered in [[#Expansion of Existing Import/Export |Expansion of Existing Import/Export]]
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in given they would be single model exports. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "Users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has Participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

To reach the above-mentioned Import/Export features from Home, the admin can navigate to:

1. Manage -> Users

2. Manage -> Assignments -> Edit an Assignment -> Topics

3. Manage -> Assignments -> Edit an Assignment -> Etc. -> Create Teams

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system handles both exporting single models and exporting sub-models that have <code> has_many</code> or <code>belongs_to</code> relations. This can be triggered by the "export related sub-models" switch on the frontend.

The graph export section lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

= Additional Changes =

Some QoL updates were made within this project, to allow easier functional tests. This was with respect to user and team editing. When performing initial checkout, roles were not appearing within the front-end Users table, so this project addresses that by adjusting the user_serializer.rb wiring to allow role, parent, and institution to be correctly viewable.

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-03-31T02:16:41Z

Kdnadkar: /* Approach */

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Importing Teams and Topics is covered in [[#Expansion of Existing Import/Export |Expansion of Existing Import/Export]]
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in, not the graph-based system. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "Users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has Participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

To reach the above-mentioned Import/Export features from Home, the admin can navigate to:

1. Manage -> Users

2. Manage -> Assignments -> Edit an Assignment -> Topics

3. Manage -> Assignments -> Edit an Assignment -> Etc. -> Create Teams

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system handles both exporting single models and exporting sub-models that have <code> has_many</code> or <code>belongs_to</code> relations. This can be triggered by the "export related sub-models" switch on the frontend.

The graph export section lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

= Additional Changes =

Some QoL updates were made within this project, to allow easier functional tests. This was with respect to user and team editing. When performing initial checkout, roles were not appearing within the front-end Users table, so this project addresses that by adjusting the user_serializer.rb wiring to allow role, parent, and institution to be correctly viewable.

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-03-31T02:16:24Z

Kdnadkar: /* Approach */

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Importing Teams and Topics is covered in [[#Expansion of Existing Import/Export |Expansion of Existing Import/Export]]
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Importing/Exporting Teams & Topics required minimal changes to the existing mix-in.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in, not the graph-based system. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "Users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has Participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

To reach the above-mentioned Import/Export features from Home, the admin can navigate to:

1. Manage -> Users

2. Manage -> Assignments -> Edit an Assignment -> Topics

3. Manage -> Assignments -> Edit an Assignment -> Etc. -> Create Teams

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system handles both exporting single models and exporting sub-models that have <code> has_many</code> or <code>belongs_to</code> relations. This can be triggered by the "export related sub-models" switch on the frontend.

The graph export section lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

= Additional Changes =

Some QoL updates were made within this project, to allow easier functional tests. This was with respect to user and team editing. When performing initial checkout, roles were not appearing within the front-end Users table, so this project addresses that by adjusting the user_serializer.rb wiring to allow role, parent, and institution to be correctly viewable.

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-03-31T01:49:37Z

Kdnadkar: /* Expansion of Existing Import/Export */

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in, not the graph-based system. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "Users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has Participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

To reach the above-mentioned Import/Export features from Home, the admin can navigate to:

1. Manage -> Users

2. Manage -> Assignments -> Edit an Assignment -> Topics

3. Manage -> Assignments -> Edit an Assignment -> Etc. -> Create Teams

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

= Additional Changes =

Some QoL updates were made within this project, to allow easier functional tests. This was with respect to user and team editing. When performing initial checkout, roles were not appearing within the front-end Users table, so this project addresses that by adjusting the user_serializer.rb wiring to allow role, parent, and institution to be correctly viewable.

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-03-31T01:49:05Z

Kdnadkar:

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in, not the graph-based system. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "Users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has Participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

To reach the above-mentioned Import/Export features from Home, the admin can navigate to:

(1) Manage -> Users

(2) Manage -> Assignments -> Edit an Assignment -> Topics

(3) Manage -> Assignments -> Edit an Assignment -> Etc. -> Create Teams

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

= Additional Changes =

Some QoL updates were made within this project, to allow easier functional tests. This was with respect to user and team editing. When performing initial checkout, roles were not appearing within the front-end Users table, so this project addresses that by adjusting the user_serializer.rb wiring to allow role, parent, and institution to be correctly viewable.

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-03-31T01:32:35Z

Kdnadkar: /* Expansion of Existing Import/Export */

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in, not the graph-based system. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "Users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has Participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-03-31T01:32:18Z

Kdnadkar: /* Expansion of Existing Import/Export */

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in, not the graph-based system. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "Users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-03-31T01:31:52Z

Kdnadkar:

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in, not the graph-based system. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

CSC/ECE 517 Spring 2026 - E2606. Finishing Import and Export helper module

2026-03-31T01:30:58Z

Kdnadkar:

= Problem Statement =

Project E2560 introduced a generic framework for import and export functionality in Expertiza. Before this, each entity such as users, teams, or questionnaires had its own separate import/export logic, even though they all performed similar tasks like reading CSV files, mapping fields, validating data, and handling duplicates. This led to repeated and tightly coupled code.

E2560 solved this by creating a unified structure using a reusable mixin (ImportableExportable), a service layer (ImportExportManager), configurable field mappings, and strategy-based duplicate handling. Models define their required fields and duplicate rules, while the service layer handles the CSV processing in a consistent and reusable way.

The current E2560 import system shows almost every field that exists in the backend. This means users see many technical or system-controlled fields, such as internal IDs, timestamps, and tokens, which they do not need to understand or import. Showing all these fields makes the screen confusing and harder to use. It also increases the risk of users selecting or mapping fields that should actually be managed only by the system. In addition, the duplicate handling logic is very general and does not clearly explain how duplicates are treated for different types of data. The system is also not easily extendable to support importing other entities like Teams or Topics.

In the current project, the import feature should be redesigned to support multiple entity types in a clean and scalable way. For each entity, we should clearly define which fields are mandatory, which are optional, and which are system-managed. Only the necessary and user-editable fields should be displayed in the UI. Duplicate handling rules should be defined separately for each entity type, with a short and clear explanation for each rule. This will make the system easier to use, reduce confusion, improve data quality, and allow future expansion without major rework.

You should write code to import users, teams, topics, and questionnaires with their associated “advice.”

You should write code to export assignment grades, author-feedback grades, teammate-review grades, users, teams, topics, and questionnaires with their associated “advice.”

= Previous Work =

This project builds heavily on [[CSC/ECE 517 Fall 2025 - E2560. Framework for Import and Export]]. Please go through the page to have a better idea of the working of the system.

= Approach =

* Importing/Exporting Users existed beforehand.
* Exporting Questionnaires with their associated QuestionAdvices involves exporting a model along with its constituent models. We used the system described in the section [[#Graph-based Export System|Graph-based Export System]] on this page to achieve this.
* Exporting Grades involves exporting data that doesn't persist in the DB or is a combination of different models in the db. We used the approach described in the section [[#Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data|Model Spoofing for Exporting Non-Persistent/Calculated DB data]].
* Hiding fields from the user is covered in [[#Hidden Fields|Hidden Fields]].

= Graph-based Export System =

The previous version of the model export system only exported single models. The current graph export system lives in backend/app/helpers/export_helper.rb and works in two stages:

1. Build a relationship graph from a root class

2. Export one CSV payload per class discovered in that graph

This separation allows the system to first understand the structure of model relationships before delegating the actual CSV generation to the existing export pipeline. It keeps concerns clean: graph construction is handled independently from data export.

== Graph Building ==

<code>export_has_many_graph(root_class)</code> starts by calling <code>build_export_graph(root_class)</code>.

Each graph node looks like this:

<pre> { class_name: ..., headers: ..., has_many: [...] } </pre>

For each class, <code>build_export_graph</code>:

1. records the class name so it can be referenced later in the export phase

2. determines headers with <code>mandatory_headers_for</code>, which defines the base set of fields for export

3. follows direct <code>has_many</code> associations declared on the model

4. also finds descendant models that <code>belongs_to</code> the current class using <code>descendants_with_belongs_to_parent</code>

This means the graph is not limited to explicitly declared <code>has_many</code> relationships. It also captures implicit reverse relationships through <code>belongs_to</code>, ensuring broader coverage of related data.

So even when a relationship is discovered through <code>belongs_to</code>, it still gets stored under the <code>has_many</code> array in the graph. This creates a consistent structure where all outward relationships are treated uniformly, simplifying traversal and later processing.

== Cycle Protection ==

The graph builder uses a visited set to avoid infinite recursion.

If a class is revisited, it returns a shortened node like:

<pre> { class_name: ..., headers: ..., cyclic_reference: true, has_many: [] } </pre>

This keeps the graph traversal safe when models point back to each other. It prevents stack overflows and runaway processing while still indicating that a relationship exists. The <code>cyclic_reference</code> flag acts as a signal that the traversal was intentionally stopped at that point.

== Export Header Propagation ==

Once the graph is built, <code>each_graph_node_for_export</code> walks through it and computes the export headers for each class.

It does this by:

1. starting with the node’s own headers

2. passing prefixed parent headers down to child nodes so relational context is preserved

3. removing identifier-style fields like <code>id</code> and <code>*_id</code> to avoid leaking internal keys and to keep the output cleaner

Header prefixing is handled by <code>prefix_headers_with_class_name</code>, ensuring that fields inherited from parent nodes remain distinguishable and do not collide with local fields. Identifier cleanup is handled by <code>remove_identifier_fields</code>, which strips out fields that are primarily useful for database relations rather than for export consumers. This step effectively flattens relational context into a CSV-friendly structure.

== CSV Export Generation ==

After gathering headers for each class, <code>export_has_many_graph</code> calls:

<pre> Export.perform(class_name.constantize, headers, graph_export: false) </pre>

for every discovered class.

That means graph export does not generate CSV directly itself. Instead, it reuses the normal export flow for each class separately. This avoids duplication of logic and ensures consistency with standard exports. Each class is processed independently, but with headers that have been enriched by the graph traversal. The final result is an array of export payloads, one per class.

== Current Behavior in Practice ==

The behavior is covered in backend/spec/helpers/export_helper_spec.rb. For example, starting from Questionnaire, the graph export reaches related classes such as:

1. Item

2. QuestionAdvice

3. Answer

The specs then verify that the returned CSV contents include real records for each of those classes.

= Model Spoofing for Exporting Non-Persistent/Calculated/Custom DB data =

In the <code>Export</code> service, an additional redirection was added to the code that performs the exporting. Instead of going through <code>model.all</code> and adding each row to the csv, the export code calls a <code>filter</code> function which retrieves data from the model. By default it is the <code>all</code> function, but it can be set to any custom function inside the model that aggregates or transforms data across one or more tables and returns it in a structured, row-based format.

This allows the export system to work not only with persisted database records, but also with computed, derived, or combined datasets that do not exist as a single table in the schema. This data can then be safely exported to the end-user using the same export pipeline, without requiring changes to the core CSV generation logic.

The requirements for spoofing a model is to:

1. Not inherit from <code>ActiveRecord</code> (this model does not correspond to a real database table and should remain decoupled from persistence concerns).

2. Have it be <code>ImportableExportable</code> (so it conforms to the interface expected by the export system).

3. Define its <code>self.column_names</code> (We're essentially faking a model so these are the attributes that it would supposedly have).

4. Define the row format as a struct (ensuring each returned row has a consistent shape and can be treated like a typical model instance).

5. Define its <code>mandatory_fields</code> and <code>optional_fields</code> (so the export system knows which fields must always be included versus those that are conditional or supplementary).

6. Define the method for aggregating the data as a static method (this method is responsible for collecting, joining, and shaping the data into exportable rows).

7. Set <code>filter -> { method_name }</code> (this tells the export system to use the custom aggregation method instead of <code>all</code> when retrieving records).

This approach effectively “spoofs” a model, allowing complex or computed datasets to be exported as if they were standard ActiveRecord-backed models.

See the implementation of models/grades.rb as an example.

= Hidden Fields =

It was also mentioned in the project description that the end user has to be insulated from accessing certain fields, for less confusion, as well as for increased security. To achieve this, a new type of field apart from <code>mandatory_fields</code> and <code>optional_fields</code> have been added, called <code>hidden_fields</code>. These fields define what is effectively completely hidden from the end user.

As defined in the E2560 project methodology, when exporting or importing, the frontend first sends a request to the backend for the metadata to show which fields should be exported or not. This project completely insulates the end user from fields that need to be hidden by removing the <code>hidden_fields</code> metadata completely from the mandatory and optional fields that are sent to the frontend.

Hidden fields take precedence over mandatory fields. If a hidden field is found in mandatory fields, it will first and foremost be considered as a hidden field, and not a mandatory field.

= Expansion of Existing Import/Export =

The Teams and Topics models utilized an expansion of the existing import/export mix-in, not the graph-based system. That being said, Teams CSV handling was reworked to use a unique team name and be formatted keying off of participants. This specifically scopes the import/export functionality to the assignment, because all teams would be associated within an assignment. On the front-end, the buttons and import-export pop-up were changed to be nearly identical to the "users" interface. Wiring was also established between the back-end and front-end for teams as it looked like E2560 established front-end-only mock examples of teams without referencing any data in the db.

Topics are more isolated, they are also associated with an assignment, but there are no joined models (Teams has participants for example). The main change for topics was naming and additional migration being run to support SignUpTopic being changed to ProjectTopic. Import/export behavior was added, as well as a serializer to support cleaner data for the front-end. Lastly for the front-end, the buttons/UI for import/export were changed to be similar to "Users" and "Teams".