Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Spring 2024 - E2401 Implementing and testing import & export controllers

2024-03-27T09:16:25Z

Jrnolan2: Add picture to description of project

This wiki page is for the description of changes made under E2401 OSS assignment for Spring 2024, CSC/ECE 517.

== Team==
=== Mentor ===
* Ed Gehringer

=== Team Members ===
* Gwen Mason - cpmason@ncsu.edu
* John Nolan - jrnolan2@ncsu.edu
* Meet Patel - mpatel29@ncsu.edu

== Expertiza Background ==
Expertiza is an open source Ruby on Rails application for teachers to manage classrooms. Expertiza is maintained by the teaching staff at NC State along with the very students who use it. Expertiza has many different models saved inside it including assignments, questions, and users. Relevant to our project, these models can be imported or exported through the website as CSV files for ease of uploading or editing many objects at once.

== Description of project ==
[[File:Team E2401 Import Shared Functionality.PNG|thumb|right|Import page is now common to all models. See the model type is a parameter in the web address bar.]]
Our project's description changed late into the project submission period. We were originally tasked with using the new implementation of the import_file_controller.rb [[CSC517 Spring 2019/E1923 New Framework For Import/Export|created by team E1923]]. Their project involved consolidating all the import functions into one common functionality. Before team 1923, each model used their own import code. However, these import codes shared a lot of functionality and were ripe for refactoring. E1923's import_file_controller.rb used polymorphism to fix this duplicate code. Each model had import specific methods that the import file controller calls and receives the information necessary to create and save the respective models. We spent the early stages of the project understanding the previous team's code. Our original goal was to take several models not yet utilizing this import_file_controller.rb and extending them to use the common code.

However, later into the project we learned that unfortunately, [https://github.com/expertiza/expertiza/pull/1438 team E1923's code was never merged into the main Expertiza repository]. In 2019, the project was tagged with merge, however, it never actually was and later closed in 2021. Upon this discovery, our project's mission changed dramatically. Instead of getting other models to work with the existing import_file_controller.rb, our goal was to redo the changes of team 1923. Our goal is to get justice for the work of E1923 by getting their changes into Expertiza. However, our project was not as easy as just copying over the changes from E1923. The codebase has changed significantly over the previous 5 years which meant that much of their code and tests had to be fast forwarded to integrate into the current repository. Additionally, E1923 finished their project without the static analysis tools that currently check for styling and vulnerabilities. There were 100s of errors on GitHub Actions related to styling, test failures, and vulnerabilities that we had to correct given the scale of E1923's changes. Because of the late-term revelation, Dr. Ed graciously extended our deadline by a few days to allow our new goal to be completed.

== Our changes ==
Our changes can be split into three main phases. All changes worked towards a common goal to have E1923's code integrated into the current repository.

=== 1. Copying changes over ===
Under our mentor's advice, we could not directly merge E1923's code into the current Expertiza. This is simply because there have been so many changes over the last 5 years that the merge conflicts would be infeasible to sift through. Instead, we had to manually go file by file and make the changes matching E1923. We had to pay attention to two main diffs for each of the 47 files that E1923 changed. We first compared the changes that E1923 made to the 2019 Expertiza repo then compare the original 2019 repo against the modern repo. We had to resolve all the conflicts that arose from changes in these files throughout the years. For example, if a method was renamed during the last 5 years, we had to make sure the copied code was changed to use that new method.

=== 2. Test fixing ===
Upon completion of the copying phase, we discovered that many tests were failing. Luckily, many of these tests had common causes (originally just a few compilation errors). Despite our scrupulousness, a few bugs had still crawled into our updated code. We had to add some missing methods. Eventually, we got to the bugs not caused by simple copy errors as well. Some bugs were caused by outdated tests and deeper routed issues that occurred from trying to integrate the 2019 and current codebases. We rewrote tests and had to refactor E1923's code until we were finally to have all tests fixing.

=== 3. Static analysis fixing ===
[[File:Team E2401 Codeclimate Errors.png|thumb|right|Original codeclimate errors after copying and integrating E1923's code]]
As aforementioned, the previous team's project occurred at a time before GitHub Actions verified many aspects of the code using static analysis. This resulted in many, many errors being brought forth when we copied the previous team's code. Most severe were a handful of security vulnerabilities that existed in E1923's code. All the found security vulnerabilities existed in import_file_controller.rb. The first, involved constantize. The error existed because constantize was being run on user provided input, therefore, a malicious actor could provide unintended classes to the import method and receive information about these classes and maybe even have these other classes added to the website's database. To fix this error, we created a simple ALLOWED_MODELS whitelist array that includes the strings of all the permitted class types. We then call constantize on the match one of these strings to avoid the GitHub vulnerability message. This way, only the classes we desire to be imported will actually be allowed by the allowed_model method. The second vulnerability, involved the eval method. Upon further research, this is a '''terrible''' method to use in general in Ruby. The eval method will run the code of whatever is inside the string provided to it. While this is powerful, a malicious user can use this to run arbitrary code on Expertiza. The eval method was overkill for what it was really trying to do, parse a hash. We replaced the eval function with JSON.parse() which fixes this vulnerability. After the security vulnerabilities, we had 404 style errors thrown by codeclimate. Many of these changes were simple yet very tedious. For example, many suggested change double quotes to single quotes, removing trailing or excess whitespace, correcting indentation, removing unnecessary .to_s method calls, and adding spaces around the curly braces of hashes. After these changes, only 33 remain. Reviewing the GitHub Actions, we recognize that codeclimate is one the checks that is not required and we would recommend that many of the remaining suggestions are unhelpful and some may be suppressed. For example, the recommended fixes for "Use the new Ruby 1.9 hash syntax" involves changing the rocket (=>) to a colon (:), however, this change in our context will actually cause a compilation error. Furthermore, some of the refactoring suggestions are unnecessary. For example, the branch condition size was determined to high for the get_topic_attributes method in sign_up_topic.rb, however, the method is simply only adding fields if they exist and splitting this code might make it more confusing to read, not less.

== Future improvements ==
Like our preceding team, E1923, we have not touched the exporting code. Exporting could receive a mirroring change to that of importing to allow for a similar sharing of code. Additionally, as mentioned in the static analysis fixing section, there are some codeclimate errors remaining in our final pull request. If the teaching staff believes all of these should be fixed as the automated system describes, this would be a future necessary task, and potentially a good option as a good continuation for our team into Project 4.

File:Team E2401 Import Shared Functionality.PNG

2024-03-27T09:14:14Z

Jrnolan2:

CSC/ECE 517 Spring 2024 - E2401 Implementing and testing import & export controllers

2024-03-27T09:10:02Z

Jrnolan2: /* 3. Static analysis fixing */ Changing webp to png file (webp can be deleted)

This wiki page is for the description of changes made under E2401 OSS assignment for Spring 2024, CSC/ECE 517.

== Team==
=== Mentor ===
* Ed Gehringer

=== Team Members ===
* Gwen Mason - cpmason@ncsu.edu
* John Nolan - jrnolan2@ncsu.edu
* Meet Patel - mpatel29@ncsu.edu

== Expertiza Background ==
Expertiza is an open source Ruby on Rails application for teachers to manage classrooms. Expertiza is maintained by the teaching staff at NC State along with the very students who use it. Expertiza has many different models saved inside it including assignments, questions, and users. Relevant to our project, these models can be imported or exported through the website as CSV files for ease of uploading or editing many objects at once.

== Description of project ==
Our project's description changed late into the project submission period. We were originally tasked with using the new implementation of the import_file_controller.rb [[CSC517 Spring 2019/E1923 New Framework For Import/Export|created by team E1923]]. Their project involved consolidating all the import functions into one common functionality. Before team 1923, each model used their own import code. However, these import codes shared a lot of functionality and were ripe for refactoring. E1923's import_file_controller.rb used polymorphism to fix this duplicate code. Each model had import specific methods that the import file controller calls and receives the information necessary to create and save the respective models. We spent the early stages of the project understanding the previous team's code. Our original goal was to take several models not yet utilizing this import_file_controller.rb and extending them to use the common code.

However, later into the project we learned that unfortunately, [https://github.com/expertiza/expertiza/pull/1438 team E1923's code was never merged into the main Expertiza repository]. In 2019, the project was tagged with merge, however, it never actually was and later closed in 2021. Upon this discovery, our project's mission changed dramatically. Instead of getting other models to work with the existing import_file_controller.rb, our goal was to redo the changes of team 1923. Our goal is to get justice for the work of E1923 by getting their changes into Expertiza. However, our project was not as easy as just copying over the changes from E1923. The codebase has changed significantly over the previous 5 years which meant that much of their code and tests had to be fast forwarded to integrate into the current repository. Additionally, E1923 finished their project without the static analysis tools that currently check for styling and vulnerabilities. There were 100s of errors on GitHub Actions related to styling, test failures, and vulnerabilities that we had to correct given the scale of E1923's changes. Because of the late-term revelation, Dr. Ed graciously extended our deadline by a few days to allow our new goal to be completed.

== Our changes ==
Our changes can be split into three main phases. All changes worked towards a common goal to have E1923's code integrated into the current repository.

=== 1. Copying changes over ===
Under our mentor's advice, we could not directly merge E1923's code into the current Expertiza. This is simply because there have been so many changes over the last 5 years that the merge conflicts would be infeasible to sift through. Instead, we had to manually go file by file and make the changes matching E1923. We had to pay attention to two main diffs for each of the 47 files that E1923 changed. We first compared the changes that E1923 made to the 2019 Expertiza repo then compare the original 2019 repo against the modern repo. We had to resolve all the conflicts that arose from changes in these files throughout the years. For example, if a method was renamed during the last 5 years, we had to make sure the copied code was changed to use that new method.

=== 2. Test fixing ===
Upon completion of the copying phase, we discovered that many tests were failing. Luckily, many of these tests had common causes (originally just a few compilation errors). Despite our scrupulousness, a few bugs had still crawled into our updated code. We had to add some missing methods. Eventually, we got to the bugs not caused by simple copy errors as well. Some bugs were caused by outdated tests and deeper routed issues that occurred from trying to integrate the 2019 and current codebases. We rewrote tests and had to refactor E1923's code until we were finally to have all tests fixing.

=== 3. Static analysis fixing ===
[[File:Team E2401 Codeclimate Errors.png|thumb|right|Original codeclimate errors after copying and integrating E1923's code]]
As aforementioned, the previous team's project occurred at a time before GitHub Actions verified many aspects of the code using static analysis. This resulted in many, many errors being brought forth when we copied the previous team's code. Most severe were a handful of security vulnerabilities that existed in E1923's code. All the found security vulnerabilities existed in import_file_controller.rb. The first, involved constantize. The error existed because constantize was being run on user provided input, therefore, a malicious actor could provide unintended classes to the import method and receive information about these classes and maybe even have these other classes added to the website's database. To fix this error, we created a simple ALLOWED_MODELS whitelist array that includes the strings of all the permitted class types. We then call constantize on the match one of these strings to avoid the GitHub vulnerability message. This way, only the classes we desire to be imported will actually be allowed by the allowed_model method. The second vulnerability, involved the eval method. Upon further research, this is a '''terrible''' method to use in general in Ruby. The eval method will run the code of whatever is inside the string provided to it. While this is powerful, a malicious user can use this to run arbitrary code on Expertiza. The eval method was overkill for what it was really trying to do, parse a hash. We replaced the eval function with JSON.parse() which fixes this vulnerability. After the security vulnerabilities, we had 404 style errors thrown by codeclimate. Many of these changes were simple yet very tedious. For example, many suggested change double quotes to single quotes, removing trailing or excess whitespace, correcting indentation, removing unnecessary .to_s method calls, and adding spaces around the curly braces of hashes. After these changes, only 33 remain. Reviewing the GitHub Actions, we recognize that codeclimate is one the checks that is not required and we would recommend that many of the remaining suggestions are unhelpful and some may be suppressed. For example, the recommended fixes for "Use the new Ruby 1.9 hash syntax" involves changing the rocket (=>) to a colon (:), however, this change in our context will actually cause a compilation error. Furthermore, some of the refactoring suggestions are unnecessary. For example, the branch condition size was determined to high for the get_topic_attributes method in sign_up_topic.rb, however, the method is simply only adding fields if they exist and splitting this code might make it more confusing to read, not less.

== Future improvements ==
Like our preceding team, E1923, we have not touched the exporting code. Exporting could receive a mirroring change to that of importing to allow for a similar sharing of code. Additionally, as mentioned in the static analysis fixing section, there are some codeclimate errors remaining in our final pull request. If the teaching staff believes all of these should be fixed as the automated system describes, this would be a future necessary task, and potentially a good option as a good continuation for our team into Project 4.

File:Team E2401 Codeclimate Errors.png

2024-03-27T09:09:40Z

Jrnolan2:

CSC/ECE 517 Spring 2024 - E2401 Implementing and testing import & export controllers

2024-03-27T09:08:14Z

Jrnolan2: /* 3. Static analysis fixing */ Add image of errors

This wiki page is for the description of changes made under E2401 OSS assignment for Spring 2024, CSC/ECE 517.

== Team==
=== Mentor ===
* Ed Gehringer

=== Team Members ===
* Gwen Mason - cpmason@ncsu.edu
* John Nolan - jrnolan2@ncsu.edu
* Meet Patel - mpatel29@ncsu.edu

== Expertiza Background ==
Expertiza is an open source Ruby on Rails application for teachers to manage classrooms. Expertiza is maintained by the teaching staff at NC State along with the very students who use it. Expertiza has many different models saved inside it including assignments, questions, and users. Relevant to our project, these models can be imported or exported through the website as CSV files for ease of uploading or editing many objects at once.

== Description of project ==
Our project's description changed late into the project submission period. We were originally tasked with using the new implementation of the import_file_controller.rb [[CSC517 Spring 2019/E1923 New Framework For Import/Export|created by team E1923]]. Their project involved consolidating all the import functions into one common functionality. Before team 1923, each model used their own import code. However, these import codes shared a lot of functionality and were ripe for refactoring. E1923's import_file_controller.rb used polymorphism to fix this duplicate code. Each model had import specific methods that the import file controller calls and receives the information necessary to create and save the respective models. We spent the early stages of the project understanding the previous team's code. Our original goal was to take several models not yet utilizing this import_file_controller.rb and extending them to use the common code.

However, later into the project we learned that unfortunately, [https://github.com/expertiza/expertiza/pull/1438 team E1923's code was never merged into the main Expertiza repository]. In 2019, the project was tagged with merge, however, it never actually was and later closed in 2021. Upon this discovery, our project's mission changed dramatically. Instead of getting other models to work with the existing import_file_controller.rb, our goal was to redo the changes of team 1923. Our goal is to get justice for the work of E1923 by getting their changes into Expertiza. However, our project was not as easy as just copying over the changes from E1923. The codebase has changed significantly over the previous 5 years which meant that much of their code and tests had to be fast forwarded to integrate into the current repository. Additionally, E1923 finished their project without the static analysis tools that currently check for styling and vulnerabilities. There were 100s of errors on GitHub Actions related to styling, test failures, and vulnerabilities that we had to correct given the scale of E1923's changes. Because of the late-term revelation, Dr. Ed graciously extended our deadline by a few days to allow our new goal to be completed.

== Our changes ==
Our changes can be split into three main phases. All changes worked towards a common goal to have E1923's code integrated into the current repository.

=== 1. Copying changes over ===
Under our mentor's advice, we could not directly merge E1923's code into the current Expertiza. This is simply because there have been so many changes over the last 5 years that the merge conflicts would be infeasible to sift through. Instead, we had to manually go file by file and make the changes matching E1923. We had to pay attention to two main diffs for each of the 47 files that E1923 changed. We first compared the changes that E1923 made to the 2019 Expertiza repo then compare the original 2019 repo against the modern repo. We had to resolve all the conflicts that arose from changes in these files throughout the years. For example, if a method was renamed during the last 5 years, we had to make sure the copied code was changed to use that new method.

=== 2. Test fixing ===
Upon completion of the copying phase, we discovered that many tests were failing. Luckily, many of these tests had common causes (originally just a few compilation errors). Despite our scrupulousness, a few bugs had still crawled into our updated code. We had to add some missing methods. Eventually, we got to the bugs not caused by simple copy errors as well. Some bugs were caused by outdated tests and deeper routed issues that occurred from trying to integrate the 2019 and current codebases. We rewrote tests and had to refactor E1923's code until we were finally to have all tests fixing.

=== 3. Static analysis fixing ===
As aforementioned, the previous team's project occurred at a time before GitHub Actions verified many aspects of the code using static analysis. This resulted in many, many errors being brought forth when we copied the previous team's code. Most severe were a handful of security vulnerabilities that existed in E1923's code. All the found security vulnerabilities existed in import_file_controller.rb. The first, involved constantize. The error existed because constantize was being run on user provided input, therefore, a malicious actor could provide unintended classes to the import method and receive information about these classes and maybe even have these other classes added to the website's database. To fix this error, we created a simple ALLOWED_MODELS whitelist array that includes the strings of all the permitted class types. We then call constantize on the match one of these strings to avoid the GitHub vulnerability message. This way, only the classes we desire to be imported will actually be allowed by the allowed_model method. The second vulnerability, involved the eval method. Upon further research, this is a '''terrible''' method to use in general in Ruby. The eval method will run the code of whatever is inside the string provided to it. While this is powerful, a malicious user can use this to run arbitrary code on Expertiza. The eval method was overkill for what it was really trying to do, parse a hash. We replaced the eval function with JSON.parse() which fixes this vulnerability. After the security vulnerabilities, we had 404 style errors thrown by codeclimate. Many of these changes were simple yet very tedious. For example, many suggested change double quotes to single quotes, removing trailing or excess whitespace, correcting indentation, removing unnecessary .to_s method calls, and adding spaces around the curly braces of hashes. After these changes, only 33 remain. Reviewing the GitHub Actions, we recognize that codeclimate is one the checks that is not required and we would recommend that many of the remaining suggestions are unhelpful and some may be suppressed. For example, the recommended fixes for "Use the new Ruby 1.9 hash syntax" involves changing the rocket (=>) to a colon (:), however, this change in our context will actually cause a compilation error. Furthermore, some of the refactoring suggestions are unnecessary. For example, the branch condition size was determined to high for the get_topic_attributes method in sign_up_topic.rb, however, the method is simply only adding fields if they exist and splitting this code might make it more confusing to read, not less.
[[File:Team E2401 Codeclimate Errors.webp|thumb|left|Original codeclimate errors after copying and integrating E1923's code]]

== Future improvements ==
Like our preceding team, E1923, we have not touched the exporting code. Exporting could receive a mirroring change to that of importing to allow for a similar sharing of code. Additionally, as mentioned in the static analysis fixing section, there are some codeclimate errors remaining in our final pull request. If the teaching staff believes all of these should be fixed as the automated system describes, this would be a future necessary task, and potentially a good option as a good continuation for our team into Project 4.

File:Team E2401 Codeclimate Errors.webp

2024-03-27T09:06:45Z

Jrnolan2:

CSC/ECE 517 Spring 2024 - E2401 Implementing and testing import & export controllers

2024-03-27T08:20:17Z

Jrnolan2: Adding all major sections

This wiki page is for the description of changes made under E2401 OSS assignment for Spring 2024, CSC/ECE 517.

== Team==
=== Mentor ===
* Ed Gehringer

=== Team Members ===
* Gwen Mason - cpmason@ncsu.edu
* John Nolan - jrnolan2@ncsu.edu
* Meet Patel - mpatel29@ncsu.edu

== Expertiza Background ==
Expertiza is an open source Ruby on Rails application for teachers to manage classrooms. Expertiza is maintained by the teaching staff at NC State along with the very students who use it. Expertiza has many different models saved inside it including assignments, questions, and users. Relevant to our project, these models can be imported or exported through the website as CSV files for ease of uploading or editing many objects at once.

== Description of project ==
Our project's description changed late into the project submission period. We were originally tasked with using the new implementation of the import_file_controller.rb [[CSC517 Spring 2019/E1923 New Framework For Import/Export|created by team E1923]]. Their project involved consolidating all the import functions into one common functionality. Before team 1923, each model used their own import code. However, these import codes shared a lot of functionality and were ripe for refactoring. E1923's import_file_controller.rb used polymorphism to fix this duplicate code. Each model had import specific methods that the import file controller calls and receives the information necessary to create and save the respective models. We spent the early stages of the project understanding the previous team's code. Our original goal was to take several models not yet utilizing this import_file_controller.rb and extending them to use the common code.

However, later into the project we learned that unfortunately, [https://github.com/expertiza/expertiza/pull/1438 team E1923's code was never merged into the main Expertiza repository]. In 2019, the project was tagged with merge, however, it never actually was and later closed in 2021. Upon this discovery, our project's mission changed dramatically. Instead of getting other models to work with the existing import_file_controller.rb, our goal was to redo the changes of team 1923. Our goal is to get justice for the work of E1923 by getting their changes into Expertiza. However, our project was not as easy as just copying over the changes from E1923. The codebase has changed significantly over the previous 5 years which meant that much of their code and tests had to be fast forwarded to integrate into the current repository. Additionally, E1923 finished their project without the static analysis tools that currently check for styling and vulnerabilities. There were 100s of errors on GitHub Actions related to styling, test failures, and vulnerabilities that we had to correct given the scale of E1923's changes. Because of the late-term revelation, Dr. Ed graciously extended our deadline by a few days to allow our new goal to be completed.

== Our changes ==
Our changes can be split into three main phases. All changes worked towards a common goal to have E1923's code integrated into the current repository.

=== 1. Copying changes over ===
Under our mentor's advice, we could not directly merge E1923's code into the current Expertiza. This is simply because there have been so many changes over the last 5 years that the merge conflicts would be infeasible to sift through. Instead, we had to manually go file by file and make the changes matching E1923. We had to pay attention to two main diffs for each of the 47 files that E1923 changed. We first compared the changes that E1923 made to the 2019 Expertiza repo then compare the original 2019 repo against the modern repo. We had to resolve all the conflicts that arose from changes in these files throughout the years. For example, if a method was renamed during the last 5 years, we had to make sure the copied code was changed to use that new method.

=== 2. Test fixing ===
Upon completion of the copying phase, we discovered that many tests were failing. Luckily, many of these tests had common causes (originally just a few compilation errors). Despite our scrupulousness, a few bugs had still crawled into our updated code. We had to add some missing methods. Eventually, we got to the bugs not caused by simple copy errors as well. Some bugs were caused by outdated tests and deeper routed issues that occurred from trying to integrate the 2019 and current codebases. We rewrote tests and had to refactor E1923's code until we were finally to have all tests fixing.

=== 3. Static analysis fixing ===
As aforementioned, the previous team's project occurred at a time before GitHub Actions verified many aspects of the code using static analysis. This resulted in many, many errors being brought forth when we copied the previous team's code. Most severe were a handful of security vulnerabilities that existed in E1923's code. All the found security vulnerabilities existed in import_file_controller.rb. The first, involved constantize. The error existed because constantize was being run on user provided input, therefore, a malicious actor could provide unintended classes to the import method and receive information about these classes and maybe even have these other classes added to the website's database. To fix this error, we created a simple ALLOWED_MODELS whitelist array that includes the strings of all the permitted class types. We then call constantize on the match one of these strings to avoid the GitHub vulnerability message. This way, only the classes we desire to be imported will actually be allowed by the allowed_model method. The second vulnerability, involved the eval method. Upon further research, this is a '''terrible''' method to use in general in Ruby. The eval method will run the code of whatever is inside the string provided to it. While this is powerful, a malicious user can use this to run arbitrary code on Expertiza. The eval method was overkill for what it was really trying to do, parse a hash. We replaced the eval function with JSON.parse() which fixes this vulnerability. After the security vulnerabilities, we had 404 style errors thrown by codeclimate. Many of these changes were simple yet very tedious. For example, many suggested change double quotes to single quotes, removing trailing or excess whitespace, correcting indentation, removing unnecessary .to_s method calls, and adding spaces around the curly braces of hashes. After these changes, only 33 remain. Reviewing the GitHub Actions, we recognize that codeclimate is one the checks that is not required and we would recommend that many of the remaining suggestions are unhelpful and some may be suppressed. For example, the recommended fixes for "Use the new Ruby 1.9 hash syntax" involves changing the rocket (=>) to a colon (:), however, this change in our context will actually cause a compilation error. Furthermore, some of the refactoring suggestions are unnecessary. For example, the branch condition size was determined to high for the get_topic_attributes method in sign_up_topic.rb, however, the method is simply only adding fields if they exist and splitting this code might make it more confusing to read, not less.

== Future improvements ==
Like our preceding team, E1923, we have not touched the exporting code. Exporting could receive a mirroring change to that of importing to allow for a similar sharing of code. Additionally, as mentioned in the static analysis fixing section, there are some codeclimate errors remaining in our final pull request. If the teaching staff believes all of these should be fixed as the automated system describes, this would be a future necessary task, and potentially a good option as a good continuation for our team into Project 4.

CSC517 Spring 2019/E1923 New Framework For Import/Export

2024-03-27T06:46:54Z

Jrnolan2: /* Export Changes */ fix typo

== Project Introduction ==
=== Team ===
*Drew Marshburn, rdmarshb
*Veeha Khanna, vkhanna
*Randy Paluszkiewicz, rpalusz
*Andrew Miller, asmille5

Link to our video:
https://youtu.be/1nSnjta_zIc

== Project Purpose ==

The export/import feature is the most helpful feature for instructors to set up assignments. The instructors usually have a list of students, teams, etc from their learning management system. Being able to export/import these into expertiza saves a lot of time when setting up an assignment.

Expertiza provides multiple export and import features for eg. export students, teams etc. Essentially what it does is it fetches some data from database and save it as a file in desired format. However, same functionality is implemented multiple times for exporting and importing different things.

The aim of this project is to design a generic export/import feature. What you need to do is implement a framework which supports exporting and importing functionality based on input. There are substantial degrees of freedom in this project. You need to design the module which will take the database table names, column names from which data needs to be exported or the imported into. So that this module can be used for every export/import feature that expertiza provides.

Most imports and exports just import or export a single table. But sometimes exports involve “details.” For example, when questionnaires are imported or exported, the “advice” (guidelines on what characteristics of work merit each score) may be imported or exported along with the questionnaire. I suspect that all instances of import/export of multiple tables are called “details” in the current code. Provide a mechanism so that “details” can be imported or exported along with the main table involved.

== Existing Import Functionality ==

Existing import functionality is primarily routed through the ImportFileController and an import class method for various models.

SignUpTopic and User, rely on helper classes that extract attributes from a hash and create an ActiveRecord object.

Questionnaire relies on a helper method that can import Question objects (objects that make up a Questionnaire) from a CSV and adjust the size of the associated QuestionAdvice (the words that pop up after you pick a certain number of stars). However, these functions might be deprecated, as it appears that Question importing is now routed through the ImportFileController unsuccessfully. More detail about specific functions is provided below.

=== Controllers ===

*'''import_file_controller''', the list of methods in the controller are the following:
** File processing methods:
*** #get_delimiter - Sets proper delimiter for filetype
*** #parse_line - Processes line (row) of the file
*** #parse_to_grid - Turns file into 2D array
*** #parse_to_hash - Turns file into hash where 'header' stores header row and 'body' stores all contents.
*** #hash_rows_with_headers - Creates hash for each row of file. Keys are headers, values are row values.
** Import methods:
*** #import_from_hash - Primary import functionality. Creates objects for hashed rows (from #hash_rows_with_headers).
*** #import - Larger controller of import, sets error messages and displays.

*'''questionnaires_controller''', the list of methods in the controller are the following:
** ::import - Allows import from CSV using QuestionnaireHelper (Appears to be deprecated/unused)

=== Helpers ===
*'''import_file_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for User object from hash.
** ::create_new_user - Makes a user object in the database.

*'''import_topics_helper''', the list of methods in the file are the following:
** ::define_attributes - Sets and returns attributes for a SignUpTopic from hash.
** ::create_new_sign_up_topic - Makes SignUpTopic objects in the database.

*'''questionnaire_helper''' (Appears to be deprecated/unused), the list of methods in the file are the following:
** ::get_questions_from_csv - Allows Question and QuestionAdvice import/creation with CSV file.

=== Models ===

All these models have an ::import method called by the ImportFileController. In addition, SignUpTopic and User objects rely on their helpers, which are mentioned above.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''question'''
*'''review_response_map'''
*'''sign_up_sheet'''
*'''sign_up_topic'''
*'''user'''

== Proposed Import Changes ==

The import functionality should be routed through the ImportFileController. Everything should still work as it does now after our changes; we are cleaning up code and may move functionality to a different location but nothing will get removed permanently. Luckily, most import functionality is routed through the ImportFileController and import class methods on models that are being imported. It makes sense for each model to have its own import method because each model knows what to expect for itself. Making things overly generic will end up contradicting the DRY principle and lead to many nested if statements.

We intend to remove other helpers and files, such as the ImportFileHelper, the ImportTopicHelper, and the QuestionnaireHelper to keep import routing consistent. We will also remove the QuestionnaireController and route that import through the ImportFileController. In addition, we will update ImportFileController methods (in particular, #hash_rows_with_headers and #import_from_hash) to make better use of polymorphism and eliminate large and unnecessary if/else blocks.

We will also insert more specific object creation specifications. Consider if a user's import contains objects that do not exist in the database. We will provide an option for the user to specify whether or not new/dependent objects should be created and account for this boolean in relevant ::import functions for the models.

To summarize our changes:

* Route all import traffic through ImportFileController and ::import calls on models.
* Refactor ImportFileController.
* Insert object creation conditions into all relevant ::import functions and into the ImportFileController form.
* Add ability for ImportFileController to import Question objects for Questionnaires. (Add question view to views/import_file, introduce Question logic into the controller.)

Visually this is what is happening right now (with some files missing for brevity's sake):

[[Image: badUML.png]]

What we want it to look like:

[[Image: goodUML.png]]

== What has been done==
=== Models ===
*In the shared.js file there was a method #checkIfUserColumnDuplicate() which we changed to #checkForDuplicates(field_count). This method now checks to make sure that each column header is unique regardless of which model you are in.

*In the ImportFileController we moved the #start method to the beginning of the file for consistency and readability.

*The #import_from_hash and the #show method inside the ImportFileController is markedly shorter now then when we found it. We removed most of the case statements. The below images are in the respective order.

[[Image: importmethod.png]]

[[Image: showmethod.png]]

*Questionnaire.rb has a #import method that is now routed through the ImportFileController and the actual act of importing works. We removed all functionality related to importing from the QuestionnaireController and QuestionnaireHelper.

*We removed the ImportFileHelper and ImportTopicsHelper and moved that functionality into the corresponding models. Having the code segmented made things confusing since none of the functionality was all in one place.

*We removed import functionality from '''question.rb''' because there is no way to import a question out of context from a questionnaire. Importing a questionnaire, means importing questions to fill that questionnaire.

*SignUpSheet no longer has an import method. That functionality was never used and does not actually currently work in the production version of Expertiza. After discussing with our mentor, we were instructed to removed the import link on the front-end, the import method, and all related tests.

*The previous way of determining required import fields were to populate the @expected_fields variable which was incredibly hard to find in the code. We have eliminated the need for that variable and have removed all instances of it.

*We have made things as generic as possible in the .html.erb files so that you can be in any model and the code works on the front end seamlessly. This is done by adding these three methods to each model that has an import method:

<pre>
def self.required_import_fields
{"teammembers" => "Team Members"}
end

def self.optional_import_fields(id=nil)
{"teamname" => "Team Name"}
end

def self.import_options
{"handle_dups" => {"display" => "Handle Duplicates",
"options" => {"ignore" => "Ignore new team name",
"replace" => "Replace the existing team with the new team",
"insert" => "Insert any new team members into the existing team",
"rename" => "Rename the new team and import"}}}
end
</pre>

The above content is specific to the course_team.rb but the three method names are consistent to all the models and are called on the front end.

*The models that have a "self.import" method that does not branch out into other controllers beside ImportFileController will be looked at to make sure they are as concise as possible. None of them can be all the same because they all need to have checks specific to what they need to import. We can, however, make sure that similar models, like assignment_team and course_team, have similar imports. That is what we have done for assignment_team/course_team and assignment_participant/course_participant.

Now, these are the final models/places where a user may import a file into Expertiza:

- assignment_participant

- assignment_team

- course_participant

- course_team

- review_response_map

- metareview_response_map

- sign_up_topic

- user

- questionnaire

=== Views ===

*'''start.html.erb''' file has no more case statements by the @model type and everything has become generic.

*We removed all the partials related to import_file. Which has lead the '''show.html.erb''' file to have no more case statements and everything has become generic. The removed files are:

- _metareviewer.html.erb

- _participant.html.erb

- _reviewer.html.erb

- _sign_up_topic.html.erb

- _team.html.erb

- _user.html.erb

*We have updated some text on the front end related to questionnaire. The import link did not match the capitalization format in the rest Expertiza.

=== Code Climate ===
'''Note: Code Climate was not running on Expertiza's beta branch for the last four days of the project. We have done the best we could to removed extraneous lines and white spaces.'''

There were many code climate issues in reference to our project. We have managed to fix 46 issues as a byproduct of refactoring the code. Here is a list of them with their frequency put in parenthesis':

* Method get_questions_from_csv has a Cognitive Complexity of 62 (exceeds 5 allowed). Consider refactoring.

* Method get_questions_from_csv has 41 lines of code (exceeds 25 allowed). Consider refactoring.

* File import_file_controller.rb has 278 lines of code (exceeds 250 allowed). Consider refactoring.

* Avoid deeply nested control flow statements. (3)

* Similar blocks of code found in 2 locations. Consider refactoring. (2)

* Unescaped parameter value

* Useless assignment to variable - a.

* Cyclomatic complexity for get_questions_from_csv is too high. [18/6]

* Assignment Branch Condition size for get_questions_from_csv is too high. [43.3/15]

* Block has too many lines. [36/25]

* Avoid more than 3 levels of block nesting. (3)

* Perceived complexity for get_questions_from_csv is too high. [16/7]

* Align elsif with if.

* Space missing after comma. (13)

* Line is too long. [172/160]

* Method has too many lines. [108/60]

* Use the return of the conditional for variable assignment and comparison.

* Move @optional_count = 0 out of the conditional. (2)

* Move contents_hash = eval(params[:contents_hash]) out of the conditional. (6)

* Convert if nested inside else to elsif.

* Don't use parentheses around the condition of an if. (3)

== Export Changes ==

The goal was to get all the export functionality routed through one place, the ExportFileController. The overall effect would have been similar to what we hope to accomplish when updating the import functionality.

We ran out of time to work on this functionality.

== Test Plan ==
'''Note: Code Climate was not running on Expertiza's beta branch for the last four days of the project. We have done the best we could to removed extraneous lines and white spaces.'''

All these models have an ::import method that we amended and as a result we need to update the tests. We added or amended tests in their respective .spec files to make sure that the ::import method has 100% coverage.

*'''assignment_participant'''
*'''assignment_team'''
*'''course_participant'''
*'''course_team'''
*'''team'''
*'''metareview_response_map'''
*'''review_response_map'''
*'''sign_up_topic'''
*'''user'''

There is one failing test about exporting and we have not touched export functionality. There is another failing test about a review tab that we didn't touch.

== Deployment ==

The project is deployed on an NCSU VCL server. It can be accessed at http://152.46.17.230:8080/.

A sample instructor login is:

*Username: instructor6
*Password: password

CSC/ECE 517 Spring 2024 - E2401 Implementing and testing import & export controllers

2024-03-27T06:44:48Z

Jrnolan2: Expertiza background

== Team==
=== Mentor ===

* Ed Gehringer

=== Team Members ===

* Gwen Mason - cpmason@ncsu.edu
* John Nolan - jrnolan2@ncsu.edu
* Meet Patel - mpatel29@ncsu.edu

== Expertiza Background ==
Expertiza is an open source Ruby on Rails application for teachers to manage classrooms. Expertiza is maintained by the teaching staff at NC State along with the very students who use it. Expertiza has many different models saved inside it including assignments, questions, and users. Relevant to our project, these models can be imported or exported through the website as CSV files for ease of uploading or editing many objects at once.

CSC/ECE 517 Spring 2024 - E2401 Implementing and testing import & export controllers

2024-03-27T06:37:28Z

Jrnolan2: Team section

== Team==
=== Mentor ===

* Ed Gehringer

=== Team Members ===

* Gwen Mason - cpmason@ncsu.edu
* John Nolan - jrnolan2@ncsu.edu
* Meet Patel - mpatel29@ncsu.edu

CSC/ECE 517 Spring 2024

2024-03-27T06:24:23Z

Jrnolan2: Add E2401

* [[CSC/ECE 517 Spring 2024 - E2401 Implementing and testing import & export controllers]]
* [[CSC/ECE 517 Spring 2024 - E2405 Refactor review_mapping_helper.rb]]
* [[CSC/ECE 517 Spring 2024 - E2407 Refactor review_mapping_controller.rb]]
* [[CSC/ECE 517 Spring 2024 - E2428 Replicate Roles and Institution UIs ReactJS]]
* [[CSC/ECE 517 Spring 2024 - E2429 Reimplement student_task list]]
* [[CSC/ECE 517 Spring 2024 - E2430 Reimplement student_task view]]
* [[CSC/ECE 517 Spring 2024 - E2410. View for Results of Bidding ]]
* [[CSC/ECE 517 Spring 2024 - E2414 Grading Audit Trail]]
* [[CSC/ECE 517 Spring 2024 - NTNX-1 : Extend NDB Operator to Support Postgres HA]]
* [[CSC/ECE 517 Spring 2024 - ‬NTNX-2‬‭ : Snapshot Functionality for provisioned databases]]
* [[CSC/ECE 517 Spring 2024 - E2411 : Fix "Back" link on “New Late Policy” page]]
* [[CSC/ECE 517 Spring 2024 - E2416. Reimplement the Question hierarchy]]
* [[CSC/ECE 517 Spring 2024 - E2420. Reimplement student_quizzes_controller]]
* [[CSC/ECE 517 Spring 2024 - E2424. Reimplement the Bookmarks Controller]]
* [[CSC/ECE 517 Spring 2024 - E2426. Create a UI for Assignment Edit page "Etc" tab in ReactJS]]
* [[CSC/ECE 517 Spring 2024 - E2417. Reimplement submitted content controller.rb]]
* [[CSC/ECE 517 Spring 2024 - E2425. Create a Courses user interface in ReactJS]]
* [[CSC/ECE 517 Spring 2024 - E2431. Reimplement grades/view_team]]
* [[CSC/ECE 517 Spring 2024 - E2404 Refactor student teams functionality]]
* [[CSC/ECE 517 Spring 2024 - E2406 Refactor review_mapping_helper.rb]]
* [[CSC/ECE 517 Spring 2024 - E2403 Mentor-Meeting Management]]
* [[CSC/ECE 517 Spring 2024 - E2421. Reimplement impersonating users (within impersonate controller.rb)]]
* [[CSC/ECE 517 Spring 2024 - E2413. Testing - Answer Tagging]]
* [[CSC/ECE 517 Spring 2024 - E2412. Testing for hamer.rb]]
* [[CSC/ECE 517 Spring 2024 - E2427. UI for questionnaire.rb]]
* [[CSC/ECE 517 Spring 2024 - E2419. Reimplement duties controller.rb and badges controller.rb]]
* [[CSC/ECE 517 Spring 2024 - E2432. UI for Participants.rb]]
* [[CSC/ECE 517 Spring 2024 - G2402 Implement REST client, REST API, and Graphql API endpoint for repositories]]