CSC/ECE 517 Fall 2017/E1776 Enhance Imports

Team Contact

MEMBERS

Pushpendra Patel: ppatel16@ncsu.edu

Tanay Kothari: tkothar@ncsu.edu

Timothy Dement: tmdement@ncsu.edu

MENTOR

Ferry Pramudianto: fferry@ncsu.edu

Topic Description

Backgroud

Expertiza is an open-source web application devloped on the Ruby on Rails platform that helps students create reusable learning objects through peer review, and also supports document submission and team projects.

Expertiza includes several variations of import functionality, and allows the instructors to import following data:

• A list of users.
• A list of participants for an existing assignment.
• A list of participants for an existing course.
• A list of teams for an existing assignment
• A list of teams for an existing course.
• A list of reviewers (reviewing contributors to assignments).
• A list of meta-reviewers (reviewing reviewers).
• A list of topics for an existing assignment.

These imports are done by uploading a file containing rows of data, with each individual value of a row separated by a given delimiter.

Expertiza allows four specifications for delimiters:

1. Comma
2. Space
3. Tab
4. Other (any custom delimiter provided by the user as text)

All of these import functions are routed through the Import File Controller, which is responsible for parsing file data and delegating the import process to the appropriate model.

Problem Statement

The following is an Expertiza-based OSS project which deals primarily with the Import File Controller. This project is associated with fixing issue #110, detailed on the Expertiza Github. A significant problem with the existing import functions is that they are not implemented consistently, and there are rigid restrictions placed on how the columns should be ordered in the given import files. In the current implementation, rows are taken is arrays, with the column numbers of specific fields hard-coded into the helper methods for different imports. The aim of this project is to improve the import functionality as well as provide a flexible and user-friendly interface so that users can easily and reliably take advantage of Expertiza's various import functions.

Solution

To resolve this issue, we have enhanced the interface and the import process, keeping the core methods of import intact, but expanding its functionality to increase its reliability, robustness, and ease-of-use. In the new interface, the system will display what will be imported in an easy-to-read grid before the import is finalized, letting the user verify the correctness of the data and, if necessary, choose from a dropdown menu which columns need to be rearranged. The default ordering is the same as what is currently required by Expertiza, unless a specific order is provided by the user as a header in the import file.

In our implementation of the import method, we have made the use of hash data structure. During the import process, we create an array of hashes, in which each row of the array corresponds to a row of the original import file. In each hash, keys refer to the field name, and values refer to the actual value of that field that will be saved to the database. This greatly improved the flexibility of the import process, since the order of the data in the import files is no longer required to be in one specific order only.

Plan of Work and Implementation Process

In this project we have tried to make the import process more flexible. As stated in the previous section, in the old implementation, user had to import the files with restriction to the order of columns required. We tried to make this process flexible for the user, so that user is able to easily specify the order of columns using drop down menus.

Here is view used for the previous implementation of the User Imports -

Here is the view with our implementation of the User Imports -

On the code side of imports, we have improved the implementation significantly. Previously, the import process used hard coded index values to retrieve specific columns from the import file -

    user = User.find_by_name(row[0])

    if user.nil?
      attributes = ImportFileHelper.define_attributes(row)
      user = ImportFileHelper.create_new_user(attributes, session)
      password = user.reset_password # the password is reset
      MailerHelper.send_mail_to_user(user, "Your Expertiza account has been created.", "user_welcome", password).deliver
    else
      user.email = row[2].strip                                      # ***** Use of hard-coded index values *****
      user.fullname = row[1].strip
      user.parent_id = (session[:user]).id
      user.save
    end

This was the reason why columns were needed to be in a restricted order. But in our approach, we divide the rows of data provided in import files into hashes. Accessing data from hashes is flexible as we just need the keys to access certain values which we specify during our import process and thus we know ,before hand, what key to use to access a specific value whenever needed. Following is a snippet from the "User.rb" to demonstrate our usage of hash.

    user = User.find_by_name(row_hash[:name])

    if user.nil?
      attributes = ImportFileHelper.define_attributes(row_hash)
      user = ImportFileHelper.create_new_user(attributes, session)
      password = user.reset_password
      MailerHelper.send_mail_to_user(user, "Your Expertiza account has been created.", "user_welcome", password).deliver
    else
      user.email = row_hash[:email]                                  # ***** No hard-coded index values required *****
      user.fullname = row_hash[:fullname]                            # ***** Direct use of field name for setting up user parameters for import *****
      user.parent_id = (session[:user]).id
      user.save
    end

The import process in the previous implementation imports is as follows -

• User initiates import process via links for importing files on different views for different models.
• User is redirected to "start" view of Import File controller.
• User uploads the file containing data to be imported. User also modifies the options used for import on the "start" view.
• Once the user clicks "Import" button on this view, "import" method of the Import File controller is called which in turn calls the "import_file" method of the same controller. "import" method collects the error which might be encountered during the import process and displays it to the user.
• "import_file" converts the data from import file into a multidimensional array in which each row represents a row from the import file. Following is an example -

[ [ 'name', 'fullname', 'email' ],
  [ 'ppatel16', 'Pushpendra Patel', 'ppatel@ncsu.edu' ],
  [ 'tkothar', 'Tanay Kothari', 'tkothar@ncsu.edu' ],
  [ 'tmdement', 'Timothy Dement', 'tmdement@ncsu.edu' ] ]

• This method now checks which model is to be used to import the data and calls "import" method of the respective models.
• "import" method of respective model performs the import based on hard coded order of columns.

Import process of using our implementation is as follows -

• User initiates import process via links for importing files on different views for different models.
• User is redirected to "start" view of Import File controller.
• User uploads the file containing data to be imported. User also modifies the options used for import on the "start" view.
• When user submits the options to import, the file is parsed row-by-row. It is divided into a "header" which contains the column names and a "body" which contains all the data to be imported in form of a multi-dimensional array.
• "show" action of the Import File controller is called and user is redirected to "show" view the same controller. On this view user can use drop downs to select which field of the import data is which.
• Once the user clicks "Import" button on the "show" view, "import" method of the Import File controller is called which in turn calls the "import_from_hash" method of the same controller. "import" method collects the errors which might be encountered during the import process and displays it to the user.
• In the "import_from_hash" method, each row of the multidimensional array (body) is converted from into a hash. Following is an example of such a converted hash -

[ { :name => 'ppatel16',
    :fullname => 'Pushpendra Patel',
    :email => 'ppatel16@ncsu.edu' },
  { :name => 'tkothar',
    :fullname => 'Tanay Kothari',
    :email => 'tkothar@ncsu.edu' },
  { :name => 'tmdement',
    :fullname => 'Timothy Dement',
    :email => 'tmdement@ncsu.edu } ]

• This method now checks which model is to be used to import the data and calls "import" method of the respective models.
• "import" method of respective model performs the import based on hard coded order of columns.

Test Plan and Demonstrations

User Import

User Import Demo

This video demonstrates the enhanced import features for the User model with several varying examples, and also demonstrates that the original error messaging system is preserved with our new implementation.

Assigment Participant and Course Participant Import

Participant Import Demo

This video demonstrates the enhanced import features for the Assignment Participant and Course Participant models with several varying examples.

Assignment Team and Course Team Import

Team Import Demo

This video demonstrates the enhanced import features for the Assignment Team and Course Team models with several varying examples.

Reviewer and Meta-reviewer Import

Review Import Demo

This video demonstrates the enhanced import features for the Review Response Map and Metareview Response Map models with several varying examples.

Updates to Views

User Import

Improvements:

• Added descriptive title
• Fixed form alignment
• Added "Header" radio button
• Changed "Expected Columns" to be more legible

Old User Import View

New User Import View

Assignment Participant and Course Participant Import

NOTE: Course Participant changes are identical.

Improvements:

• Added descriptive title
• Fixed form alignment
• Added "Header" radio button
• Changed "Expected Columns" to be more legible

Old Assignment Participant Import View

New Assignment Participant Import View

Assignment Team and Course Team Import

NOTE: Course Team changes are identical.

Improvements:

• Added descriptive title
• Fixed form alignment
• Added "Header" radio button
• Added "Team Name" radio button
• Changed "Expected Columns" to be more legible

Old Assignment Team Import View

New Assignment Team Import View

Reviewer Import

Improvements:

• Added descriptive title
• Fixed form alignment
• Added "Header" radio button
• Added "Contributor" radio button
• Changed "Expected Columns" to be more legible

Old Reviewer Import View

New Reviewer Import View

Metareviewer Import

Improvements:

• Added descriptive title
• Fixed form alignment
• Added "Header" radio button
• Added "Contributor and Reviewer" radio button
• Changed "Expected Columns" to be more legible

Old Metareviewer Import View

New Metareviewer Import View

Topic Import

Improvements:

• Added descriptive title
• Fixed form alignment
• Added "Header" radio button
• Added "Optional Columns" checkbox list
• Changed "Expected Columns" to be more legible

Old Topic Import View

New Topic Import View

Files Changed

Assets

To implement the functionality of non-duplicate column names, we created two new methods checkIfUserColumnDuplicate() and checkIfParticipateColumnDuplicate() in the shared.js . We have also fixed the checkIfFileExist() method so that if the user doesn't select the file and click on import button it gives him/her an error stating that "No file has been selected to import".

Config

We needed to change the routes.rb so that the new partial views can be implemented.

Controllers

For the enhancement of the import functionality, we modified the import_file_controller. In this controller, we modified one existing method import(), other than that we had to declare some additional methods import_from_hash(), hash_rows_with_headers(), parse_to_hash(), parse_to_grid() and show().

Helpers

As we changed the original implementation from array to hash we need to also modify the import_file_helper a little bit.

Models

Here is a list of models and their respective methods that we had to modify to enhance the import functionality:

1. assignment_participant : self.import()
2. course_participant : self.import()
3. course_team : add_member()
4. metareview_response_map : self.import()
5. review_response_map : self.import()
6. team : import_team_members() and self.import()
7. user : self.import()

Views

To use the import functionality in a new and enhanced way, we modified one of the existing view and added new views, listed below:

Views Modified:

1. start.html.erb

Views Created:

1. _metareviewer.html.erb
2. _participant.html.erb
3. show.html.erb
4. _reviewer.html.erb
5. _team.html.erb
6. _user.html.erb

Navigating Expertiza

User Import

To begin the User import process, first select the "User" link from the "Manage" drop-down menu.

Manage Users

Next, scroll to the bottom of the page and click the "Import Users" link.

Import Users

You will then be redirected to the User import page.

Assignment Participant Import

To begin the Assignment Participant import process, first select the "Assignments" link from the "Manage" drop-down menu.

Manage Assignments

Make sure that "Assignments" is highlighted on the "Manage content" page, then locate the appropriate assignment and click the "Add Participants" button.

Add Assignment Participant

Finally, scroll to the bottom of the page and click the "Import assignment participants" link.

Import Assignment Participant

You will then be redirected to the Assignment Participant import page.

Course Participant Import

To begin the Course Participant import process, first select the "Courses" link from the "Manage" drop-down menu.

Manage Courses

Make sure that "Courses" is highlighted on the "Manage content" page, then locate the appropriate course and click the "Add Participants" button.

Add Course Participant

Finally, scroll to the bottom of the page and click the "Import course participants" link.

Import Course Participant

You will then be redirected to the Course Participant import page.

Assignment Team Import

To begin the Assignment Team import process, first select the "Assignments" link from the "Manage" drop-down menu.

Manage Assignments

Make sure that "Assignments" is highlighted on the "Manage content" page, then locate the appropriate assignment and click the "Create Teams" button.

Create Assignment Team

Finally, scroll to the bottom of the page and click the "Import Teams" link.

Import Assignment Teams

You will then be redirected to the Assignment Team import page.

Course Team Import

To begin the Course Team import process, first select the "Courses" link from the "Manage" drop-down menu.

Manage Courses

Make sure that "Courses" is highlighted on the "Manage content" page, then locate the appropriate assignment and click the "Create Teams" button.

Create Course Team

Finally, scroll to the bottom of the page and click the "Import Teams" link.

Import Course Teams

You will then be redirected to the Course Team import page.

Reviewer and Metareviewer Import

To begin the Reviewer and Metareviewer import process, first select the "Assignments" link from the "Manage" drop-down menu.

Manage Assignments

Make sure that "Assignments" is highlighted on the "Manage content" page, then locate the appropriate assignment and click the "Assign reviewers" button.

Assign Reviewers

Finally, scroll to the bottom of the page, and click either the "Import reviewer mappings" link or the "Import meta reviewer mappings" link.

Import Reviewer Mappings

Import Metareviewer Mappings

You will then be redirected to either the Reviewer import page or the Metareviewer import page.

Topic Import

To begin the Topic import process, first select the "Assignments" link from the "Manage" drop-down menu.

Manage Assignments

Make sure that "Assignments" is highlighted on the "Manage content" page, then locate the appropriate assignment and click the "Edit" button.

Edit Assignment

Next, select the "Topics" tab, scroll to the bottom of the page and click the "Import topics" link.

Import Topics

If topics have not been set up yet on the given assignment, you will receive an alert that will need to be accepted before continuing.

Topic Alert

You will then be redirected to the Topic import page.