CSC/ECE 517 Fall 2016/E1656. Improve imports

E1656. Improve imports

This page provides a description of the Expertiza based OSS project.

Introduction to Expertiza

Expertiza is an open source project based on Ruby on Rails framework.

Project Description

About Import

In Expertiza, various kinds of data may be imported from csv files, including users, participants in an assignemnt, topics and teams. When importing data, a corresponding csv file have to be composed. In a csv file, each new line represent a piece of datum and the different fields of a datum are separated by delimiters. But there are rough edges in the way these imports are done and certain other data that should be importable is not. This project focuses on implementing following tasks about data import functionality:

• When a team with a duplicate name is imported, one of the options is to rename the new team. There should also be an option to rename an existing team. See Issue 329.
• Review assignments (“reviewer mappings”) should be able to be imported, but cannot be imported. See Issue 711
• It should be possible to import a list of users who have signed up for topics, but this feature does not yet exist. See Issue 153.
• For every kind of data to be imported, Expertiza currently specifies the ordering of fields. This isn’t very flexible; it may require the user (an instructor) to edit an existing CSV file. It would be better if, after importing the data, Expertiza showed the data together with dropdowns containing the default field headers. (This is similar to the way Excel shows data when it is imported from a CSV file.) Then the user could change the dropdowns to cause a different ordering of fields. See Issue 110.
• Create a way to export a list of teams that have signed up for topics, and who are waitlisted for topics, as well as participants in an assignment who have not signed up for topics. Choose the format, but keep it as consistent as possible with the other export formats. This would help the professor when students ask him for advice on finding teammates. Extra Issue

The following tasks were listed as a requirement, but after careful examination it was determined that these tasks were already implemented

• The simplest fix: If I try to import topics, the import page gives me this message: “The import process expects the following columns:” but it doesn’t say what the columns are (topic number, topic name, category, number of slots, category)! It used to be in the system; please revert the change to fix this bug. See Issue 719.
• When teams are imported in order to insert new members, the members are not inserted. See Issue 328.
• It should be possible to leave off the final fields in a line of a CSV file. These fields are often not specified anyway. For example, if a password isn’t specified when a new user is created, the system generates a password (see Issue 183). When importing topics, topic categories are rarely specified. But if the final field is blank, the import requires the CSV line to end with “, “ (comma and space). This should be fixed for all imports.

About Versions Controller

Git is a popular version control system, As a part of the project, We have made use of git for the purpose of version control. Our repository is a fork of the original Expertiza repository on which we have collaborated our work. And the project has been submitted through a pull request.

About Tests

New features implemented can be tested through UI operations or add some Rspec functional tests. Through deployment link, users can do overall tests on these features, test some edge cases and add some new scenarios. Therefore, in this project, some simple scenarios are added to Rspec tests and most of the tests are done through UI operations. The deployed link for remote access is: http://104.236.1.180:3000

Handle Duplicates (Issue329)

Current Implementation

When a team with a duplicate name is imported, one of the options is to rename the new team. There should also be an option to rename an existing team. See Issue 329. The current implementation allows the user to handle duplicates, by the following techniques

• ignore the new team
• replace the existing team with the new team
• insert any new members into the existing team
• rename the new team and import
• rename the existing team and import

Functionality

The above functionalities make it easier to achieve the goal of handling duplicates while importing.

Drawbacks and Solutions

The above set of functionalities do not allow User to ever rename an existing team, rather force him rename his own team. Thus the above requirement provides the necessary freedom to the User by providing him the option to rename an already existing team.

New Implementation

The option has been implemented using the update functionality that is already present in rails models, to update existing values of Active Records. To implement this feature we have also made use of the functionality for generating team names, that is already present within the Team model.

A code snippet is given below:

 if handle_dups == "renameOldTeam" # rename: rename new team
      if teamtype.is_a?(CourseTeam)
        CourseTeam.update(team.id, :name => self.generate_team_name(Course.find(id).name))
        return name
      elsif teamtype.is_a?(AssignmentTeam)
        AssignmentTeam.update(team.id, :name => self.generate_team_name(Assignment.find(id).name))
        return name
      end
    end

UI Testing

The Above feature has been tested from the UI using the both course Teams and Assignment Teams, where new teams were added with conflicting names and the rename existing team option was selected, which lead lead to the desired behavior, that is the team that was extant was renamed and new team was imported with the same name as that specified in the CSV file

The feature can be tested from Add teams option in manage assignments. From where one can import teams and choose the required feature

Fix Reviewer and Metareviewer Mappings (Issue711)

Current Implementation

When an instructor tries to assign reviewers for an assignment, if the review strategy of the assignment is‘Instructor-Selected’, he could assign reviewers by directly importing the reviewer mappings. But now, when the instructor attempts to submit the import, the system will raise an argument error showing:

The import fails and, clearly, it's a bug in Expertiza.

New Implementation

To solve this problem, first refer to the view/review_mapping/_list_review_mappings.html.erb file and find out that the model to be imported is ReviewResponseMap. The import method in review_response_map.rb model file takes in 3 arguments which are row, _session and id. A code snippet of import_file_controller.rb (the import controller) is given below:

Before we add the code in parentheses, the ReviewResponseMap model was not explicitly specified which case it belongs to and falls into the else statement where the import method takes in 4 arguments. That’s the reason of getting the “wrong number of argument” error. To resolve the problem, a "or params[:model] == 'ReviewResponseMap'" is inserted into the parentheses to make the import method of reviewer mappings takes in 3 arguments again and then we can correctly import the reviewer mappings. The recovery method for MetareviewResponseMap is exactly the same with this one by adding "or params[:model] == 'MetareviewResponseMap'" statement into the parentheses.

UI Testing

1. Login as instructor. Click on ‘Manage Assignment’ to see all assignments in the database.

2. Among all the assignments, click the ‘Review strategy’ card in edit page and find out one assignment with 'Instructor-Selected' strategy (e.g. 'Design exercise'). Creating a new 'Instructor-Selected' assignment and add some participants to it is also applicable.

3. 'Assign reviewers' for this assignment and click on the 'Import reviewer mappings' on the bottom of the page.

4. Compose a csv file in the given format of 'Contributor, Reviewer1, Reviewer2, …, ReviewerN' and import the file. The ‘Contributor’ here is the name of a team in this assignment.

5. Go back to reviewer mappings page to check if the 'Reviewers of the Contributor' in the file are listed in the 'Reviewed By' section of the 'Contributors'.

Following are screenshots of the UI tests. Screenshot711

Import Sign Up Sheet (Issue153)

Current Implementation

When a signup sheet is used in Expertiza, users are expected to sign up for topics. But, the instructor might've taken signups offline, e.g., by passing around a signup sheet in class. If the signup are not taken online by the students, the instructor has to impersonate all the students one by one and sign them up for a topic, which is tedious.

New Implementation

To implement this new issue, a model which is in charge of creating a sign_up and SignedUpTeam should be located first. In SignUpSheet model, methods are designed for the creation of a sign_up and SignedUpTeam using the data from data base and the new implementation should create them from an imported file. Hence, an import method is added to this model and a code snippet of it is shown as:

This new method imports new sign_up data in a format of "Topic identifier, User1, User2". Users here are the ones who signed up for the topic with a certain identifier (id). Some edge cases are also taken into consideration in this method to prevent importing a non-existing team in this assignment, a non-existing user or a user who is not a participant of this assignment. Next, a link on the view page needs to be created to access this method. The instructor can view the sign up sheet of an assignment under the "topic" card of the "edit" section. An "Import sign up sheet" button is added beside the "Import topic" link

and the corresponding code should be inserted in the "view/sign_up_sheet/_add_topics/html.erb" file.

After all of this, the new feature is successfully implemented and the instructor can directly import a large offline sign up sheet without tedious impersonating. The whole process of importing sign ups works exactly the same as student online sign ups. If the assignment does not have a topic yet, the 'Import sign up sheet' link will be invisible unless a topic is created or imported. If a imported user does not have a team, he will be automatically assigned a team and sign up for the topic. If users imported exceeds the limit a certain topic can hold, the leftmost users in the file (like the earliest signed up users) will have the topic and those right(later signed up) users will be in the wait list.

UI Testing

1. Login as instructor. Click on 'Manage Assignment’ to see all assignments in the database.

2. Click on 'edit', go to 'topic' card and find the "Import sign up sheet" link.

3. Compose a csv file in the given format of 'Topic identifier, User1, User2, …' and import the file. The ‘Topic identifier’ here is the 'topic#' in the sign up sheet.

4. Go back to sign up sheet page to check if the 'Users' in the file and their 'Teams' are listed under the 'Topic names' section.

Export Assignment Participants (Extra Issue)

Current Implementation

Current Implementation just allows users to Export the set of all Assignment Participants, of a given assignment. It doesn't allow any further filtering.

Functionality

The current functionality is very rigid and doesn't allow the user to know any more than the set of assignment Participants, which is also a major drawback

New Implementation

The new implementation allows user to export Assignment Participants in four ways.

• List of All Users mapped with team number and topic name who have signed up for an assignment topic
• List of All Users mapped with team number and topic name who are wait-listed for an assignment topic
• List of All Participants who haven't yet signed up for a topic and haven't formed teams yet
• List of All Participants

The functionality is present as a set of Radio buttons, which is checked by default to the fourth option that is to export all Participants. However that can be changed according to requirement.

The above has been implemented by creating a partial for the view to add the radio button. Further and If else bock has been used in the model, to identify the option selected and accordingly, on the required set of Rows are pushed to the csv file. To know whether a User has signed up or not for a topic, The SignedUpTeam model is used and to find if the team is wait listed or not, the is_waitlisted boolean value id used to determine if the team is wait listed or not.

A small Code snippet describing how one of these option was achieved is given below:

 if options["participant_type"] == "signed"
        Team.where(:parent_id=> parent_id).find_each do |team|
          TeamsUser.where(:team_id=> team.id).find_each do |team_user|
            user = User.find(team_user.user_id)
            if SignedUpTeam.where( :team_id => team.id).where(:is_waitlisted => [false, nil," ", ""]).present?
              assignemnt_team = SignedUpTeam.where( :team_id => team.id).where(:is_waitlisted => [false, nil," ", ""]).first
              topic = SignUpTopic.find(assignemnt_team.topic_id)
            csv << [
              user.name,
              user.fullname,
              user.email,
              user.role.name,
              user.parent.name,
              user.email_on_submission,
              user.email_on_review,
              user.email_on_review_of_review,
              team.id,
              topic.topic_name,
              "handle"
            ]
            end
          end
        end

UI Testing

UI testing for the above was done by exporting Assignment Teams for a given assignment using all the above options individually and checking if the result actually matched what was expected. Example 'all' should export all the participants listed withing the assignment. 'Signed Up Team' gives all the participants who have signed up mapped to their teams and respective topics. If you are signed in as Instructor: Feature can be tested by clicking on add participants button in manage assignments and then clicking on export course participants at the bottom of the page. And then following the applicable option for testing the respective features described above.

Ordering of Imported Fields (Issue 110)

Current Implementation

Currently the order of the columns is preset and the user has to change the csv to match it.

Drawbacks and Solutions

This is inconvenient for the user to keep editing csv files. The solution is to allow users chose the order of the columns as it appears in their csv file. This is done by providing drop down fields with all the column name based on the model that the import is related to. User is free to change he order or keep it as default.

New Implementation

In the view we added the drop down fields based on the expected_fields

   *Please choose the order of the columns here:

   <%@array_expected_values.each{|field|%>
   <%=select_tag 'import_field_[field]', options_for_select(@array_expected_values.collect.with_index.to_a,field)%>

   <%}%>

In the import_file_controller.rb we made addition to the start method. Here we also return an array of values from the expected_fields. We unpack the values like Team Member1,2...N by returning the maximum number of Team Members. Full code can be found in import_file_controller.rb

   @array_expected_values = parse_line(@expected_fields,',',params)

Also in the import_file_controller.rb we added function that reorders the columns as they are expected by expertiza. As the user changed the order of the fields in the main import form this order is processed by the controller. The reorder function matches the field name in the default order with the position in the newly chosen order and reorders the data as it is read line by line from the csv file.

 def reorder_row(row,params)
   ...

   expected_fields_variable=expected_fields_variable_default.reject.with_index { |x,i| i > row.length-1 }
   expected_fields_variable.each_with_index { |field, index|
     custom_order[params["import_field_#{index}"]]=row[index]
   }
   expected_fields_variable.each_with_index { |field, index| return_row[index]=custom_order[field]}
   return_row
 end

UI Testing

To test this in the browser first need to be logged in as instructor. Then go to Manage -> Assignments -> Assignments tab. Then select to add teams to any assignment -> import teams. Here you would need to create a CSV file with the Team name and team members - separated by comma. The order of the column are arbitrary, just have to match them in the drop down fields provided. For the team members there are maximum drop down fields, but you only need to fill out the relevant once. It will know to ignore irrelevant columns.

Following are screenshots of the UI tests Screenshots

Improvement Scope

The above Import as well as export functionalities that have been implemented in due course of the project have a scope for improvement. Imports and exports are functionalities that are used very often used to insert data as well to transfer data stored into the Models to external CSV files. Thus these features can be further tested using rspec tests as well as capybara and cucumber tests so as to make it fool proof. However these tests cannot be made such that they hamper the flexibility of the implementation to achieve a particular functionality. Further We can also add options to handle duplicates as well as to filter out exports in other models as well, instead of the ones it is already present. Thus there is a scope of a lot more improvement to make Expertiza more user friendly.