CSC/ECE 517 Fall 2017/E17A9 Lazy loading (infinite scroll) for assignments courses questionnaires and user lists with Jscroll

Introduction

Problem Statement

The https://expertiza.ncsu.edu/tree_display/list page contains lists of courses, assignments, and questionnaires. This lists can be very long and takes time to load especially when the instructor also loads public assignments since all of them are loaded and rendered in the list. A more common way to accelerate this is to load the first few records that show up in the screen, and load more records as the user scroll down which is called lazy loading. We would like you to optimize this page by implementing such lazy loading using Jscroll or other plugins since the original tree_list code is written in ReactJS.

Moreover, the tree_display lists have some bugs that you should fix:

1. It always goes to assignment page first, so when a user is editing a course then click browser’s back button. It goes back to the assignment instead of the course tab. The same applies when a user is editing a questionnaire, then click back, it goes back to the assignment, instead of questionnaire. We want you to enable “back” to the correct tab.

2. Secondly, when “Include others' items” checkbox has been checked, then the user is editing an assignment or courses, then goes back. The checkbox is unchecked again. It should remember the last state and load the assignment or courses accordingly.

Task Description

Lazy loading

The following picture is what the page looks like now. We can see from firefox developer tools that getting all data from the database once costs 4303ms which is very slow. Therefore, we need to use infinite scroll to load a certain number of the record at one time e.g., record #1-#10

The instructor page load extremely slow due to too much backend Database query ran before loading the front end. We could analyze both frontend and backend of this page to figure out a best solution to improve the performance of this page. The backend part could try to translate more than a thousand SQLs to just a few by Query modification. The frontend can use infinite scroll to reduce the render consumption.

At first, we tried to solved the problem using jscroll. However, it did not make any difference and there were only a few examples and documentation online. Those resources did not help us find the bug. Therefore, because the instructor said we do not have to use jscroll to implement infinite scroll, we decided to use a react plugin called "react-infinite" instead.

Bugs

After we decided to move to another project, Prof. Gehringer mentioned two minor bugs in tree_display/list page.

Bug1

In "Questionnaires" table, after the user click one row to expand the table, he cannot click again to close it. It would raise an error "TypeError: React.addons is undefined"

Bug2

"Others' Public Assignments" list is very long and unorganized. We need to organize it first by "Instructor" and then by "Creation Date".

Plan

Steps

The majority of the logic behind the tree display page was written using ReactJS in expertiza/assets/javascript/tree_display.jsx. So we need to take the following steps:

• 1. Install the dependencies we need to add infinite scroll functionality to ReactJS.
• 2. Since there is no easy way to implement infinite scroll in a table tag of html, we should figure out a way to refactor the whole structure of tree_list.
• 3. Add react-infinite to the tree_list.
• 4. Modify the corresponding functions in assignment, course, questionnaire controllers to support lazy loading. For example, we may need to add "LIMIT" to the statement which query data from the database.

Files to be Modified

app/assets/javascript/application.js
app/assets/javascript/tree_display.jsx

app/controllers/assignment_questionnaire_controller.rb
app/controllers/assignments_controller.rb
app/controllers/course_controller.rb

Tools

react-infinite https://github.com/seatgeek/react-infinite

react-infinite is A browser-ready efficient scrolling container based on UITableView.

Here's a demo http://chairnerd.seatgeek.com/react-infinite-a-browser-ready-efficient-scrolling-container-based-on-uitableview/

Previous Test Plan

1. When a user first navigate to the page, there are 10 records and he can scroll down to see more records.
2. When a user edits a course, he will go back to course tab after clicking browser's back button .
3. When a user edits a assignment, he will go back to assignment tab after clicking browser's back button .
4. When a user edits a questionnaire, he will go back to questionnaire tab after clicking browser's back button.
5. When "Include others' items" checkbox has been checked, then the user is editing an assignment or courses, then goes back, the checkbox remains checked.
6. When "Include others' items" checkbox has not been checked, then the user is editing an assignment or courses, then goes back, the checkbox remains unchecked.

Previous Attempt

While we are working on this project, we have tried multiple approaches, but nothing had worked out.

The first one is jscroll, we had load jscroll into vendor directory using bower, and able to required in the project, but it turns out that jscroll doesn’t work in ReactJS.

Thus, we tried to find third party React Plugins to achieve the function of infinite scroll.

The first React Plugin is react-infinite, We were also able to load react-infinite into vendor directory using bower, and has no error occurred while adding to the react component.

add dependency "react-infinite" to bower.json

"react-infinite": "*"

run setup.sh to load "react-infinite" into /vendor/ modify "app/assets/javascript/application.js" to require the plugin

//= require react-infinite

add "react-infinite" tags to "app/assets/javascript/tree_display.jsx"

<react-infinite>
</react-infinite>

However, it also didn’t work out because of the main content is html table.

After that, we have asked this problem on the stackoverflow. People recommended another plugin called react-infinite-scroller, this plugin couldn’t find by bower package manager, so we have to load this package by another package manager like webpacker, but webpacker needs Rails 5 to load React, however Expertiza is based on Rails 4.2.6. We’ve also tried sprockets but we were unable to import.

Our last try is https://infinite-scroll.com/ https://github.com/metafizzy/infinite-scroll

componentWillUpdate: function() {
         console.log(1111);
         var $container = $('.scroll').infiniteScroll({
             // path is required, hack with {{#}}
             path: 'page{{#}}',
             loadOnScroll: false,
             scrollThreshold: 400
         });
         $container.on( 'scrollThreshold.infiniteScroll', function() {
             console.log(2222);
             $container.append(<ContentTableRow
                 key={1}
                 id={1}
                 name={1}
                 directory={1}
                 instructor={1}
                 institution={1}
                 creation_date={1}
                 updated_date={1}
                 actions={1}
                 is_available={1}
                 course_id={1}
                 max_team_size={1}
                 is_intelligent={1}
                 require_quiz={1}
                 dataType={1}
                 //this is just a hack. All current users courses are marked as private during fetch for display purpose.
                 private={1}
                 allow_suggestions={1}
                 has_topic={1}
                 rowClicked={1}
                 newParams={1}
             />);
         })
     },

But we still stuck at this bug.

TypeError: instance._init is not a function.

This project turns out to be really complicated to implement. So we asked to switch the project to implement Rspec integration test for the users_controller.rb

Implementation & Test Plan

Bug1

Because React.addons cannot be used here, I changed it to the function of JS.

       var index = this.state.expandedRow.indexOf(id)
       if (index > -1) {
           var list = this.state.expandedRow;
           list.splice(index,1);
         this.setState({
           expandedRow: list
         })
       }

Bug2

I sorted the list before the data is returned to the frontend in the tree_display_controller.rb. The list is sorted first in ascending order of 'instructor' and then in the descending order of 'creation_date' which requires 'creation_date' to be converted to timestamp.

res['Assignments'] = res['Assignments'].sort_by {|x| [x['instructor'],-1*(x['creation_date'].to_i)]}

users_controller.rb Test

Our new project is to write Rspec Test for the users_controller.rb.

Here is our code.

require 'pry'

describe UsersController do
 let(:admin) {build(:admin)}
 let(:superadmin) {build(:superadmin)}
 let(:instructor) {build(:instructor, id: 6)}
 let(:instructor2) {build(:instructor, id: 66)}
 let(:ta) {build(:teaching_assistant, id: 8)}
 let(:student) {build(:student)}
 before(:each) do
   allow(User).to receive(:find_by_name).with('instructor6').and_return(instructor)
   allow(User).to receive(:find_by_name).with('instructor66').and_return(instructor2)
   allow(User).to receive(:find).with('1').and_return(instructor2)
 end

 describe '#action_allowed?' do
   context 'when params action is request_new' do
     it 'allows certain action' do
       controller.params = {action: 'request_new'}
       expect(controller.send(:action_allowed?)).to be true
     end
   end

   context 'when params action is request_user_create' do
     it 'allows certain action' do
       controller.params = {action: 'request_user_create'}
       expect(controller.send(:action_allowed?)).to be true
     end
   end

   context 'when params action is review' do
     it 'allows certain action' do
       controller.params = {action: 'review'}
       stub_current_user(superadmin, superadmin.role.name, superadmin.role)
       expect(controller.send(:action_allowed?)).to be true
     end
   end

   context 'when params action is keys' do
     it 'allows certain action' do
       controller.params = {action: 'keys'}
       stub_current_user(student, student.role.name, student.role)
       expect(controller.send(:action_allowed?)).to be true
     end
   end

   context 'when the current role is administrator and params action is something else' do
     it 'allows certain action' do
       controller.params = {action: 'view'}
       stub_current_user(admin, admin.role.name, admin.role)
       expect(controller.send(:action_allowed?)).to be true
     end
   end

   context 'when the current role is superadministrator and params action is something else' do
     it 'allows certain action' do
       controller.params = {action: 'view'}
       stub_current_user(superadmin, superadmin.role.name, superadmin.role)
       expect(controller.send(:action_allowed?)).to be true
     end
   end

   context 'when the current role is instructor and params action is something else' do
     it 'allows certain action' do
       controller.params = {action: 'view'}
       stub_current_user(instructor, instructor.role.name, instructor.role)
       expect(controller.send(:action_allowed?)).to be true
     end
   end

   context 'when the current role is ta and params action is something else' do
     it 'allows certain action' do
       controller.params = {action: 'view'}
       stub_current_user(ta, ta.role.name, ta.role)
       expect(controller.send(:action_allowed?)).to be true
     end
   end
 end

 describe '#index' do
   context 'when the current user is student' do
     it 'redirect to the home page' do
       stub_current_user(student, student.role.name, student.role)
       get :index
       expect(response).to redirect_to('/')
     end
   end

   context 'when the current user is not student' do
     it 'redirect to /users/list page' do
       stub_current_user(instructor, instructor.role.name, instructor.role)
       @request.session['user'] = instructor
       get :index
       expect(response).to render_template(:list)
     end
   end
 end

 describe '#list' do
   it 'render list' do
     stub_current_user(instructor, instructor.role.name, instructor.role)
     @request.session['user'] = instructor
     get :list
     expect(response).to render_template(:list)
   end
 end
 describe '#list_pending_requested' do
   it 'render list pending' do
     stub_current_user(instructor, instructor.role.name, instructor.role)
     get :list_pending_requested
     expect(response).to render_template(:list_pending_requested)
   end
 end

 describe '#new' do
   it 'creates a new user and renders users#new page' do
     stub_current_user(instructor, instructor.role.name, instructor.role)
     @request.session['user'] = instructor
     get :new
     expect(response).to render_template(:new)
     expect(assigns(:user)).to be_a_new(User)
   end
 end

 describe '#request_new' do
   context 'when the current user is not student' do
     it 'creates a new user and renders users#new page' do
       stub_current_user(instructor, instructor.role.name, instructor.role)
       @request.session['user'] = instructor
       get :new
       expect(response).to render_template(:new)
       expect(assigns(:user)).to be_a_new(User)
     end
   end
 end

 describe '#show_selection' do
   context 'when user is not nil and parent id is nil' do
     it 'redirect to /users/show' do
       stub_current_user(instructor, instructor.role.name, instructor.role)
       get :show_selection, {user: {name: 'instructor6'}}
       expect(response).to render_template(:show)
     end
   end
 end

 describe '#show' do
   context 'when params id is nil' do
     it 'redirect to /tree_display/drill' do
       stub_current_user(instructor, instructor.role.name, instructor.role)
       get :show, id: nil
       expect(response).to redirect_to('/tree_display/drill')
     end
   end

   context 'when params id is not nil' do
     it 'render show' do
       stub_current_user(instructor, instructor.role.name, instructor.role)
       get :show, id: 1
       expect(response).to render_template(:show)
     end
   end
 end

 describe '#edit' do
   it 'can find user' do
     stub_current_user(instructor, instructor.role.name, instructor.role)
     @request.session['user'] = instructor
     get :edit, id: 1
     expect(response).to render_template(:edit)
   end
 end
end

Test Result

The test result is as below:

Reference

1. jscroll
2. react-infinite