View source for CSC/ECE 517 Fall 2017/E17A8. Use a profiler to identify the problems / pages that take some time to load & fix them

= Optimization for page loading in Expertiza =

== Introduction ==
Expertiza is an online system that is used by students to view/submit assignments and review others' work. Expertiza also provides tools to visualize the scores and gauge the improvements made during the course semester. It also facilitates and monitors team projects. It is targeted at educational and non-profit organizations. The project is funded by the National Software Foundation (NSF), NCSU Learning in a Technology-Rich Environment (LITRE) program, the NCSU Faculty Center for Teaching and Learning, the NCSU STEM Initiative, and the Center for Advanced Computing and Communication.

Expertiza is an open-source project with the source code available as a public repository on GitHub. It is developed using Ruby on Rails and is increasingly becoming robust thanks to the innumerable bugs being fixed by the community. The project has a micro-blog on SourceForge where the developer community report bugs and document updates.

== Project Description ==

1) Identifying the Expertiza pages which take time to load using Rack-mini-profiler and Flamegraphs.

2) Propose fixes which would improve the Expertiza project.

3) Optimizing the view, models and controllers for few of these corresponding fixes to improve the load time of these pages.

== Gems Installed  ==

*rack-mini-profiler :- Middleware that displays speed badge for every html page. Designed to work both in production and in development
*flamegraphs :- Flame graphs are a visualization of profiled software, allowing the most frequent code-paths to be identified quickly and accurately.
*stackprof:-A sampling call-stack profiler for ruby 2.1+.Downloaded as a dependency for rack-mini-profiler.
*fast_stack :-fast_stack is dynamically resizable data structure optimized for fast iteration over the large arrays of similar elements avoiding memory fragmentation.
*Kaminari:- A Scope & Engine based, clean, powerful, customizable and sophisticated paginator for modern web app frameworks and ORMs

== Pages which needs to be optimized ==

1) expertiza.ncsu.edu/grades/view

2) expertiza.ncsu.edu/users/list

== PROJECT DESIGN ==


For the purpose of this project, experts in the expertiza domain are Instructors. 

The following two patterns are implemented in the project - 

1.MVC Pattern – The project is implemented in Ruby on Rails that uses MVC architecture. It separates an application’s data model, user interface, and control logic into three distinct components (model, view, and controller, respectively). 

2.DRY Principle – We are trying to reuse the existing functionalities in Expertiza, thus avoiding code duplication. Whenever possible, code modification based on the existing classes, controllers, or tables will be done instead of creating the new one.

3.Optimization-In order to optimize, we need to understand the factors which are causing the lag. This can be identified by using the flamegraph generation method and rack-mini-profiler statistics.

==Approach==
To complete the given task, we identified the factors which are causing the lag. This was done by using flamegraph generation and rack-mini-profiler statistics. The statistics generated by rack miniprofiler helped us to understand the time taken by each component to get loaded. From these statistics, we identified the model/controller/view/database query which is causing this latency. From here, we identified the methods which needed to be implemented in order to optimize the webpage rendering.

Optimization approach for the pages:
*users/list :- Pagination was used to optimze. This was done using the Kaminari gem for Rails.
*grades/view : Ajax must be used for the optimization. Maximum time for the loading is required to fetch the data regarding particular teams review and score partial. So data for each team should be loaded asynchronously with ajax.

==Identification of pages ==
===Analysis of page 1: expertiza.ncsu.edu/tree_display/list ===

===FlameGraph Statistics===
The following is the flamegraph for this page:-


[[File:f.png]]

The X axis of the FlameGraph represents the time taken to load the page. It can give a clear picture of where the expertiza page is getting bogged down. The widest layers take the longest to run. They’re the areas one should look into, because speeding them up could have the biggest impact. As we can see over here, most of the time taken over here is in the controller action.

===MiniProfiler Statistics===

The following is the rack-mini-profiler statistics for the page:-


[[File:tre1.png]]


The box which is seen in the leftmost corner is the MiniProfiler statistics. MiniProfiler gives  a constant notification of how long each page takes to load. The miniprofiler statistics show the time taken to render the view as well as the corresponding SQL calls.


[[File:tree2.png]]


This image shows the time and the number of SQL calls which are required for each rendering.


[[File:tree5.png]]


Here as we can see the total number of SQL calls required for children_node_ng is 1123. This is causing the latency in the entire page by 10564.8 ms. These statistics helped us to identify the method which needs to be refactored.


[[File:tree6.png]]


The total number of SQL calls required for 2 components here is 2 each. Hence we do not need to refactor this component.

===Identified Component causing the delay===
tree_display_controller

The following methods are consuming most of the time in this code :-

1)update_fnode_children

2)initialize_fnode_update_children

3)children_node_ng

These methods are from line 172-201 in this controller.

==  Solution ==
*users/list
The time taken to load the page was high because the server had to access through hundreds of rows in the database .The page was rendering the entire row of database at the same time. Pagination helped to split the database  into manageable chunks of data.

*grades/view
The grades/view is the page where  the instructor accesses the grades statistics of each student.
The reason for  the slow down in the page is that the data for all the teams were loaded as soon as the page was loaded.The design of the page has an individual tab for each team, hence loading the entire data as soon as the page is loaded is quite redundant as the instructor needs to access the scores of each individual team at a given time.
The solution to this problem is to access the individual team statistics from the database when the instructor clicks the tab specified for the particular individual. 

Issues faced during implementation of this solution:- 
1)



2)

==Test Plan==
The project is based on optimizing the time required to load the pages which has been mentioned above,in order to test that there is an improvement in the time taken to load these pages one will have to compare the initial loading time and the time taken after the code has been modified.

===Initial Setup for testing===

1)Clone the git repository.

2)Go to the commit on 3rd November(name: initial setup),this is the code which is originally provided by the professor and has not been modified. It however consists of the two gems required to check the loading time.

3)To login as instructor :-
USERNAME :- instructor6
Password:-password

===Testing===

1)Check the time taken to load the page localhost:3000/tree_display/list on the left corner and compare the same for the latest pull,a substantial difference in the loading time.


2)Check the time taken to load the page localhost:3000/review_mapping/response_report on the left corner and compare the same for the latest pull,a substantial difference in the loading time.


3)Check the time taken to load the page localhost:3000/grades/view on the left corner and compare the same for the latest pull,a substantial difference in the loading time.


4)Check the time taken to load the page localhost:3000/users/list on the left corner and compare the same for the latest pull,a substantial difference in the loading time.