Use of bullet gem and .includes to speed up db accesses: Difference between revisions
(Created page with "==Introduction== === Team === Dr. Gehringer (mentor), * Harsh Kachhadia (hmkachha) * Tirth Patel (tdpatel2) ==N+1 problem background== ==How eager loading solves the problem...") |
(No difference)
|
Revision as of 21:05, 17 December 2021
Introduction
Team
Dr. Gehringer (mentor),
- Harsh Kachhadia (hmkachha)
- Tirth Patel (tdpatel2)
N+1 problem background
How eager loading solves the problem
To solve the N+1 problem, we use a process called 'eager loading'. In general, whenever code requests for a resource using db query, a query is fired to the database and the particular resource is fetched. In lazy loading, only the resources required at the moment are fetched and if in future more resources or entities are needed, then additional queries are fired into database. This significantly increases the number of queries fired whenever there is some loop construct in the code. This results into slow page load speed. As opposed to lazy loading, by using eager loading, we load additional required entities which we might require in near future, along with the main resource required by the query. Especially, during loops such as .each, eager loading significantly lowers the page load time, due to less queries fired resulting from additional data loaded from a single main resource query.
For instance, if we are firing a query to fetch all participants, and then loop through all the participants using .each to display their course name, then using eager loading, we can load all the course names for all participants in the same query in which all the participants are fetched from the database, as opposed to individual queries fired to access course names for individual participants in lazy loading.
How 'includes' method helps in eager loading
In Rails, 'includes' method specifies model associations to be included in the result set of the query being fired to the database.
For eg. current_user.participants.includes(:assignment, assignment: [:course]).each{|p| course_ids << p.assignment.course.id if p.assignment and p.assignment.course
What happened is Post.includes(:user) told ActiveRecord to retrieve the corresponding user records from the database immediately after the initial request for all posts. Since the records of users were already in the memory, post.user.username could be retrieved by only one query. Now, even if we have 10,000 posts in our database, we can execute the example code above by just 2 queries! This is a huge difference.
Here participants.includes(:assignment, assignment: [:course]) will retrieve all the assignment and course records relating to each participant using 2 queries(1 for participants, 1 for assignments and courses relating to participants), so that when each loop accesses the assignment and course for a participant using 'p.assignment.course' or 'p.assignment', no new query will be fired. Thus, reducing the number of queries using 'includes' method to 2 queries even if we have 1000 participants as compared to 2000 queries.
How bullet gem helps in solving N+1 problem
How to use bullet gem in rails=
Overview of changes we made during this project
With the use of above mentioned bullet gem, we found the locations in the expertiza code, where eager loading needs to be implemented, by looking at the stack trace recorded in bullet.log file by bullet gem. Bullet gem locates these code locations, as we navigate throughout the website.
For this project, we targeted the following areas of expertiza, those which loads very slow and needs eager loading.
tree_display_controller
- Courses
- Assignments
- Questionnaires
- Users
grades_controller
- view (View scores)
review_mapping_controller
- list_mappings
reports_controller
- Review report
- Author-feedback report
- Teammate-review report
- Answer-tagging report
student_task_controller
- Impersonating a student
We navigated through the UI, that use the code in above mentioned controllers, during which bullet gem located the code locations in these files where eager loading has to be implemented using 'includes' method.
Files that are modified in this project
- tree_display_controller.rb
- feedback_response_map
- instructor.rb
- student_task.rb
- user.rb
- _feedback_report.html.erb
- _flash_notifications.html.erb
- development.rb
Code changes to resolve N+1 Problem in Expertiza
Testing plan
We aim to perform automatic and manual testing for this project in order to achieve better reliability for this implementation.
As for this project, very few lines of code have been modified in the above mentioned files and also as this is more of a refactoring project, automatic testing was done by checking whether the existing tests pass or not.
Manual testing was done by navigating the UI to the locations using the code from the files that were modified. All manual tests are passed.
Important Links
- Github Pull Request
- Expertiza wiki page link (the link is mentioned here for records. It redirects to this same page.)
- Google Doc Project Description
- Github Repository Link for this expertiza fork (make sure you change branch to 'beta' branch if the page doesn't load by default to beta branch)
- Youtube Video Link