CSC/ECE 517 Fall 2016/E1701. Accelerate RSpec testing

Expertiza

Expertiza is a web application where students can submit and peer-review learning objects (articles, code, web sites, etc). It is used in select courses at NC State and by professors at several other colleges and universities.

Rspec

Rspec is a meta-gem, which depends on the rspec-core, rspec-expectations and rspec-mocks gems. Each of these can be installed separately and loaded in isolation using require. Among other benefits, this allows you to use rspec-expectations, for example, in Test::Unit::TestCase if you happen to prefer that style. Conversely, if you like RSpec's approach to declaring example groups and examples (describe and it) but prefer Test::Unit assertions and mocha, rr or flexmock for mocking, you'll be able to do that without having to install or load the components of RSpec that you're not using.

Problem Statement

Unfortunately, Expertiza tests are really slow. If you check the TravisCI, it needs nearly 9 min to run all tests.

One reason is that we use fixture to create records in test DB each time running tests.

For example, the codes in quiz_spec.rb contain too much create so that prolong testing time.

   # Create an assignment due date
    create(:deadline_type, name: "submission")
    create(:deadline_type, name: "review")
    create(:deadline_type, name: "metareview")
    create(:deadline_type, name: "drop_topic")
    create(:deadline_type, name: "signup")
    create(:deadline_type, name: "team_formation")
    create(:deadline_right)
    create(:deadline_right, name: 'Late')
    create(:deadline_right, name: 'OK')
    create :assignment_due_date, due_at: (DateTime.now + 1)

One solution is building an complete database to support all RSpec test without creating new records.

Task

Formally, we need to:

• Create the records in test DB according to the content in fixtures.
• Check each test file and delete certain DB records creation code that insert default records (eg. create(:deadline_type)).
• And keep all the test cases passing when using test DB and make sure the time running test cases is shorter than before.
• You should submit the sql file of test DB to Expertiza.

To finish these tasks, we need to modify all RSpec test files in Expertiza and using a new Database:Expertiza_test.

Refactor files

test database

Rspec tests

Design

Design idea explanation

In this design, we create a new database for testing, and we save some related information in this database. Every test just need to fetch data from this test database, and after the testing, we need to add some operation to rollback the data. For example, if the test is about creating a new quiz, in this situation, we need to delete this quiz after this test. Because of it, the state of database can be saved, and also it will not influence other test or test this test case again.

Database design

We design a expertiza_test database to save the test date used for Rspec. The test database owns the same structure as the real expertiza database, including the relations between tables and some restriction of attribute like it cannot be null or some other requirements for different attributes. Besides, the data in test database is the same as the data in factories part in spec, which includes FeedbackResponseMap.rb, Respone.rb, factories.rb, quiz_factory.rb. In this situation we do not need to create some data before testing, and we can use the data in test DB directly. Because of it, the overhead of testing cases will experience obvious decreasing. like the Rspec statements in creating some User objects.:

 factory :admin, class: User do
   sequence(:name) {|n| "admin#{n}" }
   role { Role.where(name: 'Administrator').first || association(:role_of_administrator) }
   password "password"
   password_confirmation "password"
   sequence(:fullname) {|n| "#{n}, administrator" }
   email "expertiza@mailinator.com"
   parent_id 1
   private_by_default  false
   mru_directory_path  nil
   email_on_review true
   email_on_submission true
   email_on_review_of_review true
   is_new_user false
   master_permission_granted 0
   handle "handle"
   leaderboard_privacy false
   digital_certificate nil
   timezonepref nil
   public_key nil
   copy_of_emails  false
 end

then transfer this into SQL query and then add the information the same as designed for Rspec test into expertiza_test database.

We will try two method to achieve the goal:

1. Write a script to create all test data automatically then don't clean the database.
2. We transfer the data in girls_factory into the data base, and there are some steps:

1. we delete some related information in Rails_hepler.rb

config.before(:suite) do
   DatabaseCleaner.clean_with(:truncation)
 end
 config.before(:each) do |_example|
   DatabaseCleaner.strategy = :truncation
   DatabaseCleaner.start
 end

 config.after(:each) do
   DatabaseCleaner.clean
 end

Those statements is used for adding the technology to clean the database before everytime using Rspec test, and then also clean the database(using turncate to increase the time of cleaning database). 2. We generate a new create rb file, and in this file we just put the create statements in Rspec style function, like create(:deadline_type, name: "submission") to transfer the data into database when running rspec create.rb

Feature test design

To accelerate the feature test, we have to eliminate pseudo data creation. Now all the feature tests must create their own data before each test. For an example, now a test first create a user before it can do the login operation. The creation latency is considerable. After building a test database, all the pseudo data creation statement can be removed from the test files. Instead, the tests reference data stored in the test database. This change will not only accelerate the testing time, but also dry out the code.

The way tests reference data from database is the same as it does in the development mode. To login, or do other operations, the test can invoke existing user data in test database. Using "User.find()" or "User.where()", test data can easily be invoked by tests.

For existing tests, they can directly use the test database. For new tests added later, they can use the records in test database or create specialized data use FactoryGirl. So to keep the test database up to date ,it needs to be maintained regularly.

Modify feature test

With test data in the database, we modified the RSpec feature test files: 1. Disable the data clean statement in rails_helper.rb.

 config.before(:suite) do
    DatabaseCleaner.clean_with(:truncation)
  end
  config.before(:each) do |_example|
    DatabaseCleaner.strategy = :truncation
    DatabaseCleaner.start
  end
 
  config.after(:each) do
    DatabaseCleaner.clean
  end

2. Eliminate data creation statement in each test case. Delete” create(:assignment)” like code

3. Modify url visit statement. Modify “visit '/student_teams/view?student_id=1'” to @student=User.find_by(name:”student2064”) “visit '/student_teams/view?student_id=@student.id’”

4. Recover test database before or after each change data operation For tests involved with create or delete actions, the data modified should be rollback.

The file we need to edit

airbrake_expection_errors_feature_tests_spec.rb

assignment_creation_spec.rb

calibration_spec.rb

inherit_teams_display_spec.rb

instructor_interface_spec.rb

list_teams_spec.rb

questionnaire_spec.rb

review_mapping_spec.rb

staggered_deadline_spec.rb

topic_suggestion_spec.rb

Performance improvement

We run a single rspec test "airbrake_expection_errors_feature_tests_spec.rb" to test the performance improvement when using test database.

Without a test database, the feature test costs 1:37 minutes:

Using the test database, the same test costs 1:10 minutes:

The time is decreased by about 20%. Consider the browser's response time in both trial, the improvement proportion should be higher.

Here is a video to show the improvement: https://www.youtube.com/watch?v=9y0i-zSRHe8&feature=youtu.be