CSC/ECE 517 Spring 2015/ch1b 18 AS
Apache Solr and Rails
The topic write up for this page can be found here.
Apache Solr<ref>http://lucene.apache.org/solr/</ref> is a standalone, open-source enterprise search server created by Yonik Seely. It has a REST-like API. It is a indexing and searching framework which could be deployed and used with many web frameworks like Rails, Drupal, Django etc. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results. It is a very popular, fast and scaleable open source enterprise search platform built on Apache Lucene search library. Websites using Rails can take advantage of the Solr search engine to provide very sophisticated and customizable search features. Rails integrates with Solr search server using Sunspot<ref>https://rubygems.org/gems/sunspot_rails</ref><ref>https://github.com/sunspot/sunspot</ref> gem.
Introduction
Apache Solr is a standalone enterprise search server with a REST-like API. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results. It is a popular, scalable, blazing-fast, open source enterprise search platform built on Apache Lucene. Websites using rails can take advantage of the Solr search engine to provide sophisticated and customizable search features.
Solr is built on top of Apache Lucene. It is a toolbox responsible for indexing, searching, spell-check and advance tokenization whereas Solr is a search server which inherits all the features of Lucene but also adds API integration, caching and most importantly a web admin interface. This feature makes Solr very easy to use in production environment. Both of these technology needs Java 1.4 or above.
Sunspot library interacts with Solr using a low-level interface called RSolr. It is a ruby client which integrates the Solr API's to Rails through the use of Sunspot gem. Sunspot has a drop-in ActiveRecord support. We can call Sunspot as the high level client of Solr. We will be talking about the installation and application of Solr using Sunspot gem in the section below.
Features<ref>http://lucene.apache.org/solr/features.html</ref>
Apache Solr has the following features
- Enables full-text matching, powered by Lucene software
- Built to handle high volume traffic
- Provides faceted searching
- Provides features like suggester, spellcheck, clustering, auto-complete, highlighting
- Extensible through plugins
- Supports statistical and aggregate processing of text
- Supports rich format data such as PDF, Word, Powerpoint
Rails uses the Sunspot Ruby library to integrate with the Solr search engine.
Installation
Sunspot makes it easy to do full text searching through Solr. Sunspot comes as a gem and is installed in the usual way by adding it to the Gemfile and running bundle.
gem 'sunspot_rails' gem 'sunspot_solr' # optional pre-packaged Solr distribution for use in development
bundle install
Once the gem and its dependencies have installed we will need to generate Sunspot’s configuration file which we can do by running
$ rails g sunspot_rails:install
This command creates a YML file at /config/sunspot.yml. We don’t need to make any changes to the default settings in this file. Sunspot embeds Solr inside the gem so there’s no need to install it separately. This means that it works straight out of the box which makes it far more convenient to use in development. To get it up and running we run
$ rake sunspot:solr:start
If you’re running OS X Lion and you haven’t installed a Java runtime you’ll be prompted to do so when you run this command. You may also see a deprecation warning but this can be safely ignored. The command will also create some more configuration files for advanced configuration.
Usage and Examples<ref>http://outoftime.github.io/sunspot/docs/index.html</ref>
Indexing Objects
Add a searchable block to the objects you wish to index.
class Example < ActiveRecord::Base
searchable do
text :title, :body
end
end
Text fields will be full-text searchable. Other fields which are outside the scope of searchable can be used to scope queries.
Searching Objects
Now searching can be done simply by passing the query to search method in the respective class.
Example.search do fulltext 'query' with :conditions end
We can use many variations on the search now by changing the scope variables in the query.
Example of a Blog Search
Here we want to index the text fields. So, it is kept inside the searchable block.
class Blog < ActiveRecord::Base
searchable do
text :title, :body
text :comments do
comments.map { |comment| comment.body }
end
boolean :featured
integer :blog_id
integer :author_id
integer :category_ids, :multiple => true
double :average_rating
time :published_at
time :expired_at
string :sort_title do
title.downcase.gsub(/^(an?|the)/, '')
end
end
end
Now for searching the indexed objects we can use:
Blog.search do fulltext 'best author' with :blog_id, 1 with(:published_at).less_than Time.now order_by :published_at, :desc paginate :page => 2, :per_page => 15 facet :category_ids, :author_id end
Here the text fields are full text searchable and other attributes like blog_id, page, author_id is used to scope the query.
Extensions to Sunspot
Though Sunspot is used primarily for indexing and searching, it could be further extended to support multiple features. Some of them are listed as:
Scoping scalar fields
We can put Positive and negative restrictions along with conjunctions or disjunctions to scope the query.
- Positive restrictions
#Posts with a category of 1, 3, or 5 Blog.search do with(:category_ids, [1, 3, 5]) end
- Negative restrictions
# Blogs not in category 1 or 3 Blog.search do without(:category_ids, [1, 3]) end
- Disjunctions and Conjunctions
# Blogs that do not have an expired time or have not yet expired
Blog.search do
any_of do
with(:expired_at).greater_than(Time.now)
with(:expired_at, nil)
end
end
Pagination
The search results given by Sunspot are paginated upto 30 items per page. A custom number of results per page can be specified with the :per_page option to paginate:
search = Blog.search do fulltext "pizza" paginate :page => 1, :per_page => 50 end
Faceting
It is a feature of Solr that determines the number of documents that match a given search and an additional criterion. This allows you to build powerful drill-down interfaces for search. In field facets each row represents a particular value for a given field. In query facets, each row represents an arbitrary scope.
# Blogs that match 'war' returning counts for each :author_id
search = Blog.search do
fulltext "war"
facet :author_id
end
search.facet(:author_id).rows.each do |facet|
puts "Author #{facet.value} has #{facet.count} war article!"
end
Ordering
By default, Sunspot orders results by "score": the Solr-determined relevancy metric. We can use order_by method to customize the search.
# Order by average rating, descending
Blog.search do
fulltext("war")
order_by(:average_rating, :desc)
end
Highlighting
It is the snippet of the matched part of the search result. It has to be stored in order to be produced.
search = Blog.search do
fulltext "war" do
highlight :body
end
end
It will highlight the word war in every result string.
Hits vs Results
Sunspot simply stores the type and primary key of objects in Solr. When results are retrieved, those primary keys are used to load the actual object like any relational database.
# Using #results pulls in the records from the object-relational mapper Blog.search.results.each do |result| puts result.body end
To get results without accessing the database, use hits:
Blog.search.hits.each do |hit| puts hit.stored(:body) end
Reindexing
Objects are automatically indexed to Solr as a part of the save callbacks but if there is a change in schema then reindexing is necessary.
bundle exec rake sunspot:solr:reindex
<ref>http://tech.favoritemedium.com/2010/01/full-text-search-in-rails-with-sunspot.html</ref><ref>http://www.linux-mag.com/id/7341/</ref>Other interesting reads on the topic and referral links are mentioned below.
References
<references/>