Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-05-05T16:44:39Z

Amohanr: /* Implementation */

Extending Developer Tools for Servo
__TOC__
=='''Introduction'''==
===Rust===
Rust is a general purpose, [http://en.wikipedia.org/wiki/Programming_paradigm#Multi-paradigm multi-paradigm], compiled programming language developed by [http://en.wikipedia.org/wiki/Mozilla Mozilla] Research. It is designed to be a "safe, concurrent, practical language", supporting [http://en.wikipedia.org/wiki/Purely_functional pure-functional], [http://en.wikipedia.org/wiki/Actor_model concurrent-actor], [http://en.wikipedia.org/wiki/Procedural_programming imperative-procedural], and object-oriented styles.<ref>http://en.wikipedia.org/wiki/Rust_%28programming_language%29</ref> Being a modern systems programming language focusing on safety and speed, it accomplishes these goals by being memory safe without using garbage collection.<ref>http://doc.rust-lang.org/nightly/intro.html</ref>

===Servo===
Servo is an experimental project to build a Web browser engine for a new generation of hardware: mobile devices, multi-core processors and high-performance GPUs. With Servo, we are rethinking the browser at every level of the technology stack — from input parsing to page layout to graphics rendering — to optimize for power efficiency and maximum parallelism.
Servo builds on top of Rust to provide a secure and reliable foundation. Memory safety at the core of the platform ensures a high degree of assurance in the browser’s trusted computing base. Rust’s lightweight task mechanism also promises to allow fine-grained isolation between browser components, such as tabs and extensions, without the need for expensive runtime protection schemes, like operating system process isolation.<ref>https://www.mozilla.org/en-US/research/projects/</ref>

=='''Background'''==
===Remote Developer Tools===

Firefox supports remote developer tools - ie. communicating with an arbitrary server that implements a protocol for exposing information about web content. You can use the [https://developer.mozilla.org/en-US/docs/Tools Firefox developer tools] on your desktop to debug Web sites and Web apps running in other browsers or runtimes. The other browser might be on the same device as the tools themselves or on a different device, such as a phone connected over USB.

=='''Project Description'''==
Servo implements a very basic developer tools server that currently supports executing JS remotely and investigating the DOM tree in the document inspector. We want to expand these capabilities by completing previous work that enables remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo.<ref>https://github.com/servo/servo/wiki/More-developer-tools-student-project</ref>

The initial step included changes for enabling remote logging from the [http://developer.mozilla.org/en-US/docs/Tools/Web_Console Web Console] using 'console.log'. This was completed as part of the OSS project (Pull request [http://github.com/servo/servo/pull/5229 here]).

In continuation, the objective of the project is to add support for logging HTTP requests and responses in the Web Console.

=='''Requirement Analysis'''==

To configure Firefox for remote debugging, please follow the instructions on [http://developer.mozilla.org/en-US/docs/Tools/Remote_Debugging/Thunderbird Setting up Firefox]

Alternatively, to just view the Web Console, follow instructions [http://developer.mozilla.org/en-US/docs/Tools/Web_Console here].

As mentioned above, Firefox provides the ability to debug a web page running on a remote server with its Developer Tools. The Message Display pane of the Web Console displays various kinds of messages:
* HTTP requests
* JavaScript warnings and errors
* CSS warnings, errors, and reflow events
* Security warnings and errors
* console API calls
* Input/output messages

The Message Display pane looks like this:

[[File:web_console_display_pane.jpg]]

Open the Web Console from the Developer menu (or Ctrl+Shift+K) from a version of Firefox with Developer Tools extensions. Navigate to any web page and observe messages on the Web Console (Console tab). You may have to check the Log option under "Net" (the first option in black).
The log is cleared on every redirection/reload of a page. To avoid this, you can select the "Enable Persistent Logs" option in Settings.

HTTP Requests are logged with lines that looks like this on the Web Console:

[[File:HTTP_console_log.jpg]]

Clicking on the message brings up a new window that gives more details about the HTTP request and the response.

[[File:network_action_window.jpg]]

Versions of Firefox in use today use [http://en.wikipedia.org/wiki/Gecko_%28software%29 Gecko] as the browser engine. The Developer Edition of Firefox includes complete support for the latest [https://developer.mozilla.org/en-US/docs/Tools Firefox Developer Tools]

Servo so far implements partial support for Developer Tools, which does not include the logging of HTTP requests as shown above. The goal of this project is to implement capability for logging HTTP requests and responses on the Web Console.
When we run Servo with our changes on a webpage with the --devtools argument, and connect from an instance of Firefox, we should be able to see the HTTP requests and responses from the server appear on the log in the Web Console.

=='''Implementation'''==
The following is a rough list of changes that will be required in the code to enable logging of HTTP messages. It includes adding types for HTTP request and response messages to the enum that represents messages exchanged between Servo and the debugger client. An "actor" is created to store the information that will be sent in these messages.
*Add a NetworkEventMessage variant to the DevtoolsControlMsg enum in devtools_traits/lib.rs, containing fields for the request id and the network event.
[[File:DevToolsControlMsg.png]]
**DevtoolsControlMsg is a template type for the messages exchanged between the developer tools server and the actor objects. The developer tools server runs in a loop in the run_server function in devtools/lib.rs (this will be referred to as the "main loop" in run_server). The run_server function takes as arguments a sender and receiver, that can send and receive respectively messages of type <DevtoolsControlMsg>
**A new variant NetworkEventMessage of DevtoolsControlMsg will be used for all network event messages that are sent to the developer tools server. These messages will notify the server of either an HTTP request or an HTTP response.
**NetworkEventMessage will have two fields - a String representing the unique id corresponding to a request-response pair, and an enum NetworkEvent that holds the information about the request or response.
**The HTTRequest variant of the NetworkEvent enum contains fields for the target url, the method, headers, and the request body. It uses the types that are present in the LoadData struct in components/net/resource_task.rs.
**The HTTPResponse variant of the DevtoolsControlMsg enum containing fields for the response headers, status, and body. It uses the same types that are present in the Metadata struct in components/net/resource_task.rs.
*Make the HTTP loader's load function take an optional Sender<DevtoolsControlMsg>. Use the cookies_chan argument as a model. Send HTTPRequest and HTTPResponse messages using this sender at the appropriate times.
**The HTTP loader's load function is present in components/net/http_loader.rs. This is the place in Servo from which HTTP requests are sent and responses are received. This means that the developer tools server, if it is running (i.e., Servo is run with the --devtools argument), needs to be notified of network events from this function.
**So the load function now needs a channel to send messages to the developer tools server (the run_server function).
**This is accomplished by adding an optional argument devtools_chan of type Sender<DevtoolsControlMsg> to the HTTP loader's factory. This is then passed to the load function, where it can be used to send messages to the devtools server.
**The devtools_chan parameter is passed to the HTTP loader's factory when a new resource task is created in components/net/resource_task.rs (new_resource_task)
**This devtools_chan is used to send messages of the type DevtoolsControlMsg::NetworkEventMessage from the load function.
**A message containing NetworkEvent::HttpRequest is sent with the information contained in the load_data struct of the load function.
**A new unique ID for this request is created using the uuid crate, and passed in the NetworkEventMessage. This ID is used whenever the devtools server is notified of further updates corresponding to this request or its response. A new ID is generated for every new request.
**A message containing NetworkEvent::HttpResponse is sent with the response information after the load function receives the HTTP response. The values in the fields of the HttpResponse enum come from the information contained in the metadata struct. This response is tagged with the same ID value that was generated for its request earlier.
*Create a NetworkEventActor actor in the devtools crate that stores the request and response information transmitted in the new messages. The String field in the NetworkEventMessage contains a unique ID that joins the HTTPRequest and HTTPResponse. Associate the unique IDs with the corresponding NetworkEventActor names via a hashtable.
[[File:NetworkEventActor.png]]
**A new NetworkEventActor that implements the Actor trait is added to components/devtools/actors/. The NetworkEventActor needs to implement the name() and handle_message(..) functions of the Actor trait. The NetworkEventActor stores information about the HTTP requests and responses.
**A new NetworkEventActor object is created for each network event that is a new HTTP request. The corresponding response and updates should use the same actor object, which makes it possible for the debugger client to send query messages to the actor for more information (headers, cookies, security info, etc).
**This is achieved by tracking the unique request_id of each new request when its actor object is created. The new NetworkEventActor object is assigned a name using the new_name function of the actor registry, which creates a name by appending a number to the "netevent" string that is passed to it. The (request_id, actor_name) pairs are tracked in a HashMap called actor_requests.
**When the devtools server is notified of new network event, the actor_requests hash table is first looked up with the request_id to see if an actor exists for the event. If it does, the name of the actor is retrieved from the table. Otherwise, a new NetworkEventActor is created.
*Send the networkEvent message when the HTTPRequest and HTTPResponse messages are received. Use the Firefox devtools code (onNetworkEvent) as a reference.
**When a NetworkEventMessage (HTTPRequest or HTTPResponse) is received, the NetworkEventActor corresponding to it is either created or located as described above.
**On an HTTPRequest, a networkEvent message is sent to each debugger client using the accepted_connections vector of TcpStream objects. The message is in JSON and contains the name of the NetworkEventActor associated with the event.
<pre>{
"from": "server1.conn0.consoleActor2",
"type": "networkEvent",
"eventActor": {
"actor": "server1.conn0.netEvent45",
"startedDateTime": "2015-04-22T20:47:08.545Z",
"url": "https://aus4.mozilla.org/update/3/Firefox/40.0a1/20150421152828/Darwin_x86_64-gcc3/en-US/default/Darwin%2013.4.0/default/default/update.xml?force=1",
"method": "GET",
"isXHR": true,
"private": false
}
}</pre>
**Similarly, on an HTTPResponse, a networkEventUpdate message is sent to the debugger client(s). The name of the NetworkEventActor passed in this message should be the same as of the actor that was used for the request.
<pre>{
"from": "server1.conn0.netEvent45",
"type": "networkEventUpdate",
"updateType": "responseStart",
"response": {
"httpVersion": "HTTP/1.1",
"remoteAddress": "63.245.217.43",
"remotePort": 443,
"status": "200",
"statusText": "OK",
"headersSize": 337,
"discardResponseBody": true
}
}</pre>

*Implement the getRequestHeaders, getRequestCookies, getRequestPostData, getReponseHeaders, getReponseCookies, and getResponseContent messages for NetworkEventActor.
**Messages can be sent from the debugger clients to the developer tools server via the Actor object.
**Once the client knows the name of the NetworkEventActor that is associated with a request, it can query the devtools server for more information about the request/response by sending messages with the actor name in the "to" field. These messages will be directed to the actor object with that name registered in the actors registry (The network event actors are registered upon creation).
**The handle_message function in the NetworkEventActor will handle messages that are sent to the actor object from the debugger client. It contains a parameter stream of type TcpStream that can be used to send JSON messages in reply.
**The handle_message function should return the appropriate response using the information stored in the fields of the NetworkEventActor. The struct contains private fields of type struct HttpRequest and struct HttpResponse that are populated at the time when requests and responses are received by the main loop in run_server.

Sample query message from the client:
<pre>
DBG-SERVER: Received packet 304: {
"to": "server1.conn1.child1/netEvent42",
"type": "getRequestHeaders"
}
</pre>

Sample response from the actor (truncated):
<pre>
DBG-SERVER: Received packet 305: {
"from": "server1.conn1.child1/netEvent42",
"headers": [
{
"name": "Host",
"value": "sendto.mozilla.org"
},
{
"name": "User-Agent",
"value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:40.0) Gecko/20100101 Firefox/40.0"
},
{
"name": "Accept",
"value": "image/png,image/*;q=0.8,*/*;q=0.5"
},
{
"name": "Accept-Language",
"value": "en-US,en;q=0.5"
},
</pre>

=='''Architecture'''==
The conceptual architecture of Mozilla Firefox running Gecko is represented in the following figure. This is from Grosskurth and Godfrey, A Reference Architecture for Web Browsers <ref>http://grosskurth.ca/papers/browser-refarch.pdf</ref>

[[File:mozilla_arch.jpg]]

===Component Diagrams===
Servo is designed to be highly parallel, with many components (rendering, layout, HTML parsing, image decoding, etc.) handled by fine-grained, isolated tasks.

The following figures represent the browser architecture with all these components, and the Servo architecture <ref>http://www.joshmatthews.net/fosdemservo/</ref>

Browser:

[[File:browser_arch.jpg]]

Servo Architecture:

[[File:servo_arch.jpg]]

=='''Design'''==
===enum in Rust===
Enums are datatypes with several alternate representations. A simple enum defines one or more constants with the same data type. In Rust, an enum can have complex variants though, like a struct. For example, consider an enum 'Shape' with variants 'Circle' and 'Triangle' each of which is a struct.

enum Shape {
Circle { center: Point, radius: f64 },
Triangle{ vert1: Point, vert2: Point, vert3: Point }
}

A variable of type Shape can be resolved to its appropriate variant by using a 'match'.

fn area(sh: Shape) -> f64 {
match sh {
Circle(_, size) => f64::consts::PI * size * size,
Rectangle(Point { x, y }, Point { x: x2, y: y2 }) => (x2 - x) * (y2 - y)
}
}

The Servo Developers Tools project has a 'DevtoolsControlMsg' enum which is used to instruct the devtools server to update its known actors/state according to changes in the browser. The current project requires the addition of two new variants to the 'DevtoolsControlMsg' enum, namely 'HTTRequest' and 'HTTResponse'.

===Sender and Receiver===

In Rust, a pipe is used for communication between tasks. A pipe is simply a pair of endpoints: one for sending messages and another for receiving messages(Sender and Receiver). The simplest way to create a pipe is to use the channel function to create a (Sender, Receiver) pair. In Rust parlance, a sender is a sending endpoint of a pipe, and a receiver is the receiving endpoint. A simple channel can be created as follows:
let (tx, rx) = channel();
spawn(proc() {
tx.send(10i);
});
assert_eq!(rx.recv(), 10i);

The sending half of the asynchronous channel type is Struct [https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html Sender<T>] and the receiving half is the struct [https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html Receiver<T>].
For the purpose of this project, we will be using the channel to send and receive messages. To log the HTTP requests and responses on the console, we can use a variant of the enum 'DevtoolsControlMsg'. This enum is used to send messages to the devtools server to update its actors/state according to changes in the browser. We now add HTTPRequest and HHTPResponse variant to the DevtoolsControlMsg enum. We can now use the HTTP loaders' load function to send the HTTPRequest and HHTPResponse to the main loop in the run_server, where it can be received using a receiver. This is done by adding an argument, Sender<DevtoolsControlMsg> to the load function.

=='''Design Patterns'''==
===Actor Model===
The actor model<ref>http://en.wikipedia.org/wiki/Actor_model</ref> in computer science is a mathematical model of concurrent computation that treats "actors" as the universal primitives of concurrent computation: in response to a message that it receives, an actor can make local decisions, create more actors, send more messages, and determine how to respond to the next message received.

The Actor model adopts the philosophy that everything is an actor. This is similar to the everything is an object philosophy used by some object-oriented programming languages, but differs in that object-oriented software is typically executed sequentially, while the Actor model is inherently concurrent.

An actor is a computational entity that, in response to a message it receives, can concurrently:
*send a finite number of messages to other actors;
*create a finite number of new actors;
*designate the behavior to be used for the next message it receives.

Servo uses an actor-based devtools server implementation. There are different types of actor objects created to handle different messages. The developer tools server creates a ConsoleActor that is a part of every TabActor. The ConsoleActor is responsible for handling console events and sending messages of type ConsoleMsg to the debugger clients. As part of the initial steps of this project, we extended existing work that made console.log messages appear on the Web Console.

For the logging of Http requests and responses, a new NetworkEventActor that implements the Actor trait is added. This actor sends messages to, and listens for messages from, the debugger clients. We use the NetworkEventActor to store the request and response information that is to be transmitted in these messages. A unique id is used to keep track of a request and its corresponding response (as described in the Implementation section). In the event that a client wants more information about a request/response, it can query the server using this unique id. Following is a code snippet showing the structure of the actor and its fields.

<pre>
struct HttpRequest {
url: String,
method: Method,
headers: Headers,
body: Option<Vec<u8>>,
}

struct HttpResponse {
headers: Option<Headers>,
status: Option<RawStatus>,
body: Option<Vec<u8>>
}

pub struct NetworkEventActor {
pub name: String,
request: HttpRequest,
response: HttpResponse,
}
</pre>

Examples of messages sent from and handled by the NetworkEventActor are shown in the Implementation section. Some other sample JSON messages sent from Servo to the debugger clients can be found on [https://github.com/servo/servo/wiki/More-developer-tools-student-project this page]

=='''UML Diagrams'''==

The diagrams are an approximation of UML, that are drawn by treating the 'struct's in Rust as classes.

==='''Class Diagram'''===
Traits can be used in a manner similar to interfaces in a language like C++ or Java.

[[File:Class_diagram1.png‎]]

===='''Sequence Diagram'''====

[[File:uml_seq1.jpg]]

=='''Proposed Test Cases'''==
===='''Testing the feature'''====
The feature added will be tested using the following steps:
* Build Servo successfully with the changes (./mach build)
* Run Servo on a web page with the --devtools argument specifying a port number, say 6000 (./mach run [url] --devtools 6000)
* Open an instance of a recent version Firefox configured with Remote Debugging enabled.
* Connect to Servo running the web page on port 6000.
* Enable persistent logs in Settings. Select logging of Net messages from the Console.
* Navigate to another url from the webpage running Servo.
* The HTTP request and response messages should be logged on the Console.

===='''Regression Testing'''====
We want to ensure that everything is in order after we have completed our changes; more importantly, that we have not broken any existing functionality. Mozilla has an automated workflow in place for running tests on pull requests submitted to Servo. Servo has a test suite which runs on every reviewed pull request, via [https://github.com/bors-servo bors-servo] and the buildbot.

After it sees an approval on a reviewed pull requests, bors-servo automatically merges the changes

[[File:bors-servo-merge.jpg]]

It starts off running a suite of regression tests, and any failure is reported. It can be issued a retry command once the fixes are in place.

[[File:bors-servo-fail.jpg]]

When all tests pass, the changes are successfully merged into master. The following is the set of tests that will be run after the pull request is reviewed:

[[File:bors-servo-success.jpg]]

===='''Unit Tests'''====
The unit tests can be run using the command:
<pre>
./mach test-unit
</pre>
Since the code changes involve changing the parameters needed by existing functions new_resource_task(..) and LoadData::new(..), some of the unit tests that used these functions need to be fixed.

=='''Reference'''==
<references/>

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-05-05T16:43:47Z

Amohanr: /* Implementation */

Extending Developer Tools for Servo
__TOC__
=='''Introduction'''==
===Rust===
Rust is a general purpose, [http://en.wikipedia.org/wiki/Programming_paradigm#Multi-paradigm multi-paradigm], compiled programming language developed by [http://en.wikipedia.org/wiki/Mozilla Mozilla] Research. It is designed to be a "safe, concurrent, practical language", supporting [http://en.wikipedia.org/wiki/Purely_functional pure-functional], [http://en.wikipedia.org/wiki/Actor_model concurrent-actor], [http://en.wikipedia.org/wiki/Procedural_programming imperative-procedural], and object-oriented styles.<ref>http://en.wikipedia.org/wiki/Rust_%28programming_language%29</ref> Being a modern systems programming language focusing on safety and speed, it accomplishes these goals by being memory safe without using garbage collection.<ref>http://doc.rust-lang.org/nightly/intro.html</ref>

===Servo===
Servo is an experimental project to build a Web browser engine for a new generation of hardware: mobile devices, multi-core processors and high-performance GPUs. With Servo, we are rethinking the browser at every level of the technology stack — from input parsing to page layout to graphics rendering — to optimize for power efficiency and maximum parallelism.
Servo builds on top of Rust to provide a secure and reliable foundation. Memory safety at the core of the platform ensures a high degree of assurance in the browser’s trusted computing base. Rust’s lightweight task mechanism also promises to allow fine-grained isolation between browser components, such as tabs and extensions, without the need for expensive runtime protection schemes, like operating system process isolation.<ref>https://www.mozilla.org/en-US/research/projects/</ref>

=='''Background'''==
===Remote Developer Tools===

Firefox supports remote developer tools - ie. communicating with an arbitrary server that implements a protocol for exposing information about web content. You can use the [https://developer.mozilla.org/en-US/docs/Tools Firefox developer tools] on your desktop to debug Web sites and Web apps running in other browsers or runtimes. The other browser might be on the same device as the tools themselves or on a different device, such as a phone connected over USB.

=='''Project Description'''==
Servo implements a very basic developer tools server that currently supports executing JS remotely and investigating the DOM tree in the document inspector. We want to expand these capabilities by completing previous work that enables remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo.<ref>https://github.com/servo/servo/wiki/More-developer-tools-student-project</ref>

The initial step included changes for enabling remote logging from the [http://developer.mozilla.org/en-US/docs/Tools/Web_Console Web Console] using 'console.log'. This was completed as part of the OSS project (Pull request [http://github.com/servo/servo/pull/5229 here]).

In continuation, the objective of the project is to add support for logging HTTP requests and responses in the Web Console.

=='''Requirement Analysis'''==

To configure Firefox for remote debugging, please follow the instructions on [http://developer.mozilla.org/en-US/docs/Tools/Remote_Debugging/Thunderbird Setting up Firefox]

Alternatively, to just view the Web Console, follow instructions [http://developer.mozilla.org/en-US/docs/Tools/Web_Console here].

As mentioned above, Firefox provides the ability to debug a web page running on a remote server with its Developer Tools. The Message Display pane of the Web Console displays various kinds of messages:
* HTTP requests
* JavaScript warnings and errors
* CSS warnings, errors, and reflow events
* Security warnings and errors
* console API calls
* Input/output messages

The Message Display pane looks like this:

[[File:web_console_display_pane.jpg]]

Open the Web Console from the Developer menu (or Ctrl+Shift+K) from a version of Firefox with Developer Tools extensions. Navigate to any web page and observe messages on the Web Console (Console tab). You may have to check the Log option under "Net" (the first option in black).
The log is cleared on every redirection/reload of a page. To avoid this, you can select the "Enable Persistent Logs" option in Settings.

HTTP Requests are logged with lines that looks like this on the Web Console:

[[File:HTTP_console_log.jpg]]

Clicking on the message brings up a new window that gives more details about the HTTP request and the response.

[[File:network_action_window.jpg]]

Versions of Firefox in use today use [http://en.wikipedia.org/wiki/Gecko_%28software%29 Gecko] as the browser engine. The Developer Edition of Firefox includes complete support for the latest [https://developer.mozilla.org/en-US/docs/Tools Firefox Developer Tools]

Servo so far implements partial support for Developer Tools, which does not include the logging of HTTP requests as shown above. The goal of this project is to implement capability for logging HTTP requests and responses on the Web Console.
When we run Servo with our changes on a webpage with the --devtools argument, and connect from an instance of Firefox, we should be able to see the HTTP requests and responses from the server appear on the log in the Web Console.

=='''Implementation'''==
The following is a rough list of changes that will be required in the code to enable logging of HTTP messages. It includes adding types for HTTP request and response messages to the enum that represents messages exchanged between Servo and the debugger client. An "actor" is created to store the information that will be sent in these messages.
*Add a NetworkEventMessage variant to the DevtoolsControlMsg enum in devtools_traits/lib.rs, containing fields for the request id and the network event.
[[File:DevtoolsControlMsg.png]]
**DevtoolsControlMsg is a template type for the messages exchanged between the developer tools server and the actor objects. The developer tools server runs in a loop in the run_server function in devtools/lib.rs (this will be referred to as the "main loop" in run_server). The run_server function takes as arguments a sender and receiver, that can send and receive respectively messages of type <DevtoolsControlMsg>
**A new variant NetworkEventMessage of DevtoolsControlMsg will be used for all network event messages that are sent to the developer tools server. These messages will notify the server of either an HTTP request or an HTTP response.
**NetworkEventMessage will have two fields - a String representing the unique id corresponding to a request-response pair, and an enum NetworkEvent that holds the information about the request or response.
**The HTTRequest variant of the NetworkEvent enum contains fields for the target url, the method, headers, and the request body. It uses the types that are present in the LoadData struct in components/net/resource_task.rs.
**The HTTPResponse variant of the DevtoolsControlMsg enum containing fields for the response headers, status, and body. It uses the same types that are present in the Metadata struct in components/net/resource_task.rs.
*Make the HTTP loader's load function take an optional Sender<DevtoolsControlMsg>. Use the cookies_chan argument as a model. Send HTTPRequest and HTTPResponse messages using this sender at the appropriate times.
**The HTTP loader's load function is present in components/net/http_loader.rs. This is the place in Servo from which HTTP requests are sent and responses are received. This means that the developer tools server, if it is running (i.e., Servo is run with the --devtools argument), needs to be notified of network events from this function.
**So the load function now needs a channel to send messages to the developer tools server (the run_server function).
**This is accomplished by adding an optional argument devtools_chan of type Sender<DevtoolsControlMsg> to the HTTP loader's factory. This is then passed to the load function, where it can be used to send messages to the devtools server.
**The devtools_chan parameter is passed to the HTTP loader's factory when a new resource task is created in components/net/resource_task.rs (new_resource_task)
**This devtools_chan is used to send messages of the type DevtoolsControlMsg::NetworkEventMessage from the load function.
**A message containing NetworkEvent::HttpRequest is sent with the information contained in the load_data struct of the load function.
**A new unique ID for this request is created using the uuid crate, and passed in the NetworkEventMessage. This ID is used whenever the devtools server is notified of further updates corresponding to this request or its response. A new ID is generated for every new request.
**A message containing NetworkEvent::HttpResponse is sent with the response information after the load function receives the HTTP response. The values in the fields of the HttpResponse enum come from the information contained in the metadata struct. This response is tagged with the same ID value that was generated for its request earlier.
*Create a NetworkEventActor actor in the devtools crate that stores the request and response information transmitted in the new messages. The String field in the NetworkEventMessage contains a unique ID that joins the HTTPRequest and HTTPResponse. Associate the unique IDs with the corresponding NetworkEventActor names via a hashtable.
[[File:NetworkEventActor.png]]
**A new NetworkEventActor that implements the Actor trait is added to components/devtools/actors/. The NetworkEventActor needs to implement the name() and handle_message(..) functions of the Actor trait. The NetworkEventActor stores information about the HTTP requests and responses.
**A new NetworkEventActor object is created for each network event that is a new HTTP request. The corresponding response and updates should use the same actor object, which makes it possible for the debugger client to send query messages to the actor for more information (headers, cookies, security info, etc).
**This is achieved by tracking the unique request_id of each new request when its actor object is created. The new NetworkEventActor object is assigned a name using the new_name function of the actor registry, which creates a name by appending a number to the "netevent" string that is passed to it. The (request_id, actor_name) pairs are tracked in a HashMap called actor_requests.
**When the devtools server is notified of new network event, the actor_requests hash table is first looked up with the request_id to see if an actor exists for the event. If it does, the name of the actor is retrieved from the table. Otherwise, a new NetworkEventActor is created.
*Send the networkEvent message when the HTTPRequest and HTTPResponse messages are received. Use the Firefox devtools code (onNetworkEvent) as a reference.
**When a NetworkEventMessage (HTTPRequest or HTTPResponse) is received, the NetworkEventActor corresponding to it is either created or located as described above.
**On an HTTPRequest, a networkEvent message is sent to each debugger client using the accepted_connections vector of TcpStream objects. The message is in JSON and contains the name of the NetworkEventActor associated with the event.
<pre>{
"from": "server1.conn0.consoleActor2",
"type": "networkEvent",
"eventActor": {
"actor": "server1.conn0.netEvent45",
"startedDateTime": "2015-04-22T20:47:08.545Z",
"url": "https://aus4.mozilla.org/update/3/Firefox/40.0a1/20150421152828/Darwin_x86_64-gcc3/en-US/default/Darwin%2013.4.0/default/default/update.xml?force=1",
"method": "GET",
"isXHR": true,
"private": false
}
}</pre>
**Similarly, on an HTTPResponse, a networkEventUpdate message is sent to the debugger client(s). The name of the NetworkEventActor passed in this message should be the same as of the actor that was used for the request.
<pre>{
"from": "server1.conn0.netEvent45",
"type": "networkEventUpdate",
"updateType": "responseStart",
"response": {
"httpVersion": "HTTP/1.1",
"remoteAddress": "63.245.217.43",
"remotePort": 443,
"status": "200",
"statusText": "OK",
"headersSize": 337,
"discardResponseBody": true
}
}</pre>

*Implement the getRequestHeaders, getRequestCookies, getRequestPostData, getReponseHeaders, getReponseCookies, and getResponseContent messages for NetworkEventActor.
**Messages can be sent from the debugger clients to the developer tools server via the Actor object.
**Once the client knows the name of the NetworkEventActor that is associated with a request, it can query the devtools server for more information about the request/response by sending messages with the actor name in the "to" field. These messages will be directed to the actor object with that name registered in the actors registry (The network event actors are registered upon creation).
**The handle_message function in the NetworkEventActor will handle messages that are sent to the actor object from the debugger client. It contains a parameter stream of type TcpStream that can be used to send JSON messages in reply.
**The handle_message function should return the appropriate response using the information stored in the fields of the NetworkEventActor. The struct contains private fields of type struct HttpRequest and struct HttpResponse that are populated at the time when requests and responses are received by the main loop in run_server.

Sample query message from the client:
<pre>
DBG-SERVER: Received packet 304: {
"to": "server1.conn1.child1/netEvent42",
"type": "getRequestHeaders"
}
</pre>

Sample response from the actor (truncated):
<pre>
DBG-SERVER: Received packet 305: {
"from": "server1.conn1.child1/netEvent42",
"headers": [
{
"name": "Host",
"value": "sendto.mozilla.org"
},
{
"name": "User-Agent",
"value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:40.0) Gecko/20100101 Firefox/40.0"
},
{
"name": "Accept",
"value": "image/png,image/*;q=0.8,*/*;q=0.5"
},
{
"name": "Accept-Language",
"value": "en-US,en;q=0.5"
},
</pre>

=='''Architecture'''==
The conceptual architecture of Mozilla Firefox running Gecko is represented in the following figure. This is from Grosskurth and Godfrey, A Reference Architecture for Web Browsers <ref>http://grosskurth.ca/papers/browser-refarch.pdf</ref>

[[File:mozilla_arch.jpg]]

===Component Diagrams===
Servo is designed to be highly parallel, with many components (rendering, layout, HTML parsing, image decoding, etc.) handled by fine-grained, isolated tasks.

The following figures represent the browser architecture with all these components, and the Servo architecture <ref>http://www.joshmatthews.net/fosdemservo/</ref>

Browser:

[[File:browser_arch.jpg]]

Servo Architecture:

[[File:servo_arch.jpg]]

=='''Design'''==
===enum in Rust===
Enums are datatypes with several alternate representations. A simple enum defines one or more constants with the same data type. In Rust, an enum can have complex variants though, like a struct. For example, consider an enum 'Shape' with variants 'Circle' and 'Triangle' each of which is a struct.

enum Shape {
Circle { center: Point, radius: f64 },
Triangle{ vert1: Point, vert2: Point, vert3: Point }
}

A variable of type Shape can be resolved to its appropriate variant by using a 'match'.

fn area(sh: Shape) -> f64 {
match sh {
Circle(_, size) => f64::consts::PI * size * size,
Rectangle(Point { x, y }, Point { x: x2, y: y2 }) => (x2 - x) * (y2 - y)
}
}

The Servo Developers Tools project has a 'DevtoolsControlMsg' enum which is used to instruct the devtools server to update its known actors/state according to changes in the browser. The current project requires the addition of two new variants to the 'DevtoolsControlMsg' enum, namely 'HTTRequest' and 'HTTResponse'.

===Sender and Receiver===

In Rust, a pipe is used for communication between tasks. A pipe is simply a pair of endpoints: one for sending messages and another for receiving messages(Sender and Receiver). The simplest way to create a pipe is to use the channel function to create a (Sender, Receiver) pair. In Rust parlance, a sender is a sending endpoint of a pipe, and a receiver is the receiving endpoint. A simple channel can be created as follows:
let (tx, rx) = channel();
spawn(proc() {
tx.send(10i);
});
assert_eq!(rx.recv(), 10i);

The sending half of the asynchronous channel type is Struct [https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html Sender<T>] and the receiving half is the struct [https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html Receiver<T>].
For the purpose of this project, we will be using the channel to send and receive messages. To log the HTTP requests and responses on the console, we can use a variant of the enum 'DevtoolsControlMsg'. This enum is used to send messages to the devtools server to update its actors/state according to changes in the browser. We now add HTTPRequest and HHTPResponse variant to the DevtoolsControlMsg enum. We can now use the HTTP loaders' load function to send the HTTPRequest and HHTPResponse to the main loop in the run_server, where it can be received using a receiver. This is done by adding an argument, Sender<DevtoolsControlMsg> to the load function.

=='''Design Patterns'''==
===Actor Model===
The actor model<ref>http://en.wikipedia.org/wiki/Actor_model</ref> in computer science is a mathematical model of concurrent computation that treats "actors" as the universal primitives of concurrent computation: in response to a message that it receives, an actor can make local decisions, create more actors, send more messages, and determine how to respond to the next message received.

The Actor model adopts the philosophy that everything is an actor. This is similar to the everything is an object philosophy used by some object-oriented programming languages, but differs in that object-oriented software is typically executed sequentially, while the Actor model is inherently concurrent.

An actor is a computational entity that, in response to a message it receives, can concurrently:
*send a finite number of messages to other actors;
*create a finite number of new actors;
*designate the behavior to be used for the next message it receives.

Servo uses an actor-based devtools server implementation. There are different types of actor objects created to handle different messages. The developer tools server creates a ConsoleActor that is a part of every TabActor. The ConsoleActor is responsible for handling console events and sending messages of type ConsoleMsg to the debugger clients. As part of the initial steps of this project, we extended existing work that made console.log messages appear on the Web Console.

For the logging of Http requests and responses, a new NetworkEventActor that implements the Actor trait is added. This actor sends messages to, and listens for messages from, the debugger clients. We use the NetworkEventActor to store the request and response information that is to be transmitted in these messages. A unique id is used to keep track of a request and its corresponding response (as described in the Implementation section). In the event that a client wants more information about a request/response, it can query the server using this unique id. Following is a code snippet showing the structure of the actor and its fields.

<pre>
struct HttpRequest {
url: String,
method: Method,
headers: Headers,
body: Option<Vec<u8>>,
}

struct HttpResponse {
headers: Option<Headers>,
status: Option<RawStatus>,
body: Option<Vec<u8>>
}

pub struct NetworkEventActor {
pub name: String,
request: HttpRequest,
response: HttpResponse,
}
</pre>

Examples of messages sent from and handled by the NetworkEventActor are shown in the Implementation section. Some other sample JSON messages sent from Servo to the debugger clients can be found on [https://github.com/servo/servo/wiki/More-developer-tools-student-project this page]

=='''UML Diagrams'''==

The diagrams are an approximation of UML, that are drawn by treating the 'struct's in Rust as classes.

==='''Class Diagram'''===
Traits can be used in a manner similar to interfaces in a language like C++ or Java.

[[File:Class_diagram1.png‎]]

===='''Sequence Diagram'''====

[[File:uml_seq1.jpg]]

=='''Proposed Test Cases'''==
===='''Testing the feature'''====
The feature added will be tested using the following steps:
* Build Servo successfully with the changes (./mach build)
* Run Servo on a web page with the --devtools argument specifying a port number, say 6000 (./mach run [url] --devtools 6000)
* Open an instance of a recent version Firefox configured with Remote Debugging enabled.
* Connect to Servo running the web page on port 6000.
* Enable persistent logs in Settings. Select logging of Net messages from the Console.
* Navigate to another url from the webpage running Servo.
* The HTTP request and response messages should be logged on the Console.

===='''Regression Testing'''====
We want to ensure that everything is in order after we have completed our changes; more importantly, that we have not broken any existing functionality. Mozilla has an automated workflow in place for running tests on pull requests submitted to Servo. Servo has a test suite which runs on every reviewed pull request, via [https://github.com/bors-servo bors-servo] and the buildbot.

After it sees an approval on a reviewed pull requests, bors-servo automatically merges the changes

[[File:bors-servo-merge.jpg]]

It starts off running a suite of regression tests, and any failure is reported. It can be issued a retry command once the fixes are in place.

[[File:bors-servo-fail.jpg]]

When all tests pass, the changes are successfully merged into master. The following is the set of tests that will be run after the pull request is reviewed:

[[File:bors-servo-success.jpg]]

===='''Unit Tests'''====
The unit tests can be run using the command:
<pre>
./mach test-unit
</pre>
Since the code changes involve changing the parameters needed by existing functions new_resource_task(..) and LoadData::new(..), some of the unit tests that used these functions need to be fixed.

=='''Reference'''==
<references/>

File:NetworkEventActor.png

2015-05-05T16:41:49Z

Amohanr:

File:DevToolsControlMsg.png

2015-05-05T16:38:14Z

Amohanr:

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-05-05T16:29:16Z

Amohanr: /* Actor Model */

Extending Developer Tools for Servo
__TOC__
=='''Introduction'''==
===Rust===
Rust is a general purpose, [http://en.wikipedia.org/wiki/Programming_paradigm#Multi-paradigm multi-paradigm], compiled programming language developed by [http://en.wikipedia.org/wiki/Mozilla Mozilla] Research. It is designed to be a "safe, concurrent, practical language", supporting [http://en.wikipedia.org/wiki/Purely_functional pure-functional], [http://en.wikipedia.org/wiki/Actor_model concurrent-actor], [http://en.wikipedia.org/wiki/Procedural_programming imperative-procedural], and object-oriented styles.<ref>http://en.wikipedia.org/wiki/Rust_%28programming_language%29</ref> Being a modern systems programming language focusing on safety and speed, it accomplishes these goals by being memory safe without using garbage collection.<ref>http://doc.rust-lang.org/nightly/intro.html</ref>

===Servo===
Servo is an experimental project to build a Web browser engine for a new generation of hardware: mobile devices, multi-core processors and high-performance GPUs. With Servo, we are rethinking the browser at every level of the technology stack — from input parsing to page layout to graphics rendering — to optimize for power efficiency and maximum parallelism.
Servo builds on top of Rust to provide a secure and reliable foundation. Memory safety at the core of the platform ensures a high degree of assurance in the browser’s trusted computing base. Rust’s lightweight task mechanism also promises to allow fine-grained isolation between browser components, such as tabs and extensions, without the need for expensive runtime protection schemes, like operating system process isolation.<ref>https://www.mozilla.org/en-US/research/projects/</ref>

=='''Background'''==
===Remote Developer Tools===

Firefox supports remote developer tools - ie. communicating with an arbitrary server that implements a protocol for exposing information about web content. You can use the [https://developer.mozilla.org/en-US/docs/Tools Firefox developer tools] on your desktop to debug Web sites and Web apps running in other browsers or runtimes. The other browser might be on the same device as the tools themselves or on a different device, such as a phone connected over USB.

=='''Project Description'''==
Servo implements a very basic developer tools server that currently supports executing JS remotely and investigating the DOM tree in the document inspector. We want to expand these capabilities by completing previous work that enables remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo.<ref>https://github.com/servo/servo/wiki/More-developer-tools-student-project</ref>

The initial step included changes for enabling remote logging from the [http://developer.mozilla.org/en-US/docs/Tools/Web_Console Web Console] using 'console.log'. This was completed as part of the OSS project (Pull request [http://github.com/servo/servo/pull/5229 here]).

In continuation, the objective of the project is to add support for logging HTTP requests and responses in the Web Console.

=='''Requirement Analysis'''==

To configure Firefox for remote debugging, please follow the instructions on [http://developer.mozilla.org/en-US/docs/Tools/Remote_Debugging/Thunderbird Setting up Firefox]

Alternatively, to just view the Web Console, follow instructions [http://developer.mozilla.org/en-US/docs/Tools/Web_Console here].

As mentioned above, Firefox provides the ability to debug a web page running on a remote server with its Developer Tools. The Message Display pane of the Web Console displays various kinds of messages:
* HTTP requests
* JavaScript warnings and errors
* CSS warnings, errors, and reflow events
* Security warnings and errors
* console API calls
* Input/output messages

The Message Display pane looks like this:

[[File:web_console_display_pane.jpg]]

Open the Web Console from the Developer menu (or Ctrl+Shift+K) from a version of Firefox with Developer Tools extensions. Navigate to any web page and observe messages on the Web Console (Console tab). You may have to check the Log option under "Net" (the first option in black).
The log is cleared on every redirection/reload of a page. To avoid this, you can select the "Enable Persistent Logs" option in Settings.

HTTP Requests are logged with lines that looks like this on the Web Console:

[[File:HTTP_console_log.jpg]]

Clicking on the message brings up a new window that gives more details about the HTTP request and the response.

[[File:network_action_window.jpg]]

Versions of Firefox in use today use [http://en.wikipedia.org/wiki/Gecko_%28software%29 Gecko] as the browser engine. The Developer Edition of Firefox includes complete support for the latest [https://developer.mozilla.org/en-US/docs/Tools Firefox Developer Tools]

Servo so far implements partial support for Developer Tools, which does not include the logging of HTTP requests as shown above. The goal of this project is to implement capability for logging HTTP requests and responses on the Web Console.
When we run Servo with our changes on a webpage with the --devtools argument, and connect from an instance of Firefox, we should be able to see the HTTP requests and responses from the server appear on the log in the Web Console.

=='''Implementation'''==
The following is a rough list of changes that will be required in the code to enable logging of HTTP messages. It includes adding types for HTTP request and response messages to the enum that represents messages exchanged between Servo and the debugger client. An "actor" is created to store the information that will be sent in these messages.
*Add a NetworkEventMessage variant to the DevtoolsControlMsg enum in devtools_traits/lib.rs, containing fields for the request id and the network event.
**DevtoolsControlMsg is a template type for the messages exchanged between the developer tools server and the actor objects. The developer tools server runs in a loop in the run_server function in devtools/lib.rs (this will be referred to as the "main loop" in run_server). The run_server function takes as arguments a sender and receiver, that can send and receive respectively messages of type <DevtoolsControlMsg>
**A new variant NetworkEventMessage of DevtoolsControlMsg will be used for all network event messages that are sent to the developer tools server. These messages will notify the server of either an HTTP request or an HTTP response.
**NetworkEventMessage will have two fields - a String representing the unique id corresponding to a request-response pair, and an enum NetworkEvent that holds the information about the request or response.
**The HTTRequest variant of the NetworkEvent enum contains fields for the target url, the method, headers, and the request body. It uses the types that are present in the LoadData struct in components/net/resource_task.rs.
**The HTTPResponse variant of the DevtoolsControlMsg enum containing fields for the response headers, status, and body. It uses the same types that are present in the Metadata struct in components/net/resource_task.rs.
*Make the HTTP loader's load function take an optional Sender<DevtoolsControlMsg>. Use the cookies_chan argument as a model. Send HTTPRequest and HTTPResponse messages using this sender at the appropriate times.
**The HTTP loader's load function is present in components/net/http_loader.rs. This is the place in Servo from which HTTP requests are sent and responses are received. This means that the developer tools server, if it is running (i.e., Servo is run with the --devtools argument), needs to be notified of network events from this function.
**So the load function now needs a channel to send messages to the developer tools server (the run_server function).
**This is accomplished by adding an optional argument devtools_chan of type Sender<DevtoolsControlMsg> to the HTTP loader's factory. This is then passed to the load function, where it can be used to send messages to the devtools server.
**The devtools_chan parameter is passed to the HTTP loader's factory when a new resource task is created in components/net/resource_task.rs (new_resource_task)
**This devtools_chan is used to send messages of the type DevtoolsControlMsg::NetworkEventMessage from the load function.
**A message containing NetworkEvent::HttpRequest is sent with the information contained in the load_data struct of the load function.
**A new unique ID for this request is created using the uuid crate, and passed in the NetworkEventMessage. This ID is used whenever the devtools server is notified of further updates corresponding to this request or its response. A new ID is generated for every new request.
**A message containing NetworkEvent::HttpResponse is sent with the response information after the load function receives the HTTP response. The values in the fields of the HttpResponse enum come from the information contained in the metadata struct. This response is tagged with the same ID value that was generated for its request earlier.
*Create a NetworkEventActor actor in the devtools crate that stores the request and response information transmitted in the new messages. The String field in the NetworkEventMessage contains a unique ID that joins the HTTPRequest and HTTPResponse. Associate the unique IDs with the corresponding NetworkEventActor names via a hashtable.
**A new NetworkEventActor that implements the Actor trait is added to components/devtools/actors/. The NetworkEventActor needs to implement the name() and handle_message(..) functions of the Actor trait. The NetworkEventActor stores information about the HTTP requests and responses.
**A new NetworkEventActor object is created for each network event that is a new HTTP request. The corresponding response and updates should use the same actor object, which makes it possible for the debugger client to send query messages to the actor for more information (headers, cookies, security info, etc).
**This is achieved by tracking the unique request_id of each new request when its actor object is created. The new NetworkEventActor object is assigned a name using the new_name function of the actor registry, which creates a name by appending a number to the "netevent" string that is passed to it. The (request_id, actor_name) pairs are tracked in a HashMap called actor_requests.
**When the devtools server is notified of new network event, the actor_requests hash table is first looked up with the request_id to see if an actor exists for the event. If it does, the name of the actor is retrieved from the table. Otherwise, a new NetworkEventActor is created.
*Send the networkEvent message when the HTTPRequest and HTTPResponse messages are received. Use the Firefox devtools code (onNetworkEvent) as a reference.
**When a NetworkEventMessage (HTTPRequest or HTTPResponse) is received, the NetworkEventActor corresponding to it is either created or located as described above.
**On an HTTPRequest, a networkEvent message is sent to each debugger client using the accepted_connections vector of TcpStream objects. The message is in JSON and contains the name of the NetworkEventActor associated with the event.
<pre>{
"from": "server1.conn0.consoleActor2",
"type": "networkEvent",
"eventActor": {
"actor": "server1.conn0.netEvent45",
"startedDateTime": "2015-04-22T20:47:08.545Z",
"url": "https://aus4.mozilla.org/update/3/Firefox/40.0a1/20150421152828/Darwin_x86_64-gcc3/en-US/default/Darwin%2013.4.0/default/default/update.xml?force=1",
"method": "GET",
"isXHR": true,
"private": false
}
}</pre>
**Similarly, on an HTTPResponse, a networkEventUpdate message is sent to the debugger client(s). The name of the NetworkEventActor passed in this message should be the same as of the actor that was used for the request.
<pre>{
"from": "server1.conn0.netEvent45",
"type": "networkEventUpdate",
"updateType": "responseStart",
"response": {
"httpVersion": "HTTP/1.1",
"remoteAddress": "63.245.217.43",
"remotePort": 443,
"status": "200",
"statusText": "OK",
"headersSize": 337,
"discardResponseBody": true
}
}</pre>

*Implement the getRequestHeaders, getRequestCookies, getRequestPostData, getReponseHeaders, getReponseCookies, and getResponseContent messages for NetworkEventActor.
**Messages can be sent from the debugger clients to the developer tools server via the Actor object.
**Once the client knows the name of the NetworkEventActor that is associated with a request, it can query the devtools server for more information about the request/response by sending messages with the actor name in the "to" field. These messages will be directed to the actor object with that name registered in the actors registry (The network event actors are registered upon creation).
**The handle_message function in the NetworkEventActor will handle messages that are sent to the actor object from the debugger client. It contains a parameter stream of type TcpStream that can be used to send JSON messages in reply.
**The handle_message function should return the appropriate response using the information stored in the fields of the NetworkEventActor. The struct contains private fields of type struct HttpRequest and struct HttpResponse that are populated at the time when requests and responses are received by the main loop in run_server.

Sample query message from the client:
<pre>
DBG-SERVER: Received packet 304: {
"to": "server1.conn1.child1/netEvent42",
"type": "getRequestHeaders"
}
</pre>

Sample response from the actor (truncated):
<pre>
DBG-SERVER: Received packet 305: {
"from": "server1.conn1.child1/netEvent42",
"headers": [
{
"name": "Host",
"value": "sendto.mozilla.org"
},
{
"name": "User-Agent",
"value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:40.0) Gecko/20100101 Firefox/40.0"
},
{
"name": "Accept",
"value": "image/png,image/*;q=0.8,*/*;q=0.5"
},
{
"name": "Accept-Language",
"value": "en-US,en;q=0.5"
},
</pre>

=='''Architecture'''==
The conceptual architecture of Mozilla Firefox running Gecko is represented in the following figure. This is from Grosskurth and Godfrey, A Reference Architecture for Web Browsers <ref>http://grosskurth.ca/papers/browser-refarch.pdf</ref>

[[File:mozilla_arch.jpg]]

===Component Diagrams===
Servo is designed to be highly parallel, with many components (rendering, layout, HTML parsing, image decoding, etc.) handled by fine-grained, isolated tasks.

The following figures represent the browser architecture with all these components, and the Servo architecture <ref>http://www.joshmatthews.net/fosdemservo/</ref>

Browser:

[[File:browser_arch.jpg]]

Servo Architecture:

[[File:servo_arch.jpg]]

=='''Design'''==
===enum in Rust===
Enums are datatypes with several alternate representations. A simple enum defines one or more constants with the same data type. In Rust, an enum can have complex variants though, like a struct. For example, consider an enum 'Shape' with variants 'Circle' and 'Triangle' each of which is a struct.

enum Shape {
Circle { center: Point, radius: f64 },
Triangle{ vert1: Point, vert2: Point, vert3: Point }
}

A variable of type Shape can be resolved to its appropriate variant by using a 'match'.

fn area(sh: Shape) -> f64 {
match sh {
Circle(_, size) => f64::consts::PI * size * size,
Rectangle(Point { x, y }, Point { x: x2, y: y2 }) => (x2 - x) * (y2 - y)
}
}

The Servo Developers Tools project has a 'DevtoolsControlMsg' enum which is used to instruct the devtools server to update its known actors/state according to changes in the browser. The current project requires the addition of two new variants to the 'DevtoolsControlMsg' enum, namely 'HTTRequest' and 'HTTResponse'.

===Sender and Receiver===

In Rust, a pipe is used for communication between tasks. A pipe is simply a pair of endpoints: one for sending messages and another for receiving messages(Sender and Receiver). The simplest way to create a pipe is to use the channel function to create a (Sender, Receiver) pair. In Rust parlance, a sender is a sending endpoint of a pipe, and a receiver is the receiving endpoint. A simple channel can be created as follows:
let (tx, rx) = channel();
spawn(proc() {
tx.send(10i);
});
assert_eq!(rx.recv(), 10i);

The sending half of the asynchronous channel type is Struct [https://doc.rust-lang.org/std/sync/mpsc/struct.Sender.html Sender<T>] and the receiving half is the struct [https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html Receiver<T>].
For the purpose of this project, we will be using the channel to send and receive messages. To log the HTTP requests and responses on the console, we can use a variant of the enum 'DevtoolsControlMsg'. This enum is used to send messages to the devtools server to update its actors/state according to changes in the browser. We now add HTTPRequest and HHTPResponse variant to the DevtoolsControlMsg enum. We can now use the HTTP loaders' load function to send the HTTPRequest and HHTPResponse to the main loop in the run_server, where it can be received using a receiver. This is done by adding an argument, Sender<DevtoolsControlMsg> to the load function.

=='''Design Patterns'''==
===Actor Model===
The actor model<ref>http://en.wikipedia.org/wiki/Actor_model</ref> in computer science is a mathematical model of concurrent computation that treats "actors" as the universal primitives of concurrent computation: in response to a message that it receives, an actor can make local decisions, create more actors, send more messages, and determine how to respond to the next message received.

The Actor model adopts the philosophy that everything is an actor. This is similar to the everything is an object philosophy used by some object-oriented programming languages, but differs in that object-oriented software is typically executed sequentially, while the Actor model is inherently concurrent.

An actor is a computational entity that, in response to a message it receives, can concurrently:
*send a finite number of messages to other actors;
*create a finite number of new actors;
*designate the behavior to be used for the next message it receives.

Servo uses an actor-based devtools server implementation. There are different types of actor objects created to handle different messages. The developer tools server creates a ConsoleActor that is a part of every TabActor. The ConsoleActor is responsible for handling console events and sending messages of type ConsoleMsg to the debugger clients. As part of the initial steps of this project, we extended existing work that made console.log messages appear on the Web Console.

For the logging of Http requests and responses, a new NetworkEventActor that implements the Actor trait is added. This actor sends messages to, and listens for messages from, the debugger clients. We use the NetworkEventActor to store the request and response information that is to be transmitted in these messages. A unique id is used to keep track of a request and its corresponding response (as described in the Implementation section). In the event that a client wants more information about a request/response, it can query the server using this unique id. Following is a code snippet showing the structure of the actor and its fields.

<pre>
struct HttpRequest {
url: String,
method: Method,
headers: Headers,
body: Option<Vec<u8>>,
}

struct HttpResponse {
headers: Option<Headers>,
status: Option<RawStatus>,
body: Option<Vec<u8>>
}

pub struct NetworkEventActor {
pub name: String,
request: HttpRequest,
response: HttpResponse,
}
</pre>

Examples of messages sent from and handled by the NetworkEventActor are shown in the Implementation section. Some other sample JSON messages sent from Servo to the debugger clients can be found on [https://github.com/servo/servo/wiki/More-developer-tools-student-project this page]

=='''UML Diagrams'''==

The diagrams are an approximation of UML, that are drawn by treating the 'struct's in Rust as classes.

==='''Class Diagram'''===
Traits can be used in a manner similar to interfaces in a language like C++ or Java.

[[File:Class_diagram1.png‎]]

===='''Sequence Diagram'''====

[[File:uml_seq1.jpg]]

=='''Proposed Test Cases'''==
===='''Testing the feature'''====
The feature added will be tested using the following steps:
* Build Servo successfully with the changes (./mach build)
* Run Servo on a web page with the --devtools argument specifying a port number, say 6000 (./mach run [url] --devtools 6000)
* Open an instance of a recent version Firefox configured with Remote Debugging enabled.
* Connect to Servo running the web page on port 6000.
* Enable persistent logs in Settings. Select logging of Net messages from the Console.
* Navigate to another url from the webpage running Servo.
* The HTTP request and response messages should be logged on the Console.

===='''Regression Testing'''====
We want to ensure that everything is in order after we have completed our changes; more importantly, that we have not broken any existing functionality. Mozilla has an automated workflow in place for running tests on pull requests submitted to Servo. Servo has a test suite which runs on every reviewed pull request, via [https://github.com/bors-servo bors-servo] and the buildbot.

After it sees an approval on a reviewed pull requests, bors-servo automatically merges the changes

[[File:bors-servo-merge.jpg]]

It starts off running a suite of regression tests, and any failure is reported. It can be issued a retry command once the fixes are in place.

[[File:bors-servo-fail.jpg]]

When all tests pass, the changes are successfully merged into master. The following is the set of tests that will be run after the pull request is reviewed:

[[File:bors-servo-success.jpg]]

===='''Unit Tests'''====
The unit tests can be run using the command:
<pre>
./mach test-unit
</pre>
Since the code changes involve changing the parameters needed by existing functions new_resource_task(..) and LoadData::new(..), some of the unit tests that used these functions need to be fixed.

=='''Reference'''==
<references/>

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-05-05T03:13:33Z

Amohanr: /* UML Diagrams */

File:Class diagram1.png

2015-05-05T03:11:10Z

Amohanr:

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-05-04T19:00:55Z

Amohanr: /* Sender and Receiver */

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-05-04T16:33:00Z

Amohanr: /* Actor Model */

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-05-04T16:32:30Z

Amohanr: /* Actor Model */

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-05-04T16:30:34Z

Amohanr: /* Actor Model */

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-04-21T19:21:58Z

Amohanr: /* Sender and Receiver */

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-04-21T19:20:12Z

Amohanr: /* Sender */

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-04-01T21:16:45Z

Amohanr: /* Implementation */

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-04-01T21:05:45Z

Amohanr: /* Project Description */

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-04-01T21:05:35Z

Amohanr: /* Remote Developer Tools */

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-04-01T21:04:07Z

Amohanr: /* Actor Model */

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-03-31T19:52:51Z

Amohanr: /* Design Patterns */

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-03-31T16:22:25Z

Amohanr: /* Background */

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-03-31T01:55:16Z

Amohanr: /* Background */

Extending Developer Tools for Servo
__TOC__
=='''Introduction'''==
===Servo===
===Rust===

=='''Background'''==
===Remote Developer Tools===

=='''Project Description'''==
=='''Requirement Analysis'''==
=='''Implementation'''==
=='''Architecture'''==
=='''Design Patterns'''==
=='''UML Diagrams'''==
=='''Proposed Test Cases'''==

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-03-31T01:54:38Z

Amohanr: /* Introduction */

Extending Developer Tools for Servo
__TOC__
=='''Introduction'''==
===Servo===
===Rust===

=='''Background'''==
=='''Project Description'''==
=='''Requirement Analysis'''==
=='''Implementation'''==
=='''Architecture'''==
=='''Design Patterns'''==
=='''UML Diagrams'''==
=='''Proposed Test Cases'''==

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-03-31T01:31:12Z

Amohanr:

Extending Developer Tools for Servo
__TOC__
=='''Introduction'''==
=='''Background'''==
=='''Project Description'''==
=='''Requirement Analysis'''==
=='''Implementation'''==
=='''Architecture'''==
=='''Design Patterns'''==
=='''UML Diagrams'''==
=='''Proposed Test Cases'''==

CSC/ECE 517 Spring 2015 M1503 EDTS

2015-03-31T01:27:51Z

Amohanr: Created page with "Extending Developer Tools for Servo"

Extending Developer Tools for Servo

CSC/ECE 517 Spring 2015

2015-03-31T01:27:19Z

Amohanr: /* Final Project Design Document */

==Writing Assignment 1==
*[[CSC/ECE 517 Spring 2015/ch1a 17 WL]]
*[[CSC/ECE 517 Spring 2015/ch1a 5 ZX]]
*[[CSC/ECE 517 Spring 2015/ch1a 6 TZ]]
*[[CSC/ECE 517 Spring 2015/ch1a 4 RW]]
*[[CSC/ECE 517 Spring 2015/ch1a 7 SA]]
*[[CSC/ECE 517 Spring 2015/ch1a 9 RA]]
*[[CSC/ECE 517 Spring 2015/ch1a 14 RI]]
*[[CSC/ECE 517 Spring 2015/ch1a 1 DZ]]
*[[CSC/ECE 517 Spring 2015/ch1a 20 HA]]
*[[CSC/ECE 517 Spring 2015/ch1a 3 RF]]
*[[CSC/ECE 517 Spring 2015/ch1a 12 LS]]
*[[CSC/ECE 517 Spring 2015/ch1a 13 MA]]
*[[CSC/ECE 517 Spring 2015/ch1a 2 WA]]
*[[CSC/ECE 517 Spring 2015/ch1b 21 QW]]
*[[CSC/ECE 517 Spring 2015/ch1b 23 MS]]
*[[CSC/ECE 517 Spring 2015/ch1b 10 GL]]
*[[CSC/ECE 517 Spring 2015/ch1b 27 VC]]
*[[CSC/ECE 517 Spring 2015/ch1b 22 SF]]
*[[CSC/ECE 517 Spring 2015/ch1b 15 SH]]
*[[CSC/ECE 517 Spring 2015/ch1b 18 AS]]

==Writing Assignment 2==
*[[CSC/ECE 517 Spring 2015/oss E1502 wwj]]
*[[CSC/ECE 517 Fall 2014/oss E1508 MRS]]
*[[CSC/ECE 517 Spring 2015/oss E1504 IMV]]
*[[CSC/ECE 517 Spring 2015/oss E1505 xzl]]
*[[CSC/ECE 517 Spring 2015/oss E1509 lds]]
*[[CSC/ECE 517 Spring 2015/oss E1510 FLP]]
*[[CSC/ECE 517 Spring 2015/oss E1506 SYZ]]
*[[CSC/ECE 517 Spring 2015/oss S1504 AAC]]
*[[CSC/ECE 517 Spring 2015/oss E1507 DG]]
*[[CSC/ECE 517 Spring 2015/oss M1502 GVJ]]
*[[CSC/ECE 517 Spring 2015/oss M1503 EDT]]
*[[CSC/ECE 517 Spring 2015/oss E1503 RSA]]
*[[CSC/ECE 517 Spring 2015/oss E1501 YWS]]
*[[CSC/ECE 517 Spring 2015/oss S1501 OA]]

==Final Project Design Document==

*[[CSC/ECE 517 Spring 2015 E1526 MPRI]]
*[[CSC/ECE 517 Spring 2015 S1503 LWJZ]]
*[[CSC/ECE 517 Spring 2015 S1524 FSZZ]]
*[[CSC/ECE 517 Spring 2015 M1503 EDTS]]

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-24T03:54:01Z

Amohanr:

Extending Developer Tools for Servo

Servo<ref>https://github.com/servo/servo/</ref> is a prototype web browser engine written in the [http://en.wikipedia.org/wiki/JavaScript RUST] language. Servo implements a very basic developer tools server that currently supports executing [http://en.wikipedia.org/wiki/JavaScript JavaScript] (JS) remotely and investigating the [https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model Document Object Model] (DOM) tree in the document inspector. We want to expand these capabilities by completing previous work that enables remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo.

__TOC__
=='''Introduction'''==
Servo is a prototype web browser engine and is currently developed on 64bit OS X, 64bit Linux, and Android.It is developed using the RUST<ref>http://doc.rust-lang.org/book/README.html</ref> language.

=='''Running Servo'''==
===Installation===
Debian-based Linuxes
<pre>
sudo apt-get install curl freeglut3-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
msttcorefonts gperf g++ cmake python-virtualenv \
libssl-dev libbz2-dev libosmesa6-dev
</pre>

Fedora
<pre>
sudo yum install curl freeglut-devel libtool gcc-c++ libXi-devel \
freetype-devel mesa-libGL-devel glib2-devel libX11-devel libXrandr-devel gperf \
fontconfig-devel cabextract ttmkfdir python python-virtualenv expat-devel \
rpm-build openssl-devel cmake bzip2-devel libXcursor-devel
pushd /tmp
wget http://corefonts.sourceforge.net/msttcorefonts-2.5-1.spec
rpmbuild -bb msttcorefonts-2.5-1.spec
sudo yum install $HOME/rpmbuild/RPMS/noarch/msttcorefonts-2.5-1.noarch.rpm
popd
</pre>

===Building===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.
Normal Build
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build
./mach run tests/html/about-mozilla.html
</pre>

===Running===
<pre>
./mach run [url]
</pre>

=='''Remote Developer Tools<ref>https://developer.mozilla.org/en-US/docs/Tools/Remote_Debugging</ref>'''==
You can use remote developer tools, such as the one in Firefox to debug Web sites and Web apps running on different browsers. The other browser could either be on the same device or a different device.
Servo has a basic implementation of developer tools that supports executing JS remotely and investigating the DOM tree in the document inspector. As part of the project for extending the funcionalities to enable remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo, we performed some initial additions to the code to get the web console to print out log messages.

The following changes were made to the ConsoleMsg struct in components/devtools/libs.rs:
*rename logLevel to level and make it a String
*rename timestamp to timeStamp
*add an arguments vector of Strings
*remove the message field and add the value to the arguments vector
*add filename, lineNumber, and columnNumber fields

The following code snippets show the modified version:

components/devtools/lib.rs:
<pre>
struct ConsoleMsg {
level: String,
timeStamp: u64,
arguments: Vec<String>,
filename: String,
lineNumber: u32,
columnNumber: u32,
}

fn handle_console_message(actors: Arc<Mutex<ActorRegistry>>,
id: PipelineId,
console_message: ConsoleMessage,
actor_pipelines: &HashMap<PipelineId, String>) {
let console_actor_name = find_console_actor(actors.clone(), id, actor_pipelines);
let actors = actors.lock().unwrap();
let console_actor = actors.find::<ConsoleActor>(console_actor_name.as_slice());
match console_message {
ConsoleMessage::LogMessage(message, filename, lineNumber, columnNumber) => {
let msg = ConsoleAPICall {
from: console_actor.name.clone(),
__type__: "consoleAPICall".to_string(),
message: ConsoleMsg {
level: "log".to_string(),
timeStamp: precise_time_ns(),
arguments: vec!(message),
filename: filename,
lineNumber: lineNumber,
columnNumber: columnNumber,
},
};
for stream in console_actor.streams.borrow_mut().iter_mut() {
stream.write_json_packet(&msg);
}
}
}
}
</pre>

We also update the ConsoleMessage enum in components/devtools_traits/lib.rs to add fields filename, lineNumber, columnNumber
components/devtools_traits/lib.rs
<pre>
pub enum ConsoleMessage {
// Log: message, filename, line number, column number
LogMessage(String, String, u32, u32),
//WarnMessage(String),
}
</pre>

components/script/dom/console.rs
<pre>
impl<'a> ConsoleMethods for JSRef<'a, Console> {
fn Log(self, message: DOMString) {
println!("{}", message);
//TODO: Sending fake values for filename, lineNumber and columnNumber in LogMessage; adjust later
propagate_console_msg(&self, ConsoleMessage::LogMessage(message, String::from_str("test"), 1, 1));
}
.
.
.
</pre>

==Connecting to Firefox==
Run Servo on a webpage with the --devtools 6000 argument, connect to it from a recent version of Firefox. Check [https://developer.mozilla.org/en-US/docs/Tools/Remote_Debugging/Thunderbird here] to see how to configure Firefox for this. You can now experiment with the document inspector and web console.

=='''Conclusion'''==
After these changes were made, the message now logs on the web console. A video demonstration is available [http://www.screencast.com/t/nwN23WnG9 here].

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-24T03:47:50Z

Amohanr:

Extending Developer Tools for Servo

Servo<ref>https://github.com/servo/servo/</ref> is a prototype web browser engine written in the [http://en.wikipedia.org/wiki/JavaScript RUST] language. Servo implements a very basic developer tools server that currently supports executing [http://en.wikipedia.org/wiki/JavaScript JavaScript] (JS) remotely and investigating the [https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model Document Object Model] (DOM) tree in the document inspector. We want to expand these capabilities by completing previous work that enables remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo.

__TOC__
=='''Introduction'''==
Servo is a prototype web browser engine and is currently developed on 64bit OS X, 64bit Linux, and Android.It is developed using the RUST<ref>http://doc.rust-lang.org/book/README.html</ref> language.

=='''Running Servo'''==
===Installation===
Debian-based Linuxes
<pre>
sudo apt-get install curl freeglut3-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
msttcorefonts gperf g++ cmake python-virtualenv \
libssl-dev libbz2-dev libosmesa6-dev
</pre>

Fedora
<pre>
sudo yum install curl freeglut-devel libtool gcc-c++ libXi-devel \
freetype-devel mesa-libGL-devel glib2-devel libX11-devel libXrandr-devel gperf \
fontconfig-devel cabextract ttmkfdir python python-virtualenv expat-devel \
rpm-build openssl-devel cmake bzip2-devel libXcursor-devel
pushd /tmp
wget http://corefonts.sourceforge.net/msttcorefonts-2.5-1.spec
rpmbuild -bb msttcorefonts-2.5-1.spec
sudo yum install $HOME/rpmbuild/RPMS/noarch/msttcorefonts-2.5-1.noarch.rpm
popd
</pre>

===Building===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.
Normal Build
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build
./mach run tests/html/about-mozilla.html
</pre>

===Running===
<pre>
./mach run [url]
</pre>

=='''Remote Developer Tools<ref>https://developer.mozilla.org/en-US/docs/Tools/Remote_Debugging</ref>'''==
You can use remote developer tools, such as the one in Firefox to debug Web sites and Web apps running on different browsers. The other browser could either be on the same device or a different device.
Servo has a basic implementation of developer tools that supports executing JS remotely and investigating the DOM tree in the document inspector. As part of the project for extending the funcionalities to enable remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo, we performed some initial additions to the code to get the web console to print out log messages.

The following changes were made to the ConsoleMsg struct in components/devtools/libs.rs:
*rename logLevel to level and make it a String
*rename timestamp to timeStamp
*add an arguments vector of Strings
*remove the message field and add the value to the arguments vector
*add filename, lineNumber, and columnNumber fields

The following code snippets show the modified version:

components/devtools/lib.rs:
<pre>
struct ConsoleMsg {
level: String,
timeStamp: u64,
arguments: Vec<String>,
filename: String,
lineNumber: u32,
columnNumber: u32,
}

fn handle_console_message(actors: Arc<Mutex<ActorRegistry>>,
id: PipelineId,
console_message: ConsoleMessage,
actor_pipelines: &HashMap<PipelineId, String>) {
let console_actor_name = find_console_actor(actors.clone(), id, actor_pipelines);
let actors = actors.lock().unwrap();
let console_actor = actors.find::<ConsoleActor>(console_actor_name.as_slice());
match console_message {
ConsoleMessage::LogMessage(message, filename, lineNumber, columnNumber) => {
let msg = ConsoleAPICall {
from: console_actor.name.clone(),
__type__: "consoleAPICall".to_string(),
message: ConsoleMsg {
level: "log".to_string(),
timeStamp: precise_time_ns(),
arguments: vec!(message),
filename: filename,
lineNumber: lineNumber,
columnNumber: columnNumber,
},
};
for stream in console_actor.streams.borrow_mut().iter_mut() {
stream.write_json_packet(&msg);
}
}
}
}
</pre>

We also update the ConsoleMessage enum in components/devtools_traits/lib.rs to add fields filename, lineNumber, columnNumber
components/devtools_traits/lib.rs
<pre>
pub enum ConsoleMessage {
// Log: message, filename, line number, column number
LogMessage(String, String, u32, u32),
//WarnMessage(String),
}
</pre>

components/script/dom/console.rs
<pre>
impl<'a> ConsoleMethods for JSRef<'a, Console> {
fn Log(self, message: DOMString) {
println!("{}", message);
//TODO: Sending fake values for filename, lineNumber and columnNumber in LogMessage; adjust later
propagate_console_msg(&self, ConsoleMessage::LogMessage(message, String::from_str("test"), 1, 1));
}
.
.
.
</pre>

=='''Conclusion'''==
After these changes were made, the message now logs on the web console.

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-24T00:04:48Z

Amohanr: /* Running */

Extending Developer Tools for Servo

Servo<ref>https://github.com/servo/servo/</ref> is a prototype web browser engine written in the [http://en.wikipedia.org/wiki/JavaScript RUST] language. Servo implements a very basic developer tools server that currently supports executing [http://en.wikipedia.org/wiki/JavaScript JavaScript] (JS) remotely and investigating the [https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model Document Object Model] (DOM) tree in the document inspector. We want to expand these capabilities by completing previous work that enables remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo.

__TOC__
=='''Introduction'''==
Servo is a prototype web browser engine and is currently developed on 64bit OS X, 64bit Linux, and Android.It is developed using the RUST<ref>http://doc.rust-lang.org/book/README.html</ref> language.

=='''Running Servo'''==
===Installation===
Debian-based Linuxes
<pre>
sudo apt-get install curl freeglut3-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
msttcorefonts gperf g++ cmake python-virtualenv \
libssl-dev libbz2-dev libosmesa6-dev
</pre>

Fedora
<pre>
sudo yum install curl freeglut-devel libtool gcc-c++ libXi-devel \
freetype-devel mesa-libGL-devel glib2-devel libX11-devel libXrandr-devel gperf \
fontconfig-devel cabextract ttmkfdir python python-virtualenv expat-devel \
rpm-build openssl-devel cmake bzip2-devel libXcursor-devel
pushd /tmp
wget http://corefonts.sourceforge.net/msttcorefonts-2.5-1.spec
rpmbuild -bb msttcorefonts-2.5-1.spec
sudo yum install $HOME/rpmbuild/RPMS/noarch/msttcorefonts-2.5-1.noarch.rpm
popd
</pre>

===Building===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.
Normal Build
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build
./mach run tests/html/about-mozilla.html
</pre>

===Running===
<pre>
./mach run [url]
</pre>

=='''Remote Developer Tools<ref>https://developer.mozilla.org/en-US/docs/Tools/Remote_Debugging</ref>'''==
You can use remote developer tools, such as the one in Firefox to debug Web sites and Web apps running on different browsers. The other browser could either be on the same device or a different device.
Servo has a basic implementation of developer tools that supports executing JS remotely and investigating the DOM tree in the document inspector. As part of the project for extending the funcionalities to enable remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo, we performed some initial additions to the code to get the web console to print out log messages.

The following changes were made to the ConsoleMsg struct in components/devtools/libs.rs:
*rename logLevel to level and make it a String
*rename timestamp to timeStamp
*add an arguments vector of Strings
*remove the message field and add the value to the arguments vector
*add filename, lineNumber, and columnNumber fields

The following code snippets show the modified version:

components/devtools/lib.rs:
<pre>
struct ConsoleMsg {
level: String,
timeStamp: u64,
arguments: Vec<String>,
filename: String,
lineNumber: u32,
columnNumber: u32,
}

fn handle_console_message(actors: Arc<Mutex<ActorRegistry>>,
id: PipelineId,
console_message: ConsoleMessage,
actor_pipelines: &HashMap<PipelineId, String>) {
let console_actor_name = find_console_actor(actors.clone(), id, actor_pipelines);
let actors = actors.lock().unwrap();
let console_actor = actors.find::<ConsoleActor>(console_actor_name.as_slice());
match console_message {
ConsoleMessage::LogMessage(message, filename, lineNumber, columnNumber) => {
let msg = ConsoleAPICall {
from: console_actor.name.clone(),
__type__: "consoleAPICall".to_string(),
message: ConsoleMsg {
level: "log".to_string(),
timeStamp: precise_time_ns(),
arguments: vec!(message),
filename: filename,
lineNumber: lineNumber,
columnNumber: columnNumber,
},
};
for stream in console_actor.streams.borrow_mut().iter_mut() {
stream.write_json_packet(&msg);
}
}
}
}
</pre>

components/devtools_traits/lib.rs
<pre>
pub enum ConsoleMessage {
// Log: message, filename, line number, column number
LogMessage(String, String, u32, u32),
//WarnMessage(String),
}
</pre>

components/script/dom/console.rs
<pre>
impl<'a> ConsoleMethods for JSRef<'a, Console> {
fn Log(self, message: DOMString) {
println!("{}", message);
//TODO: Sending fake values for filename, lineNumber and columnNumber in LogMessage; adjust later
propagate_console_msg(&self, ConsoleMessage::LogMessage(message, String::from_str("test"), 1, 1));
}
.
.
.
</pre>

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-24T00:03:25Z

Amohanr:

Extending Developer Tools for Servo

Servo<ref>https://github.com/servo/servo/</ref> is a prototype web browser engine written in the [http://en.wikipedia.org/wiki/JavaScript RUST] language. Servo implements a very basic developer tools server that currently supports executing [http://en.wikipedia.org/wiki/JavaScript JavaScript] (JS) remotely and investigating the [https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model Document Object Model] (DOM) tree in the document inspector. We want to expand these capabilities by completing previous work that enables remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo.

__TOC__
=='''Introduction'''==
Servo is a prototype web browser engine and is currently developed on 64bit OS X, 64bit Linux, and Android.It is developed using the RUST<ref>http://doc.rust-lang.org/book/README.html</ref> language.

=='''Running Servo'''==
===Installation===
Debian-based Linuxes
<pre>
sudo apt-get install curl freeglut3-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
msttcorefonts gperf g++ cmake python-virtualenv \
libssl-dev libbz2-dev libosmesa6-dev
</pre>

Fedora
<pre>
sudo yum install curl freeglut-devel libtool gcc-c++ libXi-devel \
freetype-devel mesa-libGL-devel glib2-devel libX11-devel libXrandr-devel gperf \
fontconfig-devel cabextract ttmkfdir python python-virtualenv expat-devel \
rpm-build openssl-devel cmake bzip2-devel libXcursor-devel
pushd /tmp
wget http://corefonts.sourceforge.net/msttcorefonts-2.5-1.spec
rpmbuild -bb msttcorefonts-2.5-1.spec
sudo yum install $HOME/rpmbuild/RPMS/noarch/msttcorefonts-2.5-1.noarch.rpm
popd
</pre>

===Building===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.
Normal Build
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build
./mach run tests/html/about-mozilla.html
</pre>

===Running===
<pre>
./mach run [url]
</pre>

=='''Remote Developer Tools'''==<ref>https://developer.mozilla.org/en-US/docs/Tools/Remote_Debugging</ref>
You can use remote developer tools, such as the one in Firefox to debug Web sites and Web apps running on different browsers. The other browser could either be on the same device or a different device.
Servo has a basic implementation of developer tools that supports executing JS remotely and investigating the DOM tree in the document inspector. As part of the project for extending the funcionalities to enable remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo, we performed some initial additions to the code to get the web console to print out log messages.

The following changes were made to the ConsoleMsg struct in components/devtools/libs.rs:
*rename logLevel to level and make it a String
*rename timestamp to timeStamp
*add an arguments vector of Strings
*remove the message field and add the value to the arguments vector
*add filename, lineNumber, and columnNumber fields

The following code snippets show the modified version:

components/devtools/lib.rs:
<pre>
struct ConsoleMsg {
level: String,
timeStamp: u64,
arguments: Vec<String>,
filename: String,
lineNumber: u32,
columnNumber: u32,
}

fn handle_console_message(actors: Arc<Mutex<ActorRegistry>>,
id: PipelineId,
console_message: ConsoleMessage,
actor_pipelines: &HashMap<PipelineId, String>) {
let console_actor_name = find_console_actor(actors.clone(), id, actor_pipelines);
let actors = actors.lock().unwrap();
let console_actor = actors.find::<ConsoleActor>(console_actor_name.as_slice());
match console_message {
ConsoleMessage::LogMessage(message, filename, lineNumber, columnNumber) => {
let msg = ConsoleAPICall {
from: console_actor.name.clone(),
__type__: "consoleAPICall".to_string(),
message: ConsoleMsg {
level: "log".to_string(),
timeStamp: precise_time_ns(),
arguments: vec!(message),
filename: filename,
lineNumber: lineNumber,
columnNumber: columnNumber,
},
};
for stream in console_actor.streams.borrow_mut().iter_mut() {
stream.write_json_packet(&msg);
}
}
}
}
</pre>

components/devtools_traits/lib.rs
<pre>
pub enum ConsoleMessage {
// Log: message, filename, line number, column number
LogMessage(String, String, u32, u32),
//WarnMessage(String),
}
</pre>

components/script/dom/console.rs
<pre>
impl<'a> ConsoleMethods for JSRef<'a, Console> {
fn Log(self, message: DOMString) {
println!("{}", message);
//TODO: Sending fake values for filename, lineNumber and columnNumber in LogMessage; adjust later
propagate_console_msg(&self, ConsoleMessage::LogMessage(message, String::from_str("test"), 1, 1));
}
.
.
.
</pre>

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-24T00:02:05Z

Amohanr:

Extending Developer Tools for Servo

Servo<ref>https://github.com/servo/servo/</ref> is a prototype web browser engine written in the [http://en.wikipedia.org/wiki/JavaScript RUST] language. Servo implements a very basic developer tools server that currently supports executing [http://en.wikipedia.org/wiki/JavaScript JavaScript] (JS) remotely and investigating the [https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model Document Object Model] (DOM) tree in the document inspector. We want to expand these capabilities by completing previous work that enables remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo.

__TOC__
=='''Introduction'''==
Servo is a prototype web browser engine and is currently developed on 64bit OS X, 64bit Linux, and Android.It is developed using the RUST<ref>http://doc.rust-lang.org/book/README.html</ref> language.

=='''Running Servo'''==
===Installation===
Debian-based Linuxes
<pre>
sudo apt-get install curl freeglut3-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
msttcorefonts gperf g++ cmake python-virtualenv \
libssl-dev libbz2-dev libosmesa6-dev
</pre>

Fedora
<pre>
sudo yum install curl freeglut-devel libtool gcc-c++ libXi-devel \
freetype-devel mesa-libGL-devel glib2-devel libX11-devel libXrandr-devel gperf \
fontconfig-devel cabextract ttmkfdir python python-virtualenv expat-devel \
rpm-build openssl-devel cmake bzip2-devel libXcursor-devel
pushd /tmp
wget http://corefonts.sourceforge.net/msttcorefonts-2.5-1.spec
rpmbuild -bb msttcorefonts-2.5-1.spec
sudo yum install $HOME/rpmbuild/RPMS/noarch/msttcorefonts-2.5-1.noarch.rpm
popd
</pre>

===Building===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.
Normal Build
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build
./mach run tests/html/about-mozilla.html
</pre>

===Running===
<pre>
./mach run [url]
</pre>

=='''Remote Developer Tools'''==
You can use remote developer tools, such as the one in Firefox to debug Web sites and Web apps running on different browsers. The other browser could either be on the same device or a different device.
Servo has a basic implementation of developer tools that supports executing JS remotely and investigating the DOM tree in the document inspector. As part of the project for extending the funcionalities to enable remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo, we performed some initial additions to the code to get the web console to print out log messages.

The following changes were made to the ConsoleMsg struct in components/devtools/libs.rs:
*rename logLevel to level and make it a String
*rename timestamp to timeStamp
*add an arguments vector of Strings
*remove the message field and add the value to the arguments vector
*add filename, lineNumber, and columnNumber fields

The following code snippets show the modified version:

components/devtools/lib.rs:
<pre>
struct ConsoleMsg {
level: String,
timeStamp: u64,
arguments: Vec<String>,
filename: String,
lineNumber: u32,
columnNumber: u32,
}

fn handle_console_message(actors: Arc<Mutex<ActorRegistry>>,
id: PipelineId,
console_message: ConsoleMessage,
actor_pipelines: &HashMap<PipelineId, String>) {
let console_actor_name = find_console_actor(actors.clone(), id, actor_pipelines);
let actors = actors.lock().unwrap();
let console_actor = actors.find::<ConsoleActor>(console_actor_name.as_slice());
match console_message {
ConsoleMessage::LogMessage(message, filename, lineNumber, columnNumber) => {
let msg = ConsoleAPICall {
from: console_actor.name.clone(),
__type__: "consoleAPICall".to_string(),
message: ConsoleMsg {
level: "log".to_string(),
timeStamp: precise_time_ns(),
arguments: vec!(message),
filename: filename,
lineNumber: lineNumber,
columnNumber: columnNumber,
},
};
for stream in console_actor.streams.borrow_mut().iter_mut() {
stream.write_json_packet(&msg);
}
}
}
}
</pre>

components/devtools_traits/lib.rs
<pre>
pub enum ConsoleMessage {
// Log: message, filename, line number, column number
LogMessage(String, String, u32, u32),
//WarnMessage(String),
}
</pre>

components/script/dom/console.rs
<pre>
impl<'a> ConsoleMethods for JSRef<'a, Console> {
fn Log(self, message: DOMString) {
println!("{}", message);
//TODO: Sending fake values for filename, lineNumber and columnNumber in LogMessage; adjust later
propagate_console_msg(&self, ConsoleMessage::LogMessage(message, String::from_str("test"), 1, 1));
}
.
.
.
</pre>

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-24T00:01:39Z

Amohanr:

Extending Developer Tools for Servo

Servo<ref>https://github.com/servo/servo/</ref> is a prototype web browser engine written in the [http://en.wikipedia.org/wiki/JavaScript RUST] language. Servo implements a very basic developer tools server that currently supports executing [http://en.wikipedia.org/wiki/JavaScript JavaScript] (JS)remotely and investigating the [https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model Document Object Model] (DOM) tree in the document inspector. We want to expand these capabilities by completing previous work that enables remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo.

__TOC__
=='''Introduction'''==
Servo is a prototype web browser engine and is currently developed on 64bit OS X, 64bit Linux, and Android.It is developed using the RUST<ref>http://doc.rust-lang.org/book/README.html</ref> language.

=='''Running Servo'''==
===Installation===
Debian-based Linuxes
<pre>
sudo apt-get install curl freeglut3-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
msttcorefonts gperf g++ cmake python-virtualenv \
libssl-dev libbz2-dev libosmesa6-dev
</pre>

Fedora
<pre>
sudo yum install curl freeglut-devel libtool gcc-c++ libXi-devel \
freetype-devel mesa-libGL-devel glib2-devel libX11-devel libXrandr-devel gperf \
fontconfig-devel cabextract ttmkfdir python python-virtualenv expat-devel \
rpm-build openssl-devel cmake bzip2-devel libXcursor-devel
pushd /tmp
wget http://corefonts.sourceforge.net/msttcorefonts-2.5-1.spec
rpmbuild -bb msttcorefonts-2.5-1.spec
sudo yum install $HOME/rpmbuild/RPMS/noarch/msttcorefonts-2.5-1.noarch.rpm
popd
</pre>

===Building===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.
Normal Build
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build
./mach run tests/html/about-mozilla.html
</pre>

===Running===
<pre>
./mach run [url]
</pre>

=='''Remote Developer Tools'''==
You can use remote developer tools, such as the one in Firefox to debug Web sites and Web apps running on different browsers. The other browser could either be on the same device or a different device.
Servo has a basic implementation of developer tools that supports executing JS remotely and investigating the DOM tree in the document inspector. As part of the project for extending the funcionalities to enable remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo, we performed some initial additions to the code to get the web console to print out log messages.

The following changes were made to the ConsoleMsg struct in components/devtools/libs.rs:
*rename logLevel to level and make it a String
*rename timestamp to timeStamp
*add an arguments vector of Strings
*remove the message field and add the value to the arguments vector
*add filename, lineNumber, and columnNumber fields

The following code snippets show the modified version:

components/devtools/lib.rs:
<pre>
struct ConsoleMsg {
level: String,
timeStamp: u64,
arguments: Vec<String>,
filename: String,
lineNumber: u32,
columnNumber: u32,
}

fn handle_console_message(actors: Arc<Mutex<ActorRegistry>>,
id: PipelineId,
console_message: ConsoleMessage,
actor_pipelines: &HashMap<PipelineId, String>) {
let console_actor_name = find_console_actor(actors.clone(), id, actor_pipelines);
let actors = actors.lock().unwrap();
let console_actor = actors.find::<ConsoleActor>(console_actor_name.as_slice());
match console_message {
ConsoleMessage::LogMessage(message, filename, lineNumber, columnNumber) => {
let msg = ConsoleAPICall {
from: console_actor.name.clone(),
__type__: "consoleAPICall".to_string(),
message: ConsoleMsg {
level: "log".to_string(),
timeStamp: precise_time_ns(),
arguments: vec!(message),
filename: filename,
lineNumber: lineNumber,
columnNumber: columnNumber,
},
};
for stream in console_actor.streams.borrow_mut().iter_mut() {
stream.write_json_packet(&msg);
}
}
}
}
</pre>

components/devtools_traits/lib.rs
<pre>
pub enum ConsoleMessage {
// Log: message, filename, line number, column number
LogMessage(String, String, u32, u32),
//WarnMessage(String),
}
</pre>

components/script/dom/console.rs
<pre>
impl<'a> ConsoleMethods for JSRef<'a, Console> {
fn Log(self, message: DOMString) {
println!("{}", message);
//TODO: Sending fake values for filename, lineNumber and columnNumber in LogMessage; adjust later
propagate_console_msg(&self, ConsoleMessage::LogMessage(message, String::from_str("test"), 1, 1));
}
.
.
.
</pre>

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-23T23:59:12Z

Amohanr:

Extending Developer Tools for Servo

Servo<ref>https://github.com/servo/servo/</ref> is a prototype web browser engine written in the [http://en.wikipedia.org/wiki/JavaScript RUST] language. Servo implements a very basic developer tools server that currently supports executing [https://github.com/rust-lang/rust JavaScript] remotely and investigating the DOM tree in the document inspector. We want to expand these capabilities by completing previous work that enables remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo.

__TOC__
=='''Introduction'''==
Servo is a prototype web browser engine and is currently developed on 64bit OS X, 64bit Linux, and Android.It is developed using the RUST<ref>http://doc.rust-lang.org/book/README.html</ref> language.

=='''Running Servo'''==
===Installation===
Debian-based Linuxes
<pre>
sudo apt-get install curl freeglut3-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
msttcorefonts gperf g++ cmake python-virtualenv \
libssl-dev libbz2-dev libosmesa6-dev
</pre>

Fedora
<pre>
sudo yum install curl freeglut-devel libtool gcc-c++ libXi-devel \
freetype-devel mesa-libGL-devel glib2-devel libX11-devel libXrandr-devel gperf \
fontconfig-devel cabextract ttmkfdir python python-virtualenv expat-devel \
rpm-build openssl-devel cmake bzip2-devel libXcursor-devel
pushd /tmp
wget http://corefonts.sourceforge.net/msttcorefonts-2.5-1.spec
rpmbuild -bb msttcorefonts-2.5-1.spec
sudo yum install $HOME/rpmbuild/RPMS/noarch/msttcorefonts-2.5-1.noarch.rpm
popd
</pre>

===Building===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.
Normal Build
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build
./mach run tests/html/about-mozilla.html
</pre>

===Running===
<pre>
./mach run [url]
</pre>

=='''Remote Developer Tools'''==
You can use remote developer tools, such as the one in Firefox to debug Web sites and Web apps running on different browsers. The other browser could either be on the same device or a different device.
Servo has a basic implementation of developer tools that supports executing JS remotely and investigating the DOM tree in the document inspector. As part of the project for extending the funcionalities to enable remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo, we performed some initial additions to the code to get the web console to print out log messages.

The following changes were made to the ConsoleMsg struct in components/devtools/libs.rs:
*rename logLevel to level and make it a String
*rename timestamp to timeStamp
*add an arguments vector of Strings
*remove the message field and add the value to the arguments vector
*add filename, lineNumber, and columnNumber fields

The following code snippets show the modified version:

components/devtools/lib.rs:
<pre>
struct ConsoleMsg {
level: String,
timeStamp: u64,
arguments: Vec<String>,
filename: String,
lineNumber: u32,
columnNumber: u32,
}

fn handle_console_message(actors: Arc<Mutex<ActorRegistry>>,
id: PipelineId,
console_message: ConsoleMessage,
actor_pipelines: &HashMap<PipelineId, String>) {
let console_actor_name = find_console_actor(actors.clone(), id, actor_pipelines);
let actors = actors.lock().unwrap();
let console_actor = actors.find::<ConsoleActor>(console_actor_name.as_slice());
match console_message {
ConsoleMessage::LogMessage(message, filename, lineNumber, columnNumber) => {
let msg = ConsoleAPICall {
from: console_actor.name.clone(),
__type__: "consoleAPICall".to_string(),
message: ConsoleMsg {
level: "log".to_string(),
timeStamp: precise_time_ns(),
arguments: vec!(message),
filename: filename,
lineNumber: lineNumber,
columnNumber: columnNumber,
},
};
for stream in console_actor.streams.borrow_mut().iter_mut() {
stream.write_json_packet(&msg);
}
}
}
}
</pre>

components/devtools_traits/lib.rs
<pre>
pub enum ConsoleMessage {
// Log: message, filename, line number, column number
LogMessage(String, String, u32, u32),
//WarnMessage(String),
}
</pre>

components/script/dom/console.rs
<pre>
impl<'a> ConsoleMethods for JSRef<'a, Console> {
fn Log(self, message: DOMString) {
println!("{}", message);
//TODO: Sending fake values for filename, lineNumber and columnNumber in LogMessage; adjust later
propagate_console_msg(&self, ConsoleMessage::LogMessage(message, String::from_str("test"), 1, 1));
}
.
.
.
</pre>

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-23T23:55:41Z

Amohanr: /* Remote Developer Tools */

Extending Developer Tools for Servo

Servo<ref>https://github.com/servo/servo/</ref> is a prototype web browser engine written in the [https://github.com/rust-lang/rust RUST] language. Servo implements a very basic developer tools server that currently supports executing JS remotely and investigating the DOM tree in the document inspector. We want to expand these capabilities by completing previous work that enables remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo.

__TOC__
=='''Introduction'''==
Servo is a prototype web browser engine and is currently developed on 64bit OS X, 64bit Linux, and Android.It is developed using the RUST<ref>http://doc.rust-lang.org/book/README.html</ref> language.

=='''Running Servo'''==
===Installation===
Debian-based Linuxes
<pre>
sudo apt-get install curl freeglut3-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
msttcorefonts gperf g++ cmake python-virtualenv \
libssl-dev libbz2-dev libosmesa6-dev
</pre>

Fedora
<pre>
sudo yum install curl freeglut-devel libtool gcc-c++ libXi-devel \
freetype-devel mesa-libGL-devel glib2-devel libX11-devel libXrandr-devel gperf \
fontconfig-devel cabextract ttmkfdir python python-virtualenv expat-devel \
rpm-build openssl-devel cmake bzip2-devel libXcursor-devel
pushd /tmp
wget http://corefonts.sourceforge.net/msttcorefonts-2.5-1.spec
rpmbuild -bb msttcorefonts-2.5-1.spec
sudo yum install $HOME/rpmbuild/RPMS/noarch/msttcorefonts-2.5-1.noarch.rpm
popd
</pre>

===Building===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.
Normal Build
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build
./mach run tests/html/about-mozilla.html
</pre>

===Running===
<pre>
./mach run [url]
</pre>

=='''Remote Developer Tools'''==
You can use remote developer tools, such as the one in Firefox to debug Web sites and Web apps running on different browsers. The other browser could either be on the same device or a different device.
Servo has a basic implementation of developer tools that supports executing JS remotely and investigating the DOM tree in the document inspector. As part of the project for extending the funcionalities to enable remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo, we performed some initial additions to the code to get the web console to print out log messages.

The following changes were made to the ConsoleMsg struct in components/devtools/libs.rs:
*rename logLevel to level and make it a String
*rename timestamp to timeStamp
*add an arguments vector of Strings
*remove the message field and add the value to the arguments vector
*add filename, lineNumber, and columnNumber fields

The following code snippets show the modified version:

components/devtools/lib.rs:
<pre>
struct ConsoleMsg {
level: String,
timeStamp: u64,
arguments: Vec<String>,
filename: String,
lineNumber: u32,
columnNumber: u32,
}

fn handle_console_message(actors: Arc<Mutex<ActorRegistry>>,
id: PipelineId,
console_message: ConsoleMessage,
actor_pipelines: &HashMap<PipelineId, String>) {
let console_actor_name = find_console_actor(actors.clone(), id, actor_pipelines);
let actors = actors.lock().unwrap();
let console_actor = actors.find::<ConsoleActor>(console_actor_name.as_slice());
match console_message {
ConsoleMessage::LogMessage(message, filename, lineNumber, columnNumber) => {
let msg = ConsoleAPICall {
from: console_actor.name.clone(),
__type__: "consoleAPICall".to_string(),
message: ConsoleMsg {
level: "log".to_string(),
timeStamp: precise_time_ns(),
arguments: vec!(message),
filename: filename,
lineNumber: lineNumber,
columnNumber: columnNumber,
},
};
for stream in console_actor.streams.borrow_mut().iter_mut() {
stream.write_json_packet(&msg);
}
}
}
}
</pre>

components/devtools_traits/lib.rs
<pre>
pub enum ConsoleMessage {
// Log: message, filename, line number, column number
LogMessage(String, String, u32, u32),
//WarnMessage(String),
}
</pre>

components/script/dom/console.rs
<pre>
impl<'a> ConsoleMethods for JSRef<'a, Console> {
fn Log(self, message: DOMString) {
println!("{}", message);
//TODO: Sending fake values for filename, lineNumber and columnNumber in LogMessage; adjust later
propagate_console_msg(&self, ConsoleMessage::LogMessage(message, String::from_str("test"), 1, 1));
}
.
.
.
</pre>

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-23T23:53:29Z

Amohanr: /* Remote Developer Tools */

Extending Developer Tools for Servo

Servo<ref>https://github.com/servo/servo/</ref> is a prototype web browser engine written in the [https://github.com/rust-lang/rust RUST] language. Servo implements a very basic developer tools server that currently supports executing JS remotely and investigating the DOM tree in the document inspector. We want to expand these capabilities by completing previous work that enables remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo.

__TOC__
=='''Introduction'''==
Servo is a prototype web browser engine and is currently developed on 64bit OS X, 64bit Linux, and Android.It is developed using the RUST<ref>http://doc.rust-lang.org/book/README.html</ref> language.

=='''Running Servo'''==
===Installation===
Debian-based Linuxes
<pre>
sudo apt-get install curl freeglut3-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
msttcorefonts gperf g++ cmake python-virtualenv \
libssl-dev libbz2-dev libosmesa6-dev
</pre>

Fedora
<pre>
sudo yum install curl freeglut-devel libtool gcc-c++ libXi-devel \
freetype-devel mesa-libGL-devel glib2-devel libX11-devel libXrandr-devel gperf \
fontconfig-devel cabextract ttmkfdir python python-virtualenv expat-devel \
rpm-build openssl-devel cmake bzip2-devel libXcursor-devel
pushd /tmp
wget http://corefonts.sourceforge.net/msttcorefonts-2.5-1.spec
rpmbuild -bb msttcorefonts-2.5-1.spec
sudo yum install $HOME/rpmbuild/RPMS/noarch/msttcorefonts-2.5-1.noarch.rpm
popd
</pre>

===Building===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.
Normal Build
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build
./mach run tests/html/about-mozilla.html
</pre>

===Running===
<pre>
./mach run [url]
</pre>

=='''Remote Developer Tools'''==
You can use remote developer tools, such as the one in Firefox to debug Web sites and Web apps running on different browsers. The other browser could either be on the same device or a different device.
Servo has a basic implementation of developer tools that supports executing JS remotely and investigating the DOM tree in the document inspector. As part of the project for extending the funcionalities to enable remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo, we performed some initial additions to the code to get the web console to print out log messages.

The following changes were made to the ConsoleMsg struct in components/devtools/libs.rs:
*rename logLevel to level and make it a String
*rename timestamp to timeStamp
*add an arguments vector of Strings
*remove the message field and add the value to the arguments vector
*add filename, lineNumber, and columnNumber fields

The following code snippets show the modified version:

components/devtools/lib.rs:
<pre>
struct ConsoleMsg {
level: String,
timeStamp: u64,
arguments: Vec<String>,
filename: String,
lineNumber: u32,
columnNumber: u32,
}

fn handle_console_message(actors: Arc<Mutex<ActorRegistry>>,
id: PipelineId,
console_message: ConsoleMessage,
actor_pipelines: &HashMap<PipelineId, String>) {
let console_actor_name = find_console_actor(actors.clone(), id, actor_pipelines);
let actors = actors.lock().unwrap();
let console_actor = actors.find::<ConsoleActor>(console_actor_name.as_slice());
match console_message {
ConsoleMessage::LogMessage(message, filename, lineNumber, columnNumber) => {
let msg = ConsoleAPICall {
from: console_actor.name.clone(),
__type__: "consoleAPICall".to_string(),
message: ConsoleMsg {
level: "log".to_string(),
timeStamp: precise_time_ns(),
arguments: vec!(message),
filename: filename,
lineNumber: lineNumber,
columnNumber: columnNumber,
},
};
for stream in console_actor.streams.borrow_mut().iter_mut() {
stream.write_json_packet(&msg);
}
}
}
}
</pre>

components/devtools_traits/lib.rs
<pre>
pub enum ConsoleMessage {
// Log: message, filename, line number, column number
LogMessage(String, String, u32, u32),
//WarnMessage(String),
}
</pre>

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-23T23:48:01Z

Amohanr: /* Remote Developer Tools */

Extending Developer Tools for Servo

Servo<ref>https://github.com/servo/servo/</ref> is a prototype web browser engine written in the [https://github.com/rust-lang/rust RUST] language. Servo implements a very basic developer tools server that currently supports executing JS remotely and investigating the DOM tree in the document inspector. We want to expand these capabilities by completing previous work that enables remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo.

__TOC__
=='''Introduction'''==
Servo is a prototype web browser engine and is currently developed on 64bit OS X, 64bit Linux, and Android.It is developed using the RUST<ref>http://doc.rust-lang.org/book/README.html</ref> language.

=='''Running Servo'''==
===Installation===
Debian-based Linuxes
<pre>
sudo apt-get install curl freeglut3-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
msttcorefonts gperf g++ cmake python-virtualenv \
libssl-dev libbz2-dev libosmesa6-dev
</pre>

Fedora
<pre>
sudo yum install curl freeglut-devel libtool gcc-c++ libXi-devel \
freetype-devel mesa-libGL-devel glib2-devel libX11-devel libXrandr-devel gperf \
fontconfig-devel cabextract ttmkfdir python python-virtualenv expat-devel \
rpm-build openssl-devel cmake bzip2-devel libXcursor-devel
pushd /tmp
wget http://corefonts.sourceforge.net/msttcorefonts-2.5-1.spec
rpmbuild -bb msttcorefonts-2.5-1.spec
sudo yum install $HOME/rpmbuild/RPMS/noarch/msttcorefonts-2.5-1.noarch.rpm
popd
</pre>

===Building===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.
Normal Build
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build
./mach run tests/html/about-mozilla.html
</pre>

===Running===
<pre>
./mach run [url]
</pre>

=='''Remote Developer Tools'''==
You can use remote developer tools, such as the one in Firefox to debug Web sites and Web apps running on different browsers. The other browser could either be on the same device or a different device.
Servo has a basic implementation of developer tools that supports executing JS remotely and investigating the DOM tree in the document inspector. As part of the project for extending the funcionalities to enable remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo, we performed some initial additions to the code to get the web console to print out log messages.

The following changes were made to the ConsoleMsg struct in components/devtools/libs.rs:
*rename logLevel to level and make it a String
*rename timestamp to timeStamp
*add an arguments vector of Strings
*remove the message field and add the value to the arguments vector
*add filename, lineNumber, and columnNumber fields

The following code snippets show the modified version:

<pre>
components/devtools/lib.rs:

struct ConsoleMsg {
level: String,
timeStamp: u64,
arguments: Vec<String>,
filename: String,
lineNumber: u32,
columnNumber: u32,
}

fn handle_console_message(actors: Arc<Mutex<ActorRegistry>>,
id: PipelineId,
console_message: ConsoleMessage,
actor_pipelines: &HashMap<PipelineId, String>) {
let console_actor_name = find_console_actor(actors.clone(), id, actor_pipelines);
let actors = actors.lock().unwrap();
let console_actor = actors.find::<ConsoleActor>(console_actor_name.as_slice());
match console_message {
ConsoleMessage::LogMessage(message, filename, lineNumber, columnNumber) => {
let msg = ConsoleAPICall {
from: console_actor.name.clone(),
__type__: "consoleAPICall".to_string(),
message: ConsoleMsg {
level: "log".to_string(),
timeStamp: precise_time_ns(),
arguments: vec!(message),
filename: filename,
lineNumber: lineNumber,
columnNumber: columnNumber,
},
};
for stream in console_actor.streams.borrow_mut().iter_mut() {
stream.write_json_packet(&msg);
}
}
}
}

</pre>

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-23T23:08:54Z

Amohanr: /* Remote Developer Tools */

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-23T23:03:42Z

Amohanr:

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-23T22:10:32Z

Amohanr:

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-23T21:39:49Z

Amohanr:

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-22T16:54:12Z

Amohanr:

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-22T16:51:12Z

Amohanr:

CSC/ECE 517 Spring 2015/oss M1503 EDT

2015-03-22T16:50:32Z

Amohanr: Created page with "Extending Developer Tools for Servo Servo<ref>https://github.com/servo/servo</ref> is a prototype web browser engine written in the <span class="pla..."

Extending Developer Tools for Servo

Servo<ref>https://github.com/servo/servo</ref> is a prototype web browser engine written in the [https://github.com/rust-lang/rust RUST] language. Servo implements a very basic developer tools server that currently supports executing JS remotely and investigating the DOM tree in the document inspector. We want to expand these capabilities by completing previous work that enables remote logging from web content, and add new capabilities to log HTTP requests and responses to allow for easier debugging of network-related problems in Servo.

CSC/ECE 517 Spring 2015

2015-03-22T16:19:54Z

Amohanr:

==Writing Assignment 1==
*[[CSC/ECE 517 Spring 2015/ch1a 17 WL]]
*[[CSC/ECE 517 Spring 2015/ch1a 5 ZX]]
*[[CSC/ECE 517 Spring 2015/ch1a 6 TZ]]
*[[CSC/ECE 517 Spring 2015/ch1a 4 RW]]
*[[CSC/ECE 517 Spring 2015/ch1a 7 SA]]
*[[CSC/ECE 517 Spring 2015/ch1a 9 RA]]
*[[CSC/ECE 517 Spring 2015/ch1a 14 RI]]
*[[CSC/ECE 517 Spring 2015/ch1a 1 DZ]]
*[[CSC/ECE 517 Spring 2015/ch1a 20 HA]]
*[[CSC/ECE 517 Spring 2015/ch1a 3 RF]]
*[[CSC/ECE 517 Spring 2015/ch1a 12 LS]]
*[[CSC/ECE 517 Spring 2015/ch1a 13 MA]]
*[[CSC/ECE 517 Spring 2015/ch1a 2 WA]]
*[[CSC/ECE 517 Spring 2015/ch1b 21 QW]]
*[[CSC/ECE 517 Spring 2015/ch1b 23 MS]]
*[[CSC/ECE 517 Spring 2015/ch1b 10 GL]]
*[[CSC/ECE 517 Spring 2015/ch1b 27 VC]]
*[[CSC/ECE 517 Spring 2015/ch1b 22 SF]]
*[[CSC/ECE 517 Spring 2015/ch1b 15 SH]]
*[[CSC/ECE 517 Spring 2015/ch1b 18 AS]]

==Writing Assignment 2==
*[[CSC/ECE 517 Fall 2014/oss E1502 wwj]]
*[[CSC/ECE 517 Fall 2014/oss E1508 MRS]]
*[[CSC/ECE 517 Spring 2015/oss E1504 IMV]]
*[[CSC/ECE 517 Spring 2015/oss E1505 xzl]]
*[[CSC/ECE 517 Spring 2015/oss E1509 lds]]
*[[CSC/ECE 517 Spring 2015/oss E1510 FLP]]
*[[CSC/ECE 517 Spring 2015/oss E1506 SYZ]]
*[[CSC/ECE 517 Spring 2015/oss S1504 AAC]]
*[[CSC/ECE 517 Spring 2015/oss E1507 DG]]
*[[CSC/ECE 517 Spring 2015/oss M1502 GVJ]]
*[[CSC/ECE 517 Spring 2015/oss M1503 EDT]]

CSC/ECE 517 Spring 2015/ch1b 18 AS

2015-02-18T21:32:51Z

Amohanr:

Apache Solr and Rails

The topic write up for this page can be found [https://docs.google.com/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc here].
[[File: solr.jpg|right]]
Apache Solr<ref>http://lucene.apache.org/solr/</ref> is a standalone, open-source enterprise search server, written in Java and created by Yonik Seely. It is a servlet that can run within a servlet container such as Apache Tomcat. It is a very popular, fast and scaleable open source enterprise search platform built on [http://lucene.apache.org/index.html Apache Lucene] search library.

Rails is a ruby framwork used to develop web based application that incorporates the [http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controllerRails MVC] architectural pattern. Websites using [http://rubyonrails.org/ Rails] can take advantage of the Solr search engine to provide very sophisticated and customizable search features. Rails integrates with Solr search server using Sunspot<ref>https://rubygems.org/gems/sunspot_rails</ref><ref>https://github.com/sunspot/sunspot</ref> gem.

__TOC__
=='''Introduction'''==
Apache Solr is a standalone enterprise search server with a [http://en.wikipedia.org/wiki/Representational_state_transfer REST]-like API. Solr is an indexing and searching framework which could be deployed and used with many web frameworks like Rails, Drupal, Django etc. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results.

===Technology Stack<ref>http://www.slideshare.net/dkeener/rails-and-the-apache-solr-search-engine</ref>===
Solr is built on top of Apache Lucene. It is a toolbox responsible for indexing, searching, spell-check and advance tokenization whereas Solr is a search server which inherits all the features of Lucene but also adds API integration, caching and most importantly a web admin interface. This feature makes Solr very easy to use in production environment. Both of these technology needs Java 1.4 or above.
[[File: structure_solr.png|center]]
Sunspot library interacts with Solr using a low-level interface called RSolr. It is a ruby client which integrates with the Solr API's to Rails through the use of Sunspot gem. Sunspot has a drop-in ActiveRecord support. Sunspot provides a easy abstraction to using the Solr search engine. The installation and application of Solr using Sunspot gem is discussed below.

=='''Features<ref>http://lucene.apache.org/solr/features.html</ref>'''==
Apache Solr has the following features

*Enables full-text matching, powered by Lucene software
*Built to handle high volume traffic
*Provides [http://en.wikipedia.org/wiki/Faceted_search faceted] searching
*Provides features like suggester, spellcheck, clustering, auto-complete, highlighting
*Extensible through plugins
*Supports statistical and aggregate processing of text
*Supports rich format data such as PDF, Word, Powerpoint

Rails uses the Sunspot Ruby library to integrate with the Solr search engine.
=='''Installation'''==

Sunspot makes it easy to do full text searching through Solr. Sunspot comes as a gem and is installed in the usual way by adding it to the Gemfile and running bundle.

<pre>
gem 'sunspot_rails'
gem 'sunspot_solr' # optional pre-packaged Solr distribution for use in development
</pre>

<pre>
bundle install
</pre>

Once the gem and its dependencies have installed we will need to generate Sunspot’s configuration file which we can do by running

<pre>
$ rails g sunspot_rails:install
</pre>

This command creates a YML file at /config/sunspot.yml. We don’t need to make any changes to the default settings in this file.
Sunspot embeds Solr inside the gem so there’s no need to install it separately. This means that it works straight out of the box which makes it far more convenient to use in development. To get it up and running we run

<pre>
$ rake sunspot:solr:start
</pre>

If you’re running OS X Lion and you haven’t installed a Java runtime you’ll be prompted to do so when you run this command. You may also see a deprecation warning but this can be safely ignored. The command will also create some more configuration files for advanced configuration.

=='''Usage and Examples<ref>http://outoftime.github.io/sunspot/docs/index.html</ref>'''==
=== Indexing Objects ===
Add a searchable block to the objects you wish to index.

<pre>
class Example < ActiveRecord::Base
searchable do
text :title, :body
end
end
</pre>

Text fields will be full-text searchable. Other fields which are outside the scope of searchable can be used to scope queries.

=== Searching Objects ===
Now searching can be done simply by passing the query to search method in the respective class.

<pre>
Example.search do
fulltext 'query'
with :conditions
end
</pre>

We can use many variations on the search now by changing the scope variables in the query.

=== Example of a Blog Search ===
Here we want to index the text fields. So, it is kept inside the searchable block.

<pre>
class Blog < ActiveRecord::Base
searchable do
text :title, :body
text :comments do
comments.map { |comment| comment.body }
end

boolean :featured
integer :blog_id
integer :author_id
integer :category_ids, :multiple => true
double :average_rating
time :published_at
time :expired_at

string :sort_title do
title.downcase.gsub(/^(an?|the)/, '')
end
end
end
</pre>

Now for searching the indexed objects we can use:
<pre>
Blog.search do
fulltext 'best author'

with :blog_id, 1
with(:published_at).less_than Time.now
order_by :published_at, :desc
paginate :page => 2, :per_page => 15
facet :category_ids, :author_id
end
</pre>

Here the text fields are full text searchable and other attributes like blog_id, page, author_id is used to scope the query.

=='''Extensions to Sunspot'''==
Though Sunspot is used primarily for indexing and searching, it could be further extended to support multiple features. Some of them are listed as:

=== [http://en.wikipedia.org/wiki/Scope_%28computer_science%29 Scoping] scalar fields===
We can put Positive and negative restrictions along with conjunctions or disjunctions to scope the query.

*Positive restrictions
<pre>
#Posts with a category of 1, 3, or 5
Blog.search do
with(:category_ids, [1, 3, 5])
end
</pre>

*Negative restrictions
<pre>
# Blogs not in category 1 or 3
Blog.search do
without(:category_ids, [1, 3])
end
</pre>

*Disjunctions and Conjunctions
<pre>
# Blogs that do not have an expired time or have not yet expired
Blog.search do
any_of do
with(:expired_at).greater_than(Time.now)
with(:expired_at, nil)
end
end
</pre>

=== [http://en.wikipedia.org/wiki/Pagination Pagination]===
The search results given by Sunspot are paginated upto 30 items per page.
A custom number of results per page can be specified with the :per_page option to paginate:

<pre>
search = Blog.search do
fulltext "pizza"
paginate :page => 1, :per_page => 50
end
</pre>

===Faceting===

It is a feature of Solr that determines the number of documents that match a given search and an additional criterion. This allows you to build powerful drill-down interfaces for search.
In field facets each row represents a particular value for a given field. In query facets, each row represents an arbitrary scope.

<pre>
# Blogs that match 'war' returning counts for each :author_id
search = Blog.search do
fulltext "war"
facet :author_id
end

search.facet(:author_id).rows.each do |facet|
puts "Author #{facet.value} has #{facet.count} war article!"
end
</pre>

===Ordering===
By default, Sunspot orders results by "score": the Solr-determined relevancy metric. We can use order_by method to customize the search.

<pre>
# Order by average rating, descending
Blog.search do
fulltext("war")
order_by(:average_rating, :desc)
end
</pre>

===Highlighting===
It is the snippet of the matched part of the search result. It has to be stored in order to be produced.
<pre>
search = Blog.search do
fulltext "war" do
highlight :body
end
end
</pre>

It will highlight the word war in every result string.

===Hits vs Results===

Sunspot simply stores the type and primary key of objects in Solr. When results are retrieved, those primary keys are used to load the actual object like any relational database.

<pre>
# Using #results pulls in the records from the object-relational mapper
Blog.search.results.each do |result|
puts result.body
end
</pre>

To get results without accessing the database, use hits:
<pre>
Blog.search.hits.each do |hit|
puts hit.stored(:body)
end
</pre>

===Reindexing===
Objects are automatically indexed to Solr as a part of the save callbacks but if there is a change in schema then reindexing is necessary.

<pre>
bundle exec rake sunspot:solr:reindex
</pre>

<ref>http://tech.favoritemedium.com/2010/01/full-text-search-in-rails-with-sunspot.html</ref><ref>http://www.linux-mag.com/id/7341/</ref>Other interesting reads on the topic and referral links are mentioned below.

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/ch1b 18 AS

2015-02-18T20:22:05Z

Amohanr:

Apache Solr and Rails

The topic write up for this page can be found [https://docs.google.com/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc here].
[[File: solr.jpg|right]]
Apache Solr<ref>http://lucene.apache.org/solr/</ref> is a standalone, open-source enterprise search server, written in Java and created by Yonik Seely. It is a servlet that can run within a servlet container such as Apache Tomcat. It has a [http://en.wikipedia.org/wiki/Representational_state_transfer REST]-like API. Solr is an indexing and searching framework which could be deployed and used with many web frameworks like Rails, Drupal, Django etc. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results. It is a very popular, fast and scaleable open source enterprise search platform built on [http://lucene.apache.org/index.html Apache Lucene] search library.

Rails is a ruby framwork used to develop web based application that incorporates the [http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controllerRails MVC] architectural pattern. Websites using [http://rubyonrails.org/ Rails] can take advantage of the Solr search engine to provide very sophisticated and customizable search features. Rails integrates with Solr search server using Sunspot<ref>https://rubygems.org/gems/sunspot_rails</ref><ref>https://github.com/sunspot/sunspot</ref> gem.

__TOC__
=='''Introduction'''==
Apache Solr is a standalone enterprise search server with a REST-like API. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results. It is a popular, scalable, blazing-fast, open source enterprise search platform built on Apache Lucene. Websites using rails can take advantage of the Solr search engine to provide sophisticated and customizable search features.

===Technology Stack<ref>http://www.slideshare.net/dkeener/rails-and-the-apache-solr-search-engine</ref>===
Solr is built on top of Apache Lucene. It is a toolbox responsible for indexing, searching, spell-check and advance tokenization whereas Solr is a search server which inherits all the features of Lucene but also adds API integration, caching and most importantly a web admin interface. This feature makes Solr very easy to use in production environment. Both of these technology needs Java 1.4 or above.
[[File: structure_solr.png|center]]
Sunspot library interacts with Solr using a low-level interface called RSolr. It is a ruby client which integrates with the Solr API's to Rails through the use of Sunspot gem. Sunspot has a drop-in ActiveRecord support. Sunspot provides a easy abstraction to using the Solr search engine. The installation and application of Solr using Sunspot gem is discussed below.

=='''Features<ref>http://lucene.apache.org/solr/features.html</ref>'''==
Apache Solr has the following features

*Enables full-text matching, powered by Lucene software
*Built to handle high volume traffic
*Provides [http://en.wikipedia.org/wiki/Faceted_search faceted] searching
*Provides features like suggester, spellcheck, clustering, auto-complete, highlighting
*Extensible through plugins
*Supports statistical and aggregate processing of text
*Supports rich format data such as PDF, Word, Powerpoint

Rails uses the Sunspot Ruby library to integrate with the Solr search engine.
=='''Installation'''==

Sunspot makes it easy to do full text searching through Solr. Sunspot comes as a gem and is installed in the usual way by adding it to the Gemfile and running bundle.

<pre>
gem 'sunspot_rails'
gem 'sunspot_solr' # optional pre-packaged Solr distribution for use in development
</pre>

<pre>
bundle install
</pre>

Once the gem and its dependencies have installed we will need to generate Sunspot’s configuration file which we can do by running

<pre>
$ rails g sunspot_rails:install
</pre>

This command creates a YML file at /config/sunspot.yml. We don’t need to make any changes to the default settings in this file.
Sunspot embeds Solr inside the gem so there’s no need to install it separately. This means that it works straight out of the box which makes it far more convenient to use in development. To get it up and running we run

<pre>
$ rake sunspot:solr:start
</pre>

If you’re running OS X Lion and you haven’t installed a Java runtime you’ll be prompted to do so when you run this command. You may also see a deprecation warning but this can be safely ignored. The command will also create some more configuration files for advanced configuration.

=='''Usage and Examples<ref>http://outoftime.github.io/sunspot/docs/index.html</ref>'''==
=== Indexing Objects ===
Add a searchable block to the objects you wish to index.

<pre>
class Example < ActiveRecord::Base
searchable do
text :title, :body
end
end
</pre>

Text fields will be full-text searchable. Other fields which are outside the scope of searchable can be used to scope queries.

=== Searching Objects ===
Now searching can be done simply by passing the query to search method in the respective class.

<pre>
Example.search do
fulltext 'query'
with :conditions
end
</pre>

We can use many variations on the search now by changing the scope variables in the query.

=== Example of a Blog Search ===
Here we want to index the text fields. So, it is kept inside the searchable block.

<pre>
class Blog < ActiveRecord::Base
searchable do
text :title, :body
text :comments do
comments.map { |comment| comment.body }
end

boolean :featured
integer :blog_id
integer :author_id
integer :category_ids, :multiple => true
double :average_rating
time :published_at
time :expired_at

string :sort_title do
title.downcase.gsub(/^(an?|the)/, '')
end
end
end
</pre>

Now for searching the indexed objects we can use:
<pre>
Blog.search do
fulltext 'best author'

with :blog_id, 1
with(:published_at).less_than Time.now
order_by :published_at, :desc
paginate :page => 2, :per_page => 15
facet :category_ids, :author_id
end
</pre>

Here the text fields are full text searchable and other attributes like blog_id, page, author_id is used to scope the query.

=='''Extensions to Sunspot'''==
Though Sunspot is used primarily for indexing and searching, it could be further extended to support multiple features. Some of them are listed as:

=== [http://en.wikipedia.org/wiki/Scope_%28computer_science%29 Scoping] scalar fields===
We can put Positive and negative restrictions along with conjunctions or disjunctions to scope the query.

*Positive restrictions
<pre>
#Posts with a category of 1, 3, or 5
Blog.search do
with(:category_ids, [1, 3, 5])
end
</pre>

*Negative restrictions
<pre>
# Blogs not in category 1 or 3
Blog.search do
without(:category_ids, [1, 3])
end
</pre>

*Disjunctions and Conjunctions
<pre>
# Blogs that do not have an expired time or have not yet expired
Blog.search do
any_of do
with(:expired_at).greater_than(Time.now)
with(:expired_at, nil)
end
end
</pre>

=== [http://en.wikipedia.org/wiki/Pagination Pagination]===
The search results given by Sunspot are paginated upto 30 items per page.
A custom number of results per page can be specified with the :per_page option to paginate:

<pre>
search = Blog.search do
fulltext "pizza"
paginate :page => 1, :per_page => 50
end
</pre>

===Faceting===

It is a feature of Solr that determines the number of documents that match a given search and an additional criterion. This allows you to build powerful drill-down interfaces for search.
In field facets each row represents a particular value for a given field. In query facets, each row represents an arbitrary scope.

<pre>
# Blogs that match 'war' returning counts for each :author_id
search = Blog.search do
fulltext "war"
facet :author_id
end

search.facet(:author_id).rows.each do |facet|
puts "Author #{facet.value} has #{facet.count} war article!"
end
</pre>

===Ordering===
By default, Sunspot orders results by "score": the Solr-determined relevancy metric. We can use order_by method to customize the search.

<pre>
# Order by average rating, descending
Blog.search do
fulltext("war")
order_by(:average_rating, :desc)
end
</pre>

===Highlighting===
It is the snippet of the matched part of the search result. It has to be stored in order to be produced.
<pre>
search = Blog.search do
fulltext "war" do
highlight :body
end
end
</pre>

It will highlight the word war in every result string.

===Hits vs Results===

Sunspot simply stores the type and primary key of objects in Solr. When results are retrieved, those primary keys are used to load the actual object like any relational database.

<pre>
# Using #results pulls in the records from the object-relational mapper
Blog.search.results.each do |result|
puts result.body
end
</pre>

To get results without accessing the database, use hits:
<pre>
Blog.search.hits.each do |hit|
puts hit.stored(:body)
end
</pre>

===Reindexing===
Objects are automatically indexed to Solr as a part of the save callbacks but if there is a change in schema then reindexing is necessary.

<pre>
bundle exec rake sunspot:solr:reindex
</pre>

<ref>http://tech.favoritemedium.com/2010/01/full-text-search-in-rails-with-sunspot.html</ref><ref>http://www.linux-mag.com/id/7341/</ref>Other interesting reads on the topic and referral links are mentioned below.

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/ch1b 18 AS

2015-02-18T19:40:20Z

Amohanr:

Apache Solr and Rails

The topic write up for this page can be found [https://docs.google.com/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc here].
[[File: solr.jpg|right]]
Apache Solr<ref>http://lucene.apache.org/solr/</ref> is a standalone, open-source enterprise search server, written in Java and created by Yonik Seely. It is a servlet that can run within a servlet container such as Apache Tomcat. It has a [http://en.wikipedia.org/wiki/Representational_state_transfer REST]-like API. Solr is an indexing and searching framework which could be deployed and used with many web frameworks like Rails, Drupal, Django etc. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results. It is a very popular, fast and scaleable open source enterprise search platform built on [http://lucene.apache.org/index.html Apache Lucene] search library. Websites using [http://rubyonrails.org/ Rails] can take advantage of the Solr search engine to provide very sophisticated and customizable search features. Rails integrates with Solr search server using Sunspot<ref>https://rubygems.org/gems/sunspot_rails</ref><ref>https://github.com/sunspot/sunspot</ref> gem.

__TOC__
=='''Introduction'''==
Apache Solr is a standalone enterprise search server with a REST-like API. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results. It is a popular, scalable, blazing-fast, open source enterprise search platform built on Apache Lucene. Websites using rails can take advantage of the Solr search engine to provide sophisticated and customizable search features.

===Technology Stack<ref>http://www.slideshare.net/dkeener/rails-and-the-apache-solr-search-engine</ref>===
Solr is built on top of Apache Lucene. It is a toolbox responsible for indexing, searching, spell-check and advance tokenization whereas Solr is a search server which inherits all the features of Lucene but also adds API integration, caching and most importantly a web admin interface. This feature makes Solr very easy to use in production environment. Both of these technology needs Java 1.4 or above.
[[File: structure_solr.png|center]]
Sunspot library interacts with Solr using a low-level interface called RSolr. It is a ruby client which integrates with the Solr API's to Rails through the use of Sunspot gem. Sunspot has a drop-in ActiveRecord support. Sunspot provides a easy abstraction to using the Solr search engine. The installation and application of Solr using Sunspot gem is discussed below.

=='''Features<ref>http://lucene.apache.org/solr/features.html</ref>'''==
Apache Solr has the following features

*Enables full-text matching, powered by Lucene software
*Built to handle high volume traffic
*Provides [http://en.wikipedia.org/wiki/Faceted_search faceted] searching
*Provides features like suggester, spellcheck, clustering, auto-complete, highlighting
*Extensible through plugins
*Supports statistical and aggregate processing of text
*Supports rich format data such as PDF, Word, Powerpoint

Rails uses the Sunspot Ruby library to integrate with the Solr search engine.
=='''Installation'''==

Sunspot makes it easy to do full text searching through Solr. Sunspot comes as a gem and is installed in the usual way by adding it to the Gemfile and running bundle.

<pre>
gem 'sunspot_rails'
gem 'sunspot_solr' # optional pre-packaged Solr distribution for use in development
</pre>

<pre>
bundle install
</pre>

Once the gem and its dependencies have installed we will need to generate Sunspot’s configuration file which we can do by running

<pre>
$ rails g sunspot_rails:install
</pre>

This command creates a YML file at /config/sunspot.yml. We don’t need to make any changes to the default settings in this file.
Sunspot embeds Solr inside the gem so there’s no need to install it separately. This means that it works straight out of the box which makes it far more convenient to use in development. To get it up and running we run

<pre>
$ rake sunspot:solr:start
</pre>

If you’re running OS X Lion and you haven’t installed a Java runtime you’ll be prompted to do so when you run this command. You may also see a deprecation warning but this can be safely ignored. The command will also create some more configuration files for advanced configuration.

=='''Usage and Examples<ref>http://outoftime.github.io/sunspot/docs/index.html</ref>'''==
=== Indexing Objects ===
Add a searchable block to the objects you wish to index.

<pre>
class Example < ActiveRecord::Base
searchable do
text :title, :body
end
end
</pre>

Text fields will be full-text searchable. Other fields which are outside the scope of searchable can be used to scope queries.

=== Searching Objects ===
Now searching can be done simply by passing the query to search method in the respective class.

<pre>
Example.search do
fulltext 'query'
with :conditions
end
</pre>

We can use many variations on the search now by changing the scope variables in the query.

=== Example of a Blog Search ===
Here we want to index the text fields. So, it is kept inside the searchable block.

<pre>
class Blog < ActiveRecord::Base
searchable do
text :title, :body
text :comments do
comments.map { |comment| comment.body }
end

boolean :featured
integer :blog_id
integer :author_id
integer :category_ids, :multiple => true
double :average_rating
time :published_at
time :expired_at

string :sort_title do
title.downcase.gsub(/^(an?|the)/, '')
end
end
end
</pre>

Now for searching the indexed objects we can use:
<pre>
Blog.search do
fulltext 'best author'

with :blog_id, 1
with(:published_at).less_than Time.now
order_by :published_at, :desc
paginate :page => 2, :per_page => 15
facet :category_ids, :author_id
end
</pre>

Here the text fields are full text searchable and other attributes like blog_id, page, author_id is used to scope the query.

=='''Extensions to Sunspot'''==
Though Sunspot is used primarily for indexing and searching, it could be further extended to support multiple features. Some of them are listed as:

=== [http://en.wikipedia.org/wiki/Scope_%28computer_science%29 Scoping] scalar fields===
We can put Positive and negative restrictions along with conjunctions or disjunctions to scope the query.

*Positive restrictions
<pre>
#Posts with a category of 1, 3, or 5
Blog.search do
with(:category_ids, [1, 3, 5])
end
</pre>

*Negative restrictions
<pre>
# Blogs not in category 1 or 3
Blog.search do
without(:category_ids, [1, 3])
end
</pre>

*Disjunctions and Conjunctions
<pre>
# Blogs that do not have an expired time or have not yet expired
Blog.search do
any_of do
with(:expired_at).greater_than(Time.now)
with(:expired_at, nil)
end
end
</pre>

=== [http://en.wikipedia.org/wiki/Pagination Pagination]===
The search results given by Sunspot are paginated upto 30 items per page.
A custom number of results per page can be specified with the :per_page option to paginate:

<pre>
search = Blog.search do
fulltext "pizza"
paginate :page => 1, :per_page => 50
end
</pre>

===Faceting===

It is a feature of Solr that determines the number of documents that match a given search and an additional criterion. This allows you to build powerful drill-down interfaces for search.
In field facets each row represents a particular value for a given field. In query facets, each row represents an arbitrary scope.

<pre>
# Blogs that match 'war' returning counts for each :author_id
search = Blog.search do
fulltext "war"
facet :author_id
end

search.facet(:author_id).rows.each do |facet|
puts "Author #{facet.value} has #{facet.count} war article!"
end
</pre>

===Ordering===
By default, Sunspot orders results by "score": the Solr-determined relevancy metric. We can use order_by method to customize the search.

<pre>
# Order by average rating, descending
Blog.search do
fulltext("war")
order_by(:average_rating, :desc)
end
</pre>

===Highlighting===
It is the snippet of the matched part of the search result. It has to be stored in order to be produced.
<pre>
search = Blog.search do
fulltext "war" do
highlight :body
end
end
</pre>

It will highlight the word war in every result string.

===Hits vs Results===

Sunspot simply stores the type and primary key of objects in Solr. When results are retrieved, those primary keys are used to load the actual object like any relational database.

<pre>
# Using #results pulls in the records from the object-relational mapper
Blog.search.results.each do |result|
puts result.body
end
</pre>

To get results without accessing the database, use hits:
<pre>
Blog.search.hits.each do |hit|
puts hit.stored(:body)
end
</pre>

===Reindexing===
Objects are automatically indexed to Solr as a part of the save callbacks but if there is a change in schema then reindexing is necessary.

<pre>
bundle exec rake sunspot:solr:reindex
</pre>

<ref>http://tech.favoritemedium.com/2010/01/full-text-search-in-rails-with-sunspot.html</ref><ref>http://www.linux-mag.com/id/7341/</ref>Other interesting reads on the topic and referral links are mentioned below.

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/ch1b 18 AS

2015-02-18T18:55:23Z

Amohanr:

Apache Solr and Rails

The topic write up for this page can be found [https://docs.google.com/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc here].
[[File: solr.jpg|right]]
Apache Solr<ref>http://lucene.apache.org/solr/</ref> is a standalone, open-source enterprise search server, written in Java and created by Yonik Seely. It has a [http://en.wikipedia.org/wiki/Representational_state_transfer REST]-like API. It is an indexing and searching framework which could be deployed and used with many web frameworks like Rails, Drupal, Django etc. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results. It is a very popular, fast and scaleable open source enterprise search platform built on [http://lucene.apache.org/index.html Apache Lucene] search library. Websites using [http://rubyonrails.org/ Rails] can take advantage of the Solr search engine to provide very sophisticated and customizable search features. Rails integrates with Solr search server using Sunspot<ref>https://rubygems.org/gems/sunspot_rails</ref><ref>https://github.com/sunspot/sunspot</ref> gem.

__TOC__
=='''Introduction'''==
Apache Solr is a standalone enterprise search server with a REST-like API. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results. It is a popular, scalable, blazing-fast, open source enterprise search platform built on Apache Lucene. Websites using rails can take advantage of the Solr search engine to provide sophisticated and customizable search features.

===Technology Stack<ref>http://www.slideshare.net/dkeener/rails-and-the-apache-solr-search-engine</ref>===
Solr is built on top of Apache Lucene. It is a toolbox responsible for indexing, searching, spell-check and advance tokenization whereas Solr is a search server which inherits all the features of Lucene but also adds API integration, caching and most importantly a web admin interface. This feature makes Solr very easy to use in production environment. Both of these technology needs Java 1.4 or above.
[[File: structure_solr.png|center]]
Sunspot library interacts with Solr using a low-level interface called RSolr. It is a ruby client which integrates with the Solr API's to Rails through the use of Sunspot gem. Sunspot has a drop-in ActiveRecord support. Sunspot provides a easy abstraction to using the Solr search engine. The installation and application of Solr using Sunspot gem is discussed below.

=='''Features<ref>http://lucene.apache.org/solr/features.html</ref>'''==
Apache Solr has the following features

*Enables full-text matching, powered by Lucene software
*Built to handle high volume traffic
*Provides [http://en.wikipedia.org/wiki/Faceted_search faceted] searching
*Provides features like suggester, spellcheck, clustering, auto-complete, highlighting
*Extensible through plugins
*Supports statistical and aggregate processing of text
*Supports rich format data such as PDF, Word, Powerpoint

Rails uses the Sunspot Ruby library to integrate with the Solr search engine.
=='''Installation'''==

Sunspot makes it easy to do full text searching through Solr. Sunspot comes as a gem and is installed in the usual way by adding it to the Gemfile and running bundle.

<pre>
gem 'sunspot_rails'
gem 'sunspot_solr' # optional pre-packaged Solr distribution for use in development
</pre>

<pre>
bundle install
</pre>

Once the gem and its dependencies have installed we will need to generate Sunspot’s configuration file which we can do by running

<pre>
$ rails g sunspot_rails:install
</pre>

This command creates a YML file at /config/sunspot.yml. We don’t need to make any changes to the default settings in this file.
Sunspot embeds Solr inside the gem so there’s no need to install it separately. This means that it works straight out of the box which makes it far more convenient to use in development. To get it up and running we run

<pre>
$ rake sunspot:solr:start
</pre>

If you’re running OS X Lion and you haven’t installed a Java runtime you’ll be prompted to do so when you run this command. You may also see a deprecation warning but this can be safely ignored. The command will also create some more configuration files for advanced configuration.

=='''Usage and Examples<ref>http://outoftime.github.io/sunspot/docs/index.html</ref>'''==
=== Indexing Objects ===
Add a searchable block to the objects you wish to index.

<pre>
class Example < ActiveRecord::Base
searchable do
text :title, :body
end
end
</pre>

Text fields will be full-text searchable. Other fields which are outside the scope of searchable can be used to scope queries.

=== Searching Objects ===
Now searching can be done simply by passing the query to search method in the respective class.

<pre>
Example.search do
fulltext 'query'
with :conditions
end
</pre>

We can use many variations on the search now by changing the scope variables in the query.

=== Example of a Blog Search ===
Here we want to index the text fields. So, it is kept inside the searchable block.

<pre>
class Blog < ActiveRecord::Base
searchable do
text :title, :body
text :comments do
comments.map { |comment| comment.body }
end

boolean :featured
integer :blog_id
integer :author_id
integer :category_ids, :multiple => true
double :average_rating
time :published_at
time :expired_at

string :sort_title do
title.downcase.gsub(/^(an?|the)/, '')
end
end
end
</pre>

Now for searching the indexed objects we can use:
<pre>
Blog.search do
fulltext 'best author'

with :blog_id, 1
with(:published_at).less_than Time.now
order_by :published_at, :desc
paginate :page => 2, :per_page => 15
facet :category_ids, :author_id
end
</pre>

Here the text fields are full text searchable and other attributes like blog_id, page, author_id is used to scope the query.

=='''Extensions to Sunspot'''==
Though Sunspot is used primarily for indexing and searching, it could be further extended to support multiple features. Some of them are listed as:

=== [http://en.wikipedia.org/wiki/Scope_%28computer_science%29 Scoping] scalar fields===
We can put Positive and negative restrictions along with conjunctions or disjunctions to scope the query.

*Positive restrictions
<pre>
#Posts with a category of 1, 3, or 5
Blog.search do
with(:category_ids, [1, 3, 5])
end
</pre>

*Negative restrictions
<pre>
# Blogs not in category 1 or 3
Blog.search do
without(:category_ids, [1, 3])
end
</pre>

*Disjunctions and Conjunctions
<pre>
# Blogs that do not have an expired time or have not yet expired
Blog.search do
any_of do
with(:expired_at).greater_than(Time.now)
with(:expired_at, nil)
end
end
</pre>

=== [http://en.wikipedia.org/wiki/Pagination Pagination]===
The search results given by Sunspot are paginated upto 30 items per page.
A custom number of results per page can be specified with the :per_page option to paginate:

<pre>
search = Blog.search do
fulltext "pizza"
paginate :page => 1, :per_page => 50
end
</pre>

===Faceting===

It is a feature of Solr that determines the number of documents that match a given search and an additional criterion. This allows you to build powerful drill-down interfaces for search.
In field facets each row represents a particular value for a given field. In query facets, each row represents an arbitrary scope.

<pre>
# Blogs that match 'war' returning counts for each :author_id
search = Blog.search do
fulltext "war"
facet :author_id
end

search.facet(:author_id).rows.each do |facet|
puts "Author #{facet.value} has #{facet.count} war article!"
end
</pre>

===Ordering===
By default, Sunspot orders results by "score": the Solr-determined relevancy metric. We can use order_by method to customize the search.

<pre>
# Order by average rating, descending
Blog.search do
fulltext("war")
order_by(:average_rating, :desc)
end
</pre>

===Highlighting===
It is the snippet of the matched part of the search result. It has to be stored in order to be produced.
<pre>
search = Blog.search do
fulltext "war" do
highlight :body
end
end
</pre>

It will highlight the word war in every result string.

===Hits vs Results===

Sunspot simply stores the type and primary key of objects in Solr. When results are retrieved, those primary keys are used to load the actual object like any relational database.

<pre>
# Using #results pulls in the records from the object-relational mapper
Blog.search.results.each do |result|
puts result.body
end
</pre>

To get results without accessing the database, use hits:
<pre>
Blog.search.hits.each do |hit|
puts hit.stored(:body)
end
</pre>

===Reindexing===
Objects are automatically indexed to Solr as a part of the save callbacks but if there is a change in schema then reindexing is necessary.

<pre>
bundle exec rake sunspot:solr:reindex
</pre>

<ref>http://tech.favoritemedium.com/2010/01/full-text-search-in-rails-with-sunspot.html</ref><ref>http://www.linux-mag.com/id/7341/</ref>Other interesting reads on the topic and referral links are mentioned below.

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/ch1b 18 AS

2015-02-18T18:52:35Z

Amohanr:

Apache Solr and Rails

The topic write up for this page can be found [https://docs.google.com/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc here].
[[File: solr.jpg|right]]
Apache Solr<ref>http://lucene.apache.org/solr/</ref> is a standalone, open-source enterprise search server created by Yonik Seely. It has a [http://en.wikipedia.org/wiki/Representational_state_transfer REST]-like API. It is an indexing and searching framework which could be deployed and used with many web frameworks like Rails, Drupal, Django etc. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results. It is a very popular, fast and scaleable open source enterprise search platform built on [http://lucene.apache.org/index.html Apache Lucene] search library. Websites using [http://rubyonrails.org/ Rails] can take advantage of the Solr search engine to provide very sophisticated and customizable search features. Rails integrates with Solr search server using Sunspot<ref>https://rubygems.org/gems/sunspot_rails</ref><ref>https://github.com/sunspot/sunspot</ref> gem.

__TOC__
=='''Introduction'''==
Apache Solr is a standalone enterprise search server with a REST-like API. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results. It is a popular, scalable, blazing-fast, open source enterprise search platform built on Apache Lucene. Websites using rails can take advantage of the Solr search engine to provide sophisticated and customizable search features.

===Technology Stack<ref>http://www.slideshare.net/dkeener/rails-and-the-apache-solr-search-engine</ref>===
Solr is built on top of Apache Lucene. It is a toolbox responsible for indexing, searching, spell-check and advance tokenization whereas Solr is a search server which inherits all the features of Lucene but also adds API integration, caching and most importantly a web admin interface. This feature makes Solr very easy to use in production environment. Both of these technology needs Java 1.4 or above.
[[File: structure_solr.png|center]]
Sunspot library interacts with Solr using a low-level interface called RSolr. It is a ruby client which integrates with the Solr API's to Rails through the use of Sunspot gem. Sunspot has a drop-in ActiveRecord support. Sunspot provides a easy abstraction to using the Solr search engine. The installation and application of Solr using Sunspot gem is discussed below.

=='''Features<ref>http://lucene.apache.org/solr/features.html</ref>'''==
Apache Solr has the following features

*Enables full-text matching, powered by Lucene software
*Built to handle high volume traffic
*Provides [http://en.wikipedia.org/wiki/Faceted_search faceted] searching
*Provides features like suggester, spellcheck, clustering, auto-complete, highlighting
*Extensible through plugins
*Supports statistical and aggregate processing of text
*Supports rich format data such as PDF, Word, Powerpoint

Rails uses the Sunspot Ruby library to integrate with the Solr search engine.
=='''Installation'''==

Sunspot makes it easy to do full text searching through Solr. Sunspot comes as a gem and is installed in the usual way by adding it to the Gemfile and running bundle.

<pre>
gem 'sunspot_rails'
gem 'sunspot_solr' # optional pre-packaged Solr distribution for use in development
</pre>

<pre>
bundle install
</pre>

Once the gem and its dependencies have installed we will need to generate Sunspot’s configuration file which we can do by running

<pre>
$ rails g sunspot_rails:install
</pre>

This command creates a YML file at /config/sunspot.yml. We don’t need to make any changes to the default settings in this file.
Sunspot embeds Solr inside the gem so there’s no need to install it separately. This means that it works straight out of the box which makes it far more convenient to use in development. To get it up and running we run

<pre>
$ rake sunspot:solr:start
</pre>

If you’re running OS X Lion and you haven’t installed a Java runtime you’ll be prompted to do so when you run this command. You may also see a deprecation warning but this can be safely ignored. The command will also create some more configuration files for advanced configuration.

=='''Usage and Examples<ref>http://outoftime.github.io/sunspot/docs/index.html</ref>'''==
=== Indexing Objects ===
Add a searchable block to the objects you wish to index.

<pre>
class Example < ActiveRecord::Base
searchable do
text :title, :body
end
end
</pre>

Text fields will be full-text searchable. Other fields which are outside the scope of searchable can be used to scope queries.

=== Searching Objects ===
Now searching can be done simply by passing the query to search method in the respective class.

<pre>
Example.search do
fulltext 'query'
with :conditions
end
</pre>

We can use many variations on the search now by changing the scope variables in the query.

=== Example of a Blog Search ===
Here we want to index the text fields. So, it is kept inside the searchable block.

<pre>
class Blog < ActiveRecord::Base
searchable do
text :title, :body
text :comments do
comments.map { |comment| comment.body }
end

boolean :featured
integer :blog_id
integer :author_id
integer :category_ids, :multiple => true
double :average_rating
time :published_at
time :expired_at

string :sort_title do
title.downcase.gsub(/^(an?|the)/, '')
end
end
end
</pre>

Now for searching the indexed objects we can use:
<pre>
Blog.search do
fulltext 'best author'

with :blog_id, 1
with(:published_at).less_than Time.now
order_by :published_at, :desc
paginate :page => 2, :per_page => 15
facet :category_ids, :author_id
end
</pre>

Here the text fields are full text searchable and other attributes like blog_id, page, author_id is used to scope the query.

=='''Extensions to Sunspot'''==
Though Sunspot is used primarily for indexing and searching, it could be further extended to support multiple features. Some of them are listed as:

=== [http://en.wikipedia.org/wiki/Scope_%28computer_science%29 Scoping] scalar fields===
We can put Positive and negative restrictions along with conjunctions or disjunctions to scope the query.

*Positive restrictions
<pre>
#Posts with a category of 1, 3, or 5
Blog.search do
with(:category_ids, [1, 3, 5])
end
</pre>

*Negative restrictions
<pre>
# Blogs not in category 1 or 3
Blog.search do
without(:category_ids, [1, 3])
end
</pre>

*Disjunctions and Conjunctions
<pre>
# Blogs that do not have an expired time or have not yet expired
Blog.search do
any_of do
with(:expired_at).greater_than(Time.now)
with(:expired_at, nil)
end
end
</pre>

=== [http://en.wikipedia.org/wiki/Pagination Pagination]===
The search results given by Sunspot are paginated upto 30 items per page.
A custom number of results per page can be specified with the :per_page option to paginate:

<pre>
search = Blog.search do
fulltext "pizza"
paginate :page => 1, :per_page => 50
end
</pre>

===Faceting===

It is a feature of Solr that determines the number of documents that match a given search and an additional criterion. This allows you to build powerful drill-down interfaces for search.
In field facets each row represents a particular value for a given field. In query facets, each row represents an arbitrary scope.

<pre>
# Blogs that match 'war' returning counts for each :author_id
search = Blog.search do
fulltext "war"
facet :author_id
end

search.facet(:author_id).rows.each do |facet|
puts "Author #{facet.value} has #{facet.count} war article!"
end
</pre>

===Ordering===
By default, Sunspot orders results by "score": the Solr-determined relevancy metric. We can use order_by method to customize the search.

<pre>
# Order by average rating, descending
Blog.search do
fulltext("war")
order_by(:average_rating, :desc)
end
</pre>

===Highlighting===
It is the snippet of the matched part of the search result. It has to be stored in order to be produced.
<pre>
search = Blog.search do
fulltext "war" do
highlight :body
end
end
</pre>

It will highlight the word war in every result string.

===Hits vs Results===

Sunspot simply stores the type and primary key of objects in Solr. When results are retrieved, those primary keys are used to load the actual object like any relational database.

<pre>
# Using #results pulls in the records from the object-relational mapper
Blog.search.results.each do |result|
puts result.body
end
</pre>

To get results without accessing the database, use hits:
<pre>
Blog.search.hits.each do |hit|
puts hit.stored(:body)
end
</pre>

===Reindexing===
Objects are automatically indexed to Solr as a part of the save callbacks but if there is a change in schema then reindexing is necessary.

<pre>
bundle exec rake sunspot:solr:reindex
</pre>

<ref>http://tech.favoritemedium.com/2010/01/full-text-search-in-rails-with-sunspot.html</ref><ref>http://www.linux-mag.com/id/7341/</ref>Other interesting reads on the topic and referral links are mentioned below.

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/ch1b 18 AS

2015-02-18T18:31:50Z

Amohanr:

Apache Solr and Rails

The topic write up for this page can be found [https://docs.google.com/document/d/1TgBtp7flIPKJwkkShgtcIkt--mtHuwVHsQX6Tpzj1rc here].
[[File: solr.jpg|right]]
Apache Solr<ref>http://lucene.apache.org/solr/</ref> is a standalone, open-source enterprise search server created by Yonik Seely. It has a [http://en.wikipedia.org/wiki/Representational_state_transfer REST]-like API. It is a indexing and searching framework which could be deployed and used with many web frameworks like Rails, Drupal, Django etc. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results. It is a very popular, fast and scaleable open source enterprise search platform built on [http://lucene.apache.org/index.html Apache Lucene] search library. Websites using [http://rubyonrails.org/ Rails] can take advantage of the Solr search engine to provide very sophisticated and customizable search features. Rails integrates with Solr search server using Sunspot<ref>https://rubygems.org/gems/sunspot_rails</ref><ref>https://github.com/sunspot/sunspot</ref> gem.

__TOC__
=='''Introduction'''==
Apache Solr is a standalone enterprise search server with a REST-like API. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results. It is a popular, scalable, blazing-fast, open source enterprise search platform built on Apache Lucene. Websites using rails can take advantage of the Solr search engine to provide sophisticated and customizable search features.

===Technology Stack<ref>http://www.slideshare.net/dkeener/rails-and-the-apache-solr-search-engine</ref>===
Solr is built on top of Apache Lucene. It is a toolbox responsible for indexing, searching, spell-check and advance tokenization whereas Solr is a search server which inherits all the features of Lucene but also adds API integration, caching and most importantly a web admin interface. This feature makes Solr very easy to use in production environment. Both of these technology needs Java 1.4 or above.
[[File: structure_solr.png|center]]
Sunspot library interacts with Solr using a low-level interface called RSolr. It is a ruby client which integrates with the Solr API's to Rails through the use of Sunspot gem. Sunspot has a drop-in ActiveRecord support. We can call Sunspot as the high level client of Solr. We will be talking about the installation and application of Solr using Sunspot gem in the section below.

=='''Features<ref>http://lucene.apache.org/solr/features.html</ref>'''==
Apache Solr has the following features

*Enables full-text matching, powered by Lucene software
*Built to handle high volume traffic
*Provides [http://en.wikipedia.org/wiki/Faceted_search faceted] searching
*Provides features like suggester, spellcheck, clustering, auto-complete, highlighting
*Extensible through plugins
*Supports statistical and aggregate processing of text
*Supports rich format data such as PDF, Word, Powerpoint

Rails uses the Sunspot Ruby library to integrate with the Solr search engine.
=='''Installation'''==

Sunspot makes it easy to do full text searching through Solr. Sunspot comes as a gem and is installed in the usual way by adding it to the Gemfile and running bundle.

<pre>
gem 'sunspot_rails'
gem 'sunspot_solr' # optional pre-packaged Solr distribution for use in development
</pre>

<pre>
bundle install
</pre>

Once the gem and its dependencies have installed we will need to generate Sunspot’s configuration file which we can do by running

<pre>
$ rails g sunspot_rails:install
</pre>

This command creates a YML file at /config/sunspot.yml. We don’t need to make any changes to the default settings in this file.
Sunspot embeds Solr inside the gem so there’s no need to install it separately. This means that it works straight out of the box which makes it far more convenient to use in development. To get it up and running we run

<pre>
$ rake sunspot:solr:start
</pre>

If you’re running OS X Lion and you haven’t installed a Java runtime you’ll be prompted to do so when you run this command. You may also see a deprecation warning but this can be safely ignored. The command will also create some more configuration files for advanced configuration.

=='''Usage and Examples<ref>http://outoftime.github.io/sunspot/docs/index.html</ref>'''==
=== Indexing Objects ===
Add a searchable block to the objects you wish to index.

<pre>
class Example < ActiveRecord::Base
searchable do
text :title, :body
end
end
</pre>

Text fields will be full-text searchable. Other fields which are outside the scope of searchable can be used to scope queries.

=== Searching Objects ===
Now searching can be done simply by passing the query to search method in the respective class.

<pre>
Example.search do
fulltext 'query'
with :conditions
end
</pre>

We can use many variations on the search now by changing the scope variables in the query.

=== Example of a Blog Search ===
Here we want to index the text fields. So, it is kept inside the searchable block.

<pre>
class Blog < ActiveRecord::Base
searchable do
text :title, :body
text :comments do
comments.map { |comment| comment.body }
end

boolean :featured
integer :blog_id
integer :author_id
integer :category_ids, :multiple => true
double :average_rating
time :published_at
time :expired_at

string :sort_title do
title.downcase.gsub(/^(an?|the)/, '')
end
end
end
</pre>

Now for searching the indexed objects we can use:
<pre>
Blog.search do
fulltext 'best author'

with :blog_id, 1
with(:published_at).less_than Time.now
order_by :published_at, :desc
paginate :page => 2, :per_page => 15
facet :category_ids, :author_id
end
</pre>

Here the text fields are full text searchable and other attributes like blog_id, page, author_id is used to scope the query.

=='''Extensions to Sunspot'''==
Though Sunspot is used primarily for indexing and searching, it could be further extended to support multiple features. Some of them are listed as:

=== [http://en.wikipedia.org/wiki/Scope_%28computer_science%29 Scoping] scalar fields===
We can put Positive and negative restrictions along with conjunctions or disjunctions to scope the query.

*Positive restrictions
<pre>
#Posts with a category of 1, 3, or 5
Blog.search do
with(:category_ids, [1, 3, 5])
end
</pre>

*Negative restrictions
<pre>
# Blogs not in category 1 or 3
Blog.search do
without(:category_ids, [1, 3])
end
</pre>

*Disjunctions and Conjunctions
<pre>
# Blogs that do not have an expired time or have not yet expired
Blog.search do
any_of do
with(:expired_at).greater_than(Time.now)
with(:expired_at, nil)
end
end
</pre>

=== [http://en.wikipedia.org/wiki/Pagination Pagination]===
The search results given by Sunspot are paginated upto 30 items per page.
A custom number of results per page can be specified with the :per_page option to paginate:

<pre>
search = Blog.search do
fulltext "pizza"
paginate :page => 1, :per_page => 50
end
</pre>

===Faceting===

It is a feature of Solr that determines the number of documents that match a given search and an additional criterion. This allows you to build powerful drill-down interfaces for search.
In field facets each row represents a particular value for a given field. In query facets, each row represents an arbitrary scope.

<pre>
# Blogs that match 'war' returning counts for each :author_id
search = Blog.search do
fulltext "war"
facet :author_id
end

search.facet(:author_id).rows.each do |facet|
puts "Author #{facet.value} has #{facet.count} war article!"
end
</pre>

===Ordering===
By default, Sunspot orders results by "score": the Solr-determined relevancy metric. We can use order_by method to customize the search.

<pre>
# Order by average rating, descending
Blog.search do
fulltext("war")
order_by(:average_rating, :desc)
end
</pre>

===Highlighting===
It is the snippet of the matched part of the search result. It has to be stored in order to be produced.
<pre>
search = Blog.search do
fulltext "war" do
highlight :body
end
end
</pre>

It will highlight the word war in every result string.

===Hits vs Results===

Sunspot simply stores the type and primary key of objects in Solr. When results are retrieved, those primary keys are used to load the actual object like any relational database.

<pre>
# Using #results pulls in the records from the object-relational mapper
Blog.search.results.each do |result|
puts result.body
end
</pre>

To get results without accessing the database, use hits:
<pre>
Blog.search.hits.each do |hit|
puts hit.stored(:body)
end
</pre>

===Reindexing===
Objects are automatically indexed to Solr as a part of the save callbacks but if there is a change in schema then reindexing is necessary.

<pre>
bundle exec rake sunspot:solr:reindex
</pre>

<ref>http://tech.favoritemedium.com/2010/01/full-text-search-in-rails-with-sunspot.html</ref><ref>http://www.linux-mag.com/id/7341/</ref>Other interesting reads on the topic and referral links are mentioned below.

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/ch1b 18 AS

2015-02-18T01:17:28Z

Amohanr:

Apache Solr and Rails

[[File: solr.jpg|right]]
Apache Solr<ref>http://lucene.apache.org/solr/</ref> is a standalone, open-source enterprise search server created by Yonik Seely. It has a [http://en.wikipedia.org/wiki/Representational_state_transfer REST]-like API. It is a indexing and searching framework which could be deployed and used with many web frameworks like Rails, Drupal, Django etc. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results. It is a popular, scalable, blazing-fast, open source enterprise search platform built on [http://lucene.apache.org/index.html Apache Lucene] search library. Websites using [http://rubyonrails.org/ Rails] can take advantage of the Solr search engine to provide sophisticated and customizable search features. Ruby/Rails integrates with Solr search server using Sunspot<ref>https://rubygems.org/gems/sunspot_rails</ref> library and does a full text search.

__TOC__
=='''Introduction'''==
Apache Solr is a standalone enterprise search server with a REST-like API. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results. It is a popular, scalable, blazing-fast, open source enterprise search platform built on Apache Lucene. Websites using rails can take advantage of the Solr search engine to provide sophisticated and customizable search features.

===Technology Stack<ref>http://www.slideshare.net/dkeener/rails-and-the-apache-solr-search-engine</ref>===
Solr is built on top of Apache Lucene. It is a toolbox responsible for indexing, searching, spell-check and advance tokenization whereas Solr is a search server which inherits all the features of Lucene but also adds API integration, caching and most importantly a web admin interface. This feature makes Solr very easy to use in production environment. Both of these technology needs Java 1.4 or above.
[[File: structure_solr.png|center]]
Sunspot library interacts with Solr using a low-level interface called RSolr. It is a ruby client which integrates the Solr API's to Rails through the use of Sunspot gem. Sunspot has a drop-in ActiveRecord support. We can call Sunspot as the high level client of Solr. We will be talking about the installation and application of Solr using Sunspot gem in the section below.

=='''Features<ref>http://lucene.apache.org/solr/features.html</ref>'''==
Apache Solr has the following features

*Enables full-text matching, powered by Lucence software
*Built to handle high volume traffic
*Provides [http://en.wikipedia.org/wiki/Faceted_search faceted] searching
*Provides features like suggester, spellcheck, clustering, auto-complete, highlighting
*Extensible through plugins
*Supports statistical and aggregate processing of text
*Supports rich format data such as PDF, Word, Pwerpoint

Rails uses the Sunspot Ruby library to integrate with the Solr search engine.
=='''Installation and Uses<ref>http://outoftime.github.io/sunspot/docs/index.html</ref>'''==

===Installing Sunspot Gem===
Sunspot makes it easy to do full text searching through Solr. Sunspot comes as a gem and is installed in the usual way by adding it to the Gemfile and running bundle.

<pre>
gem 'sunspot_rails'
gem 'sunspot_solr' # optional pre-packaged Solr distribution for use in development
</pre>

Bundle it.

<pre>
bundle install
</pre>

Once the gem and its dependencies have installed we will need to generate Sunspot’s configuration file which we can do by running

<pre>
$ rails g sunspot_rails:install
</pre>

This command creates a YML file at /config/sunspot.yml. We don’t need to make any changes to the default settings in this file.
Sunspot embeds Solr inside the gem so there’s no need to install it separately. This means that it works straight out of the box which makes it far more convenient to use in development. To get it up and running we run

<pre>
$ rake sunspot:solr:start
</pre>

If you’re running OS X Lion and you haven’t installed a Java runtime you’ll be prompted to do so when you run this command. You may also see a deprecation warning but this can be safely ignored. The command will also create some more configuration files for advanced configuration.

=== Indexing Objects ===
Add a searchable block to the objects you wish to index.

<pre>
class Example < ActiveRecord::Base
searchable do
text :title, :body
end
end
</pre>

Text fields will be full-text searchable. Other fields which are outside the scope of searchable can be used to scope queries.

=== Searching Objects ===
Now searching can be done simply by passing the query to search method in the respective class.

<pre>
Example.search do
fulltext 'query'
with :conditions
end
</pre>

We can use many variations on the search now by changing the scope variables in the query.

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015/ch1b 18 AS

2015-02-18T00:44:28Z

Amohanr:

Apache Solr and Rails

[[File: solr.jpg|right]]
Apache Solr<ref>http://lucene.apache.org/solr/</ref> is a standalone, open-source enterprise search server created by Yonik Seely. It has a [http://en.wikipedia.org/wiki/Representational_state_transfer REST]-like API. It is a indexing and searching framework which could be deployed and used with many web frameworks like Rails, Drupal, Django etc. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results. It is a popular, scalable, blazing-fast, open source enterprise search platform built on [http://lucene.apache.org/index.html Apache Lucene] search library. Websites using [http://rubyonrails.org/ Rails] can take advantage of the Solr search engine to provide sophisticated and customizable search features. Ruby/Rails integrates with Solr search server using Sunspot<ref>https://rubygems.org/gems/sunspot_rails</ref> library and does a full text search.

__TOC__
=='''Introduction'''==
Apache Solr is a standalone enterprise search server with a REST-like API. Indexing could be done using JSON, XML, CSV or binary over Hyper text transfer protocol. It could be then queried using HTTP with a GET method and receive the JSON, XML, CSV or binary results. It is a popular, scalable, blazing-fast, open source enterprise search platform built on Apache Lucene. Websites using rails can take advantage of the Solr search engine to provide sophisticated and customizable search features.

===Technology Stack<ref>http://www.slideshare.net/dkeener/rails-and-the-apache-solr-search-engine</ref>===
Solr is built on top of Apache Lucene. It is a toolbox responsible for indexing, searching, spell-check and advance tokenization whereas Solr is a search server which inherits all the features of Lucene but also adds API integration, caching and most importantly a web admin interface. This feature makes Solr very easy to use in production environment. Both of these technology needs Java 1.4 or above.
[[File: structure_solr.png|center]]
Solr is integrated with Rails using a client called Rsolr. It is a low level ruby client which integrates the Solr API's to Rails through the use of Sunspot gem. Sunspot has a drop-in ActiveRecord support. We can call Sunspot as the high level client of Solr. We will be talking about the installation and application of Solr using Sunspot gem in the section below.

=='''Features<ref>http://lucene.apache.org/solr/features.html</ref>'''==
Apache Solr has the following features

*Enables full-text matching, powered by Lucence software
*Built to handle high volume traffic
*Provides [http://en.wikipedia.org/wiki/Faceted_search faceted] searching
*Provides features like suggester, spellcheck, clustering, auto-complete, highlighting
*Extensible through plugins
*Supports statistical and aggregate processing of text
*Supports rich format data such as PDF, Word, Pwerpoint

Rails uses the Sunspot Ruby library to integrate with the Solr search engine.
=='''Installation and Uses<ref>http://outoftime.github.io/sunspot/docs/index.html</ref>'''==

===Installing Sunspot Gem===
Sunspot makes it easy to do full text searching through Solr. Sunspot comes as a gem and is installed in the usual way by adding it to the Gemfile and running bundle.

<pre>
gem 'sunspot_rails'
gem 'sunspot_solr' # optional pre-packaged Solr distribution for use in development
</pre>

Bundle it.

<pre>
bundle install
</pre>

Once the gem and its dependencies have installed we will need to generate Sunspot’s configuration file which we can do by running

<pre>
$ rails g sunspot_rails:install
</pre>

This command creates a YML file at /config/sunspot.yml. We don’t need to make any changes to the default settings in this file.
Sunspot embeds Solr inside the gem so there’s no need to install it separately. This means that it works straight out of the box which makes it far more convenient to use in development. To get it up and running we run

<pre>
$ rake sunspot:solr:start
</pre>

If you’re running OS X Lion and you haven’t installed a Java runtime you’ll be prompted to do so when you run this command. You may also see a deprecation warning but this can be safely ignored. The command will also create some more configuration files for advanced configuration.

=== Indexing Objects ===
Add a searchable block to the objects you wish to index.

<pre>
class Example < ActiveRecord::Base
searchable do
text :title, :body
end
end
</pre>

Text fields will be full-text searchable. Other fields which are outside the scope of searchable can be used to scope queries.

=== Searching Objects ===
Now searching can be done simply by passing the query to search method in the respective class.

<pre>
Example.search do
fulltext 'query'
with :conditions
end
</pre>

We can use many variations on the search now by changing the scope variables in the query.

=='''References'''==
<references/>