CSC/ECE 517 Fall 2015 E1589 Automating production setup and deployment

Introduction to Expertiza

The Expertiza project is a web application to create reusable learning objects through peer review. It supports various features such as team projects and the submission of various documents including URLs and wiki pages. It is being used extensively across various universities for select courses.

Project Description

Problem Statement

Currently, setting up the infrastructure for expertiza is done manually. The task is to write a Chef-solo<ref>Chef Solo</ref> recipe which will create a production ready environment for expertiza. Chef-solo recipes are Ruby<ref>Ruby</ref> code.

What needs to be done

We need to provide a single step setup of expertiza for production.

• It should include setting up the basic environment, application dependency, configuration setup and database setup.
• It should also include recipe for regular deployment.

Some Terminology

Chef Solo<ref>https://docs.chef.io/chef_solo.html</ref>

Chef is a systems and cloud infrastructure automation framework that makes it easy to deploy servers and applications to any physical, virtual, or cloud location, no matter the size of the infrastructure. Each organization is comprised of one(or more) workstations, a single server, and every node that will be configured and maintained by the chef-client. Cookbooks (and recipes) are used to tell the chef-client how each node in your organization should be configured. The chef-client(which is installed on every node) does the actual configuration.

• Node:

A node is any machine—physical, virtual, cloud, network device, etc.—that is under management by Chef.

• Recipe:

A recipe specifies the resources to use and the order in which they are to be applied. It is mostly a collection of resources, defined using patterns(resource names, attribute-value pairs, and actions). A recipe must define everything that is required to configure part of a system.

• Cookbook:

A cookbook is the fundamental unit of configuration and policy distribution. A cookbook defines a scenario and contains everything that is required to support that scenario.

• WorkStation:

A workstation is a computer that is configured to run various Chef command-line tools that synchronize with a chef-repo, author cookbooks, interact with the Chef server, interact with nodes, or applications like Chef Delivery.

The workstation is the location from which most users do most of their work.

One(or more) workstations are configured to allow users to author, test, and maintain cookbooks. Cookbooks are uploaded to the Chef server from the workstation.

• Chef Server:

The Chef server acts as a hub of information. Cookbooks and policy settings are uploaded to the Chef server by users from workstations.

• Chef-client:

The chef-client uses Ruby as its reference language for creating cookbooks and defining recipes, with an extended DSL for specific resources

Nginx

Nginx is a free, open-source, high-performance [HTTP] server and reverse proxy<ref>[1]</ref>, as well as an IMAP<ref>https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol</ref>/POP3<ref>https://en.wikipedia.org/wiki/Post_Office_Protocol</ref> proxy server. Nginx is known for its high performance, stability, rich feature set, simple configuration, and low resource consumption.<ref>Nginx Sever</ref>

Unicorn

Unicorn is an HTTP server for Rack applications designed to only serve fast clients on low-latency, high-bandwidth connections and take advantage of features in Unix/Unix-like kernels. Slow clients should only be served by placing a reverse proxy capable of fully buffering both the the request and response in between unicorn and slow clients. <ref>Unicorn Web Server</ref>

Plan of Action

Steps to Implement the Automated Deployment Process

Expertiza Virtual Infrastructure

1. Setup a virtual Vanilla<ref>Vanilla</ref> Ubuntu<ref>Ubuntu</ref> Machine.

2. Update the ubuntu to the specified version.

3. Install packages for Ruby and all the ruby dependencies.

4. Install MYSQL<ref>Mysql server</ref> and configure it.

5. Install Unicorn gem. Unicorn is a Rack<ref>Rack Server</ref> HTTP server that uses forked processes to handle multiple incoming requests concurrently. For expertiza, unicorn is being used as the web server, which will interact with the database servers to fetch the data.

6. Bundle install, install gems and migrate databases.

7. Install nginx. Nginx is a web server. It is being used in the Expertiza architecture as a reverse proxy server. Through nginx, we will also attempt to serve static pages directly.

8. Make changes in the nginx configuration.

9. Configure Unicorn and integrate it to work with nginx.

Expertiza Stack

The above visually describes the expertiza stack on the virtual machine. The stack includes nginx as the front end, web server which will redirect the requests to the unicorn server. The scope of the project includes just 1 unicorn server but the recipe will be written in a way that later modifications can be made to add more than 1 unicorn servers under a load balancer. The unicorn server will interact with the database server and write/fetch data into it.

Chef-Solo Recipe Example

Chef solo has cookbooks and recipes. An example of recipes is as follows:

   package "git-core" #apt-get install git-core

   user node[:user][:name] do
       password node[:user][:password]
       gid "admin"
       home "/home/#{node[:user][:name]}"
       supports :manage_home => true
   end

This recipe orchestrates the installation of git-core and creates a new user and sets its privileges and password along with the home directory. Similarly, the plan of action is to write recipes for downloading, installing and configuring various components(nginx, unicorn, RMV, Ruby, MYSQL etc.) which will contribute towards building the infrastructure of expertiza.

Expertiza Deployment

1. We first created a droplet( virtual private server) using digital ocean<ref>https://www.digitalocean.com/</ref>.

2. We recorded the IP address of the newly created droplet and created the associated <ipaddress>.json file.

<ipaddress>.json : This file holds all the node specific values required to install services and config node. We will set node_user , user_hash_password , ruby version , mysql root & password . The main function apart from holding node variables is to run recipes in a specific code.

The following recipes(.rb files), scripts and node specific files were created to achieve automatic deployment. Other pre-installed files and folder for Chef cookbook can be found in the git repository mentioned above.

3. Next,

setup_vps.sh : This will run the knife tool that will setup Chef-solo on the node and will execute the recipes. This needs to be from the directory where it present. The bash command in following format has to be used.

sh setup_vps.sh <node_username> <node_ip_address> <node_port>

default.rb : This updates the Ubuntu machine packages, install the list of prerequisite packages and sets the time zone.

expertiza_clone : This recipes creates a directory and clones the Expertiza git source code in that directory.

set_java_home: This recipe sets the Java configuration. It then bundle installs all the gems used by the Expertiza.

mysql.rb : It installs the MySQL server , MySQL client and its associated dependencies. The root password is preloaded and then fetched on installation by setting DEBIAN_FRONTEND environment variable.

nginx.rb : This recipe installs nginx which is web proxy server that will listen on user defined port and redirect the requests for static content (images, css, javascript) to the public folder of Experiza rails app.

unicorn.rb : It will install and configure Unicorn web server used in production environment to handle high traffic and achieve load balancing . All the requests for dynamic content will go through it.

Github link

https://github.com/amurark/expertiza

Points for the reviewer

While reviewing this project, please consider the following points:

• 1. There is no scope for testing in our project. This has been discussed and approved by the mentor.
• 2. There is no scope for UML diagrams for this project, as there are no classes/interface involved.

References

<references></references>