CSC/ECE 517 Fall 2015 E1589 Automating production setup and deployment
Introduction to Expertiza
The Expertiza project is a web application to create reusable learning objects through peer review. It supports various features such as team projects and the submission of various documents including URLs and wiki pages. It is being used extensively across various universities for select courses.
Project Description
Problem Statement
Currently, setting up the infrastructure for expertiza is done manually. The task is to write a Chef-solo<ref>Chef Solo</ref> recipe which will create a production ready environment for expertiza. Chef-solo recipes are Ruby<ref>Ruby</ref> code.
What needs to be done
We need to provide a single step setup of expertiza for production.
- It should include setting up the basic environment, application dependency, configuration setup and database setup.
- It should also include recipe for regular deployment.
Terminology
Chef Solo<ref>https://docs.chef.io/chef_solo.html</ref>
Chef is a systems and cloud infrastructure automation framework that makes it easy to deploy servers and applications to any physical, virtual, or cloud location, no matter the size of the infrastructure. Each organization is comprised of one(or more) workstations, a single server, and every node that will be configured and maintained by the chef-client. Cookbooks (and recipes) are used to tell the chef-client how each node in your organization should be configured. The chef-client(which is installed on every node) does the actual configuration.
- Node:
- A node is any machine—physical, virtual, cloud, network device, etc.—that is under management by Chef.
- Recipe:
- A recipe specifies the resources to use and the order in which they are to be applied. It is mostly a collection of resources, defined using patterns(resource names, attribute-value pairs, and actions). A recipe must define everything that is required to configure part of a system.
- Cookbook:
- A cookbook is the fundamental unit of configuration and policy distribution. A cookbook defines a scenario and contains everything that is required to support that scenario.
- WorkStation:
- A workstation is a computer that is configured to run various Chef command-line tools that synchronize with a chef-repo, author cookbooks, interact with the Chef server, interact with nodes, or applications like Chef Delivery.
- The workstation is the location from which most users do most of their work.
- One(or more) workstations are configured to allow users to author, test, and maintain cookbooks. Cookbooks are uploaded to the Chef server from the workstation.
- Chef Server:
- The Chef server acts as a hub of information. Cookbooks and policy settings are uploaded to the Chef server by users from workstations.
- Chef-client:
- The chef-client uses Ruby as its reference language for creating cookbooks and defining recipes, with an extended DSL for specific resources
Nginx
Nginx is a free, open-source, high-performance [HTTP] server and reverse proxy<ref>[1]</ref>, as well as an IMAP<ref>https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol</ref>/POP3<ref>https://en.wikipedia.org/wiki/Post_Office_Protocol</ref> proxy server. Nginx is known for its high performance, stability, rich feature set, simple configuration, and low resource consumption.<ref>Nginx Sever</ref>
Unicorn
Unicorn is an HTTP server for Rack applications designed to only serve fast clients on low-latency, high-bandwidth connections and take advantage of features in Unix/Unix-like kernels. Slow clients should only be served by placing a reverse proxy capable of fully buffering both the the request and response in between unicorn and slow clients. <ref>Unicorn Web Server</ref>
Plan of Action
Steps to Implement the Automated Deployment Process
1. Set up a virtual Vanilla<ref>Vanilla</ref> Ubuntu<ref>Ubuntu</ref> Machine.
2. Update the Ubuntu to the specified version.
3. Clone Expertiza from Github.
4. Install packages for Java, Ruby and all the ruby dependencies.
5. Bundle install for installing all the gems in Gemfile.
6. Install MYSQL<ref>Mysql server</ref>, configure it and migrate databases.
7. Install Unicorn gem. Unicorn is a Rack<ref>Rack Server</ref> HTTP server that uses forked processes to handle multiple incoming requests concurrently. For expertiza, unicorn is being used as the web server, which will interact with the database servers to fetch the data.
8. Make changes in the nginx configuration.
9. Install nginx. Nginx is a web server. It is being used in the Expertiza architecture as a reverse proxy server. Through nginx, we will also attempt to serve static pages directly.
10. Configure Unicorn and integrate it to work with nginx.
11. Start both Unicorn and Nginx server.
Expertiza Stack
The above visually describes the expertiza stack on the virtual machine. The stack includes nginx as the front end, web server which will redirect the requests to the unicorn server. The scope of the project includes just 1 unicorn server but the recipe will be written in a way that later modifications can be made to add more than one unicorn servers under a load balancer. The unicorn server will interact with the database server and write/fetch data into it.
Expertiza Deployment
1. We first created a droplet( virtual private server) using digital ocean<ref>https://www.digitalocean.com/</ref>.
2. We recorded the IP address of the newly created droplet and created the associated <ipaddress>.json file.
<ipaddress>.json : This file holds all the node specific values required to install services and config node. We will set node_user , user_hash_password , ruby version , mysql root & password . The main function apart from holding node variables is to run recipes in a specific code.
The following recipes(.rb files), scripts and node specific files were created to achieve automatic deployment. Other pre-installed files and folder for Chef cookbook can be found in the git repository mentioned above.
3. We then run the bash command in following formatsh setup_vps.sh <node_username> <node_ip_address> <node_port>
This command will run the knife tool that will set up Chef-solo on the node and will execute the recipes. This needs to be run from the directory where setup_vps.sh is present.
4. default.rb : This recipe will update the Ubuntu machine packages, install the list of prerequisite packages and set the time zone.
5. expertiza_clone : This recipe will create a directory and clone the Expertiza git source code into that directory.
6. set_java_home: This recipe will set the Java configuration. It will then bundle install all the gems used by the Expertiza.
7. mysql.rb : This recipe will install the MySQL server , MySQL client and its associated dependencies. The root password is preloaded and then fetched on installation by setting DEBIAN_FRONTEND environment variable.
8. database.rb : This recipe will execute scrubbed.sql on MySQL, which in turn will create all the Schema for production, development and test environments. It will also populate the database tables with seed (configuration) data.
9. nginx.rb : This recipe will install nginx which is a web proxy server that will listen on user defined port and redirect the requests for static content(images, css, javascript) to the public folder of the Expertiza rails app.
10. unicorn_conf.rb : This recipe will install and configure Unicorn web server used in production environment to handle requests for dynamic content passed on by the Nginx server.
11. start_server.rb : This will restart all the installed services, after which Expertiza can start taking requests.
Github link
https://github.com/amurark/knife-solo/
Points for the reviewer
While reviewing this project, please consider the following points:
- 1. There is no scope for testing in our project. This has been discussed and approved by the mentor.
- 2. There is no scope for UML diagrams for this project, as there are no classes/interface involved.
Future Scope
The cookbook has been created with the assumption of root user as the administer.For security purposes deployment should be done with another user with sudo privileges, and then root login disabled. Also a new user for MySQL with full database privileges needs to be created. But that hasn't been achieved automatically in the resent cookbook. Further, Unicorn configuration is done with a single Unicorn server in mind. It has to be configured for multiple servers for load balancing in events of heavy traffic. Currently, we are manually creating the digitalOcean droplet which is the remote server where expertiza is deployed. In future, this can be automated using a plugin provided by vagrant for digital ocean.
References
<references></references>