CSC/ECE 517 Fall 2015 E1589 Automating production setup and deployment: Difference between revisions

From Expertiza_Wiki
Jump to navigation Jump to search
 
(34 intermediate revisions by 4 users not shown)
Line 16: Line 16:
*It should also include recipe for regular deployment.
*It should also include recipe for regular deployment.


==Some Terminology==
==Terminology==
===Chef Solo<ref>https://docs.chef.io/chef_solo.html</ref>===  
===Chef Solo<ref>https://docs.chef.io/chef_solo.html</ref>===  
Chef is a systems and cloud infrastructure automation framework that makes it easy to deploy servers and applications to any physical, virtual, or cloud location, no matter the size of the infrastructure. Each organization is comprised of one(or more) workstations, a single server, and every node that will be configured and maintained by the chef-client. Cookbooks (and recipes) are used to tell the chef-client how each node in your organization should be configured. The chef-client(which is installed on every node) does the actual configuration.
Chef is a systems and cloud infrastructure automation framework that makes it easy to deploy servers and applications to any physical, virtual, or cloud location, no matter the size of the infrastructure. Each organization is comprised of one(or more) workstations, a single server, and every node that will be configured and maintained by the chef-client. Cookbooks (and recipes) are used to tell the chef-client how each node in your organization should be configured. The chef-client(which is installed on every node) does the actual configuration.
Line 42: Line 42:
===Nginx===
===Nginx===


Nginx is a free, open-source, high-performance [HTTP] server and reverse proxy<ref>[https://www.nginx.com/resources/glossary/reverse-proxy-server/]</ref>, as well as an IMAP<ref>https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol</<ref>//POP3<ref>https://en.wikipedia.org/wiki/Post_Office_Protocol</ref> proxy server. Nginx is known for its high performance, stability, rich feature set, simple configuration, and low resource consumption.<ref>[https://www.nginx.com/resources/wiki/ Nginx Sever]</ref>
Nginx is a free, open-source, high-performance [HTTP] server and reverse proxy<ref>[https://www.nginx.com/resources/glossary/reverse-proxy-server/]</ref>, as well as an IMAP<ref>https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol</ref>/POP3<ref>https://en.wikipedia.org/wiki/Post_Office_Protocol</ref> proxy server. Nginx is known for its high performance, stability, rich feature set, simple configuration, and low resource consumption.<ref>[https://www.nginx.com/resources/wiki/ Nginx Sever]</ref>


===Unicorn===
===Unicorn===
Line 55: Line 55:




1. Setup a virtual Vanilla<ref>[https://en.wikipedia.org/wiki/Vanilla_software Vanilla]</ref> Ubuntu<ref>[http://www.ubuntu.com Ubuntu]</ref> Machine.
1. Set up a virtual Vanilla<ref>[https://en.wikipedia.org/wiki/Vanilla_software Vanilla]</ref> Ubuntu<ref>[http://www.ubuntu.com Ubuntu]</ref> Machine.


2. Update the ubuntu to the specified version.
2. Update the Ubuntu to the specified version.


3. Install nginx. Nginx is a web server. It is being used in the Expertiza architecture as a reverse proxy server. Through nginx, we will also attempt to serve static pages directly.
3. Clone Expertiza from Github.


4. Make changes in the nginx configuration.
4. Install packages for Java, Ruby and all the ruby dependencies.


5. Install Ruby, RVM<ref>[https://rvm.io/ Ruby Virtual Machine]</ref> and all the ruby dependencies.
5. Bundle install for installing all the gems in Gemfile.


6. Install MYSQL<ref>[https://mysql.com Mysql server]</ref> and configure it.
6. Install MYSQL<ref>[https://mysql.com Mysql server]</ref>, configure it and migrate databases.


7. Install Unicorn gem. Unicorn is a Rack<ref>[https://en.wikipedia.org/wiki/Rack_%28web_server_interface%29 Rack Server]</ref> HTTP server that uses forked processes to handle multiple incoming requests concurrently. For expertiza, unicorn is being used as the web server, which will interact with the database servers to fetch the data.
7. Install Unicorn gem. Unicorn is a Rack<ref>[https://en.wikipedia.org/wiki/Rack_%28web_server_interface%29 Rack Server]</ref> HTTP server that uses forked processes to handle multiple incoming requests concurrently. For expertiza, unicorn is being used as the web server, which will interact with the database servers to fetch the data.


8. Configure Unicorn and integrate it to work with nginx.
8. Make changes in the nginx configuration.


9. Bundle install, install gems and migrate databases.
9. Install nginx. Nginx is a web server. It is being used in the Expertiza architecture as a reverse proxy server. Through nginx, we will also attempt to serve static pages directly.
 
10. Configure Unicorn and integrate it to work with nginx.
 
11. Start both Unicorn and Nginx server.


==Expertiza Stack==
==Expertiza Stack==
Line 77: Line 81:
[[File:Exp1.png]]
[[File:Exp1.png]]


The above visually describes the expertiza stack on the virtual machine. The stack includes nginx as the front end, web server which will redirect the requests to the unicorn server. The scope of the project includes just 1 unicorn server but the recipe will be written in a way that later modifications can be made to add more than 1 unicorn servers under a load balancer. The unicorn server will interact with the database server and write/fetch data into it.
The above visually describes the expertiza stack on the virtual machine. The stack includes nginx as the front end, web server which will redirect the requests to the unicorn server. The scope of the project includes just 1 unicorn server but the recipe will be written in a way that later modifications can be made to add more than one unicorn servers under a load balancer. The unicorn server will interact with the database server and write/fetch data into it.
 
==Expertiza Deployment==
 
 
1. We first created a droplet( virtual private server) using digital ocean<ref>https://www.digitalocean.com/</ref>.
[[File:Drop.png]]
 
2. We recorded the IP address of the newly created droplet and created the associated <ipaddress>.json file.
 
'''<ipaddress>.json''' : This file holds all the node specific values required to install services and config node.
We will set node_user , user_hash_password , ruby version , mysql root & password . The main function apart from holding node variables is to run recipes in a specific code.
 
[[File:New_ip.png]]
 
The following recipes(.rb files), scripts and node specific files were created to achieve automatic deployment. Other pre-installed files and folder for Chef cookbook can be found in the git repository mentioned above.
 


==Chef-Solo Recipe Example==
3. We then run the bash command in following format<code>sh setup_vps.sh <node_username> <node_ip_address> <node_port> </code>
Chef solo has cookbooks and recipes. An example of recipes is as follows:
    package "git-core" #apt-get install git-core


    user node[:user][:name] do
This command will run the knife tool that will set up Chef-solo on the node and will execute the recipes.
        password node[:user][:password]
This needs to be run from the directory where '''setup_vps.sh''' is present.
        gid "admin"
        home "/home/#{node[:user][:name]}"
        supports :manage_home => true
    end


This recipe orchestrates the installation of git-core and creates a new user and sets its privileges and password along with the home directory. Similarly, the plan of action is to write recipes for downloading, installing and configuring various components(nginx, unicorn, RMV, Ruby, MYSQL etc.) which will contribute towards building the infrastructure of expertiza.
[[File:New setup vps.png]]


==Expertiza Deployment ==


We first create a droplet using [digital ocean].
4. '''default.rb''' : This recipe will update the Ubuntu machine packages, install the list of prerequisite packages and set the time zone.


The following recipes (.rb files) , scripts and node specific files were created to achieve automatic deployment. Other pre-installed files and folder for Chef cookbook can be found on above git repository.
[[File:New default.png]]


'''setup_vps.sh''' : This will run the knife tool that will setup Chef-solo on the node and  will execute the recipes.
This needs to be from the directory where it present. The bash command in following format has to be used.


<code>sh setup_vps.sh <node_username> <node_ip_address> <node_port> </code>
5. '''expertiza_clone''' : This recipe will create a directory and clone the Expertiza git source code into that directory.


[[File:New expertiza clone.png ]]


[[File:Setup vps sh.JPG]]


6. '''set_java_home''': This recipe will set the Java configuration. It will then bundle install all the gems used by the Expertiza.


'''<ipaddress>.json''' : This file holds all the node specific values required to install services and config node.
[[File:New_set_java.png]]
We will set node_user , user_hash_password , ruby version , mysql root & password . The main function apart from holding node variables is to run recipes in a specific code.


[[File:Ipjson.JPG]]


7. '''mysql.rb''' : This recipe will install the MySQL server , MySQL client and its associated dependencies. The root password is preloaded and then fetched on installation by setting DEBIAN_FRONTEND environment variable.


'''default.rb''' : This updates the Ubuntu machine packages, install the list of prerequisite packages and sets the time zone.
[[File:New sql.png]]
[[File:Default rb.JPG]]




'''expertiza_clone''' : This recipes creates a directory and clones the Expertiza git source code in that directory.
8. '''database.rb''' : This recipe will execute scrubbed.sql on MySQL, which in turn will create all the Schema for production, development and test environments. It will also populate the database tables with seed (configuration) data.
[[File:Expertiza clone rb.JPG ]]


[[File:New database.png]]


'''set_java_home''': This recipe sets the Java configuration. It then bundle installs all the gems used by the Expertiza.
[[File:Set java home rb.JPG]]


9. '''nginx.rb''' : This recipe will install nginx which is a web proxy server that will listen on user defined port and redirect the requests for static content(images, css, javascript) to the public folder of the Expertiza rails app.


'''mysql.rb''' : It installs the MySQL server , MySQL client and its associated dependencies. The root password is preloaded and then fetched on installation by setting DEBIAN_FRONTEND environment variable.
[[File:New_nginx.png ]]


[[File:Mysql rb.JPG]]


10. '''unicorn_conf.rb''' : This recipe will install and configure Unicorn web server used in production environment to handle requests for dynamic content passed on by the Nginx server.


'''nginx.rb''' : This recipe installs nginx which is web proxy server that will listen on user defined port and redirect the requests for static content (images, css, javascript) to the public folder of Experiza rails app.
[[File:New_unicorn_corn.png]]


[[File:NGINX rb.JPG ]]


11. '''start_server.rb''' : This will restart all the installed services, after which Expertiza can start taking requests.


'''unicorn.rb''' : It will install and configure Unicorn web server used in production environment to handle high traffic and achieve load balancing . All the requests for dynamic content will go through it.
[[File:New start server1.png]]
[[File:Start server rb.JPG]]


==Github link==
==Github link==
https://github.com/amurark/expertiza
https://github.com/amurark/knife-solo/


==Points for the reviewer==
==Points for the reviewer==
Line 147: Line 155:
*1. There is no scope for testing in our project. This has been discussed and approved by the mentor.  
*1. There is no scope for testing in our project. This has been discussed and approved by the mentor.  
*2. There is no scope for UML diagrams for this project, as there are no classes/interface involved.
*2. There is no scope for UML diagrams for this project, as there are no classes/interface involved.
==Future Scope==
The cookbook has been created with the assumption of root user as the administer.For security purposes deployment should be done with another user with sudo privileges, and then root login disabled. Also a new user for MySQL with full database privileges needs to be created. But that hasn't been achieved automatically in the resent cookbook. Further, Unicorn configuration is done with a single Unicorn server in mind. It has to be configured for multiple servers for load balancing in events of heavy traffic.
Currently, we are manually creating the digitalOcean droplet which is the remote server where expertiza is deployed. In future, this can be automated using a plugin provided by vagrant for digital ocean.


==References==
==References==
<references></references>
<references></references>

Latest revision as of 19:54, 16 December 2015

Introduction to Expertiza

The Expertiza project is a web application to create reusable learning objects through peer review. It supports various features such as team projects and the submission of various documents including URLs and wiki pages. It is being used extensively across various universities for select courses.

Project Description

Problem Statement

Currently, setting up the infrastructure for expertiza is done manually. The task is to write a Chef-solo<ref>Chef Solo</ref> recipe which will create a production ready environment for expertiza. Chef-solo recipes are Ruby<ref>Ruby</ref> code.

What needs to be done

We need to provide a single step setup of expertiza for production.

  • It should include setting up the basic environment, application dependency, configuration setup and database setup.
  • It should also include recipe for regular deployment.

Terminology

Chef Solo<ref>https://docs.chef.io/chef_solo.html</ref>

Chef is a systems and cloud infrastructure automation framework that makes it easy to deploy servers and applications to any physical, virtual, or cloud location, no matter the size of the infrastructure. Each organization is comprised of one(or more) workstations, a single server, and every node that will be configured and maintained by the chef-client. Cookbooks (and recipes) are used to tell the chef-client how each node in your organization should be configured. The chef-client(which is installed on every node) does the actual configuration.

  • Node:
A node is any machine—physical, virtual, cloud, network device, etc.—that is under management by Chef.
  • Recipe:
A recipe specifies the resources to use and the order in which they are to be applied. It is mostly a collection of resources, defined using patterns(resource names, attribute-value pairs, and actions). A recipe must define everything that is required to configure part of a system.
  • Cookbook:
A cookbook is the fundamental unit of configuration and policy distribution. A cookbook defines a scenario and contains everything that is required to support that scenario.
  • WorkStation:
A workstation is a computer that is configured to run various Chef command-line tools that synchronize with a chef-repo, author cookbooks, interact with the Chef server, interact with nodes, or applications like Chef Delivery.
The workstation is the location from which most users do most of their work.
One(or more) workstations are configured to allow users to author, test, and maintain cookbooks. Cookbooks are uploaded to the Chef server from the workstation.
  • Chef Server:
The Chef server acts as a hub of information. Cookbooks and policy settings are uploaded to the Chef server by users from workstations.
  • Chef-client:
The chef-client uses Ruby as its reference language for creating cookbooks and defining recipes, with an extended DSL for specific resources

Nginx

Nginx is a free, open-source, high-performance [HTTP] server and reverse proxy<ref>[1]</ref>, as well as an IMAP<ref>https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol</ref>/POP3<ref>https://en.wikipedia.org/wiki/Post_Office_Protocol</ref> proxy server. Nginx is known for its high performance, stability, rich feature set, simple configuration, and low resource consumption.<ref>Nginx Sever</ref>

Unicorn

Unicorn is an HTTP server for Rack applications designed to only serve fast clients on low-latency, high-bandwidth connections and take advantage of features in Unix/Unix-like kernels. Slow clients should only be served by placing a reverse proxy capable of fully buffering both the the request and response in between unicorn and slow clients. <ref>Unicorn Web Server</ref>

Plan of Action

Steps to Implement the Automated Deployment Process

Expertiza Virtual Infrastructure


1. Set up a virtual Vanilla<ref>Vanilla</ref> Ubuntu<ref>Ubuntu</ref> Machine.

2. Update the Ubuntu to the specified version.

3. Clone Expertiza from Github.

4. Install packages for Java, Ruby and all the ruby dependencies.

5. Bundle install for installing all the gems in Gemfile.

6. Install MYSQL<ref>Mysql server</ref>, configure it and migrate databases.

7. Install Unicorn gem. Unicorn is a Rack<ref>Rack Server</ref> HTTP server that uses forked processes to handle multiple incoming requests concurrently. For expertiza, unicorn is being used as the web server, which will interact with the database servers to fetch the data.

8. Make changes in the nginx configuration.

9. Install nginx. Nginx is a web server. It is being used in the Expertiza architecture as a reverse proxy server. Through nginx, we will also attempt to serve static pages directly.

10. Configure Unicorn and integrate it to work with nginx.

11. Start both Unicorn and Nginx server.

Expertiza Stack

The above visually describes the expertiza stack on the virtual machine. The stack includes nginx as the front end, web server which will redirect the requests to the unicorn server. The scope of the project includes just 1 unicorn server but the recipe will be written in a way that later modifications can be made to add more than one unicorn servers under a load balancer. The unicorn server will interact with the database server and write/fetch data into it.

Expertiza Deployment

1. We first created a droplet( virtual private server) using digital ocean<ref>https://www.digitalocean.com/</ref>.

2. We recorded the IP address of the newly created droplet and created the associated <ipaddress>.json file.

<ipaddress>.json : This file holds all the node specific values required to install services and config node. We will set node_user , user_hash_password , ruby version , mysql root & password . The main function apart from holding node variables is to run recipes in a specific code.

The following recipes(.rb files), scripts and node specific files were created to achieve automatic deployment. Other pre-installed files and folder for Chef cookbook can be found in the git repository mentioned above.


3. We then run the bash command in following formatsh setup_vps.sh <node_username> <node_ip_address> <node_port>

This command will run the knife tool that will set up Chef-solo on the node and will execute the recipes. This needs to be run from the directory where setup_vps.sh is present.


4. default.rb : This recipe will update the Ubuntu machine packages, install the list of prerequisite packages and set the time zone.


5. expertiza_clone : This recipe will create a directory and clone the Expertiza git source code into that directory.


6. set_java_home: This recipe will set the Java configuration. It will then bundle install all the gems used by the Expertiza.


7. mysql.rb : This recipe will install the MySQL server , MySQL client and its associated dependencies. The root password is preloaded and then fetched on installation by setting DEBIAN_FRONTEND environment variable.


8. database.rb : This recipe will execute scrubbed.sql on MySQL, which in turn will create all the Schema for production, development and test environments. It will also populate the database tables with seed (configuration) data.


9. nginx.rb : This recipe will install nginx which is a web proxy server that will listen on user defined port and redirect the requests for static content(images, css, javascript) to the public folder of the Expertiza rails app.


10. unicorn_conf.rb : This recipe will install and configure Unicorn web server used in production environment to handle requests for dynamic content passed on by the Nginx server.


11. start_server.rb : This will restart all the installed services, after which Expertiza can start taking requests.

Github link

https://github.com/amurark/knife-solo/

Points for the reviewer

While reviewing this project, please consider the following points:

  • 1. There is no scope for testing in our project. This has been discussed and approved by the mentor.
  • 2. There is no scope for UML diagrams for this project, as there are no classes/interface involved.

Future Scope

The cookbook has been created with the assumption of root user as the administer.For security purposes deployment should be done with another user with sudo privileges, and then root login disabled. Also a new user for MySQL with full database privileges needs to be created. But that hasn't been achieved automatically in the resent cookbook. Further, Unicorn configuration is done with a single Unicorn server in mind. It has to be configured for multiple servers for load balancing in events of heavy traffic. Currently, we are manually creating the digitalOcean droplet which is the remote server where expertiza is deployed. In future, this can be automated using a plugin provided by vagrant for digital ocean.

References

<references></references>

Retrieved from "https://wiki.expertiza.ncsu.edu/index.php?title=CSC/ECE_517_Fall_2015_E1589_Automating_production_setup_and_deployment&oldid=100611"