Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-31T00:51:42Z

Grmenon: /* Search Added Queries */

== Introduction to Eden ==

Sahana Eden is an Open Source Humanitarian Platform which has been built specifically for Disaster Management,Development, and Environmental Management sectors. It is highly configurable so that it can be used in a wide variety of different contexts and is easy to modify to build custom solutions.<ref>Tressel, Pat, Fran Boon, Shikhar Kohli, Dominic Koenig, Belinda Lopez, Eli Lev, Michael Howden, and Anne Goldenberg(2011, October) ''Sahana Eden'' Lulu.com, Retrieved from <http://en.flossmanuals.net/sahana-eden/index/> </ref>
Sahana Eden can be accessed from the web or locally from a flash drive, allowing it to be used in environments with poor internet. Local & Web versions can be configured to synchronize to allow data to be shared between them. This helps to keep data synchronized to receive updates regularly and help in responding and recovering from disasters more effectively and efficiently.

Sahana Eden's modular design allows different pieces of functionality to be enabled and disabled as required providing flexible solutions for changing contexts.

The application can be configured to secure sensitive information, while also making data which needs to be shared available in a variety of different formats including Microsoft Excel and PDF. To ensure that Sahana Eden is accessible to every country, it can be translated into multiple languages.

== Framework - Web2py ==
Web2py is an enterprise framework for agile development of secure database-driven web-based applications, written and programmable in Python.web2py is a full-stack framework, meaning that it contains all the components you need to build fully functional web applications.<ref> Massimo Di Pierro, ''Complete Reference Manual, 6th Edition (pre-release)'' Retrieved from <http://www.web2py.com/init/default/documentation> </ref>
=== Features of Web2py: <ref name = "name"> web2py.com </ref> ===
web2py includes a Database Abstraction Layer (DAL) that writes SQL dynamically so that you, the developer, don't have to. The DAL knows how to generate SQL transparently for SQLite, MySQL, PostgreSQL, MSSQL, FireBird, Oracle, IBM DB2, Informix and Ingres.
* The DAL can also generate function calls for the Google Datastore when running on the Google App Engine.
* web2py differs from other web frameworks in that it is the only framework to fully embrace the Web 2.0 paradigm, where the web is the computer.web2py does not require installation or configuration; it runs on any architecture that can run Python (Windows, Windows CE, Mac OS X, iOS, and Unix/Linux), and the development, deployment, and maintenance phases for the applications can be done via a local or remote web interface. web2py runs with CPython (the C implementation) and PyPy (Python written in Python), on Python versions 2.5, 2.6, and 2.7.
* web2py provides a ticketing system for error events. If an error occurs, a ticket is issued to the user, and the error is logged for the administrator.
* web2py is open source and released under the LGPL version 3 license.

=== Why Web2Py? <ref name="name"> Massimo Di Pierro, ''Complete Reference Manual, 6th Edition (pre-release)'' Retrieved from <http://web2py.com/books/default/chapter/29/01/introduction> </ref>===
web2py is one of many web application frameworks, but it has compelling and unique features. web2py was originally developed as a teaching tool, with the following primary motivations:
* Easy for users to learn server-side web development without compromising functionality. For this reason, web2py requires no installation and no configuration, has no dependencies (except for the source code distribution, which requires Python 2.5 and its standard library modules), and exposes most of its functionality via a Web interface, including an Integrated Development Environment with Debugger and database interface.
* web2py has been stable from day one because it follows a top-down design; i.e., its API was designed before it was implemented. Even as new functionality has been added, web2py has never broken backwards compatibility, and it will not break compatibility when additional functionality is added in the future.
* web2py proactively addresses the most important security issues which plague many modern web applications, as determined by OWASP[owasp] below.
* web2py is lightweight. Its core libraries, including the Database Abstraction Layer, the template language, and all the helpers amount to 1.4MB. The entire source code including sample applications and images amounts to 10.4MB.
* web2py has a small footprint and is very fast. It uses the Rocket[rocket] WSGI web server developed by Timothy Farrell. It is as fast as Apache with mod_wsgi, and supports SSL and IPv6.
* web2py uses Python syntax for models, controllers, and views, but does not import models and controllers (as all the other Python frameworks do) - instead it executes them. This means that apps can be installed, uninstalled, and modified without having to restart the web server (even in production), and different apps can coexist without their modules interfering with one another.
* web2py uses a Database Abstraction Layer (DAL) instead of an Object Relational Mapper (ORM). From a conceptual point of view, this means that different database tables are mapped into different instances of one Table class and not into different classes, while records are mapped into instances of one Row class, not into instances of the corresponding table class. From a practical point of view, it means that SQL syntax maps almost one-to-one into DAL syntax, and there is no complex metaclass programming going on under the hood as in popular ORMs, which would add latency.

== S3==
The Sahana Eden framework (S3) is built on Web2Py and It is used for RAD(Papid Application Development) in all the sahana applications. Eden’s custom CRUD framework.
S3 Framework includes pre configured Filters and Reports that allow the data analysis and report generation a simple task to be performed by directly.<ref>Tressel, Pat, Fran Boon, Shikhar Kohli, Dominic Koenig, Belinda Lopez, Eli Lev, Michael Howden, and Anne Goldenberg(2011, October) "Sahana Eden." Lulu.com, . Web. Oct. 2013. Retrieved from <http://en.flossmanuals.net/sahana-eden/technical-overview/> </ref>

S3 Filter forms allow the user to apply filters to interactive views (e.g. data tables, data lists, reports etc.). S3CRUD has filter forms integrated in the standard list view (=multi-record request w/o URL method). Filter forms can be rendered by any method handler. Since filter forms will never be submitted, the method handler does not need to handle any subsequent POSTs related to the filter form, but just insert the filter form HTML into the page.

S3 Framework also includes S3Report which is a REST method handler for data analysis. S3Report generates a full page which includes a set of S3Search widgets, a pivot table (a.k.a. contingency table) from S3Resources and graphs based on the Pivot Tables. S3Report responds to the /report URL method. S3Report supports a number of aggregation functions. The functions like list,count,sun and avg have been tested so far.

Table configurations can contain a report_options item, which is a Storage object. This object is used to configure reports and report options form.
The rows, cols, facts and aggregate items are lists of available values for the user to select from.
The defaults property is a Storage object that contains the default values for the report. It can contain a value for rows, cols, fact, aggregate and totals (as described in URL Methods).
The search property is a list of S3Search widgets that will allow the report to be filtered. If no search property is specified, no filter form will be available.

== Openshift Deployment ==

OpenShift is a cloud computing platform which offers "Platform As a Service" cloud. It is based on hosting platform provided by RedHat. It supports Ruby, it is really easy to install and start working on it. OpenShift takes care of maintaining the services underlying the application and scaling the application as needed. OpenShift supports web-application frameworks by supporting each language's preferred web-integration API, with no required changes to the actual framework code. web2py works unmodified on OpenShift.

Since the software is open source, it is available on GitHub. Development can be started after deploying different web applications in different languages on the platform using [http://en.wikipedia.org/wiki/Git_(software) Git].Making Changes to your Application on OpenShift, you make code changes on your local machine, check those changes in locally, and then "push" those changes to OpenShift. One of the primary advantages of Git is that it does not require a continuous online presence in order to run. You can easily check in (in Git terminology, 'commit') and revert changes locally before deciding to upload those changes to OpenShift.

== Configuration - Twitter ==
The Sahana Eden’s social media feeds requires the user to configure the twitter settings. This is done by providing the twitter login ID, and the OAuth credentials in the Twitter Settings page

:[[File:Configure.png]]

Once the twitter settings are configured, we are all set to search twitter feeds.

== Twitter Search After Save ==

Twitter search after save allows Sahana Eden to perform a search immediately after the search query is added.Search queries can be added in Sahana Eden by clicking on the Add query button in the Twitter Queries page.

:[[File:Addquery1.png]]

The keyword to search is entered. Checking the "Search After Save?" will redirect the user to the results page once the search query is added.

:[[File:Addquery2.png]]

:[[File:Addquery3.png]]

== Twitter Search Results Filter ==
The twitter search results for the search queries can be found in the Twitter Search Results page. The search filters on the Results page allow searching of feeds by "Tweeted date" and "Tweeted by".

: [[File:New-tweeted-by.png]]

:[[File:New-tweetedby.png]]

== Twitter Search Results Report ==

The Reports page generates Reports based on date filters and "tweeted by". The report contains the total number of records, search queries, Tweeted on and language that can further be grouped by search query, tweeted on or language.

:[[ File:Report1.png]]

:[[ File:Report2.png]]

== Further Work (wrt to OSS Final Project) ==
The further work in the OSS project is listed below:
*Integration with Parsers
*rewire the 'parse_twitter' Parser
*Run Parsers on the list
*Extract Documents Parser
*Parser should run on any Message
*Parser should read attachments
*Parser should add all URLs as documents
*Parser should parse link types to see if there are Images
*optionally parser should be able to download linked images/documents into local store
*Train these parsers

== Future Work ==
*The twitter search is currently hard coded. The search accesses the msg_twitter_search table and use the keywords stored by the user through the eden portal to search and display twitter feeds on the map.
*Display the twitter icon for twitter search instead of the usual bubble
*Work on twitter links and message which would pop up contents like photos,URL etc. This work is doing the part within the tweet parsing of following embedded URLs to identify if they're Media would be really useful addition to the parser (there is a commented-out stub).
*Create a cron job for searching tweets periodically instead of manual searches as per user’s request.
*Current implementation includes integration of Sahana with Twitter. The future enhancements of the eden project includes allow Sahana Eden to be configured to have a Facebook updates in it

== Summary ==
The Sahana Eden Foundation developed for Disaster Management provides a web platform to provide solution for the relief effort. The project provides a solution to manage organizations, people, projects, inventory and assets as well as collecting information through assessments and providing situational awareness through maps.
Hence, enhancing the searches and reports will provide great help to Disaster Management.

== References ==

<references/>

== Further Reading ==
# ''Getting Started''. Retrieved from https://dev.twitter.com/start
#''Sahana Eden Project''. Retrieved from http://eden.sahanafoundation.org/wiki/WikiStart
#''Installation Guidelines''. Retrieved from http://eden.sahanafoundation.org/wiki/InstallationGuidelines

== Notes ==
* All references and further reading links are in the APA6 style
* All the content on this page is from websites that have a Attribution-NonCommercial-NoDerivs 3.0 or similar license that allows us to use their content.

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T04:58:04Z

Grmenon: Created page with "== Introduction to Eden == == Framework - Web2py == == S3 == == Openshift Deployment == == Confgiuration - Twitter == == Twitter Search Results Filter == == Twitter Search ..."

== Introduction to Eden ==

== Framework - Web2py ==

== S3 ==

== Openshift Deployment ==

== Confgiuration - Twitter ==

== Twitter Search Results Filter ==

== Twitter Search After Save ==

== Twitter Search Results Report ==

== Further Work ==

== Summary ==

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-18T16:57:12Z

Grmenon: /* Further Reading/References */

= Introduction =

Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size and complexity automated tools are becoming the norm as opposed to the exception.

Here we shall compare two popular Infrastructure Management Tools-

* Puppet
* Chef

We shall begin with an initial overview of both tools to familiarize the reader. We will then take a hypothetical scenarion and try and solve the problems in those scenario using Puppet and Chef.

We will then compare the solutions and compare how each tool handles the same problem. We do not intend to categorically state the superiority or suitability of one tool over the other but to illustrate their usage and let the reader choose them as per her/his need.

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

[[File:Puppet1.png]]

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}

An example would be -

service { 'rails s':
ensure => 'running'
}

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

[http://www.opscode.com/chef/ Opscode Chef] is an infrastructure automation and configuration management framework.

Chef is used for configuration, deployment and scalability of servers and applications for eg: it can be used to deploy and manage network in cloud, on-site or a hybrid one with n number of servers. This helps organization to save time, solve mission-critical challenges, accelerate time to market, manage scale and complexity, and safeguard systems.

Chef client works using the abstraction definitions know as cookbooks and recipes that are written in Ruby. Chef client uses this definitions to build and manage servers in an infrastructure.
== Elements ==
Basically an organization consists of following elements with chef configuration:

# [http://docs.opscode.com/essentials_node_object.html Node] : Any physical, virtual server is referred to as node.
# [http://docs.opscode.com/knife.html Knife] : An interface between the local chef-repo and the server is known as knife. It is a command line tool using which users can manage other elements of the setup.
# [http://docs.opscode.com/chef_overview.html#workstations Workstation] : It is a computer used to interact with the single server. It is also used by users to run the knife command.
# [http://docs.opscode.com/chef_quick_overview.html#repository Repository] : usually referred as chef repo is the location wher all data objects are stored like cookbooks, configuration files etc.
# [http://docs.opscode.com/chef_quick_overview.html#the-hosted-server Hoster server]: The server manages cookbooks and all the configuration details that need to be applied to nodes.
# [http://docs.opscode.com/essentials_chef_client.html Chef-client] : Chef client acts as an interface between the node and hosted server. Node gets all the configuration details from server via the client.
# [http://docs.opscode.com/chef_quick_overview.html#cookbooks Cookbook] : Cookbook stores the configuration details depending upon the scenario for eg: in order to install Apache Tomcat server it contains details about installation, upgradation, components needed to support the server.

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===
<ol style="margin-left: 5.2em;">
<li>[http://docs.opscode.com/install_server.html Install chef server] on the server machine</li>
<li>[https://learnchef.opscode.com/quickstart/workstation-setup/ Setup workstation]:</li>
:* Pre-requisites : The workstation must have a [https://www.virtualbox.org/wiki/Downloads Virtual box] , [http://downloads.vagrantup.com/ Vagrant](command line interface tool), [http://git-scm.com/ Git]( needed for Version Control Systems)
:* Use knife client command to establish an API client identity for the workstation in order to make authenticated requests to the server. Setup [https://learnchef.opscode.com/quickstart/chef-repo/ chef-repo] in the workstation.
<li>[https://learnchef.opscode.com/quickstart/chef-repo/ Create Cookbooks]:</li>
:* In order to use community cookbooks one can just run the following command:

Syntax : knife cookbook site install <cook_name>

Example: knife cookbook site install apt.

:Note: Chef has maintained a list of community cookbooks which are commonly covered under the use cases like apache2, mysql,php. So a user can leverage the community cookbooks instead of creating their own cookbooks.

:* In order to create our own cookbook:

Syntax : knife cookbook create cookbook_name

:The file default.rb in the path chef-repo/cookbooks/aliases/recipes/default.rb can be edited in order to create recipes.

:To make these cookbooks available to the nodes, it is first uploaded in the chef server.
<li>[http://www.opscode.com/chef/install/ Setup chef client]: Using knife tool in workstation, chef clients are setup on nodes.</li>
</ol>

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a Node ==

Our sample project will be using MongoDB. So the next task will be setting up the MongoDB instances.

=== Puppet ===

As we have seen in Puppet a machine’s configuration is specified in terms of resources. To tell Puppet to set up those resources on a machine we use a Manifest that consists of various resources.

But setting up a Manifest that covers all the Resources will quickly become unwieldly.

Which is why Puppet uses Classes and Modules.

A Class is a named collection of resources. A Module encloses a Class and is something that can be included in other Puppet files thus enabling reuse. There is a large collection of Modules called the Puppet Forge that has been written covering different type of uses.

Thus setting up a manifest for a MongoDB server is as simple as-

class { ‘mongodb’ :
init => ‘sysv’,
version => ‘1.1.1’ # installing a specific version of mongodb
}
include mongodb
# custom puppet code as per our requirments

Thus different Classes can be written which use different Modules for different uses. We can define a different class for our web server , for our file share and so on.

Once a Class has been defined by us that is tailor made to our uses setting up more instances is simply a question of provisioning the node and running the Puppet class on it.

This supports quick and simple scaling up.

=== Chef ===

[http://docs.opscode.com/essentials_roles.html Configuration in Chef] is done by defining [http://docs.opscode.com/essentials_roles.html# Role]. Role helps in setting required attributes and tasks for a node. A node can have 0 or more roles assigned to it. Also a role can be assigned to many nodes. Roles thus helps in defining processes for a particular node.

Let us consider to [https://learnchef.opscode.com/starter-use-cases/wordpress/ configure] a role for mongoDb server for a application:
<ol style="margin-left: 5.2em;">
<li>A role can be created by creating a file under roles directory for eg : roles/role_file.rb</li>

<li>This file is composed of
:name : defines the node.
:description: describes about the node.
:attributes: attributes specific to the node which needs to be configured.
:run lists: tasks assigned to the node.
:Sample-
[[File:codeMongo.JPG]]

</li>
<li>This file is then uploaded to the chef server using following command:</li>

knife role from file roles/redis.rb

<li>Role is assigned to the server during [http://docs.opscode.com/install_bootstrap.html bootstrap] using command :</li>

knife ec2 server create

</li>

</ol>

=== How They Compare ===

''Similarities''

* Puppet and Chef both abstract the configuration of an instance in their own way - using Resources and Recipes respectively
* Both of them take care of the issue of setting up instances of different OS's - they handle the lower level details of actually installing and setting up packages etc
* Puppet and Chef both classify or group similar configurations using Classes/Modules and Roles respectively

''Differences''

* In Chef recipes and other specifications are done in Ruby whereas Puppet has it's own language with it's own file extension - .pp
* In normal/daily use the Puppet command is sufficient but for Chef the knife tool is more frequently used.

== Setting Up A Node ==

=== Puppet ===

In Puppet once we have identified what will go into defining something that we will need an instance for - as we have seen above say configuring a MongoDB instance - we need to select a fresh instance and apply the manifest to it.

As we have seen Puppet Agents will automatically sync with the Puppet Master. So the only task left for us is to

# Provision a new instance
# Install Puppet on it
# Classify it as a particular type of node.

The last two commands can be combined using the puppet init command-

puppet node init \
--node-group=db \
…. ( login related options)
ec2-71-73-79-83.compute-1.amazonaws.com

Here we have specified the fresh instance and initialized as belonging to the group of nodes as db.

Once this process is done the instance will be classified as a db instance and will automatically begin syncing with the Puppet Master.

=== Chef ===

Pre-requisites : IP address / [http://en.wikipedia.org/wiki/Fully_qualified_domain_name FQDN] of the node

Run the [http://docs.opscode.com/install_bootstrap.html knife bootstrap command]

The knife bootstrap command requires the FQDN or the IP address for the node in order to complete the bootstrap operation. Run the knife bootstrap command. It checks for [https://learnchef.opscode.com/screencasts/manage-pem-files/ .pem] files for authentication. If not present then the node goes through the registration phase in order to establish connection between the client and the server and for exchange of authorized messages. The attributes set in the role of the node are compared against the configuration details of the node and run contents of that role’s run list are applied to the node’s configuration details.

=== How They Compare ===

'''Similarities'''

* Both support integration with cloud service providers like Amazon to support direct setup and configuration of a fresh instance.
* Both need client software - the Puppet Agent and Chef Client respectively to be installed first to complete the configuration process

'''Differences'''

* Puppet uses node-groups and classes while initializing nodes whereas Chef directly uses roles and cookbooks.
* Puppet has Forge which contains a collection of resources whereas Chef has Community Cookbooks for the same task.

== Node Management ==

=== Foreman ===

Foreman is a popular tool that is used in conjunction with Puppet. It provides a Web based GUI that lets users manage Puppet nodes.

As an example the following screenshot illustrates how Foreman can be used to manage multiple nodes across multiple environments-

[[File:env.png]]

We can apply various filters here to narrow down on a particular instance or conversely modify multiple nodes at the same time.

=== Knife ===

Node management in Chef includes users to create, read, update and delete nodes. This can be done either by using [http://docs.opscode.com/manage_server_hosted_nodes.html Management Console] or the command line tool [http://docs.opscode.com/knife_node.html Knife].

[http://docs.opscode.com/knife_node.html Knife command] the following syntax in order to manage the nodes :

$ knife node [ARGUMENT] (options)

where ARGUMENT could be any one of the management options like bulk delete, create, edit, delete,list.

For example: to perform a bulk delete, a regular expression is prepared to identify the nodes to be deleted. This is then passed as the option.

$ knife node bulk delete "node^[0-3]{6}$"

User needs to have specific user level permission in order to do the specific task. For eg: You need create permissions on the global nodes level to create a node.

=== How They Compare ===

''Similarities''

* Both provide a simple interface to manage nodes using simple command or a service running in the background.

''Differences''

* Chef enlists the details while doing the required operation whereas puppet does not. It uses command -test to enlist details.

== Config Files Templatization ==

=== Puppet ===

In Puppet files are treated as resources. So we could specify for an NFS node as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => "/some/path 192.168.100.1(rw,sync)”,
}

However the content would be different for different nodes as the environment would change.

Instead of creating a different resource for each template Puppet supports templatization of config files using Embedded Ruby (ERB).

We can re-write the same resource as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => template(“nfs/mount_conf.erb”),
}

where the mount_conf.erb file would contain Embedded Ruby code along with the appropriate Ruby variables which would automatically be compiled to give the right values to produce the conf file.

We can apply this same manifest file to all nodes across all the instances.

=== Chef ===

[http://docs.opscode.com/resource_deploy.html Deploy] is one of the [http://docs.opscode.com/resource.html Resource] which manages and controls deployment of configuration files on the nodes. It specifies action to be taken during deployment phase. These actions are then put into action by [http://docs.opscode.com/resource.html Providers] which is mapped to resource deploy.

deploy "name" do # deploy directs chef client to the provider to be used for this resource,default being Chef::Provider::Deploy::Revision.
# name is the location in which the deployment steps will occur.
attribute "value" # this represents the attributes available for this resource.
...
callback do
# callback represents additional block or link to a file to provide additional information during deployment process.
end
...
purge_before_symlink
create_dirs_before_symlink
symlink # These commands are used to create directories, link configuration files, remove directories or map them.
action :action # this specifies the action to be taken by the provider.
end

=== How They Compare ===

''Similarities''

* Both use Embedded Ruby for templatizing config files
* Both allow files to be specified as the Ruby executed version of an erb
* Both allow variables to be included in the templates to be replaced by actual files when the config file is deployed

''Differences''

* In Puppet the template is specified explicitly as part of the resource type file ( and other similar resource types) whereas in Chef it can be specified anywhere.

= Other Infrastructure Management Tools =

There are many other infrastructure management tools-

# [https://juju.ubuntu.com/ Juju] - Ubuntu juju is a tool for virtual infrastructure automation
# [http://cfengine.com Cfengine] is a much older tool that is used for large scale infrastructure automation. Puppet was originally inspired by Cfengine.
# [http://saltstack.com/community.html SaltStack] - this is another configuration management system written in Python

This is by no means an exhaustive list and is one that is growing - reflecting the trend towards heavier and more all encompassing automation.

= Conclusion =

This comparison was not meant to tout one of the tools - Puppet or Chef - over the other but merely to introduce and contrast both of them and see how they can be used with standard Infrastructure Management scenarios.

It is up to the reader to judge his/her needs and apply the tool for the job.

= Further Reading/References =

# [http://docs.opscode.com/chef_overview.html Opscode Chef] 
# [http://docs.opscode.com/knife_node.html About Knife]
# [http://www.aosabook.org/en/puppet.html Puppet Master Agent Architecture]
# [http://docs.puppetlabs.com/learning/ral.html Puppet RAL]
# [http://docs.puppetlabs.com/learning/manifests.html Puppet Manifest]
# [http://docs.puppetlabs.com/learning/modules1.html Puppet Modules and Classes]
# [http://docs.puppetlabs.com/learning/templates.html Puppet Templates]
# [http://jeff.robbins.ws/articles/nfs-tutorial NFS Mounting]
# [http://theforeman.org/ Foreman]

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-18T16:44:22Z

Grmenon: /* Master / Agent Puppet */

= Introduction =

Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size and complexity automated tools are becoming the norm as opposed to the exception.

Here we shall compare two popular Infrastructure Management Tools-

* Puppet
* Chef

We shall begin with an initial overview of both tools to familiarize the reader. We will then take a hypothetical scenarion and try and solve the problems in those scenario using Puppet and Chef.

We will then compare the solutions and compare how each tool handles the same problem. We do not intend to categorically state the superiority or suitability of one tool over the other but to illustrate their usage and let the reader choose them as per her/his need.

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

[[File:Puppet1.png]]

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}

An example would be -

service { 'rails s':
ensure => 'running'
}

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

[http://www.opscode.com/chef/ Opscode Chef] is an infrastructure automation and configuration management framework.

Chef is used for configuration, deployment and scalability of servers and applications for eg: it can be used to deploy and manage network in cloud, on-site or a hybrid one with n number of servers. This helps organization to save time, solve mission-critical challenges, accelerate time to market, manage scale and complexity, and safeguard systems.

Chef client works using the abstraction definitions know as cookbooks and recipes that are written in Ruby. Chef client uses this definitions to build and manage servers in an infrastructure.
== Elements ==
Basically an organization consists of following elements with chef configuration:

# [http://docs.opscode.com/essentials_node_object.html Node] : Any physical, virtual server is referred to as node.
# [http://docs.opscode.com/knife.html Knife] : An interface between the local chef-repo and the server is known as knife. It is a command line tool using which users can manage other elements of the setup.
# [http://docs.opscode.com/chef_overview.html#workstations Workstation] : It is a computer used to interact with the single server. It is also used by users to run the knife command.
# [http://docs.opscode.com/chef_quick_overview.html#repository Repository] : usually referred as chef repo is the location wher all data objects are stored like cookbooks, configuration files etc.
# [http://docs.opscode.com/chef_quick_overview.html#the-hosted-server Hoster server]: The server manages cookbooks and all the configuration details that need to be applied to nodes.
# [http://docs.opscode.com/essentials_chef_client.html Chef-client] : Chef client acts as an interface between the node and hosted server. Node gets all the configuration details from server via the client.
# [http://docs.opscode.com/chef_quick_overview.html#cookbooks Cookbook] : Cookbook stores the configuration details depending upon the scenario for eg: in order to install Apache Tomcat server it contains details about installation, upgradation, components needed to support the server.

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===
<ol style="margin-left: 5.2em;">
<li>[http://docs.opscode.com/install_server.html Install chef server] on the server machine</li>
<li>[https://learnchef.opscode.com/quickstart/workstation-setup/ Setup workstation]:</li>
:* Pre-requisites : The workstation must have a [https://www.virtualbox.org/wiki/Downloads Virtual box] , [http://downloads.vagrantup.com/ Vagrant](command line interface tool), [http://git-scm.com/ Git]( needed for Version Control Systems)
:* Use knife client command to establish an API client identity for the workstation in order to make authenticated requests to the server. Setup [https://learnchef.opscode.com/quickstart/chef-repo/ chef-repo] in the workstation.
<li>[https://learnchef.opscode.com/quickstart/chef-repo/ Create Cookbooks]:</li>
:* In order to use community cookbooks one can just run the following command:

Syntax : knife cookbook site install <cook_name>

Example: knife cookbook site install apt.

:Note: Chef has maintained a list of community cookbooks which are commonly covered under the use cases like apache2, mysql,php. So a user can leverage the community cookbooks instead of creating their own cookbooks.

:* In order to create our own cookbook:

Syntax : knife cookbook create cookbook_name

:The file default.rb in the path chef-repo/cookbooks/aliases/recipes/default.rb can be edited in order to create recipes.

:To make these cookbooks available to the nodes, it is first uploaded in the chef server.
<li>[http://www.opscode.com/chef/install/ Setup chef client]: Using knife tool in workstation, chef clients are setup on nodes.</li>
</ol>

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a Node ==

Our sample project will be using MongoDB. So the next task will be setting up the MongoDB instances.

=== Puppet ===

As we have seen in Puppet a machine’s configuration is specified in terms of resources. To tell Puppet to set up those resources on a machine we use a Manifest that consists of various resources.

But setting up a Manifest that covers all the Resources will quickly become unwieldly.

Which is why Puppet uses Classes and Modules.

A Class is a named collection of resources. A Module encloses a Class and is something that can be included in other Puppet files thus enabling reuse. There is a large collection of Modules called the Puppet Forge that has been written covering different type of uses.

Thus setting up a manifest for a MongoDB server is as simple as-

class { ‘mongodb’ :
init => ‘sysv’,
version => ‘1.1.1’ # installing a specific version of mongodb
}
include mongodb
# custom puppet code as per our requirments

Thus different Classes can be written which use different Modules for different uses. We can define a different class for our web server , for our file share and so on.

Once a Class has been defined by us that is tailor made to our uses setting up more instances is simply a question of provisioning the node and running the Puppet class on it.

This supports quick and simple scaling up.

=== Chef ===

[http://docs.opscode.com/essentials_roles.html Configuration in Chef] is done by defining [http://docs.opscode.com/essentials_roles.html# Role]. Role helps in setting required attributes and tasks for a node. A node can have 0 or more roles assigned to it. Also a role can be assigned to many nodes. Roles thus helps in defining processes for a particular node.

Let us consider to [https://learnchef.opscode.com/starter-use-cases/wordpress/ configure] a role for mongoDb server for a application:
<ol style="margin-left: 5.2em;">
<li>A role can be created by creating a file under roles directory for eg : roles/role_file.rb</li>

<li>This file is composed of
:name : defines the node.
:description: describes about the node.
:attributes: attributes specific to the node which needs to be configured.
:run lists: tasks assigned to the node.
:Sample-
[[File:codeMongo.JPG]]

</li>
<li>This file is then uploaded to the chef server using following command:</li>

knife role from file roles/redis.rb

<li>Role is assigned to the server during [http://docs.opscode.com/install_bootstrap.html bootstrap] using command :</li>

knife ec2 server create

</li>

</ol>

=== How They Compare ===

''Similarities''

* Puppet and Chef both abstract the configuration of an instance in their own way - using Resources and Recipes respectively
* Both of them take care of the issue of setting up instances of different OS's - they handle the lower level details of actually installing and setting up packages etc
* Puppet and Chef both classify or group similar configurations using Classes/Modules and Roles respectively

''Differences''

* In Chef recipes and other specifications are done in Ruby whereas Puppet has it's own language with it's own file extension - .pp
* In normal/daily use the Puppet command is sufficient but for Chef the knife tool is more frequently used.

== Setting Up A Node ==

=== Puppet ===

In Puppet once we have identified what will go into defining something that we will need an instance for - as we have seen above say configuring a MongoDB instance - we need to select a fresh instance and apply the manifest to it.

As we have seen Puppet Agents will automatically sync with the Puppet Master. So the only task left for us is to

# Provision a new instance
# Install Puppet on it
# Classify it as a particular type of node.

The last two commands can be combined using the puppet init command-

puppet node init \
--node-group=db \
…. ( login related options)
ec2-71-73-79-83.compute-1.amazonaws.com

Here we have specified the fresh instance and initialized as belonging to the group of nodes as db.

Once this process is done the instance will be classified as a db instance and will automatically begin syncing with the Puppet Master.

=== Chef ===

Pre-requisites : IP address / [http://en.wikipedia.org/wiki/Fully_qualified_domain_name FQDN] of the node

Run the [http://docs.opscode.com/install_bootstrap.html knife bootstrap command]

The knife bootstrap command requires the FQDN or the IP address for the node in order to complete the bootstrap operation. Run the knife bootstrap command. It checks for [https://learnchef.opscode.com/screencasts/manage-pem-files/ .pem] files for authentication. If not present then the node goes through the registration phase in order to establish connection between the client and the server and for exchange of authorized messages. The attributes set in the role of the node are compared against the configuration details of the node and run contents of that role’s run list are applied to the node’s configuration details.

=== How They Compare ===

'''Similarities'''

* Both support integration with cloud service providers like Amazon to support direct setup and configuration of a fresh instance.
* Both need client software - the Puppet Agent and Chef Client respectively to be installed first to complete the configuration process

'''Differences'''

* Puppet uses node-groups and classes while initializing nodes whereas Chef directly uses roles and cookbooks.
* Puppet has Forge which contains a collection of resources whereas Chef has Community Cookbooks for the same task.

== Node Management ==

=== Foreman ===

Foreman is a popular tool that is used in conjunction with Puppet. It provides a Web based GUI that lets users manage Puppet nodes.

As an example the following screenshot illustrates how Foreman can be used to manage multiple nodes across multiple environments-

[[File:env.png]]

We can apply various filters here to narrow down on a particular instance or conversely modify multiple nodes at the same time.

=== Knife ===

Node management in Chef includes users to create, read, update and delete nodes. This can be done either by using [http://docs.opscode.com/manage_server_hosted_nodes.html Management Console] or the command line tool [http://docs.opscode.com/knife_node.html Knife].

[http://docs.opscode.com/knife_node.html Knife command] the following syntax in order to manage the nodes :

$ knife node [ARGUMENT] (options)

where ARGUMENT could be any one of the management options like bulk delete, create, edit, delete,list.

For example: to perform a bulk delete, a regular expression is prepared to identify the nodes to be deleted. This is then passed as the option.

$ knife node bulk delete "node^[0-3]{6}$"

User needs to have specific user level permission in order to do the specific task. For eg: You need create permissions on the global nodes level to create a node.

=== How They Compare ===

''Similarities''

* Both provide a simple interface to manage nodes using simple command or a service running in the background.

''Differences''

* Chef enlists the details while doing the required operation whereas puppet does not. It uses command -test to enlist details.

== Config Files Templatization ==

=== Puppet ===

In Puppet files are treated as resources. So we could specify for an NFS node as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => "/some/path 192.168.100.1(rw,sync)”,
}

However the content would be different for different nodes as the environment would change.

Instead of creating a different resource for each template Puppet supports templatization of config files using Embedded Ruby (ERB).

We can re-write the same resource as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => template(“nfs/mount_conf.erb”),
}

where the mount_conf.erb file would contain Embedded Ruby code along with the appropriate Ruby variables which would automatically be compiled to give the right values to produce the conf file.

We can apply this same manifest file to all nodes across all the instances.

=== Chef ===

[http://docs.opscode.com/resource_deploy.html Deploy] is one of the [http://docs.opscode.com/resource.html Resource] which manages and controls deployment of configuration files on the nodes. It specifies action to be taken during deployment phase. These actions are then put into action by [http://docs.opscode.com/resource.html Providers] which is mapped to resource deploy.

deploy "name" do # deploy directs chef client to the provider to be used for this resource,default being Chef::Provider::Deploy::Revision.
# name is the location in which the deployment steps will occur.
attribute "value" # this represents the attributes available for this resource.
...
callback do
# callback represents additional block or link to a file to provide additional information during deployment process.
end
...
purge_before_symlink
create_dirs_before_symlink
symlink # These commands are used to create directories, link configuration files, remove directories or map them.
action :action # this specifies the action to be taken by the provider.
end

=== How They Compare ===

''Similarities''

* Both use Embedded Ruby for templatizing config files
* Both allow files to be specified as the Ruby executed version of an erb
* Both allow variables to be included in the templates to be replaced by actual files when the config file is deployed

''Differences''

* In Puppet the template is specified explicitly as part of the resource type file ( and other similar resource types) whereas in Chef it can be specified anywhere.

= Other Infrastructure Management Tools =

There are many other infrastructure management tools-

# [https://juju.ubuntu.com/ Juju] - Ubuntu juju is a tool for virtual infrastructure automation
# [http://cfengine.com Cfengine] is a much older tool that is used for large scale infrastructure automation. Puppet was originally inspired by Cfengine.
# [http://saltstack.com/community.html SaltStack] - this is another configuration management system written in Python

This is by no means an exhaustive list and is one that is growing - reflecting the trend towards heavier and more all encompassing automation.

= Conclusion =

This comparison was not meant to tout one of the tools - Puppet or Chef - over the other but merely to introduce and contrast both of them and see how they can be used with standard Infrastructure Management scenarios.

It is up to the reader to judge his/her needs and apply the tool for the job.

= Further Reading/References =
1. [http://docs.opscode.com/chef_overview.html Opscode Chef] 
2. [http://docs.opscode.com/knife_node.html About Knife]

File:Puppet1.png

2013-09-18T16:44:02Z

Grmenon:

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-18T16:42:14Z

Grmenon: /* Master / Agent Puppet */

= Introduction =

Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size and complexity automated tools are becoming the norm as opposed to the exception.

Here we shall compare two popular Infrastructure Management Tools-

* Puppet
* Chef

We shall begin with an initial overview of both tools to familiarize the reader. We will then take a hypothetical scenarion and try and solve the problems in those scenario using Puppet and Chef.

We will then compare the solutions and compare how each tool handles the same problem. We do not intend to categorically state the superiority or suitability of one tool over the other but to illustrate their usage and let the reader choose them as per her/his need.

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

[[File:puppet.png]]

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}

An example would be -

service { 'rails s':
ensure => 'running'
}

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

[http://www.opscode.com/chef/ Opscode Chef] is an infrastructure automation and configuration management framework.

Chef is used for configuration, deployment and scalability of servers and applications for eg: it can be used to deploy and manage network in cloud, on-site or a hybrid one with n number of servers. This helps organization to save time, solve mission-critical challenges, accelerate time to market, manage scale and complexity, and safeguard systems.

Chef client works using the abstraction definitions know as cookbooks and recipes that are written in Ruby. Chef client uses this definitions to build and manage servers in an infrastructure.
== Elements ==
Basically an organization consists of following elements with chef configuration:

# [http://docs.opscode.com/essentials_node_object.html Node] : Any physical, virtual server is referred to as node.
# [http://docs.opscode.com/knife.html Knife] : An interface between the local chef-repo and the server is known as knife. It is a command line tool using which users can manage other elements of the setup.
# [http://docs.opscode.com/chef_overview.html#workstations Workstation] : It is a computer used to interact with the single server. It is also used by users to run the knife command.
# [http://docs.opscode.com/chef_quick_overview.html#repository Repository] : usually referred as chef repo is the location wher all data objects are stored like cookbooks, configuration files etc.
# [http://docs.opscode.com/chef_quick_overview.html#the-hosted-server Hoster server]: The server manages cookbooks and all the configuration details that need to be applied to nodes.
# [http://docs.opscode.com/essentials_chef_client.html Chef-client] : Chef client acts as an interface between the node and hosted server. Node gets all the configuration details from server via the client.
# [http://docs.opscode.com/chef_quick_overview.html#cookbooks Cookbook] : Cookbook stores the configuration details depending upon the scenario for eg: in order to install Apache Tomcat server it contains details about installation, upgradation, components needed to support the server.

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===
<ol style="margin-left: 5.2em;">
<li>[http://docs.opscode.com/install_server.html Install chef server] on the server machine</li>
<li>[https://learnchef.opscode.com/quickstart/workstation-setup/ Setup workstation]:</li>
:* Pre-requisites : The workstation must have a [https://www.virtualbox.org/wiki/Downloads Virtual box] , [http://downloads.vagrantup.com/ Vagrant](command line interface tool), [http://git-scm.com/ Git]( needed for Version Control Systems)
:* Use knife client command to establish an API client identity for the workstation in order to make authenticated requests to the server. Setup [https://learnchef.opscode.com/quickstart/chef-repo/ chef-repo] in the workstation.
<li>[https://learnchef.opscode.com/quickstart/chef-repo/ Create Cookbooks]:</li>
:* In order to use community cookbooks one can just run the following command:

Syntax : knife cookbook site install <cook_name>

Example: knife cookbook site install apt.

:Note: Chef has maintained a list of community cookbooks which are commonly covered under the use cases like apache2, mysql,php. So a user can leverage the community cookbooks instead of creating their own cookbooks.

:* In order to create our own cookbook:

Syntax : knife cookbook create cookbook_name

:The file default.rb in the path chef-repo/cookbooks/aliases/recipes/default.rb can be edited in order to create recipes.

:To make these cookbooks available to the nodes, it is first uploaded in the chef server.
<li>[http://www.opscode.com/chef/install/ Setup chef client]: Using knife tool in workstation, chef clients are setup on nodes.</li>
</ol>

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a Node ==

Our sample project will be using MongoDB. So the next task will be setting up the MongoDB instances.

=== Puppet ===

As we have seen in Puppet a machine’s configuration is specified in terms of resources. To tell Puppet to set up those resources on a machine we use a Manifest that consists of various resources.

But setting up a Manifest that covers all the Resources will quickly become unwieldly.

Which is why Puppet uses Classes and Modules.

A Class is a named collection of resources. A Module encloses a Class and is something that can be included in other Puppet files thus enabling reuse. There is a large collection of Modules called the Puppet Forge that has been written covering different type of uses.

Thus setting up a manifest for a MongoDB server is as simple as-

class { ‘mongodb’ :
init => ‘sysv’,
version => ‘1.1.1’ # installing a specific version of mongodb
}
include mongodb
# custom puppet code as per our requirments

Thus different Classes can be written which use different Modules for different uses. We can define a different class for our web server , for our file share and so on.

Once a Class has been defined by us that is tailor made to our uses setting up more instances is simply a question of provisioning the node and running the Puppet class on it.

This supports quick and simple scaling up.

=== Chef ===

[http://docs.opscode.com/essentials_roles.html Configuration in Chef] is done by defining [http://docs.opscode.com/essentials_roles.html# Role]. Role helps in setting required attributes and tasks for a node. A node can have 0 or more roles assigned to it. Also a role can be assigned to many nodes. Roles thus helps in defining processes for a particular node.

Let us consider to [https://learnchef.opscode.com/starter-use-cases/wordpress/ configure] a role for mongoDb server for a application:
<ol style="margin-left: 5.2em;">
<li>A role can be created by creating a file under roles directory for eg : roles/role_file.rb</li>

<li>This file is composed of
:name : defines the node.
:description: describes about the node.
:attributes: attributes specific to the node which needs to be configured.
:run lists: tasks assigned to the node.
:Sample-
[[File:codeMongo.JPG]]

</li>
<li>This file is then uploaded to the chef server using following command:</li>

knife role from file roles/redis.rb

<li>Role is assigned to the server during [http://docs.opscode.com/install_bootstrap.html bootstrap] using command :</li>

knife ec2 server create

</li>

</ol>

=== How They Compare ===

''Similarities''

* Puppet and Chef both abstract the configuration of an instance in their own way - using Resources and Recipes respectively
* Both of them take care of the issue of setting up instances of different OS's - they handle the lower level details of actually installing and setting up packages etc
* Puppet and Chef both classify or group similar configurations using Classes/Modules and Roles respectively

''Differences''

* In Chef recipes and other specifications are done in Ruby whereas Puppet has it's own language with it's own file extension - .pp
* In normal/daily use the Puppet command is sufficient but for Chef the knife tool is more frequently used.

== Setting Up A Node ==

=== Puppet ===

In Puppet once we have identified what will go into defining something that we will need an instance for - as we have seen above say configuring a MongoDB instance - we need to select a fresh instance and apply the manifest to it.

As we have seen Puppet Agents will automatically sync with the Puppet Master. So the only task left for us is to

# Provision a new instance
# Install Puppet on it
# Classify it as a particular type of node.

The last two commands can be combined using the puppet init command-

puppet node init \
--node-group=db \
…. ( login related options)
ec2-71-73-79-83.compute-1.amazonaws.com

Here we have specified the fresh instance and initialized as belonging to the group of nodes as db.

Once this process is done the instance will be classified as a db instance and will automatically begin syncing with the Puppet Master.

=== Chef ===

Pre-requisites : IP address / [http://en.wikipedia.org/wiki/Fully_qualified_domain_name FQDN] of the node

Run the [http://docs.opscode.com/install_bootstrap.html knife bootstrap command]

The knife bootstrap command requires the FQDN or the IP address for the node in order to complete the bootstrap operation. Run the knife bootstrap command. It checks for [https://learnchef.opscode.com/screencasts/manage-pem-files/ .pem] files for authentication. If not present then the node goes through the registration phase in order to establish connection between the client and the server and for exchange of authorized messages. The attributes set in the role of the node are compared against the configuration details of the node and run contents of that role’s run list are applied to the node’s configuration details.

=== How They Compare ===

'''Similarities'''

* Both support integration with cloud service providers like Amazon to support direct setup and configuration of a fresh instance.
* Both need client software - the Puppet Agent and Chef Client respectively to be installed first to complete the configuration process

'''Differences'''

* Puppet uses node-groups and classes while initializing nodes whereas Chef directly uses roles and cookbooks.
* Puppet has Forge which contains a collection of resources whereas Chef has Community Cookbooks for the same task.

== Node Management ==

=== Foreman ===

Foreman is a popular tool that is used in conjunction with Puppet. It provides a Web based GUI that lets users manage Puppet nodes.

As an example the following screenshot illustrates how Foreman can be used to manage multiple nodes across multiple environments-

[[File:env.png]]

We can apply various filters here to narrow down on a particular instance or conversely modify multiple nodes at the same time.

=== Knife ===

Node management in Chef includes users to create, read, update and delete nodes. This can be done either by using [http://docs.opscode.com/manage_server_hosted_nodes.html Management Console] or the command line tool [http://docs.opscode.com/knife_node.html Knife].

[http://docs.opscode.com/knife_node.html Knife command] the following syntax in order to manage the nodes :

$ knife node [ARGUMENT] (options)

where ARGUMENT could be any one of the management options like bulk delete, create, edit, delete,list.

For example: to perform a bulk delete, a regular expression is prepared to identify the nodes to be deleted. This is then passed as the option.

$ knife node bulk delete "node^[0-3]{6}$"

User needs to have specific user level permission in order to do the specific task. For eg: You need create permissions on the global nodes level to create a node.

=== How They Compare ===

''Similarities''

* Both provide a simple interface to manage nodes using simple command or a service running in the background.

''Differences''

* Chef enlists the details while doing the required operation whereas puppet does not. It uses command -test to enlist details.

== Config Files Templatization ==

=== Puppet ===

In Puppet files are treated as resources. So we could specify for an NFS node as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => "/some/path 192.168.100.1(rw,sync)”,
}

However the content would be different for different nodes as the environment would change.

Instead of creating a different resource for each template Puppet supports templatization of config files using Embedded Ruby (ERB).

We can re-write the same resource as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => template(“nfs/mount_conf.erb”),
}

where the mount_conf.erb file would contain Embedded Ruby code along with the appropriate Ruby variables which would automatically be compiled to give the right values to produce the conf file.

We can apply this same manifest file to all nodes across all the instances.

=== Chef ===

[http://docs.opscode.com/resource_deploy.html Deploy] is one of the [http://docs.opscode.com/resource.html Resource] which manages and controls deployment of configuration files on the nodes. It specifies action to be taken during deployment phase. These actions are then put into action by [http://docs.opscode.com/resource.html Providers] which is mapped to resource deploy.

deploy "name" do # deploy directs chef client to the provider to be used for this resource,default being Chef::Provider::Deploy::Revision.
# name is the location in which the deployment steps will occur.
attribute "value" # this represents the attributes available for this resource.
...
callback do
# callback represents additional block or link to a file to provide additional information during deployment process.
end
...
purge_before_symlink
create_dirs_before_symlink
symlink # These commands are used to create directories, link configuration files, remove directories or map them.
action :action # this specifies the action to be taken by the provider.
end

=== How They Compare ===

''Similarities''

* Both use Embedded Ruby for templatizing config files
* Both allow files to be specified as the Ruby executed version of an erb
* Both allow variables to be included in the templates to be replaced by actual files when the config file is deployed

''Differences''

* In Puppet the template is specified explicitly as part of the resource type file ( and other similar resource types) whereas in Chef it can be specified anywhere.

= Other Infrastructure Management Tools =

There are many other infrastructure management tools-

# [https://juju.ubuntu.com/ Juju] - Ubuntu juju is a tool for virtual infrastructure automation
# [http://cfengine.com Cfengine] is a much older tool that is used for large scale infrastructure automation. Puppet was originally inspired by Cfengine.
# [http://saltstack.com/community.html SaltStack] - this is another configuration management system written in Python

This is by no means an exhaustive list and is one that is growing - reflecting the trend towards heavier and more all encompassing automation.

= Conclusion =

This comparison was not meant to tout one of the tools - Puppet or Chef - over the other but merely to introduce and contrast both of them and see how they can be used with standard Infrastructure Management scenarios.

It is up to the reader to judge his/her needs and apply the tool for the job.

= Further Reading/References =
1. [http://docs.opscode.com/chef_overview.html Opscode Chef] 
2. [http://docs.opscode.com/knife_node.html About Knife]

File:Puppet.png

2013-09-18T16:41:35Z

Grmenon:

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-18T16:26:22Z

Grmenon: /* How They Compare */

= Introduction =

Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size and complexity automated tools are becoming the norm as opposed to the exception.

Here we shall compare two popular Infrastructure Management Tools-

* Puppet
* Chef

We shall begin with an initial overview of both tools to familiarize the reader. We will then take a hypothetical scenarion and try and solve the problems in those scenario using Puppet and Chef.

We will then compare the solutions and compare how each tool handles the same problem. We do not intend to categorically state the superiority or suitability of one tool over the other but to illustrate their usage and let the reader choose them as per her/his need.

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

INSERT DIAGRAM

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}

An example would be -

service { 'rails s':
ensure => 'running'
}

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

[http://www.opscode.com/chef/ Opscode Chef] is an infrastructure automation and configuration management framework.

Chef is used for configuration, deployment and scalability of servers and applications for eg: it can be used to deploy and manage network in cloud, on-site or a hybrid one with n number of servers. This helps organization to save time, solve mission-critical challenges, accelerate time to market, manage scale and complexity, and safeguard systems.

Chef client works using the abstraction definitions know as cookbooks and recipes that are written in Ruby. Chef client uses this definitions to build and manage servers in an infrastructure.
== Elements ==
Basically an organization consists of following elements with chef configuration:

# [http://docs.opscode.com/essentials_node_object.html Node] : Any physical, virtual server is referred to as node.
# [http://docs.opscode.com/knife.html Knife] : An interface between the local chef-repo and the server is known as knife. It is a command line tool using which users can manage other elements of the setup.
# [http://docs.opscode.com/chef_overview.html#workstations Workstation] : It is a computer used to interact with the single server. It is also used by users to run the knife command.
# [http://docs.opscode.com/chef_quick_overview.html#repository Repository] : usually referred as chef repo is the location wher all data objects are stored like cookbooks, configuration files etc.
# [http://docs.opscode.com/chef_quick_overview.html#the-hosted-server Hoster server]: The server manages cookbooks and all the configuration details that need to be applied to nodes.
# [http://docs.opscode.com/essentials_chef_client.html Chef-client] : Chef client acts as an interface between the node and hosted server. Node gets all the configuration details from server via the client.
# [http://docs.opscode.com/chef_quick_overview.html#cookbooks Cookbook] : Cookbook stores the configuration details depending upon the scenario for eg: in order to install Apache Tomcat server it contains details about installation, upgradation, components needed to support the server.

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===
<ol style="margin-left: 5.2em;">
<li>[http://docs.opscode.com/install_server.html Install chef server] on the server machine</li>
<li>[https://learnchef.opscode.com/quickstart/workstation-setup/ Setup workstation]:</li>
:* Pre-requisites : The workstation must have a [https://www.virtualbox.org/wiki/Downloads Virtual box] , [http://downloads.vagrantup.com/ Vagrant](command line interface tool), [http://git-scm.com/ Git]( needed for Version Control Systems)
:* Use knife client command to establish an API client identity for the workstation in order to make authenticated requests to the server. Setup [https://learnchef.opscode.com/quickstart/chef-repo/ chef-repo] in the workstation.
<li>[https://learnchef.opscode.com/quickstart/chef-repo/ Create Cookbooks]:</li>
:* In order to use community cookbooks one can just run the following command:

Syntax : knife cookbook site install <cook_name>

Example: knife cookbook site install apt.

:Note: Chef has maintained a list of community cookbooks which are commonly covered under the use cases like apache2, mysql,php. So a user can leverage the community cookbooks instead of creating their own cookbooks.

:* In order to create our own cookbook:

Syntax : knife cookbook create cookbook_name

:The file default.rb in the path chef-repo/cookbooks/aliases/recipes/default.rb can be edited in order to create recipes.

:To make these cookbooks available to the nodes, it is first uploaded in the chef server.
<li>[http://www.opscode.com/chef/install/ Setup chef client]: Using knife tool in workstation, chef clients are setup on nodes.</li>
</ol>

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a Node ==

Our sample project will be using MongoDB. So the next task will be setting up the MongoDB instances.

=== Puppet ===

As we have seen in Puppet a machine’s configuration is specified in terms of resources. To tell Puppet to set up those resources on a machine we use a Manifest that consists of various resources.

But setting up a Manifest that covers all the Resources will quickly become unwieldly.

Which is why Puppet uses Classes and Modules.

A Class is a named collection of resources. A Module encloses a Class and is something that can be included in other Puppet files thus enabling reuse. There is a large collection of Modules called the Puppet Forge that has been written covering different type of uses.

Thus setting up a manifest for a MongoDB server is as simple as-

class { ‘mongodb’ :
init => ‘sysv’,
version => ‘1.1.1’ # installing a specific version of mongodb
}
include mongodb
# custom puppet code as per our requirments

Thus different Classes can be written which use different Modules for different uses. We can define a different class for our web server , for our file share and so on.

Once a Class has been defined by us that is tailor made to our uses setting up more instances is simply a question of provisioning the node and running the Puppet class on it.

This supports quick and simple scaling up.

=== Chef ===

[http://docs.opscode.com/essentials_roles.html Configuration in Chef] is done by defining [http://docs.opscode.com/essentials_roles.html# Role]. Role helps in setting required attributes and tasks for a node. A node can have 0 or more roles assigned to it. Also a role can be assigned to many nodes. Roles thus helps in defining processes for a particular node.

Let us consider to [https://learnchef.opscode.com/starter-use-cases/wordpress/ configure] a role for mongoDb server for a application:
<ol style="margin-left: 5.2em;">
<li>A role can be created by creating a file under roles directory for eg : roles/role_file.rb</li>

<li>This file is composed of
:name : defines the node.
:description: describes about the node.
:attributes: attributes specific to the node which needs to be configured.
:run lists: tasks assigned to the node.
:Sample-
[[File:codeMongo.JPG]]

</li>
<li>This file is then uploaded to the chef server using following command:</li>

knife role from file roles/redis.rb

<li>Role is assigned to the server during [http://docs.opscode.com/install_bootstrap.html bootstrap] using command :</li>

knife ec2 server create

</li>

</ol>

=== How They Compare ===

''Similarities''

* Puppet and Chef both abstract the configuration of an instance in their own way - using Resources and Recipes respectively
* Both of them take care of the issue of setting up instances of different OS's - they handle the lower level details of actually installing and setting up packages etc
* Puppet and Chef both classify or group similar configurations using Classes/Modules and Roles respectively

''Differences''

* In Chef recipes and other specifications are done in Ruby whereas Puppet has it's own language with it's own file extension - .pp
* In normal/daily use the Puppet command is sufficient but for Chef the knife tool is more frequently used.

== Setting Up A Node ==

=== Puppet ===

In Puppet once we have identified what will go into defining something that we will need an instance for - as we have seen above say configuring a MongoDB instance - we need to select a fresh instance and apply the manifest to it.

As we have seen Puppet Agents will automatically sync with the Puppet Master. So the only task left for us is to

# Provision a new instance
# Install Puppet on it
# Classify it as a particular type of node.

The last two commands can be combined using the puppet init command-

puppet node init \
--node-group=db \
…. ( login related options)
ec2-71-73-79-83.compute-1.amazonaws.com

Here we have specified the fresh instance and initialized as belonging to the group of nodes as db.

Once this process is done the instance will be classified as a db instance and will automatically begin syncing with the Puppet Master.

=== Chef ===

Pre-requisites : IP address / [http://en.wikipedia.org/wiki/Fully_qualified_domain_name FQDN] of the node

Run the [http://docs.opscode.com/install_bootstrap.html knife bootstrap command]

The knife bootstrap command requires the FQDN or the IP address for the node in order to complete the bootstrap operation. Run the knife bootstrap command. It checks for [https://learnchef.opscode.com/screencasts/manage-pem-files/ .pem] files for authentication. If not present then the node goes through the registration phase in order to establish connection between the client and the server and for exchange of authorized messages. The attributes set in the role of the node are compared against the configuration details of the node and run contents of that role’s run list are applied to the node’s configuration details.

=== How They Compare ===

'''Similarities'''

* Both support integration with cloud service providers like Amazon to support direct setup and configuration of a fresh instance.
* Both need client software - the Puppet Agent and Chef Client respectively to be installed first to complete the configuration process

'''Differences'''

* Puppet uses node-groups and classes while initializing nodes whereas Chef directly uses roles and cookbooks.
* Puppet has Forge which contains a collection of resources whereas Chef has Community Cookbooks for the same task.

== Node Management ==

=== Foreman ===

Foreman is a popular tool that is used in conjunction with Puppet. It provides a Web based GUI that lets users manage Puppet nodes.

As an example the following screenshot illustrates how Foreman can be used to manage multiple nodes across multiple environments-

[[File:env.png]]

We can apply various filters here to narrow down on a particular instance or conversely modify multiple nodes at the same time.

=== Knife ===

Node management in Chef includes users to create, read, update and delete nodes. This can be done either by using [http://docs.opscode.com/manage_server_hosted_nodes.html Management Console] or the command line tool [http://docs.opscode.com/knife_node.html Knife].

[http://docs.opscode.com/knife_node.html Knife command] the following syntax in order to manage the nodes :

$ knife node [ARGUMENT] (options)

where ARGUMENT could be any one of the management options like bulk delete, create, edit, delete,list.

For example: to perform a bulk delete, a regular expression is prepared to identify the nodes to be deleted. This is then passed as the option.

$ knife node bulk delete "node^[0-3]{6}$"

User needs to have specific user level permission in order to do the specific task. For eg: You need create permissions on the global nodes level to create a node.

=== How They Compare ===

''Similarities''

* Both provide a simple interface to manage nodes using simple command or a service running in the background.

''Differences''

* Chef enlists the details while doing the required operation whereas puppet does not. It uses command -test to enlist details.

== Config Files Templatization ==

=== Puppet ===

In Puppet files are treated as resources. So we could specify for an NFS node as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => "/some/path 192.168.100.1(rw,sync)”,
}

However the content would be different for different nodes as the environment would change.

Instead of creating a different resource for each template Puppet supports templatization of config files using Embedded Ruby (ERB).

We can re-write the same resource as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => template(“nfs/mount_conf.erb”),
}

where the mount_conf.erb file would contain Embedded Ruby code along with the appropriate Ruby variables which would automatically be compiled to give the right values to produce the conf file.

We can apply this same manifest file to all nodes across all the instances.

=== Chef ===

[http://docs.opscode.com/resource_deploy.html Deploy] is one of the [http://docs.opscode.com/resource.html Resource] which manages and controls deployment of configuration files on the nodes. It specifies action to be taken during deployment phase. These actions are then put into action by [http://docs.opscode.com/resource.html Providers] which is mapped to resource deploy.

deploy "name" do # deploy directs chef client to the provider to be used for this resource,default being Chef::Provider::Deploy::Revision.
# name is the location in which the deployment steps will occur.
attribute "value" # this represents the attributes available for this resource.
...
callback do
# callback represents additional block or link to a file to provide additional information during deployment process.
end
...
purge_before_symlink
create_dirs_before_symlink
symlink # These commands are used to create directories, link configuration files, remove directories or map them.
action :action # this specifies the action to be taken by the provider.
end

=== How They Compare ===

''Similarities''

* Both use Embedded Ruby for templatizing config files
* Both allow files to be specified as the Ruby executed version of an erb
* Both allow variables to be included in the templates to be replaced by actual files when the config file is deployed

''Differences''

* In Puppet the template is specified explicitly as part of the resource type file ( and other similar resource types) whereas in Chef it can be specified anywhere.

= Other Infrastructure Management Tools =

There are many other infrastructure management tools-

# [https://juju.ubuntu.com/ Juju] - Ubuntu juju is a tool for virtual infrastructure automation
# [http://cfengine.com Cfengine] is a much older tool that is used for large scale infrastructure automation. Puppet was originally inspired by Cfengine.
# [http://saltstack.com/community.html SaltStack] - this is another configuration management system written in Python

This is by no means an exhaustive list and is one that is growing - reflecting the trend towards heavier and more all encompassing automation.

= Conclusion =

This comparison was not meant to tout one of the tools - Puppet or Chef - over the other but merely to introduce and contrast both of them and see how they can be used with standard Infrastructure Management scenarios.

It is up to the reader to judge his/her needs and apply the tool for the job.

= Further Reading/References =
1. [http://docs.opscode.com/chef_overview.html Opscode Chef] 
2. [http://docs.opscode.com/knife_node.html About Knife]

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-18T16:14:38Z

Grmenon: /* Config Files Deployment */

= Introduction =

Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size and complexity automated tools are becoming the norm as opposed to the exception.

Here we shall compare two popular Infrastructure Management Tools-

* Puppet
* Chef

We shall begin with an initial overview of both tools to familiarize the reader. We will then take a hypothetical scenarion and try and solve the problems in those scenario using Puppet and Chef.

We will then compare the solutions and compare how each tool handles the same problem. We do not intend to categorically state the superiority or suitability of one tool over the other but to illustrate their usage and let the reader choose them as per her/his need.

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

INSERT DIAGRAM

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}

An example would be -

service { 'rails s':
ensure => 'running'
}

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

[http://www.opscode.com/chef/ Opscode Chef] is an infrastructure automation and configuration management framework.

Chef is used for configuration, deployment and scalability of servers and applications for eg: it can be used to deploy and manage network in cloud, on-site or a hybrid one with n number of servers. This helps organization to save time, solve mission-critical challenges, accelerate time to market, manage scale and complexity, and safeguard systems.

Chef client works using the abstraction definitions know as cookbooks and recipes that are written in Ruby. Chef client uses this definitions to build and manage servers in an infrastructure.
== Elements ==
Basically an organization consists of following elements with chef configuration:

# [http://docs.opscode.com/essentials_node_object.html Node] : Any physical, virtual server is referred to as node.
# [http://docs.opscode.com/knife.html Knife] : An interface between the local chef-repo and the server is known as knife. It is a command line tool using which users can manage other elements of the setup.
# [http://docs.opscode.com/chef_overview.html#workstations Workstation] : It is a computer used to interact with the single server. It is also used by users to run the knife command.
# [http://docs.opscode.com/chef_quick_overview.html#repository Repository] : usually referred as chef repo is the location wher all data objects are stored like cookbooks, configuration files etc.
# [http://docs.opscode.com/chef_quick_overview.html#the-hosted-server Hoster server]: The server manages cookbooks and all the configuration details that need to be applied to nodes.
# [http://docs.opscode.com/essentials_chef_client.html Chef-client] : Chef client acts as an interface between the node and hosted server. Node gets all the configuration details from server via the client.
# [http://docs.opscode.com/chef_quick_overview.html#cookbooks Cookbook] : Cookbook stores the configuration details depending upon the scenario for eg: in order to install Apache Tomcat server it contains details about installation, upgradation, components needed to support the server.

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===
<ol style="margin-left: 5.2em;">
<li>[http://docs.opscode.com/install_server.html Install chef server] on the server machine</li>
<li>[https://learnchef.opscode.com/quickstart/workstation-setup/ Setup workstation]:</li>
:* Pre-requisites : The workstation must have a [https://www.virtualbox.org/wiki/Downloads Virtual box] , [http://downloads.vagrantup.com/ Vagrant](command line interface tool), [http://git-scm.com/ Git]( needed for Version Control Systems)
:* Use knife client command to establish an API client identity for the workstation in order to make authenticated requests to the server. Setup [https://learnchef.opscode.com/quickstart/chef-repo/ chef-repo] in the workstation.
<li>[https://learnchef.opscode.com/quickstart/chef-repo/ Create Cookbooks]:</li>
:* In order to use community cookbooks one can just run the following command:

Syntax : knife cookbook site install <cook_name>

Example: knife cookbook site install apt.

:Note: Chef has maintained a list of community cookbooks which are commonly covered under the use cases like apache2, mysql,php. So a user can leverage the community cookbooks instead of creating their own cookbooks.

:* In order to create our own cookbook:

Syntax : knife cookbook create cookbook_name

:The file default.rb in the path chef-repo/cookbooks/aliases/recipes/default.rb can be edited in order to create recipes.

:To make these cookbooks available to the nodes, it is first uploaded in the chef server.
<li>[http://www.opscode.com/chef/install/ Setup chef client]: Using knife tool in workstation, chef clients are setup on nodes.</li>
</ol>

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a Node ==

Our sample project will be using MongoDB. So the next task will be setting up the MongoDB instances.

=== Puppet ===

As we have seen in Puppet a machine’s configuration is specified in terms of resources. To tell Puppet to set up those resources on a machine we use a Manifest that consists of various resources.

But setting up a Manifest that covers all the Resources will quickly become unwieldly.

Which is why Puppet uses Classes and Modules.

A Class is a named collection of resources. A Module encloses a Class and is something that can be included in other Puppet files thus enabling reuse. There is a large collection of Modules called the Puppet Forge that has been written covering different type of uses.

Thus setting up a manifest for a MongoDB server is as simple as-

class { ‘mongodb’ :
init => ‘sysv’,
version => ‘1.1.1’ # installing a specific version of mongodb
}
include mongodb
# custom puppet code as per our requirments

Thus different Classes can be written which use different Modules for different uses. We can define a different class for our web server , for our file share and so on.

Once a Class has been defined by us that is tailor made to our uses setting up more instances is simply a question of provisioning the node and running the Puppet class on it.

This supports quick and simple scaling up.

=== Chef ===

[http://docs.opscode.com/essentials_roles.html Configuration in Chef] is done by defining [http://docs.opscode.com/essentials_roles.html# Role]. Role helps in setting required attributes and tasks for a node. A node can have 0 or more roles assigned to it. Also a role can be assigned to many nodes. Roles thus helps in defining processes for a particular node.

Let us consider to [https://learnchef.opscode.com/starter-use-cases/wordpress/ configure] a role for mongoDb server for a application:
<ol style="margin-left: 5.2em;">
<li>A role can be created by creating a file under roles directory for eg : roles/role_file.rb</li>

<li>This file is composed of
:name : defines the node.
:description: describes about the node.
:attributes: attributes specific to the node which needs to be configured.
:run lists: tasks assigned to the node.
:Sample-
[[File:codeMongo.JPG]]

</li>
<li>This file is then uploaded to the chef server using following command:</li>

knife role from file roles/redis.rb

<li>Role is assigned to the server during [http://docs.opscode.com/install_bootstrap.html bootstrap] using command :</li>

knife ec2 server create

</li>

</ol>

=== How They Compare ===

''Similarities''

* Puppet and Chef both abstract the configuration of an instance in their own way - using Resources and Recipes respectively
* Both of them take care of the issue of setting up instances of different OS's - they handle the lower level details of actually installing and setting up packages etc
* Puppet and Chef both classify or group similar configurations using Classes/Modules and Roles respectively

''Differences''

* In Chef recipes and other specifications are done in Ruby whereas Puppet has it's own language with it's own file extension - .pp
* In normal/daily use the Puppet command is sufficient but for Chef the knife tool is more frequently used.

== Setting Up A Node ==

=== Puppet ===

In Puppet once we have identified what will go into defining something that we will need an instance for - as we have seen above say configuring a MongoDB instance - we need to select a fresh instance and apply the manifest to it.

As we have seen Puppet Agents will automatically sync with the Puppet Master. So the only task left for us is to

# Provision a new instance
# Install Puppet on it
# Classify it as a particular type of node.

The last two commands can be combined using the puppet init command-

puppet node init \
--node-group=db \
…. ( login related options)
ec2-71-73-79-83.compute-1.amazonaws.com

Here we have specified the fresh instance and initialized as belonging to the group of nodes as db.

Once this process is done the instance will be classified as a db instance and will automatically begin syncing with the Puppet Master.

=== Chef ===

Pre-requisites : IP address / [http://en.wikipedia.org/wiki/Fully_qualified_domain_name FQDN] of the node

Run the [http://docs.opscode.com/install_bootstrap.html knife bootstrap command]

The knife bootstrap command requires the FQDN or the IP address for the node in order to complete the bootstrap operation. Run the knife bootstrap command. It checks for [https://learnchef.opscode.com/screencasts/manage-pem-files/ .pem] files for authentication. If not present then the node goes through the registration phase in order to establish connection between the client and the server and for exchange of authorized messages. The attributes set in the role of the node are compared against the configuration details of the node and run contents of that role’s run list are applied to the node’s configuration details.

=== How They Compare ===

'''Similarities'''

* Both support integration with cloud service providers like Amazon to support direct setup and configuration of a fresh instance.
* Both need client software - the Puppet Agent and Chef Client respectively to be installed first to complete the configuration process

'''Differences'''

== Node Management ==

=== Foreman ===

Foreman is a popular tool that is used in conjunction with Puppet. It provides a Web based GUI that lets users manage Puppet nodes.

As an example the following screenshot illustrates how Foreman can be used to manage multiple nodes across multiple environments-

[[File:env.png]]

We can apply various filters here to narrow down on a particular instance or conversely modify multiple nodes at the same time.

=== Knife ===

Node management in Chef includes users to create, read, update and delete nodes. This can be done either by using [http://docs.opscode.com/manage_server_hosted_nodes.html Management Console] or the command line tool [http://docs.opscode.com/knife_node.html Knife].

[http://docs.opscode.com/knife_node.html Knife command] the following syntax in order to manage the nodes :

$ knife node [ARGUMENT] (options)

where ARGUMENT could be any one of the management options like bulk delete, create, edit, delete,list.

For example: to perform a bulk delete, a regular expression is prepared to identify the nodes to be deleted. This is then passed as the option.

$ knife node bulk delete "node^[0-3]{6}$"

User needs to have specific user level permission in order to do the specific task. For eg: You need create permissions on the global nodes level to create a node.

=== How They Compare ===

''Similarities''

* Both provide a simple interface to manage nodes using simple command or a service running in the background.

''Differences''

* Chef enlists the details while doing the required operation whereas puppet does not. It uses command -test to enlist details.

== Config Files Templatization ==

=== Puppet ===

In Puppet files are treated as resources. So we could specify for an NFS node as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => "/some/path 192.168.100.1(rw,sync)”,
}

However the content would be different for different nodes as the environment would change.

Instead of creating a different resource for each template Puppet supports templatization of config files using Embedded Ruby (ERB).

We can re-write the same resource as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => template(“nfs/mount_conf.erb”),
}

where the mount_conf.erb file would contain Embedded Ruby code along with the appropriate Ruby variables which would automatically be compiled to give the right values to produce the conf file.

We can apply this same manifest file to all nodes across all the instances.

=== Chef ===

[http://docs.opscode.com/resource_deploy.html Deploy] is one of the [http://docs.opscode.com/resource.html Resource] which manages and controls deployment of configuration files on the nodes. It specifies action to be taken during deployment phase. These actions are then put into action by [http://docs.opscode.com/resource.html Providers] which is mapped to resource deploy.

deploy "name" do # deploy directs chef client to the provider to be used for this resource,default being Chef::Provider::Deploy::Revision.
# name is the location in which the deployment steps will occur.
attribute "value" # this represents the attributes available for this resource.
...
callback do
# callback represents additional block or link to a file to provide additional information during deployment process.
end
...
purge_before_symlink
create_dirs_before_symlink
symlink # These commands are used to create directories, link configuration files, remove directories or map them.
action :action # this specifies the action to be taken by the provider.
end

=== How They Compare ===

''Similarities''

* Both use Embedded Ruby for templatizing config files
* Both allow files to be specified as the Ruby executed version of an erb
* Both allow variables to be included in the templates to be replaced by actual files when the config file is deployed

''Differences''

* In Puppet the template is specified explicitly as part of the resource type file ( and other similar resource types) whereas in Chef it can be specified anywhere.

= Other Infrastructure Management Tools =

There are many other infrastructure management tools-

# [https://juju.ubuntu.com/ Juju] - Ubuntu juju is a tool for virtual infrastructure automation
# [http://cfengine.com Cfengine] is a much older tool that is used for large scale infrastructure automation. Puppet was originally inspired by Cfengine.
# [http://saltstack.com/community.html SaltStack] - this is another configuration management system written in Python

This is by no means an exhaustive list and is one that is growing - reflecting the trend towards heavier and more all encompassing automation.

= Conclusion =

This comparison was not meant to tout one of the tools - Puppet or Chef - over the other but merely to introduce and contrast both of them and see how they can be used with standard Infrastructure Management scenarios.

It is up to the reader to judge his/her needs and apply the tool for the job.

= Further Reading/References =
1. [http://docs.opscode.com/chef_overview.html Opscode Chef] 
2. [http://docs.opscode.com/knife_node.html About Knife]

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-18T16:14:11Z

Grmenon: /* How They Compare */

= Introduction =

Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size and complexity automated tools are becoming the norm as opposed to the exception.

Here we shall compare two popular Infrastructure Management Tools-

* Puppet
* Chef

We shall begin with an initial overview of both tools to familiarize the reader. We will then take a hypothetical scenarion and try and solve the problems in those scenario using Puppet and Chef.

We will then compare the solutions and compare how each tool handles the same problem. We do not intend to categorically state the superiority or suitability of one tool over the other but to illustrate their usage and let the reader choose them as per her/his need.

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

INSERT DIAGRAM

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}

An example would be -

service { 'rails s':
ensure => 'running'
}

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

[http://www.opscode.com/chef/ Opscode Chef] is an infrastructure automation and configuration management framework.

Chef is used for configuration, deployment and scalability of servers and applications for eg: it can be used to deploy and manage network in cloud, on-site or a hybrid one with n number of servers. This helps organization to save time, solve mission-critical challenges, accelerate time to market, manage scale and complexity, and safeguard systems.

Chef client works using the abstraction definitions know as cookbooks and recipes that are written in Ruby. Chef client uses this definitions to build and manage servers in an infrastructure.
== Elements ==
Basically an organization consists of following elements with chef configuration:

# [http://docs.opscode.com/essentials_node_object.html Node] : Any physical, virtual server is referred to as node.
# [http://docs.opscode.com/knife.html Knife] : An interface between the local chef-repo and the server is known as knife. It is a command line tool using which users can manage other elements of the setup.
# [http://docs.opscode.com/chef_overview.html#workstations Workstation] : It is a computer used to interact with the single server. It is also used by users to run the knife command.
# [http://docs.opscode.com/chef_quick_overview.html#repository Repository] : usually referred as chef repo is the location wher all data objects are stored like cookbooks, configuration files etc.
# [http://docs.opscode.com/chef_quick_overview.html#the-hosted-server Hoster server]: The server manages cookbooks and all the configuration details that need to be applied to nodes.
# [http://docs.opscode.com/essentials_chef_client.html Chef-client] : Chef client acts as an interface between the node and hosted server. Node gets all the configuration details from server via the client.
# [http://docs.opscode.com/chef_quick_overview.html#cookbooks Cookbook] : Cookbook stores the configuration details depending upon the scenario for eg: in order to install Apache Tomcat server it contains details about installation, upgradation, components needed to support the server.

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===
<ol style="margin-left: 5.2em;">
<li>[http://docs.opscode.com/install_server.html Install chef server] on the server machine</li>
<li>[https://learnchef.opscode.com/quickstart/workstation-setup/ Setup workstation]:</li>
:* Pre-requisites : The workstation must have a [https://www.virtualbox.org/wiki/Downloads Virtual box] , [http://downloads.vagrantup.com/ Vagrant](command line interface tool), [http://git-scm.com/ Git]( needed for Version Control Systems)
:* Use knife client command to establish an API client identity for the workstation in order to make authenticated requests to the server. Setup [https://learnchef.opscode.com/quickstart/chef-repo/ chef-repo] in the workstation.
<li>[https://learnchef.opscode.com/quickstart/chef-repo/ Create Cookbooks]:</li>
:* In order to use community cookbooks one can just run the following command:

Syntax : knife cookbook site install <cook_name>

Example: knife cookbook site install apt.

:Note: Chef has maintained a list of community cookbooks which are commonly covered under the use cases like apache2, mysql,php. So a user can leverage the community cookbooks instead of creating their own cookbooks.

:* In order to create our own cookbook:

Syntax : knife cookbook create cookbook_name

:The file default.rb in the path chef-repo/cookbooks/aliases/recipes/default.rb can be edited in order to create recipes.

:To make these cookbooks available to the nodes, it is first uploaded in the chef server.
<li>[http://www.opscode.com/chef/install/ Setup chef client]: Using knife tool in workstation, chef clients are setup on nodes.</li>
</ol>

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a Node ==

Our sample project will be using MongoDB. So the next task will be setting up the MongoDB instances.

=== Puppet ===

As we have seen in Puppet a machine’s configuration is specified in terms of resources. To tell Puppet to set up those resources on a machine we use a Manifest that consists of various resources.

But setting up a Manifest that covers all the Resources will quickly become unwieldly.

Which is why Puppet uses Classes and Modules.

A Class is a named collection of resources. A Module encloses a Class and is something that can be included in other Puppet files thus enabling reuse. There is a large collection of Modules called the Puppet Forge that has been written covering different type of uses.

Thus setting up a manifest for a MongoDB server is as simple as-

class { ‘mongodb’ :
init => ‘sysv’,
version => ‘1.1.1’ # installing a specific version of mongodb
}
include mongodb
# custom puppet code as per our requirments

Thus different Classes can be written which use different Modules for different uses. We can define a different class for our web server , for our file share and so on.

Once a Class has been defined by us that is tailor made to our uses setting up more instances is simply a question of provisioning the node and running the Puppet class on it.

This supports quick and simple scaling up.

=== Chef ===

[http://docs.opscode.com/essentials_roles.html Configuration in Chef] is done by defining [http://docs.opscode.com/essentials_roles.html# Role]. Role helps in setting required attributes and tasks for a node. A node can have 0 or more roles assigned to it. Also a role can be assigned to many nodes. Roles thus helps in defining processes for a particular node.

Let us consider to [https://learnchef.opscode.com/starter-use-cases/wordpress/ configure] a role for mongoDb server for a application:
<ol style="margin-left: 5.2em;">
<li>A role can be created by creating a file under roles directory for eg : roles/role_file.rb</li>

<li>This file is composed of
:name : defines the node.
:description: describes about the node.
:attributes: attributes specific to the node which needs to be configured.
:run lists: tasks assigned to the node.
:Sample-
[[File:codeMongo.JPG]]

</li>
<li>This file is then uploaded to the chef server using following command:</li>

knife role from file roles/redis.rb

<li>Role is assigned to the server during [http://docs.opscode.com/install_bootstrap.html bootstrap] using command :</li>

knife ec2 server create

</li>

</ol>

=== How They Compare ===

''Similarities''

* Puppet and Chef both abstract the configuration of an instance in their own way - using Resources and Recipes respectively
* Both of them take care of the issue of setting up instances of different OS's - they handle the lower level details of actually installing and setting up packages etc
* Puppet and Chef both classify or group similar configurations using Classes/Modules and Roles respectively

''Differences''

* In Chef recipes and other specifications are done in Ruby whereas Puppet has it's own language with it's own file extension - .pp
* In normal/daily use the Puppet command is sufficient but for Chef the knife tool is more frequently used.

== Setting Up A Node ==

=== Puppet ===

In Puppet once we have identified what will go into defining something that we will need an instance for - as we have seen above say configuring a MongoDB instance - we need to select a fresh instance and apply the manifest to it.

As we have seen Puppet Agents will automatically sync with the Puppet Master. So the only task left for us is to

# Provision a new instance
# Install Puppet on it
# Classify it as a particular type of node.

The last two commands can be combined using the puppet init command-

puppet node init \
--node-group=db \
…. ( login related options)
ec2-71-73-79-83.compute-1.amazonaws.com

Here we have specified the fresh instance and initialized as belonging to the group of nodes as db.

Once this process is done the instance will be classified as a db instance and will automatically begin syncing with the Puppet Master.

=== Chef ===

Pre-requisites : IP address / [http://en.wikipedia.org/wiki/Fully_qualified_domain_name FQDN] of the node

Run the [http://docs.opscode.com/install_bootstrap.html knife bootstrap command]

The knife bootstrap command requires the FQDN or the IP address for the node in order to complete the bootstrap operation. Run the knife bootstrap command. It checks for [https://learnchef.opscode.com/screencasts/manage-pem-files/ .pem] files for authentication. If not present then the node goes through the registration phase in order to establish connection between the client and the server and for exchange of authorized messages. The attributes set in the role of the node are compared against the configuration details of the node and run contents of that role’s run list are applied to the node’s configuration details.

=== How They Compare ===

'''Similarities'''

* Both support integration with cloud service providers like Amazon to support direct setup and configuration of a fresh instance.
* Both need client software - the Puppet Agent and Chef Client respectively to be installed first to complete the configuration process

'''Differences'''

== Node Management ==

=== Foreman ===

Foreman is a popular tool that is used in conjunction with Puppet. It provides a Web based GUI that lets users manage Puppet nodes.

As an example the following screenshot illustrates how Foreman can be used to manage multiple nodes across multiple environments-

[[File:env.png]]

We can apply various filters here to narrow down on a particular instance or conversely modify multiple nodes at the same time.

=== Knife ===

Node management in Chef includes users to create, read, update and delete nodes. This can be done either by using [http://docs.opscode.com/manage_server_hosted_nodes.html Management Console] or the command line tool [http://docs.opscode.com/knife_node.html Knife].

[http://docs.opscode.com/knife_node.html Knife command] the following syntax in order to manage the nodes :

$ knife node [ARGUMENT] (options)

where ARGUMENT could be any one of the management options like bulk delete, create, edit, delete,list.

For example: to perform a bulk delete, a regular expression is prepared to identify the nodes to be deleted. This is then passed as the option.

$ knife node bulk delete "node^[0-3]{6}$"

User needs to have specific user level permission in order to do the specific task. For eg: You need create permissions on the global nodes level to create a node.

=== How They Compare ===

''Similarities''

* Both provide a simple interface to manage nodes using simple command or a service running in the background.

''Differences''

* Chef enlists the details while doing the required operation whereas puppet does not. It uses command -test to enlist details.

== Config Files Deployment ==

=== Puppet ===

In Puppet files are treated as resources. So we could specify for an NFS node as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => "/some/path 192.168.100.1(rw,sync)”,
}

However the content would be different for different nodes as the environment would change.

Instead of creating a different resource for each template Puppet supports templatization of config files using Embedded Ruby (ERB).

We can re-write the same resource as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => template(“nfs/mount_conf.erb”),
}

where the mount_conf.erb file would contain Embedded Ruby code along with the appropriate Ruby variables which would automatically be compiled to give the right values to produce the conf file.

We can apply this same manifest file to all nodes across all the instances.

=== Chef ===

[http://docs.opscode.com/resource_deploy.html Deploy] is one of the [http://docs.opscode.com/resource.html Resource] which manages and controls deployment of configuration files on the nodes. It specifies action to be taken during deployment phase. These actions are then put into action by [http://docs.opscode.com/resource.html Providers] which is mapped to resource deploy.

deploy "name" do # deploy directs chef client to the provider to be used for this resource,default being Chef::Provider::Deploy::Revision.
# name is the location in which the deployment steps will occur.
attribute "value" # this represents the attributes available for this resource.
...
callback do
# callback represents additional block or link to a file to provide additional information during deployment process.
end
...
purge_before_symlink
create_dirs_before_symlink
symlink # These commands are used to create directories, link configuration files, remove directories or map them.
action :action # this specifies the action to be taken by the provider.
end

=== How They Compare ===

''Similarities''

* Both use Embedded Ruby for templatizing config files
* Both allow files to be specified as the Ruby executed version of an erb
* Both allow variables to be included in the templates to be replaced by actual files when the config file is deployed

''Differences''

* In Puppet the template is specified explicitly as part of the resource type file ( and other similar resource types) whereas in Chef it can be specified anywhere.

= Other Infrastructure Management Tools =

There are many other infrastructure management tools-

# [https://juju.ubuntu.com/ Juju] - Ubuntu juju is a tool for virtual infrastructure automation
# [http://cfengine.com Cfengine] is a much older tool that is used for large scale infrastructure automation. Puppet was originally inspired by Cfengine.
# [http://saltstack.com/community.html SaltStack] - this is another configuration management system written in Python

This is by no means an exhaustive list and is one that is growing - reflecting the trend towards heavier and more all encompassing automation.

= Conclusion =

This comparison was not meant to tout one of the tools - Puppet or Chef - over the other but merely to introduce and contrast both of them and see how they can be used with standard Infrastructure Management scenarios.

It is up to the reader to judge his/her needs and apply the tool for the job.

= Further Reading/References =
1. [http://docs.opscode.com/chef_overview.html Opscode Chef] 
2. [http://docs.opscode.com/knife_node.html About Knife]

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-18T16:11:45Z

Grmenon: /* How They Compare */

= Introduction =

Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size and complexity automated tools are becoming the norm as opposed to the exception.

Here we shall compare two popular Infrastructure Management Tools-

* Puppet
* Chef

We shall begin with an initial overview of both tools to familiarize the reader. We will then take a hypothetical scenarion and try and solve the problems in those scenario using Puppet and Chef.

We will then compare the solutions and compare how each tool handles the same problem. We do not intend to categorically state the superiority or suitability of one tool over the other but to illustrate their usage and let the reader choose them as per her/his need.

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

INSERT DIAGRAM

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}

An example would be -

service { 'rails s':
ensure => 'running'
}

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

[http://www.opscode.com/chef/ Opscode Chef] is an infrastructure automation and configuration management framework.

Chef is used for configuration, deployment and scalability of servers and applications for eg: it can be used to deploy and manage network in cloud, on-site or a hybrid one with n number of servers. This helps organization to save time, solve mission-critical challenges, accelerate time to market, manage scale and complexity, and safeguard systems.

Chef client works using the abstraction definitions know as cookbooks and recipes that are written in Ruby. Chef client uses this definitions to build and manage servers in an infrastructure.
== Elements ==
Basically an organization consists of following elements with chef configuration:

# [http://docs.opscode.com/essentials_node_object.html Node] : Any physical, virtual server is referred to as node.
# [http://docs.opscode.com/knife.html Knife] : An interface between the local chef-repo and the server is known as knife. It is a command line tool using which users can manage other elements of the setup.
# [http://docs.opscode.com/chef_overview.html#workstations Workstation] : It is a computer used to interact with the single server. It is also used by users to run the knife command.
# [http://docs.opscode.com/chef_quick_overview.html#repository Repository] : usually referred as chef repo is the location wher all data objects are stored like cookbooks, configuration files etc.
# [http://docs.opscode.com/chef_quick_overview.html#the-hosted-server Hoster server]: The server manages cookbooks and all the configuration details that need to be applied to nodes.
# [http://docs.opscode.com/essentials_chef_client.html Chef-client] : Chef client acts as an interface between the node and hosted server. Node gets all the configuration details from server via the client.
# [http://docs.opscode.com/chef_quick_overview.html#cookbooks Cookbook] : Cookbook stores the configuration details depending upon the scenario for eg: in order to install Apache Tomcat server it contains details about installation, upgradation, components needed to support the server.

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===
<ol style="margin-left: 5.2em;">
<li>[http://docs.opscode.com/install_server.html Install chef server] on the server machine</li>
<li>[https://learnchef.opscode.com/quickstart/workstation-setup/ Setup workstation]:</li>
:* Pre-requisites : The workstation must have a [https://www.virtualbox.org/wiki/Downloads Virtual box] , [http://downloads.vagrantup.com/ Vagrant](command line interface tool), [http://git-scm.com/ Git]( needed for Version Control Systems)
:* Use knife client command to establish an API client identity for the workstation in order to make authenticated requests to the server. Setup [https://learnchef.opscode.com/quickstart/chef-repo/ chef-repo] in the workstation.
<li>[https://learnchef.opscode.com/quickstart/chef-repo/ Create Cookbooks]:</li>
:* In order to use community cookbooks one can just run the following command:

Syntax : knife cookbook site install <cook_name>

Example: knife cookbook site install apt.

:Note: Chef has maintained a list of community cookbooks which are commonly covered under the use cases like apache2, mysql,php. So a user can leverage the community cookbooks instead of creating their own cookbooks.

:* In order to create our own cookbook:

Syntax : knife cookbook create cookbook_name

:The file default.rb in the path chef-repo/cookbooks/aliases/recipes/default.rb can be edited in order to create recipes.

:To make these cookbooks available to the nodes, it is first uploaded in the chef server.
<li>[http://www.opscode.com/chef/install/ Setup chef client]: Using knife tool in workstation, chef clients are setup on nodes.</li>
</ol>

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a Node ==

Our sample project will be using MongoDB. So the next task will be setting up the MongoDB instances.

=== Puppet ===

As we have seen in Puppet a machine’s configuration is specified in terms of resources. To tell Puppet to set up those resources on a machine we use a Manifest that consists of various resources.

But setting up a Manifest that covers all the Resources will quickly become unwieldly.

Which is why Puppet uses Classes and Modules.

A Class is a named collection of resources. A Module encloses a Class and is something that can be included in other Puppet files thus enabling reuse. There is a large collection of Modules called the Puppet Forge that has been written covering different type of uses.

Thus setting up a manifest for a MongoDB server is as simple as-

class { ‘mongodb’ :
init => ‘sysv’,
version => ‘1.1.1’ # installing a specific version of mongodb
}
include mongodb
# custom puppet code as per our requirments

Thus different Classes can be written which use different Modules for different uses. We can define a different class for our web server , for our file share and so on.

Once a Class has been defined by us that is tailor made to our uses setting up more instances is simply a question of provisioning the node and running the Puppet class on it.

This supports quick and simple scaling up.

=== Chef ===

[http://docs.opscode.com/essentials_roles.html Configuration in Chef] is done by defining [http://docs.opscode.com/essentials_roles.html# Role]. Role helps in setting required attributes and tasks for a node. A node can have 0 or more roles assigned to it. Also a role can be assigned to many nodes. Roles thus helps in defining processes for a particular node.

Let us consider to [https://learnchef.opscode.com/starter-use-cases/wordpress/ configure] a role for mongoDb server for a application:
<ol style="margin-left: 5.2em;">
<li>A role can be created by creating a file under roles directory for eg : roles/role_file.rb</li>

<li>This file is composed of
:name : defines the node.
:description: describes about the node.
:attributes: attributes specific to the node which needs to be configured.
:run lists: tasks assigned to the node.
:Sample-
[[File:codeMongo.JPG]]

</li>
<li>This file is then uploaded to the chef server using following command:</li>

knife role from file roles/redis.rb

<li>Role is assigned to the server during [http://docs.opscode.com/install_bootstrap.html bootstrap] using command :</li>

knife ec2 server create

</li>

</ol>

=== How They Compare ===

''Similarities''

* Puppet and Chef both abstract the configuration of an instance in their own way - using Resources and Recipes respectively
* Both of them take care of the issue of setting up instances of different OS's - they handle the lower level details of actually installing and setting up packages etc
* Puppet and Chef both classify or group similar configurations using Classes/Modules and Roles respectively

''Differences''

* In Chef recipes and other specifications are done in Ruby whereas Puppet has it's own language with it's own file extension - .pp
* In normal/daily use the Puppet command is sufficient but for Chef the knife tool is more frequently used.

== Setting Up A Node ==

=== Puppet ===

In Puppet once we have identified what will go into defining something that we will need an instance for - as we have seen above say configuring a MongoDB instance - we need to select a fresh instance and apply the manifest to it.

As we have seen Puppet Agents will automatically sync with the Puppet Master. So the only task left for us is to

# Provision a new instance
# Install Puppet on it
# Classify it as a particular type of node.

The last two commands can be combined using the puppet init command-

puppet node init \
--node-group=db \
…. ( login related options)
ec2-71-73-79-83.compute-1.amazonaws.com

Here we have specified the fresh instance and initialized as belonging to the group of nodes as db.

Once this process is done the instance will be classified as a db instance and will automatically begin syncing with the Puppet Master.

=== Chef ===

Pre-requisites : IP address / [http://en.wikipedia.org/wiki/Fully_qualified_domain_name FQDN] of the node

Run the [http://docs.opscode.com/install_bootstrap.html knife bootstrap command]

The knife bootstrap command requires the FQDN or the IP address for the node in order to complete the bootstrap operation. Run the knife bootstrap command. It checks for [https://learnchef.opscode.com/screencasts/manage-pem-files/ .pem] files for authentication. If not present then the node goes through the registration phase in order to establish connection between the client and the server and for exchange of authorized messages. The attributes set in the role of the node are compared against the configuration details of the node and run contents of that role’s run list are applied to the node’s configuration details.

=== How They Compare ===

== Node Management ==

=== Foreman ===

Foreman is a popular tool that is used in conjunction with Puppet. It provides a Web based GUI that lets users manage Puppet nodes.

As an example the following screenshot illustrates how Foreman can be used to manage multiple nodes across multiple environments-

[[File:env.png]]

We can apply various filters here to narrow down on a particular instance or conversely modify multiple nodes at the same time.

=== Knife ===

Node management in Chef includes users to create, read, update and delete nodes. This can be done either by using [http://docs.opscode.com/manage_server_hosted_nodes.html Management Console] or the command line tool [http://docs.opscode.com/knife_node.html Knife].

[http://docs.opscode.com/knife_node.html Knife command] the following syntax in order to manage the nodes :

$ knife node [ARGUMENT] (options)

where ARGUMENT could be any one of the management options like bulk delete, create, edit, delete,list.

For example: to perform a bulk delete, a regular expression is prepared to identify the nodes to be deleted. This is then passed as the option.

$ knife node bulk delete "node^[0-3]{6}$"

User needs to have specific user level permission in order to do the specific task. For eg: You need create permissions on the global nodes level to create a node.

=== How They Compare ===

''Similarities''

* Both provide a simple interface to manage nodes using simple command or a service running in the background.

''Differences''

* Chef enlists the details while doing the required operation whereas puppet does not. It uses command -test to enlist details.

== Config Files Deployment ==

=== Puppet ===

In Puppet files are treated as resources. So we could specify for an NFS node as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => "/some/path 192.168.100.1(rw,sync)”,
}

However the content would be different for different nodes as the environment would change.

Instead of creating a different resource for each template Puppet supports templatization of config files using Embedded Ruby (ERB).

We can re-write the same resource as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => template(“nfs/mount_conf.erb”),
}

where the mount_conf.erb file would contain Embedded Ruby code along with the appropriate Ruby variables which would automatically be compiled to give the right values to produce the conf file.

We can apply this same manifest file to all nodes across all the instances.

=== Chef ===

[http://docs.opscode.com/resource_deploy.html Deploy] is one of the [http://docs.opscode.com/resource.html Resource] which manages and controls deployment of configuration files on the nodes. It specifies action to be taken during deployment phase. These actions are then put into action by [http://docs.opscode.com/resource.html Providers] which is mapped to resource deploy.

deploy "name" do # deploy directs chef client to the provider to be used for this resource,default being Chef::Provider::Deploy::Revision.
# name is the location in which the deployment steps will occur.
attribute "value" # this represents the attributes available for this resource.
...
callback do
# callback represents additional block or link to a file to provide additional information during deployment process.
end
...
purge_before_symlink
create_dirs_before_symlink
symlink # These commands are used to create directories, link configuration files, remove directories or map them.
action :action # this specifies the action to be taken by the provider.
end

=== How They Compare ===

''Similarities''

* Both use Embedded Ruby for templatizing config files
* Both allow files to be specified as the Ruby executed version of an erb
* Both allow variables to be included in the templates to be replaced by actual files when the config file is deployed

''Differences''

* In Puppet the template is specified explicitly as part of the resource type file ( and other similar resource types) whereas in Chef it can be specified anywhere.

= Other Infrastructure Management Tools =

There are many other infrastructure management tools-

# [https://juju.ubuntu.com/ Juju] - Ubuntu juju is a tool for virtual infrastructure automation
# [http://cfengine.com Cfengine] is a much older tool that is used for large scale infrastructure automation. Puppet was originally inspired by Cfengine.
# [http://saltstack.com/community.html SaltStack] - this is another configuration management system written in Python

This is by no means an exhaustive list and is one that is growing - reflecting the trend towards heavier and more all encompassing automation.

= Conclusion =

This comparison was not meant to tout one of the tools - Puppet or Chef - over the other but merely to introduce and contrast both of them and see how they can be used with standard Infrastructure Management scenarios.

It is up to the reader to judge his/her needs and apply the tool for the job.

= Further Reading/References =
1. [http://docs.opscode.com/chef_overview.html Opscode Chef] 
2. [http://docs.opscode.com/knife_node.html About Knife]

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-18T15:32:33Z

Grmenon: /* How They Compare */

= Introduction =

Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size and complexity automated tools are becoming the norm as opposed to the exception.

Here we shall compare two popular Infrastructure Management Tools-

* Puppet
* Chef

We shall begin with an initial overview of both tools to familiarize the reader. We will then take a hypothetical scenarion and try and solve the problems in those scenario using Puppet and Chef.

We will then compare the solutions and compare how each tool handles the same problem. We do not intend to categorically state the superiority or suitability of one tool over the other but to illustrate their usage and let the reader choose them as per her/his need.

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

INSERT DIAGRAM

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}

An example would be -

service { 'rails s':
ensure => 'running'
}

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

[http://www.opscode.com/chef/ Opscode Chef] is an infrastructure automation and configuration management framework.

Chef is used for configuration, deployment and scalability of servers and applications for eg: it can be used to deploy and manage network in cloud, on-site or a hybrid one with n number of servers. This helps organization to save time, solve mission-critical challenges, accelerate time to market, manage scale and complexity, and safeguard systems.

Chef client works using the abstraction definitions know as cookbooks and recipes that are written in Ruby. Chef client uses this definitions to build and manage servers in an infrastructure.
== Elements ==
Basically an organization consists of following elements with chef configuration:

# [http://docs.opscode.com/essentials_node_object.html Node] : Any physical, virtual server is referred to as node.
# [http://docs.opscode.com/knife.html Knife] : An interface between the local chef-repo and the server is known as knife. It is a command line tool using which users can manage other elements of the setup.
# [http://docs.opscode.com/chef_overview.html#workstations Workstation] : It is a computer used to interact with the single server. It is also used by users to run the knife command.
# [http://docs.opscode.com/chef_quick_overview.html#repository Repository] : usually referred as chef repo is the location wher all data objects are stored like cookbooks, configuration files etc.
# [http://docs.opscode.com/chef_quick_overview.html#the-hosted-server Hoster server]: The stores manages cookbooks and all the configuration details that need to be applied to nodes.
# [http://docs.opscode.com/essentials_chef_client.html Chef-client] : Chef client acts as an interface between the node and hosted server. Node gets all the configuration details from server via the client.
# [http://docs.opscode.com/chef_quick_overview.html#cookbooks Cookbook] : Cookbook stores the configuration details depending upon the scenario for eg: in order to install Apache Tomcat server it contains details about installation, upgradation, components needed to support the server.

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===
<ol style="margin-left: 5.2em;">
<li>[http://docs.opscode.com/install_server.html Install chef server] on the server machine</li>
<li>[https://learnchef.opscode.com/quickstart/workstation-setup/ Setup workstation]:</li>
:* Pre-requisites : The workstation must have a [https://www.virtualbox.org/wiki/Downloads Virtual box] , [http://downloads.vagrantup.com/ Vagrant](command line interface tool), [http://git-scm.com/ Git]( needed for Version Control Systems)
:* Use knife client command to establish an API client identity for the workstation in order to make authenticated requests to the server. Setup [https://learnchef.opscode.com/quickstart/chef-repo/ chef-repo] in the workstation.
<li>[https://learnchef.opscode.com/quickstart/chef-repo/ Create Cookbooks]:</li>
:* In order to use community cookbooks one can just run the following command:

Syntax : knife cookbook site install <cook_name>

Example: knife cookbook site install apt.

:Note: Chef has maintained a list of community cookbooks which are commonly covered under the use cases like apache2, mysql,php. So a user can leverage the community cookbooks instead of creating their own cookbooks.

:* In order to create our own cookbook:

Syntax : knife cookbook create cookbook_name

:The file default.rb in the path chef-repo/cookbooks/aliases/recipes/default.rb can be edited in order to create recipes.

:To make these cookbooks available to the nodes, it is first uploaded in the chef server.
<li>[http://www.opscode.com/chef/install/ Setup chef client]: Using knife tool in workstation, chef clients are setup on nodes.</li>
</ol>

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a Node ==

Our sample project will be using MongoDB. So the next task will be setting up the MongoDB instances.

=== Puppet ===

As we have seen in Puppet a machine’s configuration is specified in terms of resources. To tell Puppet to set up those resources on a machine we use a Manifest that consists of various resources.

But setting up a Manifest that covers all the Resources will quickly become unwieldly.

Which is why Puppet uses Classes and Modules.

A Class is a named collection of resources. A Module encloses a Class and is something that can be included in other Puppet files thus enabling reuse. There is a large collection of Modules called the Puppet Forge that has been written covering different type of uses.

Thus setting up a manifest for a MongoDB server is as simple as-

class { ‘mongodb’ :
init => ‘sysv’,
version => ‘1.1.1’ # installing a specific version of mongodb
}
include mongodb
# custom puppet code as per our requirments

Thus different Classes can be written which use different Modules for different uses. We can define a different class for our web server , for our file share and so on.

Once a Class has been defined by us that is tailor made to our uses setting up more instances is simply a question of provisioning the node and running the Puppet class on it.

This supports quick and simple scaling up.

=== Chef ===

[http://docs.opscode.com/essentials_roles.html Configuration in Chef] is done by defining [http://docs.opscode.com/essentials_roles.html# Role]. Role helps in setting required attributes and tasks for a node. A node can have 0 or more roles assigned to it. Also a role can be assigned to many nodes. Roles thus helps in defining processes for a particular node.

Let us consider to [https://learnchef.opscode.com/starter-use-cases/wordpress/ configure] a role for mongoDb server for a application:
<ol style="margin-left: 5.2em;">
<li>A role can be created by creating a file under roles directory for eg : roles/role_file.rb</li>

<li>This file is composed of
:name : defines the node.
:description: describes about the node.
:attributes: attributes specific to the node which needs to be configured.
:run lists: tasks assigned to the node.
:Sample-
[[File:codeMongo.JPG]]

</li>
<li>This file is then uploaded to the chef server using following command:</li>

knife role from file roles/redis.rb

<li>Role is assigned to the server during [http://docs.opscode.com/install_bootstrap.html bootstrap] using command :</li>

knife ec2 server create

</li>

</ol>

=== How They Compare ===

''Similarities''

* Puppet and Chef both abstract the configuration of an instance in their own way - using Resources and Recipes respectively
* Both of them take care of the issue of setting up instances of different OS's - they handle the lower level details of actually installing and setting up packages etc
* Puppet and Chef both classify or group similar configurations using Classes/Modules and Roles respectively

''Differences''

* In Chef recipes and other specifications are done in Ruby whereas Puppet has it's own language with it's own file extension - .pp
* In normal/daily use the Puppet command is sufficient but for Chef the knife tool is more frequently used.

== Setting Up A Node ==

=== Puppet ===

In Puppet once we have identified what will go into defining something that we will need an instance for - as we have seen above say configuring a MongoDB instance - we need to select a fresh instance and apply the manifest to it.

As we have seen Puppet Agents will automatically sync with the Puppet Master. So the only task left for us is to

# Provision a new instance
# Install Puppet on it
# Classify it as a particular type of node.

The last two commands can be combined using the puppet init command-

puppet node init \
--node-group=db \
…. ( login related options)
ec2-71-73-79-83.compute-1.amazonaws.com

Here we have specified the fresh instance and initialized as belonging to the group of nodes as db.

Once this process is done the instance will be classified as a db instance and will automatically begin syncing with the Puppet Master.

=== Chef ===

Pre-requisites : IP address / FQDN of the node

Run the knife bootstrap command

The knife bootstrap command requires the FQDN or the IP address for the node in order to complete the bootstrap operation. Run the knife bootstrap command. It checks for .pem files for authentication. If not present then the node goes through the registration phase in order to establish connection between the client and the server and for exchange of authorized messages. The attributes set in the role of the node are compared against the configuration details of the node and run contents of that role’s run list are applied to the node’s configuration details.

=== How They Compare ===

== Node Management ==

=== Foreman ===

Foreman is a popular tool that is used in conjunction with Puppet. It provides a Web based GUI that lets users manage Puppet nodes.

As an example the following screenshot illustrates how Foreman can be used to manage multiple nodes across multiple environments-

[[File:env.png]]

We can apply various filters here to narrow down on a particular instance or conversely modify multiple nodes at the same time.

=== Knife ===

Node management in Chef includes users to create, read, update and delete nodes. This can be done either by using Management Console or the command line tool Knife.

Knife command the following syntax in order to manage the nodes :

$ knife node [ARGUMENT] (options)

where ARGUMENT could be any one of the management options like bulk delete, create, edit, delete,list.

For example: to perform a bulk delete, a regular expression is prepared to identify the nodes to be deleted. This is then passed as the option.

$ knife node bulk delete "node^[0-3]{6}$"

User needs to have specific user level permission in order to do the specific task. For eg: You need create permissions on the global nodes level to create a node.

=== How They Compare ===

''Similarities''

* Both provide a simple and user friendly interface to manage nodes

''Differences''

* Puppet uses the concept of templatization for reuse of scripts in order to create node. Chef create node page enlists all the recipes and roles in the organization which can be used to create a node.

== Config Files Deployment ==

=== Puppet ===

In Puppet files are treated as resources. So we could specify for an NFS node as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => "/some/path 192.168.100.1(rw,sync)”,
}

However the content would be different for different nodes as the environment would change.

Instead of creating a different resource for each template Puppet supports templatization of config files using Embedded Ruby (ERB).

We can re-write the same resource as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => template(“nfs/mount_conf.erb”),
}

where the mount_conf.erb file would contain Embedded Ruby code along with the appropriate Ruby variables which would automatically be compiled to give the right values to produce the conf file.

We can apply this same manifest file to all nodes across all the instances.

=== Chef ===

Deploy is one of the resource which manages and controls deployment of configuration files on the nodes. It specifies action to be taken during deployment phase. These actions are then put into action by Providers which is mapped to resource deploy.

deploy "name" do # deploy directs chef client to the provider to be used for this resource,default being Chef::Provider::Deploy::Revision.
# name is the location in which the deployment steps will occur.
attribute "value" # this represents the attributes available for this resource.
...
callback do
# callback represents additional block or link to a file to provide additional information during deployment process.
end
...
purge_before_symlink
create_dirs_before_symlink
symlink # These commands are used to create directories, link configuration files, remove directories or map them.
action :action # this specifies the action to be taken by the provider.
end

=== How They Compare ===

= Other Infrastructure Management Tools =

There are many other infrastructure management tools-

# [https://juju.ubuntu.com/ Juju] - Ubuntu juju is a tool for virtual infrastructure automation
# [http://cfengine.com Cfengine] is a much older tool that is used for large scale infrastructure automation. Puppet was originally inspired by Cfengine.
# [http://saltstack.com/community.html SaltStack] - this is another configuration management system written in Python

This is by no means an exhaustive list and is one that is growing - reflecting the trend towards heavier and more all encompassing automation.

= Conclusion =

This comparison was not meant to tout one of the tools - Puppet or Chef - over the other but merely to introduce and contrast both of them and see how they can be used with standard Infrastructure Management scenarios.

It is up to the reader to judge his/her needs and apply the tool for the job.

= Further Reading/References =
1. [http://docs.opscode.com/chef_overview.html Opscode Chef] 
2. [http://docs.opscode.com/knife_node.html About Knife]

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T01:38:41Z

Grmenon: /* Conclusion */

= Introduction =

Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size and complexity automated tools are becoming the norm as opposed to the exception.

Here we shall compare two popular Infrastructure Management Tools-

* Puppet
* Chef

We shall begin with an initial overview of both tools to familiarize the reader. We will then take a hypothetical scenarion and try and solve the problems in those scenario using Puppet and Chef.

We will then compare the solutions and compare how each tool handles the same problem. We do not intend to categorically state the superiority or suitability of one tool over the other but to illustrate their usage and let the reader choose them as per her/his need.

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

INSERT DIAGRAM

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}

An example would be -

service { 'rails s':
ensure => 'running'
}

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

Opscode Chef is an infrastructure automation and configuration management framework.

Chef is used for configuration, deployment and scalability of servers and applications for eg: it can be used to deploy and manage network in cloud, on-site or a hybrid one with n number of servers. This helps organization to save time, solve mission-critical challenges, accelerate time to market, manage scale and complexity, and safeguard systems.

Chef client works using the abstraction definitions know as cookbooks and recipes that are written in Ruby. Chef client uses this definitions to build and manage servers in an infrastructure.
== Elements ==
Basically an organization consists of following elements with chef configuration:

# Node : Any physical, virtual server is referred to as node.
# Knife : An interface between the local chef-repo and the server is known as knife. It is a command line tool using which users can manage other elements of the setup.
# Workstation : It is a computer used to interact with the single server. It is also used by users to run the knife command.
# Repository : usually referred as chef repo is the location wher all data objects are stored like cookbooks, configuration files etc.
# Hoster server: The stores manages cookbooks and all the configuration details that need to be applied to nodes.
# Chef-client : Chef client acts as an interface between the node and hosted server. Node gets all the configuration details from server via the client.
# Cookbook : Cookbook stores the configuration details depending upon the scenario for eg: in order to install Apache Tomcat server it contains details about installation, upgradation, components needed to support the server.

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===
<ol style="margin-left: 5.2em;">
<li>Install chef server on the server machine</li>
<li>Setup workstation:</li>
:* Pre-requisites : The workstation must have a Virtual box , vagrant(command line interface tool), Git( needed for Version Control Systems)
:* Use knife client command to establish an API client identity for the workstation in order to make authenticated requests to the server. Setup chef-repo in the workstation.
<li>Create Cookbooks:</li>
:* In order to use community cookbooks one can just run the following command:

Syntax : knife cookbook site install <cook_name>

Example: knife cookbook site install apt.

:Note: Chef has maintained a list of community cookbooks which are commonly covered under the use cases like apache2, mysql,php. So a user can leverage the community cookbooks instead of creating their own cookbooks.

:* In order to create our own cookbook:

Syntax : knife cookbook create cookbook_name

:The file default.rb in the path chef-repo/cookbooks/aliases/recipes/default.rb can be edited in order to create recipes.

:To make these cookbooks available to the nodes, it is first uploaded in the chef server.
<li>Setup chef client: Using knife tool in workstation, chef clients are setup on nodes.</li>
</ol>

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a Node ==

Our sample project will be using MongoDB. So the next task will be setting up the MongoDB instances.

=== Puppet ===

As we have seen in Puppet a machine’s configuration is specified in terms of resources. To tell Puppet to set up those resources on a machine we use a Manifest that consists of various resources.

But setting up a Manifest that covers all the Resources will quickly become unwieldly.

Which is why Puppet uses Classes and Modules.

A Class is a named collection of resources. A Module encloses a Class and is something that can be included in other Puppet files thus enabling reuse. There is a large collection of Modules called the Puppet Forge that has been written covering different type of uses.

Thus setting up a manifest for a MongoDB server is as simple as-

class { ‘mongodb’ :
init => ‘sysv’,
version => ‘1.1.1’ # installing a specific version of mongodb
}
include mongodb
# custom puppet code as per our requirments

Thus different Classes can be written which use different Modules for different uses. We can define a different class for our web server , for our file share and so on.

Once a Class has been defined by us that is tailor made to our uses setting up more instances is simply a question of provisioning the node and running the Puppet class on it.

This supports quick and simple scaling up.

=== Chef ===

Configuration in Chef is done by defining Role. Role helps in setting required attributes and tasks for a node. A node can have 0 or more roles assigned to it. Also a role can be assigned to many nodes. Roles thus helps in defining processes for a particular node.

Let us consider to configure a role for mongoDb server for a application:
<ol style="margin-left: 5.2em;">
<li>A role can be created by creating a file under roles directory for eg : roles/role_file.rb</li>

<li>This file is composed of
:name : defines the node.
:description: describes about the node.
:attributes: attributes specific to the node which needs to be configured.
:run lists: tasks assigned to the node.
:Sample-
[[File:codeMongo.JPG]]

</li>
<li>This file is then uploaded to the chef server using following command:</li>

knife role from file roles/redis.rb

<li>Role is assigned to the server during bootstrap(explained in next section) using command :</li>

knife ec2 server create

</li>

</ol>

=== How They Compare ===

== Setting Up A Node ==

=== Puppet ===

In Puppet once we have identified what will go into defining something that we will need an instance for - as we have seen above say configuring a MongoDB instance - we need to select a fresh instance and apply the manifest to it.

As we have seen Puppet Agents will automatically sync with the Puppet Master. So the only task left for us is to

# Provision a new instance
# Install Puppet on it
# Classify it as a particular type of node.

The last two commands can be combined using the puppet init command-

puppet node init \
--node-group=db \
…. ( login related options)
ec2-71-73-79-83.compute-1.amazonaws.com

Here we have specified the fresh instance and initialized as belonging to the group of nodes as db.

Once this process is done the instance will be classified as a db instance and will automatically begin syncing with the Puppet Master.

=== Chef ===

Pre-requisites : IP address / FQDN of the node

Run the knife bootstrap command

The knife bootstrap command requires the FQDN or the IP address for the node in order to complete the bootstrap operation. Run the knife bootstrap command. It checks for .pem files for authentication. If not present then the node goes through the registration phase in order to establish connection between the client and the server and for exchange of authorized messages. The attributes set in the role of the node are compared against the configuration details of the node and run contents of that role’s run list are applied to the node’s configuration details.

=== How They Compare ===

== Node Management ==

=== Foreman ===

Foreman is a popular tool that is used in conjunction with Puppet. It provides a Web based GUI that lets users manage Puppet nodes.

As an example the following screenshot illustrates how Foreman can be used to manage multiple nodes across multiple environments-

[[File:env.png]]

We can apply various filters here to narrow down on a particular instance or conversely modify multiple nodes at the same time.

=== Knife ===

Node management in Chef includes users to create, read, update and delete nodes. This can be done either by using Management Console or the command line tool Knife.

Knife command the following syntax in order to manage the nodes :

$ knife node [ARGUMENT] (options)

where ARGUMENT could be any one of the management options like bulk delete, create, edit, delete,list.

For example: to perform a bulk delete, a regular expression is prepared to identify the nodes to be deleted. This is then passed as the option.

$ knife node bulk delete "node^[0-3]{6}$"

User needs to have specific user level permission in order to do the specific task. For eg: You need create permissions on the global nodes level to create a node.

=== How They Compare ===

== Config Files Deployment ==

=== Puppet ===

In Puppet files are treated as resources. So we could specify for an NFS node as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => "/some/path 192.168.100.1(rw,sync)”,
}

However the content would be different for different nodes as the environment would change.

Instead of creating a different resource for each template Puppet supports templatization of config files using Embedded Ruby (ERB).

We can re-write the same resource as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => template(“nfs/mount_conf.erb”),
}

where the mount_conf.erb file would contain Embedded Ruby code along with the appropriate Ruby variables which would automatically be compiled to give the right values to produce the conf file.

We can apply this same manifest file to all nodes across all the instances.

=== Chef ===

=== How They Compare ===

= Other Infrastructure Management Tools =

There are many other infrastructure management tools-

# [https://juju.ubuntu.com/ Juju] - Ubuntu juju is a tool for virtual infrastructure automation
# [http://cfengine.com Cfengine] is a much older tool that is used for large scale infrastructure automation. Puppet was originally inspired by Cfengine.
# [http://saltstack.com/community.html SaltStack] - this is another configuration management system written in Python

This is by no means an exhaustive list and is one that is growing - reflecting the trend towards heavier and more all encompassing automation.

= Conclusion =

This comparison was not meant to tout one of the tools - Puppet or Chef - over the other but merely to introduce and contrast both of them and see how they can be used with standard Infrastructure Management scenarios.

It is up to the reader to judge his/her needs and apply the tool for the job.

= Further Reading/References =

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T01:25:55Z

Grmenon: /* Further Reading */

= Introduction =

Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size and complexity automated tools are becoming the norm as opposed to the exception.

Here we shall compare two popular Infrastructure Management Tools-

* Puppet
* Chef

We shall begin with an initial overview of both tools to familiarize the reader. We will then take a hypothetical scenarion and try and solve the problems in those scenario using Puppet and Chef.

We will then compare the solutions and compare how each tool handles the same problem. We do not intend to categorically state the superiority or suitability of one tool over the other but to illustrate their usage and let the reader choose them as per her/his need.

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

INSERT DIAGRAM

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}

An example would be -

service { 'rails s':
ensure => 'running'
}

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

Opscode Chef is an infrastructure automation and configuration management framework.

Chef is used for configuration, deployment and scalability of servers and applications for eg: it can be used to deploy and manage network in cloud, on-site or a hybrid one with n number of servers. This helps organization to save time, solve mission-critical challenges, accelerate time to market, manage scale and complexity, and safeguard systems.

Chef client works using the abstraction definitions know as cookbooks and recipes that are written in Ruby. Chef client uses this definitions to build and manage servers in an infrastructure.
== Elements ==
Basically an organization consists of following elements with chef configuration:

# Node : Any physical, virtual server is referred to as node.
# Knife : An interface between the local chef-repo and the server is known as knife. It is a command line tool using which users can manage other elements of the setup.
# Workstation : It is a computer used to interact with the single server. It is also used by users to run the knife command.
# Repository : usually referred as chef repo is the location wher all data objects are stored like cookbooks, configuration files etc.
# Hoster server: The stores manages cookbooks and all the configuration details that need to be applied to nodes.
# Chef-client : Chef client acts as an interface between the node and hosted server. Node gets all the configuration details from server via the client.
# Cookbook : Cookbook stores the configuration details depending upon the scenario for eg: in order to install Apache Tomcat server it contains details about installation, upgradation, components needed to support the server.

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===
<ol style="margin-left: 5.2em;">
<li>Install chef server on the server machine</li>
<li>Setup workstation:</li>
:* Pre-requisites : The workstation must have a Virtual box , vagrant(command line interface tool), Git( needed for Version Control Systems)
:* Use knife client command to establish an API client identity for the workstation in order to make authenticated requests to the server. Setup chef-repo in the workstation.
<li>Create Cookbooks:</li>
:* In order to use community cookbooks one can just run the following command:

Syntax : knife cookbook site install <cook_name>

Example: knife cookbook site install apt.

:Note: Chef has maintained a list of community cookbooks which are commonly covered under the use cases like apache2, mysql,php. So a user can leverage the community cookbooks instead of creating their own cookbooks.

:* In order to create our own cookbook:

Syntax : knife cookbook create cookbook_name

:The file default.rb in the path chef-repo/cookbooks/aliases/recipes/default.rb can be edited in order to create recipes.

:To make these cookbooks available to the nodes, it is first uploaded in the chef server.
<li>Setup chef client: Using knife tool in workstation, chef clients are setup on nodes.</li>
</ol>

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a Node ==

Our sample project will be using MongoDB. So the next task will be setting up the MongoDB instances.

=== Puppet ===

As we have seen in Puppet a machine’s configuration is specified in terms of resources. To tell Puppet to set up those resources on a machine we use a Manifest that consists of various resources.

But setting up a Manifest that covers all the Resources will quickly become unwieldly.

Which is why Puppet uses Classes and Modules.

A Class is a named collection of resources. A Module encloses a Class and is something that can be included in other Puppet files thus enabling reuse. There is a large collection of Modules called the Puppet Forge that has been written covering different type of uses.

Thus setting up a manifest for a MongoDB server is as simple as-

class { ‘mongodb’ :
init => ‘sysv’,
version => ‘1.1.1’ # installing a specific version of mongodb
}
include mongodb
# custom puppet code as per our requirments

Thus different Classes can be written which use different Modules for different uses. We can define a different class for our web server , for our file share and so on.

Once a Class has been defined by us that is tailor made to our uses setting up more instances is simply a question of provisioning the node and running the Puppet class on it.

This supports quick and simple scaling up.

=== Chef ===

Configuration in Chef is done by defining Role. Role helps in setting required attributes and tasks for a node. A node can have 0 or more roles assigned to it. Also a role can be assigned to many nodes. Roles thus helps in defining processes for a particular node.

Let us consider to configure a role for mongoDb server for a application:
<ol style="margin-left: 5.2em;">
<li>A role can be created by creating a file under roles directory for eg : roles/role_file.rb</li>

<li>This file is composed of
:name : defines the node.
:description: describes about the node.
:attributes: attributes specific to the node which needs to be configured.
:run lists: tasks assigned to the node.
:Sample-
[[File:codeMongo.JPG]]

</li>
<li>This file is then uploaded to the chef server using following command:</li>

knife role from file roles/redis.rb

<li>Role is assigned to the server during bootstrap(explained in next section) using command :</li>

knife ec2 server create

</li>

</ol>

=== How They Compare ===

== Setting Up A Node ==

=== Puppet ===

In Puppet once we have identified what will go into defining something that we will need an instance for - as we have seen above say configuring a MongoDB instance - we need to select a fresh instance and apply the manifest to it.

As we have seen Puppet Agents will automatically sync with the Puppet Master. So the only task left for us is to

# Provision a new instance
# Install Puppet on it
# Classify it as a particular type of node.

The last two commands can be combined using the puppet init command-

puppet node init \
--node-group=db \
…. ( login related options)
ec2-71-73-79-83.compute-1.amazonaws.com

Here we have specified the fresh instance and initialized as belonging to the group of nodes as db.

Once this process is done the instance will be classified as a db instance and will automatically begin syncing with the Puppet Master.

=== Chef ===

Pre-requisites : IP address / FQDN of the node

Run the knife bootstrap command

The knife bootstrap command requires the FQDN or the IP address for the node in order to complete the bootstrap operation. Run the knife bootstrap command. It checks for .pem files for authentication. If not present then the node goes through the registration phase in order to establish connection between the client and the server and for exchange of authorized messages. The attributes set in the role of the node are compared against the configuration details of the node and run contents of that role’s run list are applied to the node’s configuration details.

=== How They Compare ===

== Node Management ==

=== Foreman ===

Foreman is a popular tool that is used in conjunction with Puppet. It provides a Web based GUI that lets users manage Puppet nodes.

As an example the following screenshot illustrates how Foreman can be used to manage multiple nodes across multiple environments-

[[File:env.png]]

We can apply various filters here to narrow down on a particular instance or conversely modify multiple nodes at the same time.

=== Knife ===

=== How They Compare ===

== Config Files Deployment ==

=== Puppet ===

In Puppet files are treated as resources. So we could specify for an NFS node as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => "/some/path 192.168.100.1(rw,sync)”,
}

However the content would be different for different nodes as the environment would change.

Instead of creating a different resource for each template Puppet supports templatization of config files using Embedded Ruby (ERB).

We can re-write the same resource as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => template(“nfs/mount_conf.erb”),
}

where the mount_conf.erb file would contain Embedded Ruby code along with the appropriate Ruby variables which would automatically be compiled to give the right values to produce the conf file.

We can apply this same manifest file to all nodes across all the instances.

=== Chef ===

=== How They Compare ===

= Other Infrastructure Management Tools =

There are many other infrastructure management tools-

# [https://juju.ubuntu.com/ Juju] - Ubuntu juju is a tool for virtual infrastructure automation
# [http://cfengine.com Cfengine] is a much older tool that is used for large scale infrastructure automation. Puppet was originally inspired by Cfengine.
# [http://saltstack.com/community.html SaltStack] - this is another configuration management system written in Python

This is by no means an exhaustive list and is one that is growing - reflecting the trend towards heavier and more all encompassing automation.

= Conclusion =
= Further Reading/References =

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T01:25:43Z

Grmenon: /* References */

= Introduction =

Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size and complexity automated tools are becoming the norm as opposed to the exception.

Here we shall compare two popular Infrastructure Management Tools-

* Puppet
* Chef

We shall begin with an initial overview of both tools to familiarize the reader. We will then take a hypothetical scenarion and try and solve the problems in those scenario using Puppet and Chef.

We will then compare the solutions and compare how each tool handles the same problem. We do not intend to categorically state the superiority or suitability of one tool over the other but to illustrate their usage and let the reader choose them as per her/his need.

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

INSERT DIAGRAM

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}

An example would be -

service { 'rails s':
ensure => 'running'
}

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

Opscode Chef is an infrastructure automation and configuration management framework.

Chef is used for configuration, deployment and scalability of servers and applications for eg: it can be used to deploy and manage network in cloud, on-site or a hybrid one with n number of servers. This helps organization to save time, solve mission-critical challenges, accelerate time to market, manage scale and complexity, and safeguard systems.

Chef client works using the abstraction definitions know as cookbooks and recipes that are written in Ruby. Chef client uses this definitions to build and manage servers in an infrastructure.
== Elements ==
Basically an organization consists of following elements with chef configuration:

# Node : Any physical, virtual server is referred to as node.
# Knife : An interface between the local chef-repo and the server is known as knife. It is a command line tool using which users can manage other elements of the setup.
# Workstation : It is a computer used to interact with the single server. It is also used by users to run the knife command.
# Repository : usually referred as chef repo is the location wher all data objects are stored like cookbooks, configuration files etc.
# Hoster server: The stores manages cookbooks and all the configuration details that need to be applied to nodes.
# Chef-client : Chef client acts as an interface between the node and hosted server. Node gets all the configuration details from server via the client.
# Cookbook : Cookbook stores the configuration details depending upon the scenario for eg: in order to install Apache Tomcat server it contains details about installation, upgradation, components needed to support the server.

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===
<ol style="margin-left: 5.2em;">
<li>Install chef server on the server machine</li>
<li>Setup workstation:</li>
:* Pre-requisites : The workstation must have a Virtual box , vagrant(command line interface tool), Git( needed for Version Control Systems)
:* Use knife client command to establish an API client identity for the workstation in order to make authenticated requests to the server. Setup chef-repo in the workstation.
<li>Create Cookbooks:</li>
:* In order to use community cookbooks one can just run the following command:

Syntax : knife cookbook site install <cook_name>

Example: knife cookbook site install apt.

:Note: Chef has maintained a list of community cookbooks which are commonly covered under the use cases like apache2, mysql,php. So a user can leverage the community cookbooks instead of creating their own cookbooks.

:* In order to create our own cookbook:

Syntax : knife cookbook create cookbook_name

:The file default.rb in the path chef-repo/cookbooks/aliases/recipes/default.rb can be edited in order to create recipes.

:To make these cookbooks available to the nodes, it is first uploaded in the chef server.
<li>Setup chef client: Using knife tool in workstation, chef clients are setup on nodes.</li>
</ol>

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a Node ==

Our sample project will be using MongoDB. So the next task will be setting up the MongoDB instances.

=== Puppet ===

As we have seen in Puppet a machine’s configuration is specified in terms of resources. To tell Puppet to set up those resources on a machine we use a Manifest that consists of various resources.

But setting up a Manifest that covers all the Resources will quickly become unwieldly.

Which is why Puppet uses Classes and Modules.

A Class is a named collection of resources. A Module encloses a Class and is something that can be included in other Puppet files thus enabling reuse. There is a large collection of Modules called the Puppet Forge that has been written covering different type of uses.

Thus setting up a manifest for a MongoDB server is as simple as-

class { ‘mongodb’ :
init => ‘sysv’,
version => ‘1.1.1’ # installing a specific version of mongodb
}
include mongodb
# custom puppet code as per our requirments

Thus different Classes can be written which use different Modules for different uses. We can define a different class for our web server , for our file share and so on.

Once a Class has been defined by us that is tailor made to our uses setting up more instances is simply a question of provisioning the node and running the Puppet class on it.

This supports quick and simple scaling up.

=== Chef ===

Configuration in Chef is done by defining Role. Role helps in setting required attributes and tasks for a node. A node can have 0 or more roles assigned to it. Also a role can be assigned to many nodes. Roles thus helps in defining processes for a particular node.

Let us consider to configure a role for mongoDb server for a application:
<ol style="margin-left: 5.2em;">
<li>A role can be created by creating a file under roles directory for eg : roles/role_file.rb</li>

<li>This file is composed of
:name : defines the node.
:description: describes about the node.
:attributes: attributes specific to the node which needs to be configured.
:run lists: tasks assigned to the node.
:Sample-
[[File:codeMongo.JPG]]

</li>
<li>This file is then uploaded to the chef server using following command:</li>

knife role from file roles/redis.rb

<li>Role is assigned to the server during bootstrap(explained in next section) using command :</li>

knife ec2 server create

</li>

</ol>

=== How They Compare ===

== Setting Up A Node ==

=== Puppet ===

In Puppet once we have identified what will go into defining something that we will need an instance for - as we have seen above say configuring a MongoDB instance - we need to select a fresh instance and apply the manifest to it.

As we have seen Puppet Agents will automatically sync with the Puppet Master. So the only task left for us is to

# Provision a new instance
# Install Puppet on it
# Classify it as a particular type of node.

The last two commands can be combined using the puppet init command-

puppet node init \
--node-group=db \
…. ( login related options)
ec2-71-73-79-83.compute-1.amazonaws.com

Here we have specified the fresh instance and initialized as belonging to the group of nodes as db.

Once this process is done the instance will be classified as a db instance and will automatically begin syncing with the Puppet Master.

=== Chef ===

Pre-requisites : IP address / FQDN of the node

Run the knife bootstrap command

The knife bootstrap command requires the FQDN or the IP address for the node in order to complete the bootstrap operation. Run the knife bootstrap command. It checks for .pem files for authentication. If not present then the node goes through the registration phase in order to establish connection between the client and the server and for exchange of authorized messages. The attributes set in the role of the node are compared against the configuration details of the node and run contents of that role’s run list are applied to the node’s configuration details.

=== How They Compare ===

== Node Management ==

=== Foreman ===

Foreman is a popular tool that is used in conjunction with Puppet. It provides a Web based GUI that lets users manage Puppet nodes.

As an example the following screenshot illustrates how Foreman can be used to manage multiple nodes across multiple environments-

[[File:env.png]]

We can apply various filters here to narrow down on a particular instance or conversely modify multiple nodes at the same time.

=== Knife ===

=== How They Compare ===

== Config Files Deployment ==

=== Puppet ===

In Puppet files are treated as resources. So we could specify for an NFS node as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => "/some/path 192.168.100.1(rw,sync)”,
}

However the content would be different for different nodes as the environment would change.

Instead of creating a different resource for each template Puppet supports templatization of config files using Embedded Ruby (ERB).

We can re-write the same resource as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => template(“nfs/mount_conf.erb”),
}

where the mount_conf.erb file would contain Embedded Ruby code along with the appropriate Ruby variables which would automatically be compiled to give the right values to produce the conf file.

We can apply this same manifest file to all nodes across all the instances.

=== Chef ===

=== How They Compare ===

= Other Infrastructure Management Tools =

There are many other infrastructure management tools-

# [https://juju.ubuntu.com/ Juju] - Ubuntu juju is a tool for virtual infrastructure automation
# [http://cfengine.com Cfengine] is a much older tool that is used for large scale infrastructure automation. Puppet was originally inspired by Cfengine.
# [http://saltstack.com/community.html SaltStack] - this is another configuration management system written in Python

This is by no means an exhaustive list and is one that is growing - reflecting the trend towards heavier and more all encompassing automation.

= Conclusion =
= Further Reading =

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T01:23:25Z

Grmenon: /* Other Infrastructure Management Tools */

= Introduction =

Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size and complexity automated tools are becoming the norm as opposed to the exception.

Here we shall compare two popular Infrastructure Management Tools-

* Puppet
* Chef

We shall begin with an initial overview of both tools to familiarize the reader. We will then take a hypothetical scenarion and try and solve the problems in those scenario using Puppet and Chef.

We will then compare the solutions and compare how each tool handles the same problem. We do not intend to categorically state the superiority or suitability of one tool over the other but to illustrate their usage and let the reader choose them as per her/his need.

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

INSERT DIAGRAM

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}

An example would be -

service { 'rails s':
ensure => 'running'
}

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

Opscode Chef is an infrastructure automation and configuration management framework.

Chef is used for configuration, deployment and scalability of servers and applications for eg: it can be used to deploy and manage network in cloud, on-site or a hybrid one with n number of servers. This helps organization to save time, solve mission-critical challenges, accelerate time to market, manage scale and complexity, and safeguard systems.

Chef client works using the abstraction definitions know as cookbooks and recipes that are written in Ruby. Chef client uses this definitions to build and manage servers in an infrastructure.
== Elements ==
Basically an organization consists of following elements with chef configuration:

# Node : Any physical, virtual server is referred to as node.
# Knife : An interface between the local chef-repo and the server is known as knife. It is a command line tool using which users can manage other elements of the setup.
# Workstation : It is a computer used to interact with the single server. It is also used by users to run the knife command.
# Repository : usually referred as chef repo is the location wher all data objects are stored like cookbooks, configuration files etc.
# Hoster server: The stores manages cookbooks and all the configuration details that need to be applied to nodes.
# Chef-client : Chef client acts as an interface between the node and hosted server. Node gets all the configuration details from server via the client.
# Cookbook : Cookbook stores the configuration details depending upon the scenario for eg: in order to install Apache Tomcat server it contains details about installation, upgradation, components needed to support the server.

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===
<ol style="margin-left: 5.2em;">
<li>Install chef server on the server machine</li>
<li>Setup workstation:</li>
:* Pre-requisites : The workstation must have a Virtual box , vagrant(command line interface tool), Git( needed for Version Control Systems)
:* Use knife client command to establish an API client identity for the workstation in order to make authenticated requests to the server. Setup chef-repo in the workstation.
<li>Create Cookbooks:</li>
:* In order to use community cookbooks one can just run the following command:

Syntax : knife cookbook site install <cook_name>

Example: knife cookbook site install apt.

:Note: Chef has maintained a list of community cookbooks which are commonly covered under the use cases like apache2, mysql,php. So a user can leverage the community cookbooks instead of creating their own cookbooks.

:* In order to create our own cookbook:

Syntax : knife cookbook create cookbook_name

:The file default.rb in the path chef-repo/cookbooks/aliases/recipes/default.rb can be edited in order to create recipes.

:To make these cookbooks available to the nodes, it is first uploaded in the chef server.
<li>Setup chef client: Using knife tool in workstation, chef clients are setup on nodes.</li>
</ol>

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a Node ==

Our sample project will be using MongoDB. So the next task will be setting up the MongoDB instances.

=== Puppet ===

As we have seen in Puppet a machine’s configuration is specified in terms of resources. To tell Puppet to set up those resources on a machine we use a Manifest that consists of various resources.

But setting up a Manifest that covers all the Resources will quickly become unwieldly.

Which is why Puppet uses Classes and Modules.

A Class is a named collection of resources. A Module encloses a Class and is something that can be included in other Puppet files thus enabling reuse. There is a large collection of Modules called the Puppet Forge that has been written covering different type of uses.

Thus setting up a manifest for a MongoDB server is as simple as-

class { ‘mongodb’ :
init => ‘sysv’,
version => ‘1.1.1’ # installing a specific version of mongodb
}
include mongodb
# custom puppet code as per our requirments

Thus different Classes can be written which use different Modules for different uses. We can define a different class for our web server , for our file share and so on.

Once a Class has been defined by us that is tailor made to our uses setting up more instances is simply a question of provisioning the node and running the Puppet class on it.

This supports quick and simple scaling up.

=== Chef ===

Configuration in Chef is done by defining Role. Role helps in setting required attributes and tasks for a node. A node can have 0 or more roles assigned to it. Also a role can be assigned to many nodes. Roles thus helps in defining processes for a particular node.

Let us consider to configure a role for mongoDb server for a application:
<ol style="margin-left: 5.2em;">
<li>A role can be created by creating a file under roles directory for eg : roles/role_file.rb</li>

<li>This file is composed of
:name : defines the node.
:description: describes about the node.
:attributes: attributes specific to the node which needs to be configured.
:run lists: tasks assigned to the node.
:Sample-
[[File:codeMongo.JPG]]

</li>
<li>This file is then uploaded to the chef server using following command:</li>

knife role from file roles/redis.rb

<li>Role is assigned to the server during bootstrap(explained in next section) using command :</li>

knife ec2 server create

</li>

</ol>

=== How They Compare ===

== Setting Up A Node ==

=== Puppet ===

In Puppet once we have identified what will go into defining something that we will need an instance for - as we have seen above say configuring a MongoDB instance - we need to select a fresh instance and apply the manifest to it.

As we have seen Puppet Agents will automatically sync with the Puppet Master. So the only task left for us is to

# Provision a new instance
# Install Puppet on it
# Classify it as a particular type of node.

The last two commands can be combined using the puppet init command-

puppet node init \
--node-group=db \
…. ( login related options)
ec2-71-73-79-83.compute-1.amazonaws.com

Here we have specified the fresh instance and initialized as belonging to the group of nodes as db.

Once this process is done the instance will be classified as a db instance and will automatically begin syncing with the Puppet Master.

=== Chef ===

Pre-requisites : IP address / FQDN of the node

Run the knife bootstrap command

The knife bootstrap command requires the FQDN or the IP address for the node in order to complete the bootstrap operation. Run the knife bootstrap command. It checks for .pem files for authentication. If not present then the node goes through the registration phase in order to establish connection between the client and the server and for exchange of authorized messages. The attributes set in the role of the node are compared against the configuration details of the node and run contents of that role’s run list are applied to the node’s configuration details.

=== How They Compare ===

== Node Management ==

=== Foreman ===

Foreman is a popular tool that is used in conjunction with Puppet. It provides a Web based GUI that lets users manage Puppet nodes.

As an example the following screenshot illustrates how Foreman can be used to manage multiple nodes across multiple environments-

[[File:env.png]]

We can apply various filters here to narrow down on a particular instance or conversely modify multiple nodes at the same time.

=== Knife ===

=== How They Compare ===

== Config Files Deployment ==

=== Puppet ===

In Puppet files are treated as resources. So we could specify for an NFS node as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => "/some/path 192.168.100.1(rw,sync)”,
}

However the content would be different for different nodes as the environment would change.

Instead of creating a different resource for each template Puppet supports templatization of config files using Embedded Ruby (ERB).

We can re-write the same resource as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => template(“nfs/mount_conf.erb”),
}

where the mount_conf.erb file would contain Embedded Ruby code along with the appropriate Ruby variables which would automatically be compiled to give the right values to produce the conf file.

We can apply this same manifest file to all nodes across all the instances.

=== Chef ===

=== How They Compare ===

= Other Infrastructure Management Tools =

There are many other infrastructure management tools-

# [https://juju.ubuntu.com/ Juju] - Ubuntu juju is a tool for virtual infrastructure automation
# [http://cfengine.com Cfengine] is a much older tool that is used for large scale infrastructure automation. Puppet was originally inspired by Cfengine.
# [http://saltstack.com/community.html SaltStack] - this is another configuration management system written in Python

This is by no means an exhaustive list and is one that is growing - reflecting the trend towards heavier and more all encompassing automation.

= Conclusion =
= Further Reading =
= References =

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T01:23:03Z

Grmenon: /* Other Infrastructure Management Tools */

= Introduction =

Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size and complexity automated tools are becoming the norm as opposed to the exception.

Here we shall compare two popular Infrastructure Management Tools-

* Puppet
* Chef

We shall begin with an initial overview of both tools to familiarize the reader. We will then take a hypothetical scenarion and try and solve the problems in those scenario using Puppet and Chef.

We will then compare the solutions and compare how each tool handles the same problem. We do not intend to categorically state the superiority or suitability of one tool over the other but to illustrate their usage and let the reader choose them as per her/his need.

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

INSERT DIAGRAM

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}

An example would be -

service { 'rails s':
ensure => 'running'
}

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

Opscode Chef is an infrastructure automation and configuration management framework.

Chef is used for configuration, deployment and scalability of servers and applications for eg: it can be used to deploy and manage network in cloud, on-site or a hybrid one with n number of servers. This helps organization to save time, solve mission-critical challenges, accelerate time to market, manage scale and complexity, and safeguard systems.

Chef client works using the abstraction definitions know as cookbooks and recipes that are written in Ruby. Chef client uses this definitions to build and manage servers in an infrastructure.
== Elements ==
Basically an organization consists of following elements with chef configuration:

# Node : Any physical, virtual server is referred to as node.
# Knife : An interface between the local chef-repo and the server is known as knife. It is a command line tool using which users can manage other elements of the setup.
# Workstation : It is a computer used to interact with the single server. It is also used by users to run the knife command.
# Repository : usually referred as chef repo is the location wher all data objects are stored like cookbooks, configuration files etc.
# Hoster server: The stores manages cookbooks and all the configuration details that need to be applied to nodes.
# Chef-client : Chef client acts as an interface between the node and hosted server. Node gets all the configuration details from server via the client.
# Cookbook : Cookbook stores the configuration details depending upon the scenario for eg: in order to install Apache Tomcat server it contains details about installation, upgradation, components needed to support the server.

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===
<ol style="margin-left: 5.2em;">
<li>Install chef server on the server machine</li>
<li>Setup workstation:</li>
:* Pre-requisites : The workstation must have a Virtual box , vagrant(command line interface tool), Git( needed for Version Control Systems)
:* Use knife client command to establish an API client identity for the workstation in order to make authenticated requests to the server. Setup chef-repo in the workstation.
<li>Create Cookbooks:</li>
:* In order to use community cookbooks one can just run the following command:

Syntax : knife cookbook site install <cook_name>

Example: knife cookbook site install apt.

:Note: Chef has maintained a list of community cookbooks which are commonly covered under the use cases like apache2, mysql,php. So a user can leverage the community cookbooks instead of creating their own cookbooks.

:* In order to create our own cookbook:

Syntax : knife cookbook create cookbook_name

:The file default.rb in the path chef-repo/cookbooks/aliases/recipes/default.rb can be edited in order to create recipes.

:To make these cookbooks available to the nodes, it is first uploaded in the chef server.
<li>Setup chef client: Using knife tool in workstation, chef clients are setup on nodes.</li>
</ol>

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a Node ==

Our sample project will be using MongoDB. So the next task will be setting up the MongoDB instances.

=== Puppet ===

As we have seen in Puppet a machine’s configuration is specified in terms of resources. To tell Puppet to set up those resources on a machine we use a Manifest that consists of various resources.

But setting up a Manifest that covers all the Resources will quickly become unwieldly.

Which is why Puppet uses Classes and Modules.

A Class is a named collection of resources. A Module encloses a Class and is something that can be included in other Puppet files thus enabling reuse. There is a large collection of Modules called the Puppet Forge that has been written covering different type of uses.

Thus setting up a manifest for a MongoDB server is as simple as-

class { ‘mongodb’ :
init => ‘sysv’,
version => ‘1.1.1’ # installing a specific version of mongodb
}
include mongodb
# custom puppet code as per our requirments

Thus different Classes can be written which use different Modules for different uses. We can define a different class for our web server , for our file share and so on.

Once a Class has been defined by us that is tailor made to our uses setting up more instances is simply a question of provisioning the node and running the Puppet class on it.

This supports quick and simple scaling up.

=== Chef ===

Configuration in Chef is done by defining Role. Role helps in setting required attributes and tasks for a node. A node can have 0 or more roles assigned to it. Also a role can be assigned to many nodes. Roles thus helps in defining processes for a particular node.

Let us consider to configure a role for mongoDb server for a application:
<ol style="margin-left: 5.2em;">
<li>A role can be created by creating a file under roles directory for eg : roles/role_file.rb</li>

<li>This file is composed of
:name : defines the node.
:description: describes about the node.
:attributes: attributes specific to the node which needs to be configured.
:run lists: tasks assigned to the node.
:Sample-
[[File:codeMongo.JPG]]

</li>
<li>This file is then uploaded to the chef server using following command:</li>

knife role from file roles/redis.rb

<li>Role is assigned to the server during bootstrap(explained in next section) using command :</li>

knife ec2 server create

</li>

</ol>

=== How They Compare ===

== Setting Up A Node ==

=== Puppet ===

In Puppet once we have identified what will go into defining something that we will need an instance for - as we have seen above say configuring a MongoDB instance - we need to select a fresh instance and apply the manifest to it.

As we have seen Puppet Agents will automatically sync with the Puppet Master. So the only task left for us is to

# Provision a new instance
# Install Puppet on it
# Classify it as a particular type of node.

The last two commands can be combined using the puppet init command-

puppet node init \
--node-group=db \
…. ( login related options)
ec2-71-73-79-83.compute-1.amazonaws.com

Here we have specified the fresh instance and initialized as belonging to the group of nodes as db.

Once this process is done the instance will be classified as a db instance and will automatically begin syncing with the Puppet Master.

=== Chef ===

Pre-requisites : IP address / FQDN of the node

Run the knife bootstrap command

The knife bootstrap command requires the FQDN or the IP address for the node in order to complete the bootstrap operation. Run the knife bootstrap command. It checks for .pem files for authentication. If not present then the node goes through the registration phase in order to establish connection between the client and the server and for exchange of authorized messages. The attributes set in the role of the node are compared against the configuration details of the node and run contents of that role’s run list are applied to the node’s configuration details.

=== How They Compare ===

== Node Management ==

=== Foreman ===

Foreman is a popular tool that is used in conjunction with Puppet. It provides a Web based GUI that lets users manage Puppet nodes.

As an example the following screenshot illustrates how Foreman can be used to manage multiple nodes across multiple environments-

[[File:env.png]]

We can apply various filters here to narrow down on a particular instance or conversely modify multiple nodes at the same time.

=== Knife ===

=== How They Compare ===

== Config Files Deployment ==

=== Puppet ===

In Puppet files are treated as resources. So we could specify for an NFS node as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => "/some/path 192.168.100.1(rw,sync)”,
}

However the content would be different for different nodes as the environment would change.

Instead of creating a different resource for each template Puppet supports templatization of config files using Embedded Ruby (ERB).

We can re-write the same resource as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => template(“nfs/mount_conf.erb”),
}

where the mount_conf.erb file would contain Embedded Ruby code along with the appropriate Ruby variables which would automatically be compiled to give the right values to produce the conf file.

We can apply this same manifest file to all nodes across all the instances.

=== Chef ===

=== How They Compare ===

= Other Infrastructure Management Tools =

There are many other infrastructure management tools-

# [https://juju.ubuntu.com/ Juju] - Ubuntu juju is a tool for virtual infrastructure automation
# [http://cfengine.comCfengine] is a much older tool that is used for large scale infrastructure automation. Puppet was originally inspired by Cfengine.
# [http://saltstack.com/community.html SaltStack] - this is another configuration management system written in Python

This is by no means an exhaustive list and is one that is growing - reflecting the trend towards heavier and more all encompassing automation.

= Conclusion =
= Further Reading =
= References =

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T01:21:19Z

Grmenon: /* Other Infrastructure Management Tools */

= Introduction =

Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size and complexity automated tools are becoming the norm as opposed to the exception.

Here we shall compare two popular Infrastructure Management Tools-

* Puppet
* Chef

We shall begin with an initial overview of both tools to familiarize the reader. We will then take a hypothetical scenarion and try and solve the problems in those scenario using Puppet and Chef.

We will then compare the solutions and compare how each tool handles the same problem. We do not intend to categorically state the superiority or suitability of one tool over the other but to illustrate their usage and let the reader choose them as per her/his need.

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

INSERT DIAGRAM

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}

An example would be -

service { 'rails s':
ensure => 'running'
}

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

Opscode Chef is an infrastructure automation and configuration management framework.

Chef is used for configuration, deployment and scalability of servers and applications for eg: it can be used to deploy and manage network in cloud, on-site or a hybrid one with n number of servers. This helps organization to save time, solve mission-critical challenges, accelerate time to market, manage scale and complexity, and safeguard systems.

Chef client works using the abstraction definitions know as cookbooks and recipes that are written in Ruby. Chef client uses this definitions to build and manage servers in an infrastructure.
== Elements ==
Basically an organization consists of following elements with chef configuration:

# Node : Any physical, virtual server is referred to as node.
# Knife : An interface between the local chef-repo and the server is known as knife. It is a command line tool using which users can manage other elements of the setup.
# Workstation : It is a computer used to interact with the single server. It is also used by users to run the knife command.
# Repository : usually referred as chef repo is the location wher all data objects are stored like cookbooks, configuration files etc.
# Hoster server: The stores manages cookbooks and all the configuration details that need to be applied to nodes.
# Chef-client : Chef client acts as an interface between the node and hosted server. Node gets all the configuration details from server via the client.
# Cookbook : Cookbook stores the configuration details depending upon the scenario for eg: in order to install Apache Tomcat server it contains details about installation, upgradation, components needed to support the server.

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===
<ol style="margin-left: 5.2em;">
<li>Install chef server on the server machine</li>
<li>Setup workstation:</li>
:* Pre-requisites : The workstation must have a Virtual box , vagrant(command line interface tool), Git( needed for Version Control Systems)
:* Use knife client command to establish an API client identity for the workstation in order to make authenticated requests to the server. Setup chef-repo in the workstation.
<li>Create Cookbooks:</li>
:* In order to use community cookbooks one can just run the following command:

Syntax : knife cookbook site install <cook_name>

Example: knife cookbook site install apt.

:Note: Chef has maintained a list of community cookbooks which are commonly covered under the use cases like apache2, mysql,php. So a user can leverage the community cookbooks instead of creating their own cookbooks.

:* In order to create our own cookbook:

Syntax : knife cookbook create cookbook_name

:The file default.rb in the path chef-repo/cookbooks/aliases/recipes/default.rb can be edited in order to create recipes.

:To make these cookbooks available to the nodes, it is first uploaded in the chef server.
<li>Setup chef client: Using knife tool in workstation, chef clients are setup on nodes.</li>
</ol>

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a Node ==

Our sample project will be using MongoDB. So the next task will be setting up the MongoDB instances.

=== Puppet ===

As we have seen in Puppet a machine’s configuration is specified in terms of resources. To tell Puppet to set up those resources on a machine we use a Manifest that consists of various resources.

But setting up a Manifest that covers all the Resources will quickly become unwieldly.

Which is why Puppet uses Classes and Modules.

A Class is a named collection of resources. A Module encloses a Class and is something that can be included in other Puppet files thus enabling reuse. There is a large collection of Modules called the Puppet Forge that has been written covering different type of uses.

Thus setting up a manifest for a MongoDB server is as simple as-

class { ‘mongodb’ :
init => ‘sysv’,
version => ‘1.1.1’ # installing a specific version of mongodb
}
include mongodb
# custom puppet code as per our requirments

Thus different Classes can be written which use different Modules for different uses. We can define a different class for our web server , for our file share and so on.

Once a Class has been defined by us that is tailor made to our uses setting up more instances is simply a question of provisioning the node and running the Puppet class on it.

This supports quick and simple scaling up.

=== Chef ===

Configuration in Chef is done by defining Role. Role helps in setting required attributes and tasks for a node. A node can have 0 or more roles assigned to it. Also a role can be assigned to many nodes. Roles thus helps in defining processes for a particular node.

Let us consider to configure a role for mongoDb server for a application:
<ol style="margin-left: 5.2em;">
<li>A role can be created by creating a file under roles directory for eg : roles/role_file.rb</li>

<li>This file is composed of
:name : defines the node.
:description: describes about the node.
:attributes: attributes specific to the node which needs to be configured.
:run lists: tasks assigned to the node.
:Sample-
[[File:codeMongo.JPG]]

</li>
<li>This file is then uploaded to the chef server using following command:</li>

knife role from file roles/redis.rb

<li>Role is assigned to the server during bootstrap(explained in next section) using command :</li>

knife ec2 server create

</li>

</ol>

=== How They Compare ===

== Setting Up A Node ==

=== Puppet ===

In Puppet once we have identified what will go into defining something that we will need an instance for - as we have seen above say configuring a MongoDB instance - we need to select a fresh instance and apply the manifest to it.

As we have seen Puppet Agents will automatically sync with the Puppet Master. So the only task left for us is to

# Provision a new instance
# Install Puppet on it
# Classify it as a particular type of node.

The last two commands can be combined using the puppet init command-

puppet node init \
--node-group=db \
…. ( login related options)
ec2-71-73-79-83.compute-1.amazonaws.com

Here we have specified the fresh instance and initialized as belonging to the group of nodes as db.

Once this process is done the instance will be classified as a db instance and will automatically begin syncing with the Puppet Master.

=== Chef ===

Pre-requisites : IP address / FQDN of the node

Run the knife bootstrap command

The knife bootstrap command requires the FQDN or the IP address for the node in order to complete the bootstrap operation. Run the knife bootstrap command. It checks for .pem files for authentication. If not present then the node goes through the registration phase in order to establish connection between the client and the server and for exchange of authorized messages. The attributes set in the role of the node are compared against the configuration details of the node and run contents of that role’s run list are applied to the node’s configuration details.

=== How They Compare ===

== Node Management ==

=== Foreman ===

Foreman is a popular tool that is used in conjunction with Puppet. It provides a Web based GUI that lets users manage Puppet nodes.

As an example the following screenshot illustrates how Foreman can be used to manage multiple nodes across multiple environments-

[[File:env.png]]

We can apply various filters here to narrow down on a particular instance or conversely modify multiple nodes at the same time.

=== Knife ===

=== How They Compare ===

== Config Files Deployment ==

=== Puppet ===

In Puppet files are treated as resources. So we could specify for an NFS node as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => "/some/path 192.168.100.1(rw,sync)”,
}

However the content would be different for different nodes as the environment would change.

Instead of creating a different resource for each template Puppet supports templatization of config files using Embedded Ruby (ERB).

We can re-write the same resource as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => template(“nfs/mount_conf.erb”),
}

where the mount_conf.erb file would contain Embedded Ruby code along with the appropriate Ruby variables which would automatically be compiled to give the right values to produce the conf file.

We can apply this same manifest file to all nodes across all the instances.

=== Chef ===

=== How They Compare ===

= Other Infrastructure Management Tools =

There are many other infrastructure management tools-

# [https://juju.ubuntu.com/|Juju] - Ubuntu juju is a tool for virtual infrastructure automation
# [Cfengine]http://cfengine.com is a much older tool that is used for large scale infrastructure automation. Puppet was originally inspired by Cfengine.
# [SaltStack]http://saltstack.com/community.html - this is another configuration management system written in Python

This is by no means an exhaustive list and is one that is growing - reflecting the trend towards heavier and more all encompassing automation.

= Conclusion =
= Further Reading =
= References =

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T01:19:47Z

Grmenon: /* Other Infrastructure Management Tools */

= Introduction =

Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size and complexity automated tools are becoming the norm as opposed to the exception.

Here we shall compare two popular Infrastructure Management Tools-

* Puppet
* Chef

We shall begin with an initial overview of both tools to familiarize the reader. We will then take a hypothetical scenarion and try and solve the problems in those scenario using Puppet and Chef.

We will then compare the solutions and compare how each tool handles the same problem. We do not intend to categorically state the superiority or suitability of one tool over the other but to illustrate their usage and let the reader choose them as per her/his need.

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

INSERT DIAGRAM

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}

An example would be -

service { 'rails s':
ensure => 'running'
}

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

Opscode Chef is an infrastructure automation and configuration management framework.

Chef is used for configuration, deployment and scalability of servers and applications for eg: it can be used to deploy and manage network in cloud, on-site or a hybrid one with n number of servers. This helps organization to save time, solve mission-critical challenges, accelerate time to market, manage scale and complexity, and safeguard systems.

Chef client works using the abstraction definitions know as cookbooks and recipes that are written in Ruby. Chef client uses this definitions to build and manage servers in an infrastructure.
== Elements ==
Basically an organization consists of following elements with chef configuration:

# Node : Any physical, virtual server is referred to as node.
# Knife : An interface between the local chef-repo and the server is known as knife. It is a command line tool using which users can manage other elements of the setup.
# Workstation : It is a computer used to interact with the single server. It is also used by users to run the knife command.
# Repository : usually referred as chef repo is the location wher all data objects are stored like cookbooks, configuration files etc.
# Hoster server: The stores manages cookbooks and all the configuration details that need to be applied to nodes.
# Chef-client : Chef client acts as an interface between the node and hosted server. Node gets all the configuration details from server via the client.
# Cookbook : Cookbook stores the configuration details depending upon the scenario for eg: in order to install Apache Tomcat server it contains details about installation, upgradation, components needed to support the server.

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===
<ol style="margin-left: 5.2em;">
<li>Install chef server on the server machine</li>
<li>Setup workstation:</li>
:* Pre-requisites : The workstation must have a Virtual box , vagrant(command line interface tool), Git( needed for Version Control Systems)
:* Use knife client command to establish an API client identity for the workstation in order to make authenticated requests to the server. Setup chef-repo in the workstation.
<li>Create Cookbooks:</li>
:* In order to use community cookbooks one can just run the following command:

Syntax : knife cookbook site install <cook_name>

Example: knife cookbook site install apt.

:Note: Chef has maintained a list of community cookbooks which are commonly covered under the use cases like apache2, mysql,php. So a user can leverage the community cookbooks instead of creating their own cookbooks.

:* In order to create our own cookbook:

Syntax : knife cookbook create cookbook_name

:The file default.rb in the path chef-repo/cookbooks/aliases/recipes/default.rb can be edited in order to create recipes.

:To make these cookbooks available to the nodes, it is first uploaded in the chef server.
<li>Setup chef client: Using knife tool in workstation, chef clients are setup on nodes.</li>
</ol>

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a Node ==

Our sample project will be using MongoDB. So the next task will be setting up the MongoDB instances.

=== Puppet ===

As we have seen in Puppet a machine’s configuration is specified in terms of resources. To tell Puppet to set up those resources on a machine we use a Manifest that consists of various resources.

But setting up a Manifest that covers all the Resources will quickly become unwieldly.

Which is why Puppet uses Classes and Modules.

A Class is a named collection of resources. A Module encloses a Class and is something that can be included in other Puppet files thus enabling reuse. There is a large collection of Modules called the Puppet Forge that has been written covering different type of uses.

Thus setting up a manifest for a MongoDB server is as simple as-

class { ‘mongodb’ :
init => ‘sysv’,
version => ‘1.1.1’ # installing a specific version of mongodb
}
include mongodb
# custom puppet code as per our requirments

Thus different Classes can be written which use different Modules for different uses. We can define a different class for our web server , for our file share and so on.

Once a Class has been defined by us that is tailor made to our uses setting up more instances is simply a question of provisioning the node and running the Puppet class on it.

This supports quick and simple scaling up.

=== Chef ===

Configuration in Chef is done by defining Role. Role helps in setting required attributes and tasks for a node. A node can have 0 or more roles assigned to it. Also a role can be assigned to many nodes. Roles thus helps in defining processes for a particular node.

Let us consider to configure a role for mongoDb server for a application:
<ol style="margin-left: 5.2em;">
<li>A role can be created by creating a file under roles directory for eg : roles/role_file.rb</li>

<li>This file is composed of
:name : defines the node.
:description: describes about the node.
:attributes: attributes specific to the node which needs to be configured.
:run lists: tasks assigned to the node.
:Sample-
[[File:codeMongo.JPG]]

</li>
<li>This file is then uploaded to the chef server using following command:</li>

knife role from file roles/redis.rb

<li>Role is assigned to the server during bootstrap(explained in next section) using command :</li>

knife ec2 server create

</li>

</ol>

=== How They Compare ===

== Setting Up A Node ==

=== Puppet ===

In Puppet once we have identified what will go into defining something that we will need an instance for - as we have seen above say configuring a MongoDB instance - we need to select a fresh instance and apply the manifest to it.

As we have seen Puppet Agents will automatically sync with the Puppet Master. So the only task left for us is to

# Provision a new instance
# Install Puppet on it
# Classify it as a particular type of node.

The last two commands can be combined using the puppet init command-

puppet node init \
--node-group=db \
…. ( login related options)
ec2-71-73-79-83.compute-1.amazonaws.com

Here we have specified the fresh instance and initialized as belonging to the group of nodes as db.

Once this process is done the instance will be classified as a db instance and will automatically begin syncing with the Puppet Master.

=== Chef ===

Pre-requisites : IP address / FQDN of the node

Run the knife bootstrap command

The knife bootstrap command requires the FQDN or the IP address for the node in order to complete the bootstrap operation. Run the knife bootstrap command. It checks for .pem files for authentication. If not present then the node goes through the registration phase in order to establish connection between the client and the server and for exchange of authorized messages. The attributes set in the role of the node are compared against the configuration details of the node and run contents of that role’s run list are applied to the node’s configuration details.

=== How They Compare ===

== Node Management ==

=== Foreman ===

Foreman is a popular tool that is used in conjunction with Puppet. It provides a Web based GUI that lets users manage Puppet nodes.

As an example the following screenshot illustrates how Foreman can be used to manage multiple nodes across multiple environments-

[[File:env.png]]

We can apply various filters here to narrow down on a particular instance or conversely modify multiple nodes at the same time.

=== Knife ===

=== How They Compare ===

== Config Files Deployment ==

=== Puppet ===

In Puppet files are treated as resources. So we could specify for an NFS node as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => "/some/path 192.168.100.1(rw,sync)”,
}

However the content would be different for different nodes as the environment would change.

Instead of creating a different resource for each template Puppet supports templatization of config files using Embedded Ruby (ERB).

We can re-write the same resource as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => template(“nfs/mount_conf.erb”),
}

where the mount_conf.erb file would contain Embedded Ruby code along with the appropriate Ruby variables which would automatically be compiled to give the right values to produce the conf file.

We can apply this same manifest file to all nodes across all the instances.

=== Chef ===

=== How They Compare ===

= Other Infrastructure Management Tools =

There are many other infrastructure management tools-

# [Juju]https://juju.ubuntu.com/ - Ubuntu juju is a tool for virtual infrastructure automation
# [Cfengine]http://cfengine.com is a much older tool that is used for large scale infrastructure automation. Puppet was originally inspired by Cfengine.
# [SaltStack]http://saltstack.com/community.html - this is another configuration management system written in Python

This is by no means an exhaustive list and is one that is growing - reflecting the trend towards heavier and more all encompassing automation.

= Conclusion =
= Further Reading =
= References =

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T01:18:43Z

Grmenon: /* Other Infrastructure Management Tools */

= Introduction =

Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size and complexity automated tools are becoming the norm as opposed to the exception.

Here we shall compare two popular Infrastructure Management Tools-

* Puppet
* Chef

We shall begin with an initial overview of both tools to familiarize the reader. We will then take a hypothetical scenarion and try and solve the problems in those scenario using Puppet and Chef.

We will then compare the solutions and compare how each tool handles the same problem. We do not intend to categorically state the superiority or suitability of one tool over the other but to illustrate their usage and let the reader choose them as per her/his need.

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

INSERT DIAGRAM

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}

An example would be -

service { 'rails s':
ensure => 'running'
}

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

Opscode Chef is an infrastructure automation and configuration management framework.

Chef is used for configuration, deployment and scalability of servers and applications for eg: it can be used to deploy and manage network in cloud, on-site or a hybrid one with n number of servers. This helps organization to save time, solve mission-critical challenges, accelerate time to market, manage scale and complexity, and safeguard systems.

Chef client works using the abstraction definitions know as cookbooks and recipes that are written in Ruby. Chef client uses this definitions to build and manage servers in an infrastructure.
== Elements ==
Basically an organization consists of following elements with chef configuration:

# Node : Any physical, virtual server is referred to as node.
# Knife : An interface between the local chef-repo and the server is known as knife. It is a command line tool using which users can manage other elements of the setup.
# Workstation : It is a computer used to interact with the single server. It is also used by users to run the knife command.
# Repository : usually referred as chef repo is the location wher all data objects are stored like cookbooks, configuration files etc.
# Hoster server: The stores manages cookbooks and all the configuration details that need to be applied to nodes.
# Chef-client : Chef client acts as an interface between the node and hosted server. Node gets all the configuration details from server via the client.
# Cookbook : Cookbook stores the configuration details depending upon the scenario for eg: in order to install Apache Tomcat server it contains details about installation, upgradation, components needed to support the server.

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===
<ol style="margin-left: 5.2em;">
<li>Install chef server on the server machine</li>
<li>Setup workstation:</li>
:* Pre-requisites : The workstation must have a Virtual box , vagrant(command line interface tool), Git( needed for Version Control Systems)
:* Use knife client command to establish an API client identity for the workstation in order to make authenticated requests to the server. Setup chef-repo in the workstation.
<li>Create Cookbooks:</li>
:* In order to use community cookbooks one can just run the following command:

Syntax : knife cookbook site install <cook_name>

Example: knife cookbook site install apt.

:Note: Chef has maintained a list of community cookbooks which are commonly covered under the use cases like apache2, mysql,php. So a user can leverage the community cookbooks instead of creating their own cookbooks.

:* In order to create our own cookbook:

Syntax : knife cookbook create cookbook_name

:The file default.rb in the path chef-repo/cookbooks/aliases/recipes/default.rb can be edited in order to create recipes.

:To make these cookbooks available to the nodes, it is first uploaded in the chef server.
<li>Setup chef client: Using knife tool in workstation, chef clients are setup on nodes.</li>
</ol>

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a Node ==

Our sample project will be using MongoDB. So the next task will be setting up the MongoDB instances.

=== Puppet ===

As we have seen in Puppet a machine’s configuration is specified in terms of resources. To tell Puppet to set up those resources on a machine we use a Manifest that consists of various resources.

But setting up a Manifest that covers all the Resources will quickly become unwieldly.

Which is why Puppet uses Classes and Modules.

A Class is a named collection of resources. A Module encloses a Class and is something that can be included in other Puppet files thus enabling reuse. There is a large collection of Modules called the Puppet Forge that has been written covering different type of uses.

Thus setting up a manifest for a MongoDB server is as simple as-

class { ‘mongodb’ :
init => ‘sysv’,
version => ‘1.1.1’ # installing a specific version of mongodb
}
include mongodb
# custom puppet code as per our requirments

Thus different Classes can be written which use different Modules for different uses. We can define a different class for our web server , for our file share and so on.

Once a Class has been defined by us that is tailor made to our uses setting up more instances is simply a question of provisioning the node and running the Puppet class on it.

This supports quick and simple scaling up.

=== Chef ===

Configuration in Chef is done by defining Role. Role helps in setting required attributes and tasks for a node. A node can have 0 or more roles assigned to it. Also a role can be assigned to many nodes. Roles thus helps in defining processes for a particular node.

Let us consider to configure a role for mongoDb server for a application:
<ol style="margin-left: 5.2em;">
<li>A role can be created by creating a file under roles directory for eg : roles/role_file.rb</li>

<li>This file is composed of
:name : defines the node.
:description: describes about the node.
:attributes: attributes specific to the node which needs to be configured.
:run lists: tasks assigned to the node.
:Sample-
[[File:codeMongo.JPG]]

</li>
<li>This file is then uploaded to the chef server using following command:</li>

knife role from file roles/redis.rb

<li>Role is assigned to the server during bootstrap(explained in next section) using command :</li>

knife ec2 server create

</li>

</ol>

=== How They Compare ===

== Setting Up A Node ==

=== Puppet ===

In Puppet once we have identified what will go into defining something that we will need an instance for - as we have seen above say configuring a MongoDB instance - we need to select a fresh instance and apply the manifest to it.

As we have seen Puppet Agents will automatically sync with the Puppet Master. So the only task left for us is to

# Provision a new instance
# Install Puppet on it
# Classify it as a particular type of node.

The last two commands can be combined using the puppet init command-

puppet node init \
--node-group=db \
…. ( login related options)
ec2-71-73-79-83.compute-1.amazonaws.com

Here we have specified the fresh instance and initialized as belonging to the group of nodes as db.

Once this process is done the instance will be classified as a db instance and will automatically begin syncing with the Puppet Master.

=== Chef ===

Pre-requisites : IP address / FQDN of the node

Run the knife bootstrap command

The knife bootstrap command requires the FQDN or the IP address for the node in order to complete the bootstrap operation. Run the knife bootstrap command. It checks for .pem files for authentication. If not present then the node goes through the registration phase in order to establish connection between the client and the server and for exchange of authorized messages. The attributes set in the role of the node are compared against the configuration details of the node and run contents of that role’s run list are applied to the node’s configuration details.

=== How They Compare ===

== Node Management ==

=== Foreman ===

Foreman is a popular tool that is used in conjunction with Puppet. It provides a Web based GUI that lets users manage Puppet nodes.

As an example the following screenshot illustrates how Foreman can be used to manage multiple nodes across multiple environments-

[[File:env.png]]

We can apply various filters here to narrow down on a particular instance or conversely modify multiple nodes at the same time.

=== Knife ===

=== How They Compare ===

== Config Files Deployment ==

=== Puppet ===

In Puppet files are treated as resources. So we could specify for an NFS node as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => "/some/path 192.168.100.1(rw,sync)”,
}

However the content would be different for different nodes as the environment would change.

Instead of creating a different resource for each template Puppet supports templatization of config files using Embedded Ruby (ERB).

We can re-write the same resource as follows-

file {'mount_conf':
path => '/etc/exports',
ensure => present,
mode => 0777,
content => template(“nfs/mount_conf.erb”),
}

where the mount_conf.erb file would contain Embedded Ruby code along with the appropriate Ruby variables which would automatically be compiled to give the right values to produce the conf file.

We can apply this same manifest file to all nodes across all the instances.

=== Chef ===

=== How They Compare ===

= Other Infrastructure Management Tools =

There are many other infrastructure management tools-

# Juju - Ubuntu juju is a tool for virtual infrastructure automation
# Cfengine is a much older tool that is used for large scale infrastructure automation. Puppet was originally inspired by Cfengine.
# SaltStack - this is another configuration management system written in Python

This is by no means an exhaustive list and is one that is growing - reflecting the trend towards heavier and more all encompassing automation.

= Conclusion =
= Further Reading =
= References =

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T00:27:06Z

Grmenon: /* How They Compare */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T00:22:32Z

Grmenon: /* Authentication Information Deployment */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T00:21:13Z

Grmenon: /* How They Work */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T00:20:55Z

Grmenon: /* How They Work */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T00:20:03Z

Grmenon: /* Chef */

CSC/ECE 517 Fall 2013/rails infrastructure management tools

2013-09-17T00:18:46Z

Grmenon: /* Overview of Chef */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T00:12:43Z

Grmenon: /* Resources */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T00:10:38Z

Grmenon: /* Puppet */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T00:10:14Z

Grmenon: /* Puppet */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-17T00:09:20Z

Grmenon: /* Overview of Chef */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-16T23:54:38Z

Grmenon: /* Puppet */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-16T23:54:18Z

Grmenon: /* Puppet */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-16T23:53:50Z

Grmenon: /* Puppet */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-16T23:34:28Z

Grmenon: /* Foreman */

File:Env.png

2013-09-16T23:34:07Z

Grmenon:

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-16T23:26:12Z

Grmenon: /* Foreman */

File:Host listing.png

2013-09-16T23:25:54Z

Grmenon:

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-16T23:24:23Z

Grmenon: /* Foreman */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-16T22:37:12Z

Grmenon: /* Authentication and key deployment */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-16T22:36:41Z

Grmenon: /* Config files deployment */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-16T22:36:08Z

Grmenon: /* Node management - foreman and chef */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-16T22:35:02Z

Grmenon: /* Overview of Puppet */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-16T22:34:11Z

Grmenon: /* Configure a Node */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-16T22:33:38Z

Grmenon: /* Provision and setup a node */

CSC/ECE 517 Fall 2013/ch1 1w10 ga

2013-09-15T21:30:57Z

Grmenon: Created page with "= Introduction = Infrastructure Managment Tools are becoming more widely used today and it is a skill set that is growing rapidly in development. With systems growing in size an..."

CSC/ECE 517 Fall 2013

2013-09-15T21:30:11Z

Grmenon:

* [[ CSC/ECE 517 Fall 2012/ch1 1w23 ph ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w30 nn]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w21 w]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w01 aj]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w24 nv]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w29 rkld]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w25 avam]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w30 ps]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w19 rj]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w18 bs]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w17 pk]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w22 ss]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w12 vn]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w14 st]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w08 cc]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w09 hs ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w10 ga ]]

CSC/ECE 517 Fall 2013/rails infrastructure management tools

2013-09-15T04:56:49Z

Grmenon: /* Configure a role */

CSC/ECE 517 Fall 2013/rails infrastructure management tools

2013-09-15T04:53:52Z

Grmenon: /* Introduction */

CSC/ECE 517 Fall 2013/rails infrastructure management tools

2013-09-15T04:45:16Z

Grmenon: /* Headline text */

= Introduction =

= Overview of Puppet =

Puppet is a Infrastructure Management tool written in Ruby. It's primary use is to manage the setup and configurations of machines or instances. For this reason it is widely used by system administrators and DevOps personnel to setup and maintain systems.

Puppet works in two fundamental modes -

== Serverless Puppet ==

In Serverless Puppet each instance with Puppet installed is standalone i.e is independent and running without external supervision.

Effectively there are two processes-

# A master process
# An agent process

The master process consumes the configuration requirements fed it via Puppet and the agent applies it. Here the particular instance running Puppet has complete access to it's own configuration information.

== Master / Agent Puppet ==

This is a typical client server pattern.

Here there is a central Master Puppet instance. This master instance contains the configuration information for all the Agent Puppet instances.

The Agent puppets are configured to periodically inform the Master with “Facts” which is basically information about the current state of the Puppet configuration.

The Master Puppet compares the state of each Agent with what it actually should be and sends it a “Catalog” which tells the Agent how it should configure itself.

After the Agent receives the Catalog and does the required changes it Reports back to the Master informing it of the status of the changes.

A point here to note is that no Agent can see the configuration information of other Agents except it's own which it receives via the Catalog from the Master

INSERT DIAGRAM

== Resources ==

Up till now we have been discussing the “configuration” a machine in abstract terms. This is where one of the key features of Puppet comes up.

Puppet treats the configuration of each machine as being made up of atomic units called Resources.

A resource can be -

*A file that needs to present at a particular location, with particular permissions, having some content etc
*A package that needs to be installed
*A service that needs to be running

and so on.

Puppet has a declarative DSL wherein a resource can be described which is in the following format-

<nowiki>resource_type { 'resource_tile':
attribute_1 => value_1,
attribute_2 => value_2,
…..
}</nowiki>

An example would be -

<nowiki>service { 'rails s':
ensure => 'running'
}</nowiki>

We would include this resource as part of our Rails server configuration to ensure that our server is always running.

The attributes vary depending on the type of the resource and the values belong to range depending on the attribute and the resource type.

A very key feature to be noted here that Puppet focuses on the WHAT instead of the HOW.

What does this mean? Let us take the above example – the command to start a process would be different depending on the OS. But Puppet abstracts that complexity for us – we need only tell it HOW we want the machine to be i.e we want a particular process running and Puppet handles the rest – we tell it the state we want it to be and Puppet makes it so.

= Overview of Chef =

= The Scenario =

The scenario we will be using for the purposes of this comparison will be one that is a very prevalent one - a large scale web application infrastructure. We shall assume that the web application under discussion is being developed used Rails though this does not overly affect the infrastructure management - it is equally applicable to other frameworks.

The specific requirements of this sample project are as follows-

* Zero downtime - the website's operation is a critical part of the client's business model. They require that the website should have zero downtime which means the infrastrucutre must handle failures, deployment and maintenance without interrupting the site's operation.

* Large scale - The client anticipates a very high and dynamic load on the site which means the site should adapt to user demand and the infrastructure should be able to handle the required performance.

* System Integration - The project is being built to be integrated with the client's legacy systems and need to integrate with them. The infrastructure must support the connection between these systems.

= Tasks: Use Cases =

To achieve the above goals various tasks need to be accomplished. We shall look at each of these tasks in turn and see how they can be accomplished using Puppet and Chef and compare them in each use case.

== Installation and Setup ==

As part of our example scenario we shall assume that our environment consists of a collection of Amazon Web Services (AWS) instances.

=== Puppet ===

Puppet can be installed via packages for debian and yum based systems. The package source need only be configured the normal way and the package to be installed.

A particular instance needs to be chosen as the Puppet Master and the puppet-server package needs to be installed on it.

The remaining instances need to have the puppet package installed on it.

Once that is the Puppet Master needs to be configured with a valid host name. Then the agent nodes need to be configured to point to the host name of the server.

Finally the Puppet Master needs to sign of on the certificate of each Puppet Agent that will communicate with it as all the communication between the Puppet nodes is encrypted and the services of the master and agent nodes need to be started

=== Chef ===

# Install chef server on the server machine
# Setup workstation:
## Setup chef-repo in the workstation.
## The machine needs to have access to the server, as well one of the chef clients machine which can act as the first node.
# Setup chef client: Using knife tool in workstation, chef clients are setup on nodes.

=== How They Compare ===

''Similarities''

* Separate packages need to be installed for the client/agent and server/master for Chef and Puppet respectively
* Both use SSL encryption to communicate

''Differences''

* Puppet uses hostnames to uniquely identify whereas Chef uses Fully Qualified Domain Names
* With Puppet each individual node has to be accessed for the installation.However with Chef installation of an agent node can be done remotely
* With Chef a chef-repo needs to be installed on a workstation to use various Chef tools but with Puppet the puppet package is sufficient

== Configure a role ==
== Provision and setup a node ==
== Node management - foreman and chef ==
== Config files deployment ==

- NFS mounting,network, security-configuration of netowrk restrictions and Ip restrictions.

== Authentication and key deployment ==

Environment setup

= Other Infrastructure Management Tools =
= Conclusion =
= Further Reading =
= References =

CSC/ECE 517 Fall 2013/rails infrastructure management tools

2013-09-15T04:42:39Z

Grmenon: /* How They Compare */