Oscon2011 tutorial

OSCON 2011: Getting Started with Chef
Joshua Timberman
joshua@opscode.com, @jtimberman

Aaron Peterson
aaron@opscode.com, @metaxis

http://opscode.com

Monday, July 25, 2011

Meta Information

• OSCON tutorials are recorded
• Rate the tutorial and comment
• http://bit.ly/chef-oscon2011
• Twitter:
• #oscon
• @opscode, #opschef
• @jtimberman, @metaxis
• Slides and Code will be posted


Who are we?

• Joshua Timberman
• Aaron Peterson


Who are you?

• System administrators?
• Developers?
• “Business” People?

http://www.ﬂickr.com/photos/timyates/2854357446/sizes/l/


Agenda

• Tutorial Logistics
• Hows and whys
• Getting Started
• Anatomy of a Chef Run
• Hands on configuring a node
• Common patterns & best practices
• Question/Answer

http://www.ﬂickr.com/photos/koalazymonkey/3590953001/

What are we talking
about here?

http://www.ﬂickr.com/photos/peterkaminski/2174679908/


Managing Infrastructure is Hard

• Sysadmins:
• Setup production machines
• Manage deployed application(s)


System administrators...

• Install packages
• Configure running services
• OS settings
• User management
• Monitoring and trending integration


Managing Infrastructure is Hard

• Developers:
• Setup local machine
• Deploy application for testing


Developers...

• Developers want self-service
• Full application stack
• Abstract the details


Automation is Good


Automation is Good

• Operable
• Reasonable
• Flexible
• Repeatable


The Chef Framework

• Reasonability
• Flexibility
• Library & Primitives
• TIMTOWTDI


The Chef Tool(s)

• ohai - information gathering
• chef-client - configuration agent
• knife - command-line API tool
• shef - console debugger


The Chef API

• RSA key authentication w/ Signed Headers
• RESTful API w/ JSON
• Search Service
• Derivative Services


The Chef Community

• Apache License, Version 2.0
• 360+ Individual contributors
• 70+ Corporate contributors
• Dell, Rackspace,VMware, RightScale,
Basho Technologies, and more
• http://community.opscode.com
• 260+ cookbooks


Getting Started with Chef

git clone git://github.com/opscode/oscon2011-chef-repo

Required Software

• SSH/SCP
• Git
• Build toolchain (gcc and friends)
• Ruby (1.8.7 or 1.9.2)
• RubyGems (1.3.7+)
• Chef (0.10.0+)


Why Opscode Hosted Chef?

• Limited time for tutorial
• Free up to 5 nodes
• Chef Server API
• Open Source Chef Server


Source Code Repository

• Chef Repository for OSCON 2011
• git clone git://github.com/opscode/oscon2011-chef-repo


Files from Opscode Hosted Chef Signup

• Knife configuration
• .chef/knife.rb
• User certificate
• .chef/USER.pem
• Validation certificate
• .chef/ORGNAME-validator.pem


Verify Access

% knife client list
oscon2011-validator

% knife node from file dummy.example.com.json
Updated Node dummy.example.com!

% knife node list
dummy.example.com

% knife node show dummy.example.com
Node Name: dummy.example.com
Environment: _default
FQDN: dummy.example.com
IP: 10.1.1.1
Run List:
Roles:
Recipes
Platform: centos 5.5


Virtual Machine Setup

• Setup outside scope of this tutorial
• Linux Virtual Machine or Cloud Instance
• SSH access as root or user w/ sudo


A quick tour of Chef


Chef runs on your systems


API Clients authenticate to the
Chef Server


Each system running Chef is a
managed Node


Nodes have attributes and a
list of things to run


Roles are a description of what
a node should be


Chef configures Resources on
your systems


Recipes are lists of resources


Cookbooks are packages for
Recipes and related files


Let’s manage some
infrastructure...


Managing Infrastructure

• Write or download cookbooks
• Create a role that uses the cookbooks
• Deploy cookbooks and role to Chef Server
• Apply the role to a node
• Run Chef on the node


Anatomy of a Chef Run


Profile the Node with Ohai


Run Ohai

• Run `ohai | less` on your system.
• Marvel at the amount of data it returns.


Authenticate


Retrieve Node from Chef
Server


Sync Cookbooks from Chef
Server


Load Cookbooks


Load Recipes


Converge


Save Node to Chef Server


Break Time

• Questions from 1st half
• Hands on in 2nd half

http://www.ﬂickr.com/photos/refractedmoments/65794219/


Questions?

http://www.ﬂickr.com/photos/oberazzi/318947873/

Reasoning about Infrastructure



• Break down complexity into components
you can think about.
• Think about commonality and differences
between systems and applications.
• Capture these in roles.



• For a given application, think about
requirements to fulfill its job.
• Think about how to meet the requirements.


Concrete use case

• Stand in for common patterns
• Things we want on all systems in the
infrastructure.
• User management
• Essential network service (NTP)


Upload Chef Repository

% knife role from file base.rb

% knife cookbook upload -a

% knife data bag create users

% knife data bag from file users luke.json

% knife data bag from file users leia.json


Configure a node

• Invoke action from the local workstation to
happen on a remote machine over SSH.
• Virtual Machine IP address
• SSH key or password for root/privileged
(sudo) user
• Optional: Use a cloud computing provider
(See README.md)


Knife Bootstrap

knife bootstrap FQDN (options)
-d DISTRO Target a specific distro (default ubuntu)
-i IDENTITY_FILE SSH identity file for authentication
-r RUN_LIST Run list for the node
-P PASSWORD The ssh password
-x USERNAME The ssh username (default root)
--sudo Execute bootstrap with sudo

% knife bootstrap --help

% knife help bootstrap # full man page!


Bootstrap Cloud Instances

• Knife works with Cloud providers
through plugins
• Knife Cloud plugins use Fog
• Cloud instances are launched via their
API then provisioned with bootstrap
• Additional RubyGems
• knife-ec2, knife-rackspace, etc
• Additional Knife Configuration


Configure a node

# Append -Ppassword or -i ~/.ssh/ssh-private-key-for-you to ssh
# Ubuntu:
knife bootstrap $IPADDRESS -r 'role[base]'
knife bootstrap $IPADDRESS -r 'role[base]' -x ubuntu --sudo

# Debian 6:
knife bootstrap $IPADDRESS -r 'role[base]' -x root
knife bootstrap $IPADDRESS -r 'role[base]' -x username --sudo

# CentOS 5.x:
knife bootstrap $IPADDRESS -r 'role[base]' -d centos5-gems
knife bootstrap $IPADDRESS -r 'role[base]' -d centos5-gems -x username --sudo

# Scientific Linux 6.x:
knife bootstrap $IPADDRESS -r 'role[base]' -d scientific6-gems
knife bootstrap $IPADDRESS -r 'role[base]' -d scientific6-gems -x username --sudo

# Example (Ubuntu 10.04):
knife bootstrap 172.16.156.130 -r 'role[base]' -x jtimberman --sudo -Poscon2011


What happened on the node?


recipe[ntp]

INFO: Processing package[ntp] action install (ntp::default line 27)
INFO: package[ntp] installed version 1:4.2.4p8+dfsg-1ubuntu2.1
INFO: Processing package[ntp] action install (ntp::default line 27)
INFO: package[ntp] installed version 1:4.2.4p8+dfsg-1ubuntu2.1
INFO: Processing template[/etc/ntp.conf] action create (ntp::default line 31)
INFO: template[/etc/ntp.conf] backed up to /var/chef/backup/etc/
ntp.conf.chef-20110717131907
INFO: template[/etc/ntp.conf] mode changed to 644
INFO: template[/etc/ntp.conf] updated content
INFO: Processing service[ntp] action enable (ntp::default line 39)
INFO: Processing service[ntp] action start (ntp::default line 39)

[ ... end of run (delayed) ... ]
INFO: template[/etc/ntp.conf] sending restart action to service[ntp] (delayed)
INFO: Processing service[ntp] action restart (ntp::default line 39)
INFO: service[ntp] restarted


SSH to the Node and inspect

% ssh 172.16.156.130

% dpkg -l ntp
ii ntp 1:4.2.4p8+dfsg Network Time Protocol daemon and

% grep server /etc/ntp.conf
server 0.pool.ntp.org
server 1.pool.ntp.org

% /etc/init.d/ntp status
* NTP server is running

% ls /etc/rc2.d/*ntp
/etc/rc2.d/S23ntp


recipe[users::sysadmin]

INFO: Processing user[luke] action create (users::sysadmins line 41)
INFO: user[luke] created
INFO: Processing directory[/home/luke/.ssh] action create (users::sysadmins line 51)
INFO: directory[/home/luke/.ssh] created directory /home/luke/.ssh
INFO: directory[/home/luke/.ssh] owner changed to 2001
INFO: directory[/home/luke/.ssh] group changed to 2001
INFO: directory[/home/luke/.ssh] mode changed to 700
INFO: Processing template[/home/luke/.ssh/authorized_keys] action create
(users::sysadmins line 57)
INFO: template[/home/luke/.ssh/authorized_keys] owner changed to 2001
INFO: template[/home/luke/.ssh/authorized_keys] owner changed to 2001
INFO: template[/home/luke/.ssh/authorized_keys] updated content

INFO: Processing user[leia] action create (users::sysadmins line 41)
INFO: user[leia] created
INFO: Processing directory[/home/leia/.ssh] action create (users::sysadmins line 51)
INFO: directory[/home/leia/.ssh] created directory /home/leia/.ssh
INFO: directory[/home/leia/.ssh] owner changed to 2002
INFO: directory[/home/leia/.ssh] group changed to 2002
INFO: directory[/home/leia/.ssh] mode changed to 700
INFO: Processing template[/home/leia/.ssh/authorized_keys] action create
(users::sysadmins line 57)
INFO: template[/home/leia/.ssh/authorized_keys] owner changed to 2002
INFO: template[/home/leia/.ssh/authorized_keys] owner changed to 2002
INFO: template[/home/leia/.ssh/authorized_keys] updated content
INFO: Processing group[sysadmin] action create (users::sysadmins line 66)
INFO: group[sysadmin] created


recipe[users::sysadmins]

% ssh 172.16.156.130

% getent passwd luke leia
luke:x:2001:2001:Force is strong with this one:/home/luke:/bin/bash
leia:x:2002:2002:There is another:/home/leia:/bin/bash

# ls ~{luke,leia}/.ssh
/home/luke/.ssh:
authorized_keys

/home/leia/.ssh:
authorized_keys


recipe[sudo]

INFO: Processing package[sudo] action upgrade (sudo::default
line 20)
INFO: Processing template[/etc/sudoers] action create
(sudo::default line 24)
INFO: template[/etc/sudoers] backed up to /var/chef/backup/
etc/sudoers.chef-20110717131908
INFO: template[/etc/sudoers] mode changed to 440
INFO: template[/etc/sudoers] updated content


recipe[sudo]

# grep ALL /etc/sudoers
root ALL=(ALL) ALL
%sysadmin ALL=(ALL) ALL


What happened on the Chef
Server?


Chef Repository on Chef Server

% knife role list
base

% knife cookbook list
ntp 1.0.0
sudo 1.0.0
users 1.0.0

% knife data bag list
users

% knife data bag show users
leia
luke


Base Role

% knife role show base
chef_type: role
default_attributes: {}
description: Base role applied to all systems
env_run_lists: {}
json_class: Chef::Role
name: base
override_attributes: {}
run_list: recipe[ntp], recipe[users::sysadmins],
recipe[sudo]


NTP Cookbook

package "ntp" do
action :install
end

template "/etc/ntp.conf" do
source "ntp.conf.erb"
owner "root"
group "root"
mode 0644
notifies :restart, "service[ntp]"
end

service "ntp" do
action [:enable, :start]
end


NTP configuration (template)

source "ntp.conf.erb"
owner "root"
group "root"
mode 0644
end

Template source:
<% node[:ntp][:servers].each do |ntpserver| -%>
server <%= ntpserver %>
<% end -%>
<% end -%>

Cookbook Attributes:
default[:ntp][:servers] = ["0.pool.ntp.org", "1.pool.ntp.org"]


NTP service management

# ...
end

service "ntp" do
action [:enable, :start]
end


Sysadmin users data bag items

% cat data_bags/users/luke.json
{
"id": "luke",
"ssh_keys": "ssh-rsa For example purposes only",
"groups": "sysadmin",
"uid": 2001,
"shell": "/bin/bash",
"comment": "Force is strong with this one"
}

% cat data_bags/users/leia.json
{
"id": "leia",
"ssh_keys": "ssh-rsa For example purposes only",
"groups": "sysadmin",
"uid": 2002,
"shell": "/bin/bash",
"comment": "There is another"
}


users::sysadmins recipe

search(:users, 'groups:sysadmin') do |u|

user u['id'] do
uid u['uid']
gid u['id']
shell u['shell']
comment u['comment']
supports :manage_home => true
home "/home/#{u['uid']}"
end

directory "#{home_dir}/.ssh" do
owner u['id']
group u['id']
mode "0700"
end

template "#{home_dir}/.ssh/authorized_keys" do
source "authorized_keys.erb"
owner u['id']
group u['id']
mode "0600"
variables :ssh_keys => u['ssh_keys']
end
end


Sudo cookbook

package "sudo" do
action :upgrade
end

template "/etc/sudoers" do
source "sudoers.erb"
mode 0440
owner "root"
group "root"
variables(
:sudoers_groups => node['authorization']['sudo']['groups'],
:sudoers_users => node['authorization']['sudo']['users'],
:passwordless => node['authorization']['sudo']['passwordless']
)
end


Sudoers template

Template source:

root ALL=(ALL) ALL
%sysadmin ALL=(ALL) <%= "NOPASSWD:" if @passwordless %>ALL

Cookbook attributes:

default['authorization']['sudo']['passwordless'] = false

Rendered content:

root ALL=(ALL) ALL
%sysadmin ALL=(ALL) ALL


Nodes

% knife node list
dummy.example.com
ubuntu1004test.example.com

% knife node show ubuntu1004test.example.com
Node Name: ubuntu1004test.example.com
Environment: _default
FQDN: ubuntu1004test.example.com
IP: 172.16.156.130
Run List: role[base]
Roles: base
Recipes ntp, users::sysadmins, sudo
Platform: ubuntu 10.04

% knife node show --help
% knife help node


Searching the Server

# Search nodes:
% knife search node "role:base"
% knife search node "platform:ubuntu"
% knife search node "platform:centos"

# Search roles:
% knife search role "run_list:recipe[users*"

# Search data bags (bag name is the index):
% knife search users "groups:sysadmin"
% knife search users "shell:*bash"


Common Patterns
and
Best Practices


Common Patterns

• Install a package
• Update a configuration file
• Restart a service


Common Patterns

• Search for nodes with a particular role
• Search for data bag items
• Make decisions or render templates based
on search results.


Search example in a recipe

pool_members = search("node", "role:webserver")

template "/etc/haproxy/haproxy.cfg" do
source "haproxy-app_lb.cfg.erb"
owner "root"
group "root"
mode 0644
variables :pool_members => pool_members.uniq
notifies :restart, "service[haproxy]"
end


Common Patterns

• Ask questions about the infrastructure.
• Target a subset of servers and take action.
• Search with Roles
• Search with Node Attributes
• Parallel execution of commands.


Operational Use Case

% knife ssh platform:ubuntu 'vmstat'
xwing.example.com procs -----------memory---------- ...
xwing.example.com r b swpd free buff cache ...
xwing.example.com 0 0 0 684804 461656 6052916 ...
tiefighter.example.com procs -----------memory---------- ...
tiefighter.example.com r b swpd free buff cache ...
tiefighter.example.com 0 0 0 169020 708844 6120008 ...


Best Practices: Cookbooks

• Publicly shared cookbooks:
• http://community.opscode.com
• Create your own
• knife cookbook create foo
• $EDITOR cookbooks/foo/recipes/default.rb


Getting Community Cookbooks

# Install apache2 cookbook from site in Git chef-repo
% knife cookbook site install apache2

# Download and install apache2 cookbook in non-Git chef-repo
% knife cookbook site download apache2

% tar -zxf apache2-VERSION.tar.gz -C cookbooks


Best Practices: Cookbooks

• Cookbook for each service
• Recipe for each component or deployment
of the service
• Set sane defaults in attributes files
• Modify attributes through roles for specific
usage requirements


Best Practices: Roles

• Roles are descriptions
• webserver
• database_master
• load_balancer
• Set role-specific attributes when necessary
• listen ports, deploy locations, etc


Best Practices: Nodes

• Use “Just Enough OS”
• Use fully updated systems
• Kickstart, AMI, etc
• Ensure system clock is synchronized
• Be ready to deploy from scratch


Managing Resources

• Chef’s primary purpose is managing
resources on nodes.
• Think in terms of resources vs commands
• Chef comes with 28 kinds of resources
• You can create your own resources in
cookbooks


Thinking in terms of resources

• package vs yum install
• service vs chkconfig
• template vs echo ‘coolstuff’ >> /etc/config
• or sed ‘s/badstuff/coolstuff/’...
• mode, owner and group parameters vs
chmod/chown
• http://wiki.opscode.com/display/chef/Resources


FAQ: Chef vs [Other Tool]


http://www.ﬂickr.com/photos/gesika22/4458155541/


FAQ: How do you test
recipes?


FAQ: Testing

• You launch cloud instances and watch
them converge.
• You use Vagrant with a Chef
Provisioner


FAQ: Testing

• You buy Stephen Nelson-Smith’s book!


FAQ: How does Chef
scale?


FAQ: Scale

• The Chef Server is a publishing
system.
• Nodes do the heavy lifting.
• Chef scales like a service-oriented
web application.
• Opscode Hosted Chef was designed
and built for massive scale.

http://www.ﬂickr.com/photos/amagill/61205408/


Questions?

• http://bit.ly/chef-oscon2011
• http://opscode.com
• @opscode, #opschef
• irc.freenode.net, #chef, #chef-hacking
• http://lists.opscode.com

http://www.ﬂickr.com/photos/oberazzi/318947873/

Thanks!

http://opscode.com
@opscode
#opschef


Oscon2011 tutorial

Recommended

Recommended

More Related Content

Similar to Oscon2011 tutorial

Similar to Oscon2011 tutorial (20)

More from jtimberman

More from jtimberman (11)

Recently uploaded

Recently uploaded (20)

Oscon2011 tutorial