Cloud Foundry BOSH Where the Platform rubber meets the Infrastructure road - ChefConf

Cloud Foundry BOSH
Where the Platform rubber
meets the Infrastructure road

Patrick Chanezon with slides from Martin Englund
Senior Director Cloud Foundry Engineering
Developer Relations
chanezonp@vmware.com @pmenglund
http://twitter.com/chanezon ChefConf, San Francisco, May 2012
Wednesday, May 16, 12

What is Cloud Computing?

Cloud According to my daughter Eliette

2


Cloud Stack - Classic Pyramid

Software
As A Service

Platform As A Service

Infrastructure As A Service

3


Cloud Stack - Developer Perspective

Software
As A Service

Platform As A Service

Infrastructure
As A Service

4


Infrastructure

5


IaaS/Virtualization getting mainstream

§ AWS, Joyent, Rackspace,...
§ Open Source projects: OpenStack, DeltaCloud, Eucalyptus
§ Automation: Chef, Juju
§ Standardization? DMTF
§ Inside the Firewall, Virtualization: VMware, Microsoft, Xen, KVM
§ 50% of workloads are virtualized
§ Easy to provision, manage instance...BUT
§ Still need to manage backups, software stacks, monitor, upgrades

6


With Infrastructure, you still need to build your own platform

§ Need to build a distributed platform on top of you infrastructure
§ Story of the AWS meltdown from last summer
• http://blog.reddit.com/2011/03/why-reddit-was-down-for-6-of-last-24.html
• http://www.readwriteweb.com/cloud/2010/12/chaos-monkey-how-netflix-
uses.php
• http://news.ycombinator.com/item?id=2477296
• http://stu.mp/2011/04/the-cloud-is-not-a-silver-bullet.html
§ Twilio, Smugmug, SimpleGeo survived it because they built their
own distributed platform on top of IaaS
§ Enterprise customers want to consider Infrastructure like CDNs
• Multi Cloud usage
• Based on Open Source de facto standards, or full standards whenever that
happens

7


Platforms

Let the wild rumpus begin!

8


Platforms

§ Raise the Unit of currency to be application & services instead of
infrastructure
§ Google App Engine, Cloud Foundry, SalesForce Heroku, Cloudbees,
Amazon Elastic Beanstalk, Microsoft ex-Azure, AppFog
§ Single or a few languages, services
§ Start of Multi language Polyglot platforms
§ Enabler for Agile Developers -> Create Business value faster
§ Lack of standards: risk, vendor lock-in
§ Enterprise needs:
• Control, customizability
• Private/Hybrid Cloud
• Avoid lock-in

9


Main Risk: Lock-In

Welcome to the hotel california
Such a lovely place
Such a lovely face
Plenty of room at the hotel california
Any time of year, you can ﬁnd it here

Last thing I remember, I was
Running for the door
I had to ﬁnd the passage back
To the place I was before
’relax,’ said the night man,
We are programmed to receive.
You can checkout any time you like,
But you can never leave!

10


Cloud Foundry: The Open PaaS

• Open Source: Apache 2 Licensed
• multi language/frameworks
• multi services
• multi cloud

Ap

ce
pli
vFabric

rfa
Postgres Private
ca

nte
Data
Services tio Clouds

rI
n

e
Se
Public

vid
vFabric
RabbitMQTM rvi
Cloud

ro
Msg Services
ce

dP
Micro

ou
Other
Cloud
Cl
Services

11


Open Source

12


Open Source Advantage: moves faster

§ http://code.google.com/p/googleappengine/issues/detail?id=13

• https://github.com/cloudfoundry/vcap/pull/25

13


Open Source Advantage: diverse communities

§ Juju Charms to setup a multi node Cloud Foundry in 10 minutes
§ See Brian Thomason & Juan Negron’s UDS 2011 session

http://cloud.ubuntu.com/2011/09/from-zero-to-drawbridge-via-ubuntu-
server-ensemble-and-cloudfoundry-in-less-than-10-minutes/

14


Open Source Advantage: more innovation

15


CloudFoundry.org

16


Cloud Foundry Open Source Workflow

cloudfoundry.com

production updated ~2x/week

github
github.com/cloudfoundry

on +2 and ✓ change pushed to github

Reviewers +1/-1 Gerrit Code Review +2/-2 Committers
reviews.cloudfoundry.org

test veriﬁcation score ✓ ✗ git push triggers test execution

Jenkins CI
ci.cloudfoundry.org
17


Cloud Foundry is 1 year old

§ Services
• Caldecott – tunnel into your services, explore with standard client tools
• PostgreSQL, RabbitMQ

§ Frameworks, Runtimes, and Tools
• Java and Ruby Auto-Reconfiguration
• Scala, node.JS 0.6.*, Erlang, JRuby, PHP, Python, .NET, Spring 3.1, Grails 2.0,
Play 2.0
• Multi-Node Chef based deployment tools
• Maven Plugin, Eclipse Integration
• VMC manifests, Java Debugging, Rails Console
• Standalone applications

§ Micro Cloud Foundry x 3
18


Cloud Foundry Ecosystem

19


Cloud Foundry Ecosystem

20


Register today

http://cloudfoundry.com/signup
Use Promo Code ChefConf
To avoid approval queue waiting time

21


Logical View

Browser
VMC client STS plugin
(user app access)

Routers

CloudControllers App App

HealthManager
Services DEA Pool

Messaging

22


Cloud Foundry
BOSH

23


Chef at VMware

§ Chef is used internally in several projects at VMware
§ To manage the Cloud Foundry deployment and Life Cycle, we
developed a more specific system: BOSH

24


Production Grade Cloud Foundry Clusters

§ 500 – 5,000 VMs cloudfoundry.com

§ 40+ unique node types
§ 75+ unique software packages
§ 75+ unique environments
§ 2x/week cf.com updates
§ 24x7x365 non-stop operation
§ No-downtime deployments
§ Reliable, robust, repeatable
deployments, updates, capacity
adjustments
§ Small teams manage many
instances production, staging, stress, qa, dev

Google style problem è Google style solution
25


Cloud Foundry BOSH

§ Cloud Foundry BOSH is an open source tool-chain for release
engineering, deployment, and lifecycle management of large scale
distributed services
• Prescriptive way of creating releases and managing systems and services
• It is not a collection of shell scripts, not a pile of Perl

§ Built to deploy and manage production-class, large scale clusters
• Production grade Cloud Foundry clusters: 500+ VMs, 40+ jobs, 75+ packages
• Multi-node, multi-tier, complex clusters: e.g., our Gerrit/Jenkins Cluster

§ Built for devops usage and scale by a crack team of veterans
• A project, not a product: command line interface, YAML, etc.
• Continuous improvement, iterative development, rough edges

github.com/cloudfoundry/bosh
26


BOSH Agenda

§ Background
§ Concepts
§ Demo
§ Workflow
§ Under the hood

27


Background

§ Built from the need to operate cloudfoundry.com
• Over 40 different types of VMs
• Hundreds of running instances of VMs
• Pushing at least 2 changes per week
§ End-to-end management
• Release engineering
• Deployment
• Lifecycle management
§ Generic solution
• Any service
• Any cloud
• Any operating system

28


Concepts

§ Packages
§ Jobs
§ Releases
• http://github.com/cloudfoundry/bosh-sample-release
§ Stemcells
§ Deployments

29


Packages

§ Source / blobs
• Don’t commit large objects in the SCM
§ Dependencies
• Compile time dependencies
§ Versioning
• Auto-increment
§ Pre-packaging
• Preparation for compilation
§ Packaging
• Compiling the source

30


Packages (nginx)

src/nginx
-rw-r--r-- 1 martin staff 25490 Apr 11 19:17 headers-more-v0.15rc1.tgz
-rw-r--r-- 1 martin staff 691501 Apr 11 19:17 nginx-1.0.11.tar.gz
-rw-r--r-- 1 martin staff 1556329 Apr 11 19:17 pcre-8.21.tar.gz

31


Packages (nginx)

packages/nginx/spec
---
name: nginx
files:
- nginx/headers-more-v0.15rc1.tgz
- nginx/nginx-1.0.11.tar.gz
- nginx/pcre-8.21.tar.gz

32


Packages (nginx)

packages/nginx/packaging
# abort script on any command that exit with a non zero value
set -e

echo "Extracting pcre..."
tar xzf nginx/pcre-8.21.tar.gz

echo "Extracting headers-more module..."
tar xzf nginx/headers-more-v0.15rc1.tgz

echo "Extracting nginx..."
tar xzf nginx/nginx-1.0.11.tar.gz

echo "Building nginx..."
cd nginx-1.0.11
./configure
--prefix=${BOSH_INSTALL_TARGET}
--with-pcre=../pcre-8.21
--with-http_ssl_module
--add-module=../headers-more-v0.15rc1
make
make install

33


Jobs

§ Packages
• Runtime dependencies
§ Templates
• Converting a generalized form to an instance specific file
§ Versioned
• Auto increments
§ Supervision
• Through monit
§ Monitoring
• Reported to the health monitor
§ Lifecycle hooks
• E.g. drain workload from job

34


Jobs (nginx)

jobs/nginx/spec
---
name: nginx

templates:
nginx_ctl: bin/nginx_ctl
nginx.conf.erb: config/nginx.conf
mime.types: config/mime.types

packages:
- nginx

35


Jobs (nginx)

jobs/nginx/monit
check process nginx
with pidfile /var/vcap/sys/run/nginx/nginx.pid
start program "/var/vcap/jobs/nginx/bin/nginx_ctl start"
stop program "/var/vcap/jobs/nginx/bin/nginx_ctl stop"
group vcap

36


Jobs (nginx)

jobs/nginx/templates/nginx.conf.erb
user root root;
worker_processes <%= properties.nginx.workers || 1 %>;

error_log /var/vcap/sys/log/nginx/nginx.error.log;
pid /var/vcap/sys/run/nginx/nginx.pid;

events {
worker_connections 8192;
use epoll;
}
...
upstream wordpress {
<% properties.wordpress.servers.each do |server| %>
server <%= server %>:<%= properties.wordpress.port %>;
<% end %>
}

server {
listen 80;
server_name "<%= properties.wordpress.servername %>";
server_name_in_redirect off;
...

37


Releases

§ Packages
§ Jobs
§ Versioned
§ Self contained
§ Incremental
§ Published

» bosh create release
» bosh upload release
» bosh delete release
» bosh releases

38


Stemcells

§ VM template
§ BOSH Agent
§ Versioned
§ Built using vmbuilder
§ IaaS Plugin

» bosh upload stemcell
» bosh delete stemcell
» bosh stemcells

39


Deployments

§ Release
§ Networks
§ Resource pools
§ Compilation workers
§ Jobs
§ Update concurrency
§ Properties
§ Cloud properties

40


Deployment (wordpress)

wordpress.yml
---
name: bosh-wordpress
release:
name: wordpress
version: 3

compilation:
workers: 4
network: default
cloud_properties:
ram: 2048
disk: 8096
cpu: 2

update:
canaries: 1
canary_watch_time: 60000
update_watch_time: 30000
max_in_flight: 4
max_errors: 1

41



wordpress.yml (continued)
networks:
- name: default
subnets:
- reserved:
- 172.31.220.2 - 172.31.220.10
static:
- 172.31.220.11 - 172.31.220.100
range: 172.31.220.0/22
gateway: 172.31.220.1
dns:
- 172.30.22.153
cloud_properties:
name: VLAN2220
- name: dmz
subnets:
- static:
- 172.28.5.209 - 172.28.5.210
range: 172.28.5.208/28
dns:
- 172.22.22.153
cloud_properties:
name: VLAN3093

42



resource_pools:
- name: infrastructure
network: default
size: 6
stemcell:
name: bosh-stemcell
version: 0.4.6
cloud_properties:
cpu: 1
disk: 8192
ram: 4096

43



jobs:
- name: mysql
template: mysql
instances: 1
resource_pool: infrastructure
persistent_disk: 16384
networks:
- name: default
static_ips:
- 172.31.220.20

- name: wordpress
template: wordpress
instances: 4
networks:
- name: default
static_ips:
- 172.31.220.30 - 172.31.220.33

44



- name: nginx
template: nginx
instances: 1
networks:
- name: default
default: [dns, gateway]
static_ips:
- 172.31.220.40
- name: dmz
static_ips:
- 172.28.5.209

45



properties:
wordpress:
admin: wordpress@cloudfoundry.com
port: 8008
servers:
- 172.31.220.30
- 172.31.220.31
- 172.31.220.32
- 172.31.220.33
servername: wordpress.cloudfoundry.com
db:
name: wp
user: wordpress
pass: w0rdpr3ss
auth_key: random key
secure_auth_key: random key
logged_in_key: random key
mysql:
address: 172.31.220.20
port: 3306
password: rootpass
nginx:
workers: 1

46


BOSH Demo

47


Dev Workflow

bosh deployment
bosh target dev
~/dev.yml

write code

run tests bosh create release

git commit bosh deploy bosh upload release

48


QA Workflow

bosh deployment
bosh target qa
~/qa.yml

bug reports to dev git pull

run tests bosh create release

bosh create
release --final
bosh deploy bosh upload release

git commit

49


Ops Workflow

bosh deployment
bosh target prod
~/prod.yml

bug reports to dev git pull

run tests bosh upload release

bosh deploy
done!

50


Control

§ Repeatable deployments
§ Predictable rollout

51


Consistency

§ Dev / Staging / Production
§ BOSH targets

52


Components

§ Director / Workers
§ Agent
§ Monitor
§ CLI
§ NATS
§ Redis
§ Database
§ Blobstore

53


BOSH: under the hood

BOSH User
“BOSH is deployed by BOSH”

bosh cli director healthmon
cloudfoundry.com

db redis nats

workers blobs
active jobs

agent
IaaS CPI stemcell disk

54


IaaS neutral by design

vSphere: battle tested implementation,
thousands of deployments CPI: code complete
vCloud Director: “work in progress”, 2H 2012 functional status: “work in progress”

Cloud Foundry BOSH

Cloud Provider Interface (CPI)

github.com/piston/openstack-bosh-
cpi

contribute: github.com/cloudfoundry/bosh

55


Cloud Provider Interface

Stemcell
create_stemcell(image, cloud_properties)
delete_stemcell(stemcell)

VM
create_vm(agent_id, stemcell, resource_pool, networks,
disk_locality, env)
delete_vm(vm)
configure_networks(vm, networks)

Disk
create_disk(size, vm_locality)
delete_disk(disk)
attach_disk(vm, disk)
detach_disk(vm, disk)

56


BOSH and Chef?

§ How should they work together?
§ Talk to us on public mailing lists:
• http://groups.google.com/a/cloudfoundry.org/group/bosh-users
• http://groups.google.com/a/cloudfoundry.org/group/bosh-dev

57


Reference / Q&A

§ Mailing lists
• http://groups.google.com/a/cloudfoundry.org/group/bosh-users
• http://groups.google.com/a/cloudfoundry.org/group/bosh-dev
§ File a Bug
• http://cloudfoundry.atlassian.net
§ Browse, Fork & Contribute
• https://github.com/cloudfoundry/bosh
• gem install gerrit-cli
• gerrit clone ssh://reviews.cloudfoundry.org:29418/bosh

58


BOSH Community Resources
@DrNic
http://drnicwilliams.com/2012/04/16/creating-a-bosh-from-scratch-on-aws/

@BrianMMcClain
http://www.brianmmcclain.com/2012/05/08/using-bosh-with-vsphere-part-2/

https://github.com/cloudfoundry/bosh

https://groups.google.com/a/cloudfoundry.org/group/bosh-users/topics

https://github.com/piston/openstack-bosh-cpi

59


Acknowledgement

§ Drawings from my daughters Eliette
§ Slides from Mark Lucovsky, Dave McCrory, Derek Collison, Martin
Englund

60


Cloud Foundry BOSH Where the Platform rubber meets the Infrastructure road - ChefConf

Recommended

Recommended

More Related Content

More from Patrick Chanezon

More from Patrick Chanezon (20)

Recently uploaded

Recently uploaded (20)

Cloud Foundry BOSH Where the Platform rubber meets the Infrastructure road - ChefConf