1. 1
OpenStack Summit
May 12-16, 2014
Atlanta, Georgia
Optimizing OpenStack for large scale
Cloud Foundry deployments
Animesh Singh, Daniel Krook, Jason Anderson, Kalonji Bankole
@animeshsingh @danielkrook @andersonljason @k_bankole

2. 2 © 2014 IBM Corporation
OAuth
OSLC
Infrastructure
as a Service
Platform
as a Service
Software
as a Service
API
economy
Cloud
operating
environment
Software-
defined
environment
TOSCA
http://www.ibm.com/developerworks/cloud/library/cl-open-architecture/
IBM cloud and open technologies
At all tiers, IBM is committed to building its cloud on an open cloud architecture
Products and services built on open source and open standards benefit IBM and customers

3. 3 © 2014 IBM Corporation
Gold Sponsors
IBM is #2 in contributions to
integrated projects
IBM is working to accelerate OpenStack
Foundation success…
Because an open interoperable Cloud
is critical for flexible cloud deployment
and customer success…
IBM has 15 core contributors15
IBMers working on OpenStack – from formation of
the Foundation to Code Quality & New Function380
IBMers active in the projects
2
107
Mar
2013
May
2014859
Contributors
8,500 Individuals
2130
Contributors
16,100+ Individuals
Exponential growth
Platinum Sponsors
OpenStack Participant Growth

4. 4 © 2014 IBM Corporation4
©2014 IBM
Corporation
…And embracing Cloud Foundry as an open source PaaS
Continuing our history of strengthening and extending open source
TOTAL CONTRIBUTORS
LINES OF CODE
PULL REQ / WK COMPANIES
1,274
Average per month
2013 average: 133
12mo average: 98
Lifetime average: 58
711k
50+ 43

5. 5 © 2014 IBM Corporation
Cloud Foundry PaaS
Cloud Foundry services registry and runtime
management layer.
Components are dynamically discoverable
and loosely coupled, exposing health
through HTTP endpoints so agents can
collect state and act on it.
Cloud Foundry architectural overview
User Authentication and Authorization
Router
DEA Pool
Service Gateway Apps
Service Connector
Health Manager
Messaging
Cloud Controller
Build Packs
Cloud Foundry BOSH
Micro
Clouds
Private
Clouds
Public
Clouds
Service Nodes

6. 6 © 2014 IBM Corporation
Integrate!

7. 7 © 2014 IBM Corporation
PaaS
BOSH Cloud Provider Interface (CPI)
Initially written for VMware and Amazon Web Services
OpenStack CPI
• OpenStack CPI written by PistonCloud Apache2 license
• PistonCloud - An enterprise OpenStack company, founded by few of OpenStack founders
Cloud Foundry – OpenStack integration
OpenStack
Cloud
Servers
Networking
Storage
Integration
IaaS
Hardware

8. 8 © 2014 IBM Corporation
BOSH deployment process
Deployment Manifest
• Release
name/version
• # VMs, job params
• Stemcells to use
Stemcell
• Base OS
• BOSH agent
Release
• Name
Jobs
• Software packages
• Config templates
• Scripts
BOSH
Deployed
Environment Virtual Machine
• Configuration
• Software Packages
Virtual Machine
• Configuration
• Software Packages
Virtual Machine
• Configuration
• Software Packages
Virtual Machine
• Configuration
• Software packages

9. 9 © 2014 IBM Corporation
Sample manifest

10. 10 © 2014 IBM Corporation
10
Cloud Provider Interface for OpenStack
• OpenStack CPI is an implementation of the BOSH CPI.
• Leverages the Fog Ruby gem for OpenStack
• OpenStack CPI gem (Ruby package), the
OpenStack Registry and Agents all reside on the
Cloud Foundry Bosh repository
https://github.com/cloudfoundry/bosh
• Stemcell Image (Base OS Image) and sample
manifest files for OpenStack exist

11. 11 © 2014 IBM Corporation
Requirements:
• Static / floating ips
• Persistent disks
• Outbound Internet connectivity
• Custom flavors
• Increased quota
• Security groups
11
Integration requirements for OpenStack

12. 12 © 2014 IBM Corporation
Automate!

13. 13 © 2014 IBM Corporation
OpenStack discovery:
• Extend the Fog OpenStack gem used by CPI to discover and OpenStack artifacts in an automated
manner in the Cloud Foundry manifest
• Pass OpenStack credentials and discover OpenStack compute and network information.
Deployment automation: Tip one
Discover Flavors
Discover Networks
Discover VM Subnet
Discover DHCP IP
Discover Keypairs
Discover Gateway IP
Discover Security groups

14. 14 © 2014 IBM Corporation
Deployment automation: Tip two
OpenStack pre-req setup:
• Extend the Fog OpenStack gem used by CPI to setup Cloud Foundry OpenStack pre-reqs
• Setup tenant quota
• Keypairs
• Custom Flavors
• Security Groups setup
Create Keypair
Create Flavors
Router, DEA, Cloud
Controller, Service
Nodes
Create Security
Groups
Cf-private, cf-public
Setup tenant quota

15. 15 © 2014 IBM Corporation
Deployment automation: Tip three
Cloud Foundry manifest generation:
• Hard to manually create and edit Cloud Foundry manifests – which can extend up to 1000+ lines
• Automation around template creation – an extension from the previous efforts of OpenStack Discovery
and Cloud Foundry pre-req setup.
• Can create templates for both nova-network and neutron based environments
• Iterates through jobs and assigns static IPs to them if we are using an environment without PowerDNS.

16. 16 © 2014 IBM Corporation
Deployment automation: Tip four
Stemcell automation:
• Community stemcell image has empty fstab and this makes the root partition auto detection to fail, and
consequently provisioning to fail.
Multiple Solutions:
1. Inject file into stemcell
• Mount stemcell image using qemu-nbd
• Inject /etc/fstab into mounted os
• Create new image based on snapshot of modified os
2. Modify Openstack Configuration:
• Change “libvirt_inject_partition” parameter in nova.conf from -1 (autodetect) to 1 (use first partition)
3. Modify bosh “Stemcell Builder” script
• Script injects /etc/fstab into CentOS, but not Ubuntu (as seen here)
4. Install cloud-init in the image
• Best solution to make it work in conjunction with metadata service
Stemcell
• Base OS
• BOSH Agent

17. 17 © 2014 IBM Corporation
Scale!

18. 18 © 2014 IBM Corporation
Sample CF sizing for 1,000 small applications
~60 virtual machines
• 20 high memory
DEAs
• 11 CF fabric
components
• 26 service broker and
service nodes
160 CPU
~500 GB memory
~1.5 TB VM disk
~200 GB block storage

19. 19 © 2014 IBM Corporation
Controller Node
Our initial OpenStack cluster was built on powerful
hardware…
…but it was not ideal for Cloud Foundry
Compute Nodes
Compute Nodes
Compute Node
Nova Compute
Nova Network
Storage Node
Cinder (volume)
Glance (image)
1x
20x
Dedicated
Servers
(bare metal)
Nova API
Nova Scheduler
Keystone
Horizon
Database (MySQL)
Messaging (Qpid)
1x
128 GB memory
6 cores
36 TB disk
each

20. 20 © 2014 IBM Corporation
Weaknesses in the OpenStack layout affected
the stability of our Cloud Foundry configuration
 Single points of failure in the networking, messaging, and storage
configuration made BOSH unhappy.
‒ Qpid on Folsom memory leaks made the head node a bottleneck.
‒ Lost messages affected VM and storage creation, leaving many
resources in stuck states.
‒ Single Nova API process design was overloaded with BOSH traffic
‒ VLAN per tenant design routed much traffic through the single head
node, impacting both the fabric and brokered services.
‒ Cinder and Glance storage and APIs were located on a single
node.
 Bare metal allocation of resources was also inefficient
‒ Every OpenStack component ran as a process directly on the OS
‒ Made reacting to workload changes difficult, i.e., difficult to
redistribute OpenStack components to react to Cloud Foundry
demands

21. 21 © 2014 IBM Corporation
We‟ve addressed the major issues OpenStack by distributing
components more intelligently
Moving some components to VMs, assigning controller, compute, data,
storage roles to groups of components improves availability, flexibility,
scalability and maintainability
Compute Nodes
Compute Nodes
Compute Node
Nova Compute
Cloud Controller
Data Node
database (mysql)
messaging (qpid)
Cloud Controller
Master Node
nova-api
nova-scheduler
Keystone
Horizon
Cloud Controller
Data Node
database (MySql)
messaging (Qpid)
Cloud Controller
Storage Node
Cinder (volume)
Glance (image) Nova Network
Controller Node
Nova API
Nova Scheduler
Keystone
Horizon
Storage Node
Cinder (volume)
Glance (image)
2x
Data Node
Database (MySQL)
Messaging (RabbitMQ)
2x 15x
Load Balancer
Node
HA Proxy
Keepalive
Load Balancer
Node
HA Proxy
Keepalive
Load Balancer
Node
HA Proxy
KeepAlive
6x3x
Virtual
Machines
(VMs)
Dedicated
Servers
(bare metal)

22. 22 © 2014 IBM Corporation
Matching the right hardware for OpenStack to host Cloud Foundry
Learn more about these topics at sessions from our team Wednesday

23. 23 © 2014 IBM Corporation
Optimize!

24. 24 © 2014 IBM Corporation
Deploy optimization tips
• Increase OpenStack API rate limits (/etc/nova/api-paste.ini)
[filter:ratelimit]
paste.filter_factory = nova.api.openstack.compute.limits:RateLimitingMiddleware.factory
limits = (POST, "*", .*, 9999, MINUTE);
(POST, "*/servers", ^/servers, 9999, DAY);
(PUT, "*", .*, 9999, MINUTE);
(GET, "*changes-since*", .*changes-since.*, 9999, MINUTE);
(DELETE, "*", .*, 9999, MINUTE)
BOSH
BOSH
Administrator
NATS
PostGres
Redis
Blob Store
Health Monitor
Director
Registry
CloudProviderInterface(CPI)
Cloud Foundry Environment
mongo_gw
mysql_gw
redis_gw
postgres_gw
cloud_ctrler
ccdb_ng
router
nats
debian_nfs
dea
uaa_db
uaa
loggregator
health_mgr rabbit_gw
mongo_node
mysql_node
redis_node
postgres_node
rabbit_node
OpenStack
Environment
OpenStack
API

25. 25 © 2014 IBM Corporation
Deploy optimization tips
• Increase BOSH‟s NATS timeout
• A large number of BOSH agents can overwhelm NATS
• Place the following in your MicroBOSH manifest file:
apply_spec:
properties:
nats:
ping_interval: 30
ping_max_outstanding: 30
BOSH
NATS
mongo_gwmysql_gw redis_gwpostgres_gw
cloud_ctrler
ccdb_ng
router
nats
debian_nfs
dea
uaa_db
uaa
loggregator
health_mgr
rabbit_gw
mongo_nodemysql_node redis_nodepostgres_node rabbit_node

26. 26 © 2014 IBM Corporation
Deploy optimization tips
• Avoid name based security groups with nova-network
• Name based security groups require message bus activity and database updates
proportional to the number of existing VMs
• Use an OpenStack scheduler which distributes VM load
• Ensure that jobs are distributed across compute nodes instead of grouping
• The default scheduler has the correct behavior
compute_scheduler_driver = nova.scheduler.filter_scheduler.FilterScheduler

27. 27 © 2014 IBM Corporation
Security optimization tips
Guiding principle: Use the most limited permissions required to complete the job
1. Use tenant credentials
• Do not use full admin credentials in your BOSH manifest
2. Only open the security ports which are required

28. 28 © 2014 IBM Corporation
Security optimization tips
3. Separate your VM network and OpenStack management network
• One challenge with BOSH VMs deployed on OpenStack
• Setup pin holes for those particular VMs (Director, OpenStack Registry)
3. Separate Cloud Foundry‟s core fabric from other brokered services
• Services connected using the service broker should be on a separate network
• Prevents compromised services from effecting Cloud Foundry

29. 29 © 2014 IBM Corporation
What next? Join us at the Cloud Foundry
Summit

30. 30 © 2014 IBM Corporation
Monday, May 12 – Room B314
12:05-12:45
Wednesday, May 14 - Room B312
9:00-9:40
9:50-10:30
11:00-11:40
11:50-12:30
OpenStack is Rockin‟ the OpenCloud Movement! Who„s Next to Join the Band ?
Angel Diaz, VP Open Technology and Cloud Labs
David Lindquist, IBM Fellow, VP, CTO Cloud & Smarter Infrastructure
Getting from enterprise ready to enterprise bliss - why OpenStack and IBM is a
match made in Cloud heaven.
Todd Moore - Director, Open Technologies and Partnerships
Taking OpenStack beyond Infrastructure with IBM SmartCloud Orchestrator.
Andrew Trossman - Distinguished Engineer, IBM Common Cloud Stack and
SmartCloud Orchestrator
IBM, SoftLayer and OpenStack - present and future
Michael Fork - Cloud Architect
IBM and OpenStack: Enabling Enterprise Cloud Solutions Now.
Tammy Van Hove -Distinguished Engineer, Software Defined Systems
IBM Sponsored Sessions

31. 31 © 2014 IBM Corporation
Monday, May 12
3:40 - 4:20
3:40 - 4:20
Tuesday, May 13
11:15 - 11:55
2:00 - 2:40
5:30 - 6:10
5:30 - 6:10
Wednesday, May14
9:50 - 10:30
2:40 - 3:20
Thursday, May 15
9:50 - 10:30
1:30 - 2:10
2:20 - 3:00
IBM Technical Sessions

32. 32
Be sure to stop by the IBM booth to see some demos and get your rockin‟
OpenStack t-shirt while they last.
Thank you !