This is a presentation to JClouds Meetup @ Twitter's HQ. Presented by Toni Batchelli. It discusses the methodologies that the GoGrid engineering team is employing to quickly and accurately produce timely builds.

1. Complex Infrastructure Made Easy™
Agile Development at GoGrid with
Pallet and JClouds
Presented to JClouds Meetup @ Twitter’s HQ
Toni Batchelli, Consultant and Entrepreneur wannabe
tbatchelli@gmail.com, @tbatchelli, @disclojure
February 2, 2011
02/01/11

2. GoGrid’s Challenges
GoGrid is a cloud provider, building their own
software
Developing and testing their cloud
management software is complex. They need
to create and maintain realistic environments
in order to load them with their latest software
builds
~10 different servers types
~20 different services

3. The problem?
Current time to setup a development/QA
environment is 2 days
• Mostly manual work
Building the environments manually is very
error prone
A high level of interconnectivity between
services

4. Goal
To be able to create development and
testing environments dynamically, in a
customizable way, and on cloud
infrastructure
A Cloud in a Cloud

5. Additional Challenges
Environments need to be verified as fully functional before
deploying software on them
Must support multiple locations
Must be able to scale dynamically
Must support concurrent development
• Different feature branches
Must manage/version non-software components in a
lockstep with the software:
• Network configuration
• System-level configurations
• Application-level configurations

6. tools
02/01/11

7. Abstraction layer on resource management
If you code your infrastructure management
code to the jclouds API, you can port your
code to all cloud providers
JVM-based
Provides access to GoGrid’s infrastructure by
abstracting the communication GoGrid’s
Cloud API

8. Uses JClouds to create and manage nodes
Provisions nodes with software and
configuration
Verifies node and service correct operation
Operates the services in the nodes
Nodes are grouped by groups (formerly
known as ‘tags’). All nodes in a group are
equivalent

9. Clojure
Pallet is built on Clojure
Clojure is a JVM-based lisp with superb
integration with Java and concurrency
Can use any existing Java library (Tomcat, jdbc,
etc...)
Provides a high level of abstraction, which is
what is needed in this case
Growing community: there are 7 books on Clojure
either published or in preparation

10. how?
02/01/11

11. GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
Service Service Service
Service Service
Service
Service Service
Service

12. GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
jclouds Service Service Service
Service Service
Service
Service Service
Service

13. GoGrid Cloud Infrastructure
gimme
Service Service
nodes!
GoGrid Cloud API
jclouds Service Service Service
Service Service
Service
Service Service
Service

14. GoGrid Cloud Infrastructure
gimme
Service Service
nodes!
GoGrid Cloud API
jclouds Service Service Service
Service Service
Service
Service Service
Service

15. GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
Service Service Service
Service Service
Service
Service Service
Service

16. GoGrid Cloud Infrastructure
Service Service
et GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service

17. GoGrid Cloud Infrastructure
do stuff! Service Service
et GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service

18. GoGrid Cloud Infrastructure
do stuff! Service Service
et GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service

19. GoGrid Cloud Infrastructure
GoGrid Cloud API

20. GoGrid Cloud Infrastructure
GoGrid Cloud API
jclouds

21. GoGrid Cloud Infrastructure
make me
a dev cloud!
GoGrid Cloud API
jclouds

22. GoGrid Cloud Infrastructure
make me Service Service
a dev cloud!
GoGrid Cloud API
jclouds Service Service Service
Service Service
Service
Service Service
Service

23. GoGrid Cloud Infrastructure
Service Service
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service

24. GoGrid Cloud Infrastructure
install cloud
Service Service
infrastructure!
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service

25. GoGrid Cloud Infrastructure
install cloud
Service Service
infrastructure!
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service

26. GoGrid Cloud Infrastructure
Service Service
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service

27. GoGrid Cloud Infrastructure
is all working Service Service
fine?
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service

28. GoGrid Cloud Infrastructure
is all working Service Service
fine?
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
OK!
Service Service
Service

29. GoGrid Cloud Infrastructure
Service Service
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service

30. GoGrid Cloud Infrastructure
deploy Service Service
new build!
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service

31. GoGrid Cloud Infrastructure
deploy Service Service
new build!
et
GoGrid Cloud API
pall
Service Service Service
pallet Service Service
Service
Service Service
Service

32. GoGrid Cloud Infrastructure
deploy GoGrid Cloud Infrastructure
Service Service
new build!
et
GoGrid Cloud API
pall
GoGrid Cloud API
Service Service Service
pallet Service Service
Service
Service Service
Service

33. GoGrid Cloud Infrastructure
GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
Service Service
Service
Service Service
Service

34. GoGrid Cloud Infrastructure
awesome! GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
Service Service
Service
Service Service
Service

35. GoGrid Cloud Infrastructure
GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
Service Service
Service
Service Service
Service

36. GoGrid Cloud Infrastructure
GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
jclouds Service Service
Service
Service Service
Service

37. GoGrid Cloud Infrastructure
let’s test:
GoGrid Cloud Infrastructure
Service Service
gimme nodes!
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
jclouds Service Service
Service
Service Service
Service

38. GoGrid Cloud Infrastructure
let’s test:
GoGrid Cloud Infrastructure
Service Service
gimme nodes!
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
jclouds Service Service
Service
Service Service
Service

39. GoGrid Cloud Infrastructure
GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
Service Service
Service
Service Service
Service

40. GoGrid Cloud Infrastructure
GoGrid Cloud Infrastructure
Service Service
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
jclouds Service Service
Service
Service Service
Service

41. GoGrid Cloud Infrastructure
I’m done, GoGrid Cloud Infrastructure
Service Service
thanks!
GoGrid Cloud API
GoGrid Cloud API
Service Service Service
jclouds Service Service
Service
Service Service
Service

42. GoGrid Cloud Infrastructure
I’m done,
thanks!
GoGrid Cloud API
jclouds

43. GoGrid Cloud Infrastructure
GoGrid Cloud API

44. environment
GoGrid Cloud Infrastructure
for branch A?
GoGrid Cloud API

45. environment
GoGrid Cloud Infrastructure
for branch A?
GoGrid Cloud Infrastructure branch A
GoGrid Cloud API
GoGrid Cloud API

46. environment
GoGrid Cloud Infrastructure
for branch A?
GoGrid Cloud Infrastructure branch A
environment
GoGrid Cloud API
GoGrid Cloud API
for branch B?

47. environment
GoGrid Cloud Infrastructure
for branch A?
GoGrid Cloud Infrastructure branch A
GoGrid Cloud Infrastructure branch B
environment
GoGrid Cloud API
GoGrid Cloud API
for branch B?
GoGrid Cloud API

48. environment
GoGrid Cloud Infrastructure
for branch A?
GoGrid Cloud Infrastructure branch A
GoGrid Cloud Infrastructure branch B
environment
GoGrid Cloud API
GoGrid Cloud API
for branch B?
GoGrid Cloud API
environment
for QA please?

49. environment
GoGrid Cloud Infrastructure
for branch A?
GoGrid Cloud Infrastructure branch A
GoGrid Cloud Infrastructure branch B
GoGrid Cloud Infrastructure QA
environment
GoGrid Cloud API
GoGrid Cloud API
for branch B?
GoGrid Cloud API
GoGrid Cloud API
environment
for QA please?

50. development
02/01/11

51. GoGrid’s Development Process
Create a feature branch
Create an environment (cloud infrastructure) for
this branch
Verify new environment (end-to-end tests)
Automatically build and deploy the new software
every time new code is committed to the branch.
Destroy environment and free resources when
done

52. An Environment
Usually one environment for active
development branch
Each environment is specified by:
• The development branch it tracks
• Service types and node count for each service type
• Network configuration
• External services/DB provisioning

53. Development Environments
Each development environment tracks a
development feature branch
Each environment has a node with Hudson
CI, configured to track the environment’s
branch
When new code is pushed to the development
branch, the environment’s Hudson server
builds the software and updates the
environment with it.

54. more on pallet
02/01/11

55. Why Pallet?
There are well known solutions for node
provisioning in the cloud, why pallet?
• It integrates with infrastructure management
• Service interconnections are first class entities in
the framework
• Scalable programming model
• Integrates well with SCM and with other OPS tools
• Comprehensive support for all development cycles

56. ✓ No central server
✓ No agents on nodes
✓ Works with standard and custom images
✓ First-class support for complex multi-service
configurations
✓ All node configuration done remotely via
generated scripts over SSH
✓ DSL for generating scripts, resources
✓ Testable

57. crates
Pallet’s basic configuration unit
A crate is a grouping of functions around a
service
Functions usually:
• manipulate/create internal representations
• result in resources deployed or scripts being run in
the target nodes

58. proxied
Custom Crates
tomcat reverse
bootstrap tomcat
deploy proxy
Standard Crates
automated haproxy
admin user
java tomcat

59. A Node
(core/defnode
proxied
"Basic
web
app,
served
by
tomcat"
{:os-­‐family
:ubuntu
:os-­‐description-­‐matches
"10.04"
:inbound-­‐ports
[8080
22]}
:bootstrap
(resource/phase
(crates/bootstrap))
:configure
(resource/phase
(crates/tomcat))
:deploy
(resource/phase
(crates/tomcat-­‐deploy
"webapp.war"))
:restart-­‐tomcat
(resource/phase
(service/service
"tomcat6"
:action
:restart)))

60. jclouds section of a node
(core/defnode
proxied
"Basic
web
app,
served
by
tomcat"
{:os-­‐family
:ubuntu
:os-­‐description-­‐matches
"10.04"
:inbound-­‐ports
[8080
22]}
:bootstrap
(resource/phase
(crates/bootstrap))
:configure
(resource/phase
(crates/tomcat))
:deploy
(resource/phase
(crates/tomcat-­‐deploy
"webapp.war"))
:restart-­‐tomcat
(resource/phase
(service/service
"tomcat6"
:action
:restart)))

61. Execution Phases
(core/defnode
proxied
"Basic
web
app,
served
by
tomcat"
{:os-­‐family
:ubuntu
:os-­‐description-­‐matches
"10.04"
:inbound-­‐ports
[8080
22]}
:bootstrap
(resource/phase
(crates/bootstrap))
:configure
(resource/phase
(crates/tomcat))
:deploy
(resource/phase
(crates/tomcat-­‐deploy
"webapp.war"))
:restart-­‐tomcat
(resource/phase
(service/service
"tomcat6"
:action
:restart)))

62. Cloud Infrastructure
Crates proxy proxy
Cloud API
Pallet webapp webapp webapp
Node
definitions DB DB

63. Cloud Infrastructure
Crates proxy proxy
Cloud API
Pallet webapp webapp webapp
Node
definitions DB DB

64. Cloud Infrastructure
Crates proxy proxy
Cloud API
Pallet webapp webapp webapp
Node
definitions DB DB

65. Cloud Infrastructure
Crates proxy proxy
jclouds
Cloud API
Pallet webapp webapp webapp
Node
definitions DB DB

66. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
jclouds
Cloud API
Pallet webapp webapp webapp
Node
definitions DB DB

67. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
jclouds
Cloud API
webapp webapp webapp
Pallet webapp webapp webapp
Node
definitions DB DB

68. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
jclouds
Cloud API
webapp webapp webapp
Pallet webapp webapp webapp
Node
DB
DB DB
DB
definitions

69. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
Cloud API
webapp webapp webapp
Pallet webapp webapp webapp
Node
DB
DB DB
DB
definitions

70. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
squid squid
Cloud API
webapp webapp webapp
Pallet webapp webapp webapp
Node
DB
DB DB
DB
definitions

71. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
squid squid
Cloud API
webapp webapp webapp
Pallet webapp webapp webapp
Node
DB
DB DB
DB
definitions

72. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
squid squid
Cloud API
webapp webapp webapp
Pallet webapp webapp webapp
tomcat tomcat tomcat
Node
DB
DB DB
DB
definitions

73. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
squid squid
Cloud API
webapp webapp webapp
Pallet webapp webapp webapp
tomcat tomcat tomcat
Node
DB
DB DB
DB
definitions

74. Cloud Infrastructure
proxy
proxy proxy
proxy
Crates
squid squid
Cloud API
webapp webapp webapp
Pallet webapp webapp webapp
tomcat tomcat tomcat
Node
DB
DB DB
DB
definitions
mysql mysql

75. How are systems interconnected?
Phase 1
• Retrieve and/or create needed nodes
• For each node in each group: run phases for crates
and
a) Collect configuration info
b) Schedule configuration actions to be performed
on node
Phase 2
• Generate and execute scripts and resources

76. how does this look
today?
02/01/11

77. Environment Definition
(def branch-1-environment
{:branch "http://svn....../branches/branch-1"
:compute-service {:provider "gogrid"
:identity "<username>"
:credential "<password>"
:endpoint "http://10...:8080/api/"}
:db-name "branch-1-db"
:db-properties env-network-properties
:pre-bootstrap-fn #'setup-environment-db
:bootstrap-tags {com.gogrid.nodes.services/LDAP 1
com.gogrid.nodes.services/Cache 1}
:bootstrap-phases [:restart-cache :restart-LDAP]
:verify-phases [:verify]})

78. Operation
user => (start branch-1-environment)
user => (verify branch-1-environment)
user => (destroy branch-1-environment)

79. Challenge with Pallet
It’s a newcomer in the field.
• Documentation needs work
• Untested support for some combinations of flavors
and versions of Linux
Steep learning curve, a new language
OMG! It’s A Lisp!

81. Some pointers...
http://jclouds.org Adrian Cole et al.
http://palletops.com Hugo Duncan et al.
http://gogrid.com For more info on GoGrid
services please contact:
Paul Lappas
VP Engineering & Co-Founder of GoGrid
paul@gogrid.com
GoGrid is hiring devops and SW engineers
for their SF office (contact Paul too)

82. Questions?
Toni Batchelli
tbatchelli@gmail.com
@tbatchelli
@disclojure
02/01/11