Tomas Doran talks about Deploying Puppet Code at Light Speed at Puppet Camp Silicon Valley

2. Deploying Puppet Code At Light Speed
Tomas Doran
@bobtfish
tdoran@yelp.com
2014-01-27

3. Key lessons
•Speed of light is (still) slow
– West coast => EU slow
– East coast => Asia slow
•git <3
– Shipping minimal changes = fast
•Eventual consistency FTW
– Centrally orchestrating a global network = LOL
– Parts of the network will be down
•mcollective <3
– Yes, it’s a chainsaw
– Decoupling good!

4. What we did
•New puppet deployment system
• svn => git
• ssh for loop => parallel mcollective
• All users sudo root => mcollective policies
• push => pull
•Winning!
– 4m deployment
– 4s deployment
– 32 puppetmasters globally

5. Why?
•Pull models > Push models
– Eventual consistency FTW
– In a large network, you will have failure - don’t rely on
100% up to ship new code
– Just cron updates to environments you care about
•Environments
•
•
•
•
Personal branches
Demo/test your code with —environment
Different teams can own different parts of the code
Staged rollouts by merging changes between
branches

6. How?
• Environments in puppet.conf
• For modules
• modulepath = /etc/puppet/environments/
$environment/modules
• For site.pp
• manifest = /etc/puppet/environments/
$environment/manifests/site.pp
• For hiera data
• datadir = /etc/puppet/environments
• %{::environment}/common.yaml
• For manifests/ - refactor!!!

7. Segue - ‘refactor’
•Originally it meant…
• Change form
• But not function
• incrementally
• Supported by tests
•I’m giving up on that meaning…
• So abused by everyone, all the time
• It just means ‘change shit’
• See also hacker/cracker…
• We won that one, right?

8. How?
•Custom mcollective agent
–Every git branch => puppet environment
– 205 lines of code
– 215 lines of tests ;)
•Cron job
–
–
–
–
mcollective agent also ships a CLI (local) client
Just cron updates to all the branches you care about
(or all branches)
Eventual consistency!

9. Detached work trees
•/etc/puppet/puppet.git
git clone —bare
git —git-dir=/etc/puppet/puppet.git
—worktree=/etc/puppet/environments/my
_example checkout --detach
--force master

10. Gitolite
•Allows role users
– Generate ‘puppetupdate’ ssh key.
– Allow this to READ the puppet code.
– Distribute to puppet masters
•All the access controls
–
–
–
–
Multiple puppetupdate ssh keys.
Allow different keys different branches
dev/stage/prod
I don’t need this _yet_, but it’s there!

11. Gotchas
•ssh concurrency limits
– Defaults are conservative
– Limit number of processes in ‘preauth’
•Scaling MOAR
– If you have 100s of puppet masters
– You’re gonna want to have multiple git servers
– Still easy, just 2 step orchestration:
– Pull to all slave git servers
– puppetupdate all the masters

12. TODOs
•Documentation not on slideshare
– I did fix the README, it’s still not awesome.
– Only 250 lines of code, just reading it isn’t hard ;)
•Better application
– Nicer user display of status
•Extend mcollective
– mco plugin package only bundles mco bits
– You need to ship /usr/local/sbin/puppetupdate
yourself

13. Link me?
• https://github.com/Yelp/puppetupdate
• Slides: http://slideshare.net/bobtfish
• Tweet me @bobtfish
• Guess what?
• We’re hiring!!!
• SF + Palo Alto
• (+ London, + Hamburg, + Dublin)