Slides for my DevSecOps Leadership forum in Amsterdam

1. Devops is a securityDevops is a security
RequirementRequirement
@KrisBuytaert
May 2018, Amsterdam

2. Kris BuytaertKris Buytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● Even did Security (OSSTM)Even did Security (OSSTM)
● Chief Trolling Ofcer and Open SourceChief Trolling Ofcer and Open Source
Consultant @inuits.euConsultant @inuits.eu
● Everything is an freaking DNS ProblemEverything is an freaking DNS Problem
● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore
● Some books, some papers, some blogsSome books, some papers, some blogs
● eToo many conferences. #devopsdays,eToo many conferences. #devopsdays,
#loadays, #cfgmgmtcamp#loadays, #cfgmgmtcamp

3. Who has upgraded his business criticalWho has upgraded his business critical
applications over the past 12 months ?applications over the past 12 months ?

4. Why not ?Why not ?

5. What's this Devops thing really about ?What's this Devops thing really about ?

6. World , 200X-2009World , 200X-2009
Patrick Debois, Gildas Le Nadan, Andrew Clay Shafer, Kris Buytaert, JezzPatrick Debois, Gildas Le Nadan, Andrew Clay Shafer, Kris Buytaert, Jezz
Humble, Lindsay Holmwood, John Willis, Chris Read, Julian Simpson, andHumble, Lindsay Holmwood, John Willis, Chris Read, Julian Simpson, and
lots of others ..lots of others ..
Gent , October 2009Gent , October 2009
Mountain View , June 2010Mountain View , June 2010
Hamburg , October 2010Hamburg , October 2010
Boston, March 2011Boston, March 2011
Mountain View, June 2011Mountain View, June 2011
Bangalore, Melbourne,Bangalore, Melbourne,
Goteborg , October 2011Goteborg , October 2011

7. C(L)AMSC(L)AMS
● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● MeasurementMeasurement
● SecuritySecurity
Damon Edwards and John WillisDamon Edwards and John Willis

8. Debunking the CriticsDebunking the Critics
Security not included ?Security not included ? Everyone is Included:Everyone is Included:
security, dba, devs,security, dba, devs,
ops, designer, analysts,ops, designer, analysts,
We are solving a busines problem,We are solving a busines problem,
Not a technology problemNot a technology problem

9. *ops*ops
*.**.*

10. Frank BreedijkFrank Breedijk
@seccubus@seccubus
● Http → httpsHttp → https
● Imap → imapsImap → imaps
● Pop3 → pop3sPop3 → pop3s
● Devop → devopSDevop → devopS

11. ““DevOps is a cultural andDevOps is a cultural and
professional movement”professional movement”
Adam JacobAdam Jacob

12. How did we get here ?How did we get here ?

13. The(se)(Old) DaysThe(se)(Old) Days
● ““Put this Code Live, here's a tarball/container”Put this Code Live, here's a tarball/container”
● What dependencies ?What dependencies ?
● No machines available ?No machines available ?
● What database ?What database ?
● Security ?Security ?
● High Availability ?High Availability ?
● Scalability ?Scalability ?
● My computer can't install this ?My computer can't install this ?

14. Devs vs OpsDevs vs Ops

15. People hated SysadminsPeople hated Sysadmins
BecauseBecause
●
They slow stuf downThey slow stuf down
●
The say noThe say no
●
They say no againThey say no again
●
They refuse to break stufThey refuse to break stuf
●
They care about uptimeThey care about uptime
●
They don't care about fancy newThey don't care about fancy new
featuresfeatures

17. People hate SecurityPeople hate Security
BecauseBecause
●
They slow stuf downThey slow stuf down
●
The say noThe say no
●
They say no againThey say no again
●
They refuse to leave holes openThey refuse to leave holes open
●
They care about securityThey care about security
●
They don't care about fancy newThey don't care about fancy new
featuresfeatures
Security Ofcers have an expiry dateSecurity Ofcers have an expiry date

19. 10 days into operation10 days into operation
● What High Load ? What Memory usage ?What High Load ? What Memory usage ?
● Are these Logs ? Or this is actualy customerAre these Logs ? Or this is actualy customer
data ?data ?
● How many users are there , should they launchHow many users are there , should they launch
100 queries each ?? Oh we're having 10K100 queries each ?? Oh we're having 10K
usersusers
● Why is debugging enabled ?Why is debugging enabled ?
● Who wrote this ?Who wrote this ?
● Does this user belong here ?Does this user belong here ?

20. 11 days into operations11 days into operations

21. 12 days into operations12 days into operations

22. 13 days into operations13 days into operations

23. 14 days into operations14 days into operations

24. Tomorrow :)Tomorrow :)

25. We can solve this !We can solve this !
● We are not here toWe are not here to
blockblock
● Some people thinkSome people think
the Security /the Security /
Operations workOperations work
starts on deploymentstarts on deployment
● It starts much earlierIt starts much earlier
● Start talking asapStart talking asap

26. Culture,Culture,
automation,automation,
Measturement,Measturement,
sharingsharing

27. Breaking the SilosBreaking the Silos
Getting AlongGetting AlongOpsOpsDevsDevs

28. ● Who is in charge of security ?Who is in charge of security ?
● What do your developers think about security ?What do your developers think about security ?
● When do you think about security ?When do you think about security ?
● The problem with security is it doesn'tThe problem with security is it doesn't
generate revenuegenerate revenue
● Security needs to become part of your DNA.Security needs to become part of your DNA.

29. With great power ...With great power ...
Your code will go to production..Your code will go to production..
You will be able to fx it ..You will be able to fx it ..
You will have access to the logsYou will have access to the logs
Access to the metrics...Access to the metrics...

31. https://www.slideshare.net/jedi4ever/from-devops-to-devops-what-a-diference-one-character-makes/75/

32. Devops is a ReorgDevops is a Reorg
● New role for Change ManagementNew role for Change Management
● New role for Security OfcersNew role for Security Ofcers
● Added roles for TestersAdded roles for Testers
● Shift LeftShift Left

34. Whats in it for you ?Whats in it for you ?
•Faster time to marketFaster time to market
•Features go live in hours vs yearsFeatures go live in hours vs years
•In a more safe (Secure)In a more safe (Secure)
•Reliable fashionReliable fashion
•Fully automatedFully automated
•More happyMore happy {customers,developers,managers,investors}{customers,developers,managers,investors}

35. Culture,Culture,
Automation,Automation,
Measurement,Measurement,
SharingSharing

36. " Our job as engineers (and ops, dev-ops, QA," Our job as engineers (and ops, dev-ops, QA,
support, everyone in the company actually) is tosupport, everyone in the company actually) is to
enable the business goals. We strongly feel thatenable the business goals. We strongly feel that
in order to do that you must havein order to do that you must have the ability tothe ability to
deploy code quickly and safelydeploy code quickly and safely. Even if the. Even if the
business goals are to deploy strongly QA’d codebusiness goals are to deploy strongly QA’d code
once a month at 3am (it’s not for us, we push allonce a month at 3am (it’s not for us, we push all
the time), having a reliable and easythe time), having a reliable and easy
deployment should bedeployment should be non-negotiablenon-negotiable."."
Etsy Blog upon releasing DeployinatorEtsy Blog upon releasing Deployinator
http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/

37. This is not Continuous DeploymentThis is not Continuous Deployment
@stahnma@stahnma
@#devopsdays Ohio@#devopsdays Ohio

38. Continuous Delivery is aContinuous Delivery is a
Security RequirementSecurity Requirement

39. MTTR ~> 0MTTR ~> 0

40. How do we get fromHow do we get from
We don’t dare to patchWe don’t dare to patch
ToTo
All systems green , lets goAll systems green , lets go

41. It's too riskyIt's too risky
•We deployed 6 months ago, it wasWe deployed 6 months ago, it was
painfull, we needed 3 weeks aftercarepainfull, we needed 3 weeks aftercare
•There's 3576 changes in the newThere's 3576 changes in the new
deploy, we have no clue what causeddeploy, we have no clue what caused
this problemthis problem
•We need 20 people in a room for 8-12We need 20 people in a room for 8-12
hourshours
•I have no clue why I wrote that line ofI have no clue why I wrote that line of
code 3 months agocode 3 months ago
•The person who wrote this left 2The person who wrote this left 2
weeks agoweeks ago
•Ooops we forgot to delete that featureOoops we forgot to delete that feature
they don't want anymore.they don't want anymore.
•We deploy automaticaly,We deploy automaticaly,
•I clearly remember what we fxedI clearly remember what we fxed
yesterdayyesterday
•And that's the only thing that hasAnd that's the only thing that has
changed in the last commitchanged in the last commit
•The person who wrote the code is stillThe person who wrote the code is still
in the buildingin the building
•We really need this feature now, weWe really need this feature now, we
can remove it latercan remove it later

43. Every commitEvery commit withwith successful testssuccessful tests
willwill automaticallyautomatically bebe deployeddeployed
productionproduction

44. Every commitEvery commit with successful testwith successful test
will automatically be deployed towill automatically be deployed to
productionproduction
Version controlVersion control
Who, changed what, why and whenWho, changed what, why and when

45. Every commit withEvery commit with successful testssuccessful tests
will automatically be deployed towill automatically be deployed to
productionproduction
Automated testing strategy, is keyAutomated testing strategy, is key
Successful tests, no bypassing of theSuccessful tests, no bypassing of the
teststests

46. Test all the thingsTest all the things
•Unit testsUnit tests
•Integration TestsIntegration Tests
•System TestsSystem Tests
•Acceptance TestsAcceptance Tests
•Security TestsSecurity Tests
•Performance TestsPerformance Tests
•Regression TestsRegression Tests
•Functional TestsFunctional Tests

47. Every commit with successful testsEvery commit with successful tests
willwill automaticallyautomatically be deployed tobe deployed to
productionproduction
Automate all the things !Automate all the things !
No humans involved,No humans involved,
Less error proneLess error prone
Less boringLess boring

48. Every commit with successful testsEvery commit with successful tests
will automatically bewill automatically be deployed todeployed to
productionproduction
Deployed code does not meanDeployed code does not mean
enabled feature.enabled feature.

49. Auditors / ComplianceAuditors / Compliance
•We do the same, just automatedWe do the same, just automated
•Separation of DutiesSeparation of Duties
●
Man vs MachineMan vs Machine
•Authentication and Audit TrailAuthentication and Audit Trail
•Full automation, Git logs, Deploy logs,Full automation, Git logs, Deploy logs,
no more manual actionsno more manual actions
•Have you tried talking to them ?Have you tried talking to them ?

51. What's in your Pipeline ?What's in your Pipeline ?

52. A pipelineA pipeline
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● PackagePackage
● Upload to RepoUpload to Repo

53. A pipeline++A pipeline++
● Checkout codeCheckout code
● SyntaxSyntax
● StyleStyle
● Code CoverageCode Coverage
● TestsTests
● BuildBuild
● More TestsMore Tests
● PackagePackage
● Upload to RepoUpload to Repo
● Deploy on TestDeploy on Test
● ……
● Insert SECURITYInsert SECURITY
TESTS !TESTS !

54. Attack yourselve onAttack yourselve on
every buildevery build
● Gauntlt , write security testsGauntlt , write security tests
● Vulnerability scans (Arachni)Vulnerability scans (Arachni)
● OpenVASOpenVAS
● OWASP DevSlop Tool ProjectOWASP DevSlop Tool Project
● The OWASP AppSec Rugged DevOps PipelineThe OWASP AppSec Rugged DevOps Pipeline
ProjectProject
● Zed Attack Proxy (ZAP)Zed Attack Proxy (ZAP)

55. Leverage InfrastructureLeverage Infrastructure
as Codeas Code
● Confgure 1000 nodes,Confgure 1000 nodes,
● Modify 2000 fles,Modify 2000 fles,
● TogetherTogether
● Think :Think :
●
Cfengine,Puppet, Chef, SaltCfengine,Puppet, Chef, Salt
● Put confgs under version controlPut confgs under version control
● Please don't roll your own ...Please don't roll your own ...

56. Puppet in ActionPuppet in Action

57. Policies/Hardening withPolicies/Hardening with
Dev-sec.ioDev-sec.io

58. OrchestrationOrchestration
● Fix security issues with 1 commandFix security issues with 1 command
● mco package bind upgrademco package bind upgrade
● Write Ansible role to upgrade XYZWrite Ansible role to upgrade XYZ

59. Culture,Culture,
Automation,Automation,
Measurement :Measurement :
measure all the thingsmeasure all the things
SharingSharing

60. Logstash in ActionLogstash in Action
Screenshot dating 2012 ON purpose .. this is NOT bleeding edge technology

61. S in devops/Clams ?S in devops/Clams ?
● Version control => AuditingVersion control => Auditing
● CI/CDCI/CD
•
Add security IN the pipelineAdd security IN the pipeline
•
Reduce MTTMReduce MTTM
● Confguration MgmtConfguration Mgmt
●
Auditing & EnforcingAuditing & Enforcing
●
Policy DefnitionPolicy Defnition
● Monitoring : Find the anomaliesMonitoring : Find the anomalies

62. It's not about the toolsIt's not about the tools
It's about changeIt's about change
It's about the peopleIt's about the people

63. {devops/security }{devops/security }
is not a product you can buy,is not a product you can buy,
It's a lifestyleIt's a lifestyle

64. ContactContact
Kris Buytaert Kris.Buytaert@inuits.euKris Buytaert Kris.Buytaert@inuits.eu
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.eu/http://www.inuits.eu/
InuitsInuits
Essensesteenweg 31Essensesteenweg 31
2930 Brasschaat2930 Brasschaat
BelgiumBelgium
891.514.231891.514.231
+32 475 961221+32 475 961221