ADDO_2022_CICID_Tom_Halpin.pdf

TRACK: CI/CD CONTINUOUS EVERYTHING
NOVEMBER 10, 2022
Tom Halpin, Distinguished Engineer
Codified Manifestation of an
Agile Release Train - Enabling
CI/CD for Highly Regulated
Environments

Introduction

Tom Halpin
 Enablement - Distinguished Engineer working in a
DevOps enablement role.
 Coaching -Teams to transform how they work to a DevOps
model in support of product-aligned value streams.
 Facilitating - Scaled adaption of the associated culture,
practices, and tools across organizations.
 Third time presenting at ADDO highly recommend the
experience.

DevOps
An Imperfect Search for Perfection

• Three Ways – A Principle-based DevOps Framework
• First Way
• Principles of flow - work always flows in one direction – downstream.
• Second Way
• Principles of feedback - create, shorten and amplify feedback loops.
• Third Way
• Principles of continuous learning - continued experimentation, learn from
mistakes, and achieve mastery.
The Phoenix Project - Gene Kim, Kevin Behr, George Spafford
DevOps - Underlying Principles

You build it, you run it
"Giving developers operational responsibilities has greatly enhanced the quality of the
services, both from a customer and a technology point of view.
The traditional model is that you take your software to the wall that separates
development and operations and throw it over and then forget about it.
Not at Amazon. You build it, you run it. This brings developers into contact with the day-
to-day operation of their software. It also brings them into day-to-day contact with the
customer.
This customer feedback loop is essential for improving the quality of the service.“
Amazon CTO Werner Vogels - 2006 Interview
DevOps - Definition of Awesome

Dev-HRE-Ops
Dev – Highly Regulated Environment - Ops

“Highly Regulated / Restricted Environment (HRE) - a physical or digital
environment characterized by: air-gapped physical spaces, air-gapped
computer systems, heightened access controls, segregation of duties, inability
to discuss certain topics outside of specific physical spaces, and an inability to
transport certain artifacts off premise.”
DEVOPS BLOG - Jose Morales
Highly Regulated Environment (HRE) -
Definition

• Processes tightly controlled.
• You build it, you run it remains the objective but is not always possible.
• Regulatory environment drives both business investment and technical
choices.
• Security controls and governance processes mandated by regulatory
environment.
• Segregation of duties often a requirement.
• Deploying changes more frequently often viewed as a risk to security and
governance controls.
• Application architecture can drive Dev-HRE-Ops outcomes.
Dev-HRE-Ops - Implications

• Common misconception arises that HRE means you cannot “do” DevOps.
• Dev-"HRE"-Ops is possible and desirable in support of HRE mandated
processes.
• Enablement of operation & support teams key to success in the last mile.
• Cultural challenge as much as a technical challenge.
• Important to include all stake-holders: squad members, operators, auditors,
regulators & change control in Dev-HRE-Ops enablement efforts.
Dev-HRE-Ops - Enabling

• Last mile challenges
• Product development squads not allowed access to production
systems.
• Air-gapped server rooms & computer systems.
• Restrictions can include
• In region or in country personnel required.
• Security clearance required.
• Screen sharing prohibited.
• Log sharing prohibited.
• Access via secure devices.
• Limited access to network services and associated resources.
Dev-HRE-Ops - Last Mile Challenges

• Application architecture - microservices, loosely coupled, APIs etc.
• Automate E2E value streams – using CI & CD codified pipelines
• Have an everything as code mindset
• PRs, CI/CD, Test Automation, Security Controls, Issues,
Documentation, Infrastructure & Change Controls.
• Codify compliance, access and regulatory requirements.
• Make use of the available tools
• Source Code Management - GitHub, GitLab, BitBucket etc.
• Agile - Jira, Trello etc.
• Orchestration - Jenkins, CircleCI, Travis CI, Cloud Native etc.
• Binary Repositories - Artifactory, Nexus, Docker Registries etc.
Dev-HRE-Ops - Approach

Dev-HRE-Ops - Enablement
Application Architecture

• Application architecture has evolved over the past decade led by tech
companies such as Meta, Netflix, AWS & others.
• Increased demands on responsiveness, scalability & resilience drove
a move to Cloud Native applications underpinned by DevOps
methodologies & tools.
• Applications developed were typically for green field environments,
however majority of HRE applications are in brown field environments.
• DevOps methodologies & tools can still be used to gain efficiencies
when developing for HRE environments.
Application Architecture - Evolution

"A microservice architecture – is an architectural pattern that arranges
an application as a collection of loosely-coupled, fine-grained services,
communicating through lightweight protocols."
- Wikipedia
Application Architecture - Microservices

Microservices typically
 are run in dedicated processes.
 can communicate via API over standard industry protocols.
 address specific business capabilities.
 are independently buildable.
 are independently deployable typically using containers.
 allow the choice of programming language and the choice of storage
medium based on capabilities of squad and/or requirements of the
business.

Microservices Benefits
 Scalability.
 Resiliency.
 Development Velocity.
 Continuous Delivery / Deployment.

• Great example of a Lambda
Dance at scale.
• Lambdas are used to build a
landing page tailored to each
and every customer upon
authentication.
Microservice Scaling - AWS Homepage

Pipelines As Code

 How do we build for and deploy to very complex environments with
many moving parts?
 CI/CD Pipelines allow squads to integrate & deliver code which
implements business features as they are developed
• CI - Continuous Integration
• CD - Continuous Deployment / Continuous Delivery
 Pipelines furthermore can be
• Implemented as code.
• Stored in the application source code management repository.
• Developed using the squads preferred development tools &
methodologies.
Pipelines as Code

Pipelines as Code

 Development squad using an Agile backlog
• All code in a centralized source code management system.
• Build quality in to applications via high degree of automated testing.
• Work in small batches using short lived (one to several days) feature
branches when developing a capability or fixing a bug.
• Continuously submit changes to main branch via Pull Requests (PRs).
• Every code commit automatically triggers CI pipeline which builds the
code, runs automated unit tests, code quality checks & static
application security tests etc.
• Pipeline run results (pass/fail) results are directly available from the PR
providing timely and accurate feedback to the squad.
Pipelines as Code - Continuous Integration

• Person reviewing the PR will not merge the request to master if any of
the checks or tests ran from the CI pipeline fail.
• When a pull request is merged to the main branch the pipeline can
– Build a binary artifact.
– Deploy the artifact to a binary package repository, a docker registry or
even to a pre-production staging server.
– Run additional test cycles.
Pipelines as Code - Continuous Integration

• Continuous Deployment is where every change merged to master
following the successful execution of a CI pipeline results in the build,
test and deployment of a binary package to production.
• This is not acceptable in the majority of HRE environments, where
unplanned changes are not allowed.
• HRE environments typically prefer a more predictable release cycle,
ideally with the release change logs available for inspection and
approval in advance.
Pipelines as Code - Continuous Deployment

 Similar to Continuous Deployment but where a release to production
requires a business decision and human intervention
• Approvals can be managed via PRs created in an agreed repository.
 Continuous Delivery is both aligned with the deployment
requirements for HRE environments and the 3 ways of DevOps
• Errors detected early in the pipelines.
• Packages only built when pipelines succeed.
• Always have a package ready for production.
• Change control and release controls codified.
• Culture of experimentation fostered.
• Feedback acted upon via the squad's Agile backlog.
Pipelines as Code - Continuous Delivery

Scaling - Codified Release Train

 Pipeline as code approach scales well for HRE environments.
 Multiple squads working in an Agile manner on different capabilities.
 Feedback used to drive innovation & to build the functionality the
business requires.
Scaling Pipelines as Code - Release Train

Scaling Pipelines as Code - Release Train

Conclusions

• Dev-"HRE"-Ops is both possible and desirable.
• Last mile poses challenges to DevOps definition of awesome.
• Operational & support team enablement essential to success.
• Cultural challenge as much as a technical challenge.
• Everything as code mindset important
• Make use of the available tools.
• Pipelines as code automate delivery end to end.
• Codify compliance and regulatory requirements.
• Scalable to the Release Train level.
Dev-HRE-Ops - Conclusions

• Key takeaway - it’s a journey - an imperfect search for perfection.
This is the "3" ways.
Dev-HRE-Ops - Conclusions

Q&A
That’s All Folks

ADDO_2022_CICID_Tom_Halpin.pdf

Recommended

Recommended

More Related Content

Similar to ADDO_2022_CICID_Tom_Halpin.pdf

Similar to ADDO_2022_CICID_Tom_Halpin.pdf (20)

Recently uploaded

Recently uploaded (20)

ADDO_2022_CICID_Tom_Halpin.pdf