OSDC 2019 | Introducing Kudo – Kubernetes Operators the easy way by Matt Jarvis

© 2019 Mesosphere, Inc. All Rights Reserved. 1
Introducing Kudo :
Kubernetes Operators the Easy Way
Matt Jarvis
@mattj_io

❏ Senior Director - Community and Evangelism @ Mesosphere
❏ Building stuﬀ with open source software for ~20 years
❏ Ops, Dev and Dev/Ops
Matt Jarvis, Senior Director - Community and Evangelism
@mattj_io

Stateless Applications
Service v1.0
Service v1.0LoadBalancer
Client
● No state persisted

Service v1.0
Client
Service v1.0
● Easy to scale up/down

Service v2.0
Client
Service v1.0

Service v2.0
Client
Service v2.0
● Straightforward to upgrade

Stateful Applications
ServiceClient
Data

Stateful Applications
Instance
Instance
Client
Instance
Service

Kubernetes
Focused initially for purely stateless workloads
Scheduler can move pods around

Kubernetes - StatefulSets
StatefulSets are valuable for applications that require one or
more of the following.
● Stable, unique network identifiers.
● Stable, persistent storage.
● Ordered, graceful deployment and scaling.
● Ordered, graceful deletion and termination.
● Ordered, automated rolling updates.

Node
Master
Node
Slave
Node
Slave

Node
Master
Node
Master
Node
Slave

Node
Slave
Node
Master
Node
Slave

Operators
● Orchestrate applications using k8s API
● Extend API using Custom Resource Definitions
● Encode domain specific operational knowledge
○ Upgrades
○ Failure and Recovery Scenarios
○ Scaling up/down
● Purpose built per application

Operators
● Operator manages and monitors lifecycle
● CRD’s represent application elements/actions
Operator
CRD CRD CRD
apiVersion: mysql.presslabs.org/v1alpha1
kind: MysqlCluster
metadata:
name: my-cluster
spec:
replicas: 2
secretName: my-secret
kubectl apply -f mysql-cluster.yaml

Developing Operators
Operator Framework
● Redhat/IBM project
● Implement using Ansible,
Helm charts, or Go
● Existing implementations
often don’t cover the entire
lifecycle
● Ansible and Helm are
limited. Go requires 1,000s
of lines of controller code
Kubebuilder
● Kubernetes SIG API Machinery
sub-project
● Operators written in Go with a
focus on code generation
● Existing implementations
often don’t cover the entire
lifecycle

Developing Operators
Operators require deep knowledge of k8s internals
May require thousands of lines of code

© 2019Mesosphere, Inc. All Rights Reserved.
Two level scheduling
Mesos Master and Agents
● Abstract resources into single pool
● Offers and tracks resources
● Guarantees isolation
● Handles workload restart on failure
Mesos Framework
● Consumes resources
● Deploys tasks
● Provides application specific logic for deployment, recovery, upgrade
Zookeeper Quorum
Master
Master
(Standby)
Master
(Standby)
Frameworks
Agent
Spark
Agent
Kubernetes
Agent
Spark
Kubernetes
Agent
Kafka
Kubernetes
Executor
Task
22

Developing Frameworks ...
23
Frameworks require deep knowledge of Mesos internals
May require thousands of lines of code

Complexity but ..
24
● Commonality between lifecycle states
● Possible to build abstraction SDK

KUDO
25
● Kubernetes Universal Declarative Operator
○ Universal operator configured via YAML
○ Encodes commonality between lifecycle
operations
● OS project licensed as Apache 2.0
● Open Development model

Operator Development
26
Kubebuilder
● Kubernetes SIG API Machinery
sub-project
● Operators written in Go with a
focus on code generation
● Existing implementations often
don’t cover the entire lifecycle
Operator Framework
● Redhat/IBM project
● Implement using Ansible, Helm
charts, or Go
● Existing implementations often
don’t cover the entire lifecycle
● Ansible and Helm are limited.
Go requires 1,000s of lines of
controller code
KUDO
● Universal Operator
● Built using community projects
(Kubebuilder, Kustomize, ...)
● Write Operators as templated
YAML manifests
● Provide high level CRDs that
represent workloads
● Focused on higher level
coordination of software
lifecycles
● “Day 2 Operators”

Kudo Concepts
27
Framework
apiVersion: kudo.k8s.io/v1alpha1
kind: Framework
metadata:
labels:
controller-tools.k8s.io: "1.0"
name: kafka
● High level description of a service
● Represented as a CRD

Kudo Concepts
28
Framework
apiVersion: kudo.k8s.io/v1alpha1
kind: FrameworkVersion
metadata:
labels:
controller-tools.k8s.io: "1.0"
name: kafka-2.11-2.4.0
namespace: default
spec:
serviceSpec:
version: "2.11-2.4.0"
connectionString: ""
framework:
name: kafka
kind: Framework
parameters:
- name: BROKER_COUNT
description: "Number of brokers spun
up for Kafka"
default: "3"
displayName: "Broker Count"
...
plans:
deploy:
strategy: serial
phases:
- name: deploy-kafka
strategy: serial
steps:
- name: deploy
tasks:
- deploy● Implementation of a framework
● Contains parameters, objects, plans
Framework Version

Kudo Concepts
29
Framework
● Ties application instantiation to a framework version
Framework Version
Instance

Kudo Concepts - Plans
30
● Describe progress through operations
● 3 level hierarchy Plan - Phase - Steps
● Status captured by PlanExecution CRD
Plan foo
├─ Phase bar
│ ├─ Step qux
│ └─ Step quux
└─ Phase baz
├─ Step quuz
├─ Step corge
└─ Step grault

Kudo Concepts - Deployment Strategy
31
● Indicates the way in which a plan, or Step must be executed.
● If a Step requires another Step to complete first it is necessary
to declare them as serial
● Serial and Parallel are the defaults, but others can be defined

Kudo Concepts - Trigger
32
● Defines what actions to take when a parameter changes
● Gives additional customization capabilities
parameters:
- name: BACKUP_FILE
description: "Filename to save the
backups to"
default: "backup.sql"
displayName: "BackupFile"
trigger: backup

FrameworkVersion Spec Examples - Plans
plans:
deploy:
strategy: serial
phases:
- name: deploy
strategy: serial
steps:
- name: deploy
tasks:
- deploy
- name: init
tasks:
- init
backup:
strategy: serial
phases:
- name: backup
strategy: serial
steps:
- name: backup
tasks:
- backup
restore:
strategy: serial
phases:
- name: restore
strategy: serial
steps:
- name: restore
tasks:
- restore
Serial or parallel
Serial or parallel
Tasks can be reused in
multiple plans

FrameworkVersion Spec Examples - Tasks
tasks:
deploy:
resources:
- mysql.yaml
init:
resources:
- init.yaml
backup:
resources:
- backup-pv.yaml
- job.yaml
patches:
- backup.yaml
restore:
resources:
- job.yaml
patches:
- restore.yaml
Mustached k8s objects
Tasks can be reused in
multiple plans
Patches on the roadmap

parameters:
- name: BROKER_COUNT
description: "Number of brokers spun
up for Kafka"
default: "3"
displayName: "Broker Count"
required: false
trigger: deploy
…
statefulset.yaml: |
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{NAME}}
namespace: {{NAMESPACE}}
spec:
replicas: {{BROKER_COUNT}}
…
Deﬁne parameter
Default value
Use parameters in spec
Plan to use for upgrading value
FrameworkVersion Spec Examples - Parameters

apiVersion: maestro.k8s.io/v1alpha1
kind: Instance
metadata:
name: kafka-instance
spec:
frameworkVersion:
name: kafka-2.11-2.4.0
namespace: default
type: FrameworkVersion
parameters:
KAFKA_ZOOKEEPER_URI:
zk-zk-0.zk-hs:2181,zk-zk-1.zk-hs:2181,zk-
zk-2.zk-hs:2181
KAFKA_ZOOKEEPER_PATH: "/kafka"
BROKER_COUNT: "1"
Currently a single CRD for Instantiations of a
FrameworkVersion
Reference
FrameworkVersion to
use
Deﬁne parameters for
instance
FrameworkVersion Spec Examples - Instance

Kudo - CLI
37
● CLI extension to kubectl # Install a KUDO package from the official GitHub repo.
kubectl kudo install <name> [flags]
# View plan history of a specific package
kubectl kudo plan history <name> [flags]
# View all plan history of a specific package
kubectl kudo plan history [flags]
# List instances
kubectl kudo list instances [flags]
# View plan status
kubectl kudo plan status [flags]

Kudo Futures - Dynamic CRD’s
38
● Create new CRD’s during runtime operation
● Update existing CRD’s
$ kubectl apply -f new-plan.yaml
plan.kafka.kudo.dev/latest blue-green created
$ kubectl kudo update-parameter-trigger --framework kafka --framework-version 1.1 --parameter IMAGE_TAG blue-green
Update successful. IMAGE_TAG updates will now trigger the `blue-green` plan

Kudo Futures - Dynamic CRD’s
39
● Represent components as ﬁrst-class Kubernetes objects
● Represent operations as ﬁrst-class Kubernetes objects
○ Kind: Kafka
○ Kind: Topic
○ Kind: User
○ Backup
■ maxAge: 24h
○ Backup / Restore
○ Index

Kudo Futures - Framework Extensions
40
MySQL
“Standard” infrastructure, plans, CRDs, etc.
MySQL + GKE
Istio, Cloud Storage, GCP Security Rules,
StackDriver Monitoring, etc.
ACME Corp
ACME speciﬁc plans. Network policy, special
operations, cached queries, custom functions,
etc.
Framework Developer Maintained ACME Corp Maintained

41
Problems with Kustomize:
● YAML, YAML, and more YAML
○ Strings of Mustached YAML!
● Strategic Merge Patches for YAML
○ Patching parameters become brittle with Kustomize
● Configuration still has to be templated separately
● Mustache -> Kustomize flow can be confusing to work with
● Kustomize is designed as a CLI tool - KUDO uses a virtual filesystem
Futures - Framework Development SDK

42
● Configure, Unify, Execute
● Superset of JSON
● Heavily influenced by Borg Configuration Language (BCL)
● Data Constraints
● Basic Types
● Union types
● Disjunctions
CUE

DEMO ….

Get Involved !
https://kudo.dev/
https://groups.google.com/forum/#!forum/kudobuilder
#kudo http://slack.k8s.io/
https://github.com/kudobuilder/kudo
Community Meeting - weekly Thursdays 10am PT

Thank You for Listening !
Matt Jarvis
Twitter: @mattj-io
Email: mjarvis@mesosphere.com

OSDC 2019 | Introducing Kudo – Kubernetes Operators the easy way by Matt Jarvis

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to OSDC 2019 | Introducing Kudo – Kubernetes Operators the easy way by Matt Jarvis

Similar to OSDC 2019 | Introducing Kudo – Kubernetes Operators the easy way by Matt Jarvis (20)

Recently uploaded

Recently uploaded (20)

OSDC 2019 | Introducing Kudo – Kubernetes Operators the easy way by Matt Jarvis