DevConf.cz - Introduction to Kubernetes Operators for Databases

1. Introduction to Kubernetes
Operators for Databases
+ Oracle Database Containers and Kubernetes Operator
Oracle
Juarez Barbosa Junior
Sr. Principal Developer Evangelist

2. About me
• Juarez Barbosa Junior - @juarezjunior
• Senior Principal Java Developer Evangelist
• Over 20 years of experience in IT
• SW Engineering, Developer Relations
• Microsoft, Oracle, IBM, Nokia, Unisys, Accenture, and a
few startups
• Microsoft Azure Developer Relations Lead
• IBM Watson Tech Evangelist & Cloud Rockstar
• IBM Mobile Tech Evangelist & Global Thought Leader
• Nokia Developers Global Champion
• Lead Software/DevOps Architect
• Expertise
• Java, Cloud, DevOps, Cloud-native, Blockchain
Copyright © 2023, Oracle and/or its affiliates

3. Copyright © 2023, Oracle and/or its affiliates
Cloud Native Computing: Kubernetes at the Center
Stage
Kubernetes
• Management of containerized applications, components and services
• Declarative platform definition: YAML
• Non-terminating Control loops
• Service discovery and load balancing
• State Maintenance
• Self-healing & automation robustness
• Storage orchestration
• Automated rollouts and rollbacks
• Secret and configuration management
Observe
Analyze
Act

4. 4
Kubernetes Popularity
Source: https://www.statista.com/statistics/1233945/kubernetes-adoption-level-organization/ Source: https://www.datadoghq.com/container-report/
Copyright © 2023 Oracle and/or its affiliates

5. 5
Kubernetes deployment challenges to address
Kubernetes alternatives
1. RedHat Openshift
2. Rancher
3. Docker Swarm
4. OKE
Source: https://www.redhat.com/en/resources/kubernetes-adoption-security-market-trends-overview
Copyright © 2023 Oracle and/or its affiliates

6. 6
Container images running in Kubernetes
Copyright © 2023 Oracle and/or its affiliates

7. Oracle’s Strategy
for Cloud Native
Application
Development
7
Copyright © 2023 Oracle and/or its affiliates

8. Governance
OKE Cluster
Data Plane
Worker nodes
Container images
Kubernetes Persistent Volume Claims
Identity and Access
Networking Encryption
Seamless integration with other OCI services
8 Copyright © 2023, Oracle and/or its affiliates
Bare Metal
Compute
Virtual
Machine
OCI VCN OCI VCN Security
Lists/Groups
OCI Container
Registry
OCI VCN
Route Table Auditing
Logging
OCI IAM Groups
OCI Key Vault
Monitoring
OCI File Storage
Container Engine
For Kubernetes
Kubernetes Services type Load Balancer
OCI Block
Storage
OCI IAM Policies
3rd party identity system
OCI Load
Balancer
Application users
federation
Control Plane
Kubernetes API Server
etcd
Node
kubelet
kube-proxy
OS
• X86
• Flex
• Arm
• GPU
OCI Vulnerability
Scanning
CI/CD and Automation
OCI DevOps
OCI Bastion
Resource Manager
(Terraform)
controller
scheduler CCM

9. OCI container-based platform for app development
SECURITY & GOVERNANCE (Secure)
OBSERVABILITY & MANAGEMENT (Operate)
Bastions Certificates Vault VCN NSG Max Security Zones
Identity & Access
Mgmt.
Cloud Guard Data Guard
Logging Log Analytics Monitoring APM Java Mgmt. OS Mgmt. Operations Insights Cloud Advisor
3. Application Runtime (Run)
STORAGE
Block
Storage
Object
Storage
Caching
APPLICATION RUNTIMES
Functions
Container Engine for
Kubernetes (OKE)
APEX
Containerized Runtimes Low Code
TRAFFIC MANAGEMENT
API Gateway
Load
Balancer
1. Languages & Frameworks (Develop)
2. DevOps (Build & Deploy)
DATA & ANALYTICS
ATP ADW
Oracle Analytics
Cloud
Argo
Resource Manager
GitHub
DevOps
Service Mesh
9 Copyright © 2023, Oracle and/or its affiliates | Oracle Confidential: Highly Restricted

10. Container Runtimes for Oracle Database
10
Docker
Running Containers on Docker
Engine
Build Images (SI, Sharding, RAC &
CMAN)
Standalone Environments
OL7 Support only
Podman
Alternative to Docker
Secure than Docker
Build Images (SI, Sharding, RAC &
CMAN)
Standalone Environments
OL8 Support
Kubernetes
Container Orchestration solution
Uses Software Defined
Networking
Provide Observability and Scaling
SI & Sharding
DB Operator
(OraOperator)
DB Specific Controller
Software extension of the
Kubernetes APIs
ADB, BaseDB, SIDB, Sharding &
Multitenant
Database Observability
Ansible
Go
Helm
Podman Compose
Docker Compose
Configuration scripts
Provisioning scripts
Copyright © 2023 Oracle and/or its affiliates

11. Make Oracle Database Container-native and Kubernetes-native - Kubernetes is popular
• Modern app dev deploys application in containers for rapid CI/CD pipelines, scale, resilience
• Deploy database in containers for the same advantages
Container-native
• Deploy Oracle Database in container runtime environments (docker, podman, Kubernetes)
• Easy deployment using pre-packaged scripts in the image available for each environment
Kubernetes-native
• Database Operator for Kubernetes (OraOperator) simplifies and automate the
management of Oracle Database lifecycle using Kubernetes commands - kubectl
• Oracle Database running in Kubernetes cluster (containers) or standalone configs
11 Copyright © 2023 Oracle and/or its affiliates
Oracle Database App Simple Mission

12. Container images Docker (OL7) Podman (OL8) Oracle Kubernetes Engine (OKE) and
Oracle Cloud Native Environment(OCNE)
Single Instance DB (Includes XE, SE,
and EE)
19.16c, 21.3c 19.3c, 21.3c 19.3c, 21.3c
Sharded DB 19.16c, 21.3c 19.3c, 21.3c 19.3c, 21.3c
Real Application Clusters (RAC) 19.16c, 21.3c 19.16c, 21.3c OCNE Planned for 23c GA
Connection Manager (CMAN) 19.3c, 21.3c 19.3c, 21.3c
Oracle Container Based Runtime Environments
Production Database Release
Copyright © 2023 Oracle and/or its affiliates

13. Oracle Database is embracing the use of Containers and Kubernetes
• Supported container images
• Helm chart for configuring these images in Kubernetes
• Operator to allow Kubernetes to manage Oracle databases (in-container or external)
• Adapters for Kubernetes open-source observability and tracing frameworks
All of these are open-source (available on GitHub)
Kubernetes-native helps build:
MicroTx: Support high-end OLTP with microservices
Spring Boot Backend: Manage database within Kubernetes
Full Support for Kubernetes
13
Copyright © 2023 Oracle and/or its affiliates

14. Oracle Database
Operator for
Kubernetes:
Architecture
14
Copyright © 2023 Oracle and/or its affiliates

15. Kubernetes Built-in Objects and Controllers
• Deployment
• StatefulSet
• Endpoint
• Service
• ReplicaSet
• Job
• CronJob
• DaemonSet
• Node
• Route
• Authorization
• Service Account
• Tokens
• …
Declarative Spec of
Deployments,
Pods, Services,
Volumes
etc
YAML
Controller monitors objects
and reconciles against Specs
Deployments, Pods
Copyright © 2023, Oracle and/or its affiliates

16. Master Node
Kubernetes Cluster
Kubernetes APIs Extended with Custom Objects and
Controllers
API
Server
Operator
(Extends K8s API)
Kubectl
Custom Resources
Specifications
YAML
Custom
Resources
Oracle CloudWorld Copyright © 2022, Oracle and/or its affiliates | Confidential: Restricted

17. Copyright © 2022, Oracle and/or its affiliates
17
Pod Specification
volumeMounts
• name
• mountPath
volumes
• name
• PVC reference
PVC Specification
Storage Class
Access Mode (rw/ro)
Size File Storage Block Storage
Physical Volume
allocate
Kubernetes: Managing States

18. Why a Kubernetes Operator for Stateful Applications?
• Stateful Application
• Replicas have own/unique state and identity
• Configuration steps before deployment
• Be in some special state for patching/upgrading
• Complexity and order of special maintenance tasks
(backup, restore, compression, and so on)
• More handholding through the entire lifecycle
• Helm charts are good for deploying stateless apps, templating/reuse; can’t
handle the lifecycle of stateful applications
• In-house scripts cannot fully automate stateful applications lifecycle
• State management in Kubernetes is complex
Oracle CloudWorld Copyright © 2022, Oracle and/or its affiliates | Confidential: Restricted

19. Oracle Database Operator for Kubernetes
Make the Oracle Database Kubernetes-Native
Oracle DB Operator
Kubernetes Cluster
kubectl
Controller Controller Controller Controller
Containerized Oracle DBs
Autonomous DBs
On-Premises DBs
Copyright © 2023, Oracle and/or its affiliates
19
1
3
2 4
Base DBs
Oracle Cloud Infrastructure
Supported Database Configurations
1. Containerized Database in Kubernetes
2. On-premises Databases
3. Autonomous Database
4. Base DBs
Lifecycle Operations
• Provision / Bind / Start / Stop / Terminate
• Backup / Restore / Patch / Upgrade
• Scale
Observability
• Monitoring, Logging, Metrics
Support major K8s Distributions and Clouds
kubectl apply -f oracle-database-operator.yaml

20. Why a Kubernetes Operator for the Oracle Database?
1. The Oracle database images as supported on Containers (Docker, Podman) and
deployable on Kubernetes (using Helm Chart and/or scripts)
2. We want to go further and make Oracle Database natively part of the Kubernetes
infrastructure
• Start, stop, monitor, patch, upgrade, and other life-cycle management operations from Kubernetes
3. Databases are Stateful Kubernetes Applications; Helm Charts and Kubernetes
Deployments are not appropriate for managing states
4. Provide these capabilities whether the Oracle database is in the cloud, on-premises, in
Kubernetes, or even outside of Kubernetes
Oracle CloudWorld Copyright © 2022, Oracle and/or its affiliates | Confidential: Restricted

21. OraOperator: Why Should You Care?
• Kubernetes is primordial for your DevOps & GitOps CI/CD pipeline
• You have adopted the Kubernetes operator pattern for automating the lifecycle
management of complex and/or stateful applications e.g., databases
• You are using or plan to use Oracle databases in your DevOps
• OraOperator eliminates the dependency on a human operator or
administrator for the majority of Oracle database operations.
• Oracle is resolute in making the Oracle database Kubernetes-native
• The Oracle database Operator for Kubernetes is open-source
Oracle CloudWorld Copyright © 2022, Oracle and/or its affiliates | Confidential: Restricted

22. Controllers Features / Lifecycle Operations
In-K8s-cluster
Single Instance DB: SE,
EE, XE
Provision DB Clone DB Patch DB
(in-place/
out-of-place)
Update DB Init
Params
Update DB
config
(Flashback
archiving)
Sharded DB Deploy Shard
Topology
Add a
Shard
Delete a
Shard
Clone DB
Shard
Off-K8s-cluster
Base DB Provision Bind Scale shape
Up/Down
Liveness
Probe
On Demand
backup
Autonomous DB
(Shared & Dedicated infra
Provision ADB Bind ADB Start/ Stop
ADB
Terminate
(Soft/Hard)
Scale
(Up/Down)
Network
access
options
On-demand
Backup/Restore
Autonomous Container
Database
Provision Bind, Restart Terminate
MultiTenant
(Exadata, PDB/CDB)
Bind to a CDB Create a
PDB
Plug a PDB Unplug a PDB Delete a
PDB
Clone a
PDB
Open/Close PDB
22
OraOperator v0.2.1 – Developer Preview
Copyright © 2023 Oracle and/or its affiliates

23. Controllers Features / Lifecycle Operations
In-K8s-cluster
Single Instance DB: SE,
EE, XE, 23c Free
Provision DB Clone DB Patch DB
(in-place/
out-of-place)
Update DB Init
Params
Update DB
config
(Flashback
archiving)
Data Guard
[v1.1.0]
Sharded DB Deploy Shard
Topology
Add a
Shard
Delete a
Shard
Clone DB
Shard
Observability Log Metrics Trace Grafana
Dashboard
Off-K8s-cluster
Base DB Provision Bind Scale shape
Up/Down
Liveness
Probe
On Demand
backup
Autonomous DB
(Shared & Dedicated infra
Provision ADB Bind ADB Start/ Stop
ADB
Terminate
(Soft/Hard)
Scale
(Up/Down)
Network
access
options
On-demand
Backup/Restore
Autonomous Container
Database
Provision Bind Restart Terminate
MultiTenant
(Exadata, PDB/CDB)
Bind to a CDB Create a
PDB
Plug a PDB Unplug a PDB Delete a
PDB
Clone a
PDB
Open/Close PDB
23
OraOperator v1.0.0 – Production (coming soon)
Copyright © 2023 Oracle and/or its affiliates

24. OraOperator v1.0.0 - (Production)
• SIDB, Sharded DB, RAC, Base DB, ADB, ACD, Multitenant
• Add-on to OKE Console deployment
OraOperator Roadmap
• RAC Controller with DB 23c GA
• True Cache & CMAN Controller
• VMWare Tanzu certification [on-going]
• Verrazzano integration
• Operator image on GitHub Container Image Registry
• Publishing on operatorhub.io
• Test on other major Kubernetes distributions as well as
clouds.
• OLM integration
24
Containers (DB 19c – Production)
• SIDB, Sharded DB – Docker, Podman, OKE, OCNE
• RAC, CMAN – Docker, Podman
Containers (DB 23c Beta – Dev. Preview)
• SIDB and Sharding [Podman]
• RAC [Podman & OCNE]
• True Cache [Podman]
Roadmap
Containers (DB23c GA – Production)
• RAC on OCNE & Podman
• Sharding [Podman & OKE]
• CMAN [Podman & OKE]
• True Cache [Podman & OKE]
Oracle Database Container + OraOperator Roadmap
Copyright © 2023 Oracle and/or its affiliates

25. Certified and Supported by Red Hat
Copyright © 2023, Oracle and/or its affiliates 25
`
Oracle DB Operator certified on OpenShift
Joint effort with Oracle Product
Management and Product Development
Operator managed from OpenShift
registry and console
All database flavours
Autonomous Exadata
On-Premise PaaS
https://catalog.redhat.com/software/operators/detail/630dd7ead2868f23242cefea
Copyright © 2023 Oracle and/or its affiliates

32. 32
OraOperator Add-on to OKE Console deployment
Copyright © 2023 Oracle and/or its affiliates

33. Demo - steps
33
Copyright © 2023 Oracle and/or its affiliates

34. Use Cases shown
• Binding to an existing ADB
• Provisioning an ADB
• Scaling up a database
• Stopping a database
• Terminating a database
Managing the Autonomous Database (ADB) Lifecycle
Copyright © 2023, Oracle and/or its affiliates
34
`

35. Bind to an existing ADB
Copyright © 2023, Oracle and/or its affiliates
35
kubectl client
machine
Prepare .yaml file for
binding to an ADB
Login cloud console and click
Autonomous Transaction Processing
Click on the database name Copy database OCID
ADB bound
Use the OCID from above
Apply the .yaml file
kubectl apply –f bind_adb.yaml
Verify the resource is available using kubectl
kubectl get adb

36. Scale up an ADB
Copyright © 2023, Oracle and/or its affiliates
36
kubectl client
machine
Scale request sent
Set CPU count in the
earlier .yaml file for
scaling up/down ADB
Scaling in progress
Apply the .yaml file
kubectl apply –f bind_adb.yaml
Verify CPU count of the
database using oci cli
oci db autonomous-database list
--compartment-id $COMPARTMENT_ID
--output table
--query "data [*].{dbname:"display-name", ocpus:"cpu-core-count"}”

37. Stop an ADB
Copyright © 2023, Oracle and/or its affiliates
37
kubectl client
machine
Stop request sent
Set lifecycle state in
the earlier .yaml file for
stopping ADB
Stopping in progress
Apply the .yaml file
kubectl apply –f bind_adb.yaml
To start/terminate the database, repeat the
steps above with lifecycle state
AVAILABLE/TERMINATED
Verify state of the database
using oci cli
oci db autonomous-database list
--compartment-id $COMPARTMENT_ID
--output table
--query "data [*].{dbname:"display-name", state:"lifecycle-state"}”

38. Terminate an ADB using HardLink
Copyright © 2023, Oracle and/or its affiliates
38
kubectl client
machine
Terminate
request sent
Terminating in progress
Delete ADB from the
Kubernetes cluster
kubectl delete –f bind_adb.yaml
Set hard link to true
in the earlier .yaml
file for terminating
ADB
ADB
configured
Apply the .yaml file
kubectl apply –f bind_adb.yaml
Verify state of the database
using oci cli
oci db autonomous-database list
--compartment-id $COMPARTMENT_ID
--output table
--query "data [*].{dbname:"display-name", state:"lifecycle-state"}”

39. Cloud Native Operator Observability
ANY Oracle
Database
MetricsExport
Controller
&
LogExporter Controller
MetricsExporter
ConfigMap, Deployment,
Service, and ServiceMonitor
LogExporter
ConfigMap, Deployment
Grafana, Prometheus,
Loki, Promtail
• MetricsExporter Controller generates all resources needed to export DB metrics in Prometheus for visualization in Grafana
• LogExporter Controller generates all resources needed to export DB logs in Loki for visualization in Grafana
• Custom Resource Definitions simply reference secrets/bindings.
generates
generates
metrics scraped
logs pushed
Copyright © 2023 Oracle and/or its affiliates

40. Copyright © 2021, Oracle and/or its affiliates
40
Observability with EM DB Express console

41. Containers:
Can we run Oracle Database in Docker Containers? Yes, with 19c, and 23c Beta
Can we run Oracle Database in Podman Containers? Yes, with 19c and 23c Beta
Is Oracle Database supported in production use with Kubernetes containers? Yes, various configurations
• https://www.oracle.com/database/kubernetes-for-container-database/
Kubernetes Operator:
https://github.com/oracle/oracle-database-operator
Can we manage Oracle Database using Kubernetes? Yes, OraOperator developer preview available; production coming soon (end of May)
Licensing:
What are the licensing guidelines/restrictions when running Oracle Database in Docker/Podman containers?
None, same pricing per OCPU in containers as in standalone database configurations
Support:
What are support guidelines/restrictions when running Oracle Database in Docker/Podman containers?
Oracle Database using the underlying hardware configuration needs to be supported by Oracle
41
FAQ & References | Production releases 19c (and 23c Beta)
Copyright © 2023 Oracle and/or its affiliates

42. 3 membership tiers
Connect: @oracleace facebook.com/OracleACEs
aceprogram_ww@oracle.com
500+ technical experts &
community leaders helping peers globally
The Oracle ACE Program recognizes & rewards individuals for
their technical & community contributions to the Oracle community
Nominate
yourself or a candidate:
ace.oracle.com/nominate
Learn more - ace.oracle.com
blogs.oracle.com/ace

43. Oracle LiveLabs
Showcasing how Oracle’s solutions can
solve your business problems
500+
free workshops,
available or in
development
3.5 million
people have already visited
LiveLabs
developer.oracle.com/livelabs
learn something new …at your pace!
600+
events run
using LiveLabs
workshops
Copyright © 2023 Oracle and/or its affiliates

44. Oracle
Database
Oracle Database 23c Free – Developer Edition
available for download now
• Oracle Database 23c accelerates Oracle’s
mission to make it simple to develop and
run all data-driven apps
• Provides developers easy early access to
23c app dev features
• Limited database size, memory, CPU
threads (same limits as Express Edition)
Free Developer Release
http://oracle.com/23cFree
http://oracle.com/23cFree

45. Create your FREE
Cloud Account
• Go to
https://signup.cloud.oracle.com/
Copyright © 2023, Oracle and/or its affiliates

46. Thank you!
46
Copyright © 2023 Oracle and/or its affiliates

47. Provision an ADB
Copyright © 2023, Oracle and/or its affiliates
47
kubectl client
machine
Click on the compartment name
2. Copy compartment OCID
1. Login cloud console and click Compartments
Use the compartment OCID and the name
of the Secret
4. Prepare .yaml file for
provisioning an ADB
3. Create a K8s Secret which
holds the password
kubectl create secret generic
admin-password
--from-literal=adb-admin-password=<password-here> *
*For demo purpose only. Passing passwords on the
command-line is not recommended.
Secret
admin-password
created
(Assume there is a valid Container Database)

48. Provision an ADB (cont’d)
Copyright © 2023, Oracle and/or its affiliates
48
5. Apply the .yaml file
kubectl apply –f create_adb.yaml
Provisioning
request sent
Wait for a few seconds while
provisioning is in progress
6. Verify the state of the
new database using oci cli
oci db autonomous-database list
--compartment-id $COMPARTMENT_ID
--output table
--query "data [*].{dbname:"display-name",state:"lifecycle-state"}”

49. Use Cases Shown
• Database Docker image
• Provision or Clone Single Instance DB in K8s
• Observability with EM Express
• Patch the DB (in-place / out-of-place)
• Update DB initialization params
• Update DB config (Flashback, archiving)
Managing the Lifecycle of Single Instance DB
Copyright © 2021, Oracle and/or its affiliates
49
`

50. Copyright © 2021, Oracle and/or its affiliates
50
Build database image from GitHub.com/oracle

51. Copyright © 2021, Oracle and/or its affiliates
51
Use pre-built database image from
container-registry.oracle.com

52. Copyright © 2021, Oracle and/or its affiliates
52
Provision/Clone Single Instance DB in K8s
using singleinstancedatabase.yaml

53. Copyright © 2021, Oracle and/or its affiliates
53
Edit and apply
singleinstancedatabase_prov.yaml /
singleinstancedatabase_clone.yaml
from our yaml templates
Wait for a
healthy DB status

54. 54 Copyright © 2021, Oracle and/or its affiliates
Verify connectivity from clients
Login to OEM

55. Copyright © 2021, Oracle and/or its affiliates
55
Observability with EM Express
Basic Observability with EM Express console

56. Copyright © 2021, Oracle and/or its affiliates
56
Build a patched image using patching extension
from github.com/oracle

57. 57 Copyright © 2021, Oracle and/or its affiliates
Ensure DB is healthy
before patching
Update DB CR with
patched image
DB goes into
Patching mode
Wait for healthy DB
status post patching,
verify patched version