Get More from your Data: Accelerate Time-to-Value and Reduce TCO with Confluent Cloud on AWS

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Get more from your data
Accelerate time-to-value and reduce TCO
with Confluent Cloud on AWS
Weifan Liang
Sr. Partner Solutions Architect
Amazon Web Services (AWS)
Mike Owens
Global Advisory Architecture Leader
Confluent
Joseph Morais
Principal, Partner Solutions Engineer
Confluent

Agenda
2
q Data streaming demands
q Advanced capabilities of Confluent Cloud on AWS
q Benefits of Confluent Cloud on AWS
q Accelerate modernization journey
q Innovate together to power customer success
q Key takeaways

Data Streaming Demands
3

Common real-time analytics use cases
4
Anomaly and
fraud detection
Empowering
IoT analytics
Tailoring customer
experience in real time
Nourishing marketing
campaigns
Supporting healthcare
and emergency services
Real-time
personalization

Challenges of data streaming
Difficult to set up Tricky to scale
Hard to achieve high availability Integration requires development
Error prone and complex to manage Expensive to maintain

Advanced Capabilities
6

AWS offers capabilities to deliver end-to-end data journey
7
17+ years of data and ML
innovation and counting
A comprehensive set of data
and machine learning services
Investing in a zero-ETL future
so you can seamlessly
connect all your data

Modern Data Architecture on AWS
Non-
relational
databases
Machine
learning
Data
warehousing
Log
analytics
Big data
processing
Relational
databases
Data lake
Key pillars
• Data at any scale
• Purpose-built services
• Seamless data movement
• Unified governance
• Performant and cost effective

Sharing across data movement with Data Mesh
Data producers Data mesh Data consumers
Unique modern data architecture
Suited to business function
Teams that want to share data
Unique modern data architecture
Suited to business function
Team that runs the marketplace Teams that want to use data

Integrate all the things
CONFLUENT OUTCOMES ON AWS
Other Regions
Amazon
Redshift sink
Amazon
SageMaker
AWS
Fargate
Amazon
EMR
Amazon
QuickSight
Amazon
Kinesis

Instantly connect popular data sources & sinks
120+
prebuilt
connectors
100+ Confluent supported 20+ partner supported, Confluent verified

Kafka clients Kafka Streams ksqlDB
ConsumerRecords<String, String> records = consumer.poll(100);
Map<String, Integer> counts = new DefaultMap<String, Integer>();
for (ConsumerRecord<String, Integer> record : records) {
String key = record.key();
int c = counts.get(key)
c += record.value()
counts.put(key, c)
}
for (Map.Entry<String, Integer> entry : counts.entrySet()) {
int stateCount;
int attempts;
while (attempts++ < MAX_RETRIES) {
try {
stateCount = stateStore.getValue(entry.getKey())
stateStore.setValue(entry.getKey(), entry.getValue() +
stateCount)
break;
} catch (StateStoreException e) {
RetryUtils.backoff(attempts);
}
}
}
builder
.stream("input-stream",
Consumed.with(Serdes.String(), Serdes.String()))
.groupBy((key, value) -> value)
.count()
.toStream()
.to("counts", Produced.with(Serdes.String(), Serdes.Long()));
SELECT x, count(*) FROM stream GROUP BY x EMIT CHANGES;
Flexibility Simplicity
confluent.awsworkshop.io
3 modalities of stream processing with
Confluent

Expanding Confluent’s stream processing portfolio
Flink
Kafka Streams ksqlDB
Future
Available Now
Fully managed on Confluent Cloud
SQL at GA, Java and Python to follow
Self-managed on Confluent Cloud and Confluent
Platform
Java
Fully managed on Confluent Cloud, Self-
managed on Confluent Platform
SQL

Federated streaming,
hybrid and multi-cloud
Data syndication and replication across
and between clouds and on premises,
with self-service APIs, data governance,
and visual tooling
Reliable and real-time data streams
between all customer sites, so you can
run always-on streaming analytics on the
data of the entire enterprise, despite
regional or cloud provider outages
Global central nervous system
Cluster linking

Benefits of Confluent Cloud
on AWS
15

Benefits of fully-managed Confluent Cloud
16
●Escalating risk of costly downtime with scaling
●Resource redirection to tackle unexpected downtime
●Hidden costs of downtime including revenue loss, reputation damage, CSAT reduction, fines
and data loss
Reduce Unplanned Downtime
●Annual spending on subpar infrastructure
●Performance decline from manual configurations and updates
Migration Benefits
●Scaling production deployment takes years
●Challenges in hiring and retaining Apache Kafka talent
Accelerate Time-to-Value &
Ability to Scale
Decrease Total Cost of
Ownership
●Balancing infrastructure costs with business focus
●Opportunity cost of diverting FTEs from app development
Reallocate Resources &
Improve Productivity
Pain Points Alleviated

Accelerate Modernization Journey
17

Confluent Data In Motion Journey Maturity Model
18

Measure
Progress &
Maturity
The Data in Motion Blueprint
19
Data in Motion
Assessment
Drive Streaming
Adoption
Train & connect teams
across the business
Centers of
Excellence &
Teaming
Training &
Education
Building a
Community
Build Streaming
Capabilities
Implement, mature, and
scale use cases
Foundations - Early
Interest & POC
Early Adoption -
Use Case in
Production
Acceleration -
Multiple, Mission
Critical Uses
Strategic - Crossing
Lines of Business
Transformational -
Central Nervous
System
Governance & Security
Design Your
Approach
Create a vision & plan for
data streaming
Create a Vision for
Data Streaming
Data Streaming
Roadmap
Data Streaming
Use Case Portfolio
Business Value &
Prioritization

Activation
Workshop
Investment:
● 4-6 hours (over 2 - 3 days)
20
Engagement Outcomes
Executive Sponsorship
● Support, and direction of DIM strategy within the
organization
A collaborative effort to align Data In Motion (DIM) capability to most pressing
business objectives and enterprise data strategy
DIM Capability Exposure & Alignment
● Broad awareness and value of DIM created within
areas of the organization
● Focused capabilities aligned to business priority and
outcomes
Technical / Execution Considerations
● Ongoing initiatives, organizational commitments,
technical/operational blockers
DIM Adoption Roadmap
● Macro roadmap on key initiatives, prioritized actions,
and timeline mapping
Business Primer
working session
Executive + Champion
Art of the Possible
Working session(s)
Champion + Stakeholders
Considerations
working session(s)
Executive + Champion + Stakeholders
Roadmap
working session
1 2
3
4
5
Executive Readout
working session

21
Workstream 1: Design
your approach
Workstream 2: Build
streaming capabilities
Workstream 3: Drive
adoption
Confluent - Migration Readiness Accelerator
~ 6 Week Timeframe
• Analysis
• Migration Planning
• Move Groups
• Timelines
• Environment
• Security & Idam
• Data & Workload
Migration
• Needs Assessment
• Training Plan Dev
• Communications
• Migration Factory

Client Migration Patterns #1:
No Data Replication
Set up:
1) Service Accounts
2) ACLs/RBAC
3) Key pairs
4) Topics
5) Schemas

Migrating Clients with Cluster Linking
Set up:
1 . Service Accounts
2. ACLs/RBAC
3. Key pairs
promote

KStreams Applications

Challenge: Modernizing legacy systems for traditional banks to
enable them to innovate faster, deliver hyper-personalized customer
experiences, and compete with digital- native banks.
Solution: Deliver a cloud-native SaaS solution— powered by
Confluent Cloud’s real-time data streaming platform.
Results:
● Reduced costs with increased agility and faster time to market for
traditional banks
● Achieved better hyper-personalized experiences for banking
customers
● Delivered a resilient and highly available platform
● Enhanced enterprise-grade security
● Reduced TCO with simplified management
“Our mission is to make banking 10x better for banks, for customers,
and society. To do that, we need a cloud-native data streaming platform
that is also 10x more reliable, 10x more performant than Apache
Kafka.”
– Mark Holt, Chief Product and Engineering Officer at 10x Banking

Challenge: Escalating costs and burden of managing Kafka on their own,
while relying on Kafka services that lacked a complete platform for all their
streaming needs.
Solution: Used a combination of Confluent Cloud and Confluent Platform
to build streaming data pipelines and scale faster, govern data better, and
ultimately lower TCO by offloading the management of their data
streaming infrastructure to Confluent.
Results:
● $1M+ in TCO savings for their data streaming infra compared to
managing in-house
● The ability to seamlessly scale data 10 to 100x, a game changer for
the company
“It’s amazing how much more we can get done when we don’t have to
worry about exactly how to do things. We can trust Confluent to offer a
secure and rock-solid Kafka platform with a myriad of value-add
capabilities like security, connectors, and stream governance on top.”
– Jared Smith, Senior Director, Threat Intelligence, at SecurityScorecard

Jumpstart: Confluent Cloud on AWS
27
Request an
Activation
Workshop
Attend a
Technical
Deep Dive
(Immersion Days
and Pilot/POC)
Leverage AWS
Migration
Programs

Migrate and Modernize to fully realize the benefits
Agility and
Innovation
Total cost of
operating
Migrate Move to Managed Cloud Native
On-Prem

Innovate together to power
customer success
29

Out-of-box integration
with popular services
Certified and validated by AWS
AWS Native Services
Top-5 Global ISV for Amazon S3 Data Volume
3rd-Party ISV Services
Amazon RDS Ready
AWS Lambda Ready
Amazon Redshift Ready
AWS PrivateLink Ready
AWS Outposts Ready
Validated Service Designations
Confluent integrations with AWS
30

Confluent on AWS reference architecture to accelerate modernization
31
AWS Direct
Connect
Legacy applications
Mainframe
Legacy data systems
JDBC / CDC
connectors
On-premises AWS Cloud
Amazon Athena
AWS Glue
Amazon
SageMaker
AWS Lake
Formation
Amazon
DynamoDB
Amazon
Aurora
Data Streams
Apps
ksqlDB
Connect
Leverage 120+ Confluent
pre-built connectors
Bridge
Hybrid cloud streaming
Modernize
Value-added apps, increase
agility, reduce TCO
Cluster Linking
Amazon
Redshift Sink
AWS
Lambda Sink
Amazon
S3 Sink
Amazon
Redshift
AWS
Lambda
Amazon
S3

Robust data streaming pipelines for real-time intelligence
32

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 33
Unlocking the
potential of
generative AI
The easiest way to build with FMs
The most price-performant infrastructure
Flexibility to build with your own FMs
Generative AI-powered applications

Next steps
35
Try Confluent Cloud on AWS
for free
Learn more about activation
workshops

Learn more
Modern Data Architecture on AWS
https://go.aws/3OJDhFk
Fuel innovation with data and AI
https://aws.amazon.com/data
Set your data in motion and connect your data to AWS
https://www.confluent.io/partner/amazon-web-services/
Accelerate your cloud migration and modernization journey
with an outcome-driven methodology
https://aws.amazon.com/migration-acceleration-program/
Confluent PS Activation Workshop
https://tinyurl.com/4jf3bjpt

© 2023, Amazon Web Services, Inc. or its affiliates.
Thank you!
Weifan Liang
weifanl@amazon.com
Mike Owens
mowens@confluent.io

Kafka Cluster Migration Options
51
Cluster Linking
Replicator
MirrorMaker2
Custom solution

Schema Registry Migration Options
Replicator Schema Translation
• Confluent Cloud can only be
the destination
• SR must be in IMPORT mode
Schema Linking
• Requires SR version 7.1
source (preview with 7.0)
• If destination context is empty,
exporter sets context to
IMPORT mode
• Non-empty context can be set
to import mode with PUT
/mode?force=true, but watch
out for ID collisions
Confluent Cloud-schema-
exporter (REST API)
• Uses REST API, destination SR
must be empty and in IMPORT
mode
• Best solution for migration
from a non-Confluent SR (ex.
Apicurio)
- For REST API, IMPORT mode
can be subject level
52

Other Components Migration Consideration
53
q Migrating to managed connectors
q Migrating to managed KsqlDB
q Repointing self-managed components
q Others

Get More from your Data: Accelerate Time-to-Value and Reduce TCO with Confluent Cloud on AWS

Recommended

Recommended

More Related Content

Similar to Get More from your Data: Accelerate Time-to-Value and Reduce TCO with Confluent Cloud on AWS

Similar to Get More from your Data: Accelerate Time-to-Value and Reduce TCO with Confluent Cloud on AWS (20)

More from HostedbyConfluent

More from HostedbyConfluent (20)

Recently uploaded

Recently uploaded (20)

Get More from your Data: Accelerate Time-to-Value and Reduce TCO with Confluent Cloud on AWS