"What is one of the most popular patterns of using Kafka & AWS? Sending topic data to S3. It’s the ultimate data sink for long term storage and analytics. Naturally then, it also makes a good fit for storing Kafka topic data. Since saving data in S3 is so cheap & popular, enabling it for Kafka data opens-up many use-cases: - To drive analytics - For Disaster Recovery (DR) of Kafka - To migrate Kafka on-prem to cloud - Develop safely in separate dev Kafka cluster - To save costs Can you do it now? Yes, but it’s painful. You can use a connector and battle with a poor developer experience. And it’s possibly very expensive if you use a non open-source connector. Can we do better? Enter AWS with Lenses. In this talk we will show you: - The latest feature-packed Lenses open-source S3 Source & Sink connectors - Patterns & best practices for how to backup/restore to S3 - How to have a seamless, one-click experience to backup and restore in Lenses 5.3"

1. ADAMOS LOIZOU
Product Manager
Lenses.io
MOSTAFA MANSOUR
Product Manager
Data Streaming AWS
Kafka & S3 with
Lenses.io Connectors
& AWS MSK Connect

2. About Lenses.io
Since 2018
Biggest contributor to open-source
Apache Kafka connectors
Leading Developer Experience for
any Apache Kafka

3. Integrating data with S3
Backup/Restore
How do I backup the Kafka
payload, headers &
metadata?
Hydrating Data Lake
How do I sink to S3 an efficient
manner for analytics?
Sourcing from S3
How do I source complex
data sources including XML
and multi-line events?
Event Driven Apps
How do I integrate data to my
downstream apps without
significant latency
Enterprise grade?
Cost-efficient?
Vendor locked-in?

4. The Solution
AWS MSK
+
Lenses S3 Connectors

5. Amazon MSK
✔ Fully managed, highly available
Apache Kafka service
✔ Automate provisioning, configuring,
and tuning
✔ Compatible with open source Apache
Kafka
✔ Lower Cost
Amazon MSK Serverless
✔ No worrying about
overprovisioning
✔ Instantly scale I/O without
worrying about overprovisioning
or reassigning partitions
✔ Pay for the data volume you
stream
✔ Fully managed Kafka Connect
clusters
✔ Compatible with Kafka Connect
✔ Scales automatically in response
to increases in usage
✔ Pay only for the resources you use
Amazon MSK Connect

6. New & improved Lenses S3
Source & Sink Connectors
INTRODUCING
Fully Open Source
Apache 2 License
Offered with
Enterprise Support
Available for MSK
Connect

7. Envelope
Source Format in Kafka (JSON) Target Format in S3 (Lenses Envelope)
{
"key": {
"value": 99836,
},
"value": {
"speed": 0,
"radio": 99836
},
"metadata": {
"offset": 14821,
"partition": 4,
"timestamp":
"2023-06-05T08:44:16.218Z"
}
}
{
"key": {“value”:999836},
"value": {"speed":0,"radio":99836},
"headers": {
"header1": "...",
"header2": "..."
},
"metadata": {
"timestamp": "023-06-05T08:44:16.218Z",
"topic": "mytopic",
"partition": "4",
"offset": "14821"
}
}
Stored as: AVRO, PARQUET, or JSON
Optimised serialisation of data in S3

8. Features
Packed with capabilities
SOURCE SINK
● Multiple input formats + unstructured
data
● Scalable
○ # tasks
○ # directory partitions
● Dynamic partition discovery
● Reliable
○ Safe resume
○ Exactly-once compatible
● Multiple formats
● Flexible partitioning
○ Kafka-native
○ Content-based
○ Time-based rolling
○ Flush by space + time
● Athena + Glue native
● Reliable
○ Crash recovery
○ Exactly-once compatible

9. Architecture
MSK-C
MSK-C
Use cases:
1. Backup/restore
2. DR (active/passive)
3. Analytics DataLake
4. AI
5. Non-kafka apps
6. Other AWS services

10. Connector developed in partnership
Backup/Restore of Kafka topics &
hydration for downstream analytics

11. Next steps

12. Thank You
Visit us at our booth
&
Win an iPhone 15 Pro Max