Originally presented by Jonathan Schabowsky at Kafka Summit 2020. ** About PubSub+ Event Portal for Apache Kafka ** You know and love Apache Kafka, but have you ever tried to visualize Kafka topology, or figure out who owns what event stream in a Kafka cluster? Your event-driven architecture has evolved, and your system has grown to the point where you’re feeling a bit… out of control. You need a tool to discover your Kafka event streams, represent it in a graphical view, and make it easy to share and reuse events. Basically, you need an API portal, but for asynchronous, event-driven applications. That is why we have developed PubSub+ Event Portal. This event management toolset makes it easy for you to discover, visualize, catalog and share your Apache Kafka event streams, including those from Confluent and Amazon MSK. To learn more, visit: https://solace.com/products/portal/kafka/

1. How to Discover, Visualize,
Catalog, Share and Reuse
your Kafka Event Streams
Jonathan Schabowsky
Senior Architect, CTO, Solace

2. Agenda
2
• Your growing Kafka event streams
• Flying blind
• Event portal: an API portal for event streams
• Demo: How to discover, visualize, catalog, share and reuse
your Kafka event streams
• Summary
© Solace
2

3. Does Your Organization Look Like this?
3
© Solace
Data Stream Flow
Legacy Application Infrastructure
….
?
?
?
JSON, XML, Plain Text…
IoT/Device Management
Producers
Consumers
Connectors
Stream
Processors
Avro

4. You’re Flying Blind…
Where do you discover
event streams for reuse?
Why does a given
event stream exist?
What topic do you use
to subscribe to it?
What changed?
How do you determine the
data structure of its payload?
Who should have access to it?
Who made the
latest revision?
Who can tell you more
about an event?
Is your change backward
compatible?
Who is impacted
by your change?
Does it comply with
security policy?
4
© Solace

5. Some Try Solutions Not Intended
for Event Stream Management
• It's a pain…
– No single place to discover
and share/reuse event streams
– No best practices guidance
• You’re breaking systems and don’t
know who changed what???
– Suffer from changes with unintended
consequences
– Hard to audit changes when data is stored in
different tools
Brittle,
complex,
manual
solutions
Custom
Developed
Catalog
PowerPoint
API
Management
Spread
sheets
GitConfluence
5
© Solace

6. API Management Solved
this for REST
6
© Solace

7. They answer Who,
What, When, Where,
Why and How For
RESTful APIs…
but what about
event streams?
Inspired by API Management Platforms
7
© Solace
API
Gateway
Documentation
• Discover
Registration
• Govern
Analysis
• Improve
• Monetize
Community
• Collaborate
Architects and
Developers
(Internal, Partner
and Third-Party)
Productivity
• Generate Code
API
Portal

8. Sinks
“While most organizations have basic
event processing infrastructure - such as
notification services, message buses,
and event brokers - many don’t have the
high-level productivity tooling that helps
developers design, develop, test and
manage event-centric applications. Nor
do they have tools for governing,
publishing and managing event-based
interfaces. The market for these tools is
sparse.”
Gartner “Top 3 Trends in Application Architecture That Enable
Digital Business” Anne Thomas, Yefim Natis, Mark O’Neill, 28 Oct 2019
8
© Solace
Event
Broker
Security
Persistence
Protocol
Translation
One-to-Many
Distribution
Source
Runtime
We Need an Event Portal
for Asynchronous Interactions
Design, Develop
and Manage
Event
Portal
Documentation
• Discover
Registration
• Govern
Analysis
• Improve
• Monetize
Community
• Collaborate
Productivity
• Generate Code (code gen)
(spec)

9. 9
© Solace
Single place to design, create, catalog, visualize, discover, share, secure
and manage all events within your ecosystem
Introducing Event Portal
Event Portal
Mainframe/
SAP
Distributed
Apps IoTMicro
services
Server
Less
Event Brokers

10. Event Portal
How Users Interact with Event Portal
10
Event
Management
(Design Time)
© Solace
Your Event-Driven
Enterprise
B2B/B2C
Ecosystem
IoTMicro-
services
Server-
less
Generate
Code
REST
API
Existing Tools
• Schema registries
• Data governance
• CI/CD pipelines
Customers
& Partners
• Discover and
consume events
in real-time
Architects
• Discover, create, update, review
apps, events and payload
schemas
• Map payload schemas to events;
and events to apps
• Govern event consumption
across app domain boundaries
Developers
• Collaborate with
architects on design
• Export AsyncAPI app
contracts for each app
• Generate and implement
event-driven apps that
provide business value
Data Scientists
• Understand
event-driven data
• Discover new insights
by combining events

11. Demo
11
© Solace

12. Benefits of an Event Portal
12
Solace
Your business can…
• Understand who is consuming which
events, how much, for what
• Create value-added services and
gain insights by combining events
• Make it easy for users across app
teams, LoBs and partner ecosystem
to find and reuse events
• Monetize popular or particular
high-value event streams
Reuse/Sharing
Architects & developers can:
• Collaborate on events
and event-driven apps
• Visually see event-driven
interactions to understand
impact of changes
• Low Code: Generate app code by
exporting AsyncAPI definitions
• Benefit from best practices &
consistently apply conventions
Productivity
CDO/data governance can:
• Understand where data is
coming from (lineage) and going
• Track changes and audit for
deviations while promoting apps
and dependencies though Dev,
Test, QA, and Prod
• Use application/event
relationships to create security
policies and validate compliance
with schema
Control

13. So, if you…
Have a lot of event-
driven applications
13
© Solace
You need an event portal!
Have (or need
to define) a lot of event
streams
Want to
manage events
as easily as you
manage your APIs

14. Event Portal = API Portal for Events/Streams
One tool to design,
create, catalog,
visualize, discover,
share, secure and
manage all of your
events and streams
(including Kafka and any
other event broker)
14
© Solace
Sign up for free!
• About data
and apps, NOT
managing Kafka
infrastructure
• High-level
productivity
tooling to help
you fully
embrace EDA
• Governance
of apps, event
and data