"The benefit of real-time data can be measured by how frequently the data in question changes, nowhere is this more apparent than threat detection. Responding to an ever changing landscape of attacks and exploits requires a system that can not only handle the scale and dynamic nature of the data but also a dynamically changing set of detection rules. We developed Confluent SIGMA, an open source project built on Kafka Streams for the open SIGMA DSL, to handle real-time rule additions and modifications. In this talk we will cover: * The architecture of our Kafka Streams layer that makes it possible to use external data feeds as rule input * How we handle dynamic criteria for joins and filters * Best practices for writing dynamic rule engines in Kafka Streams * Upcoming improvements to Kafka Streams to support versioned rules Although Confluent SIGMA focuses on cyber threat detection this same pattern can also be applied to any DSL (domain specific language) that would benefit from real-time stream processing. After attending you will have the framework to drive dynamic rules through Kafka Streams for any use case that might require it."

1. Building a Dynamic DSL Rules
Engine with Kafka Streams
Will LaForest
Field CTO
Michael Peacock
Field Engineer

2. 2
Why Something Other Than These
FRAMEWORKS
ksqlDB

3. Stream Processing Frameworks General Purpose
3
Stream processing very powerful
BUT
● Uses GPLs (Java, Python) or SQL
● 99.2% of workforce are NOT
coders
● Maybe SQL is easier
○ Still general purpose
○ Impractical for some
domains
99% 1%

4. • Volume of rules high? 100s or
1000s?
• If each rule is a stream
processing job
• Each a consumer and
producer
• Same network IO 1000x
• Process overhead
• Serialization/Deserialization
Technical Challenges
4
Cartoon style. Gnome sees something gross and vomits

5. Technical Challenges
5
• Multiple dynamic output
topics?
• Change rate of ruleset high?
• Minimize time to to
processing?
• Must be dynamic
• Don’t compile and deploy
• Don’t create a new stream
processor
• Don’t restart topology if
possible

6. Anyone Know what this is?
6
to spiral
make "n 1
while [:n < 100] [
make "n :n + 5
fd :n rt 90
]
end

7. Domain Specific Languages
Purpose built for a domain
● HTML web pages
● Regex for patterns
● MATLAB for numerical analysis
● GraphQL - Querying APIs
● Sigma for cyber and log data
If serialized as JSON, YAML, XML
● DSL as Data
● Apply data oriented operations
● Validate, secure, govern as data
● Can be transformed easily
● Generated from domain specific tool
● Dynamic

8. The Cybersecurity Domain

9. A Typical SIEM Platform Architecture
Agents /
Collectors
Reports and
alerts
Search and
investigate
Monitor and
analyze
SIEM
Tool
Network traffic
Firewall logs
RDBMS
Application logs
HTTP proxy logs

10. Forensic
Archive
AI/ML
Architecture with Kafka
Curate
Real-time Detection
APP SIEM
Index
SOAR
HDFS
S3
Big Query
Syslog
Network traffic
Firewall logs
RDBMS
Application logs
HTTP proxy logs
QRadar
Arcsight
Splunk
Machine Data
spooldir (files), SNMP Traps,
Databases, Sftp, MQs, S2S
Elastic

11. Confluent Sigma
https://github.com/confluentinc/confluent-sigma

12. What is Sigma?
12
https://github.com/SigmaHQ/sigma
● Sigma is an open signature (patterns) format
● Focused on log and network
● Researchers or analysts can describe developed
detection patterns
● Shareable with others (IMPORTANT)
● Rules schema agnostic
● Platform agnostic
○ Same rule can be be used for
○ Elastic, Splunk, Azure Sentinel, (now Kafka)

13. Confluent Sigma
13
Sigma Rules
(YAML)
Source
Data
Sigma Stream
Processor
SIEM
Applications
Cold storage
High Risk Detections
SOAR
Filtered
Cold retention
Data
Rules

14. Why Kafka Streams API?
14
● Simple to build a topology
● Easy to run as app
● No required runtime
● Start small easily
App 1
● Scale massively
● All you need is Kafka
● Can run anywhere!

15. A Walkthrough of the Sigma
Project

16. A Walkthrough of the Sigma Project
16
sigma-parser
sigma-streams
sigma-streams-ui
https://github.com/confluentinc/confluent-sigma

17. Sigma Stream Topologies
Simple Topology
supports one
sub-topology to
many rules
Aggregate Topology
requires one
sub-topology for
each rule

18. Simple Topology
18
flatMapValues KStream
iterate through each rule for the
processor (product/service
filtering)
validates the streaming data
against the DSL rule and add the
results to the output list if a
match is found
optional config variable to return
after the first match or return all
matches
dynamic output topic flatMapValues
Create a new KStream by transforming the value of
each record in this stream into zero or more
values with the same key in the new stream.
{
"@stream": "dns",
"@system": "bobs.bigwheel.local",
"@proc": "zeek",
"ts": 1588205199.82437,
"uid": "Cvf4XX17hSAgXDdGEd",
"id_orig_h": "10.0.1.6",
"id_orig_p": 54243,
"id_resp_h": "10.0.0.4",
"id_resp_p": 53,
"proto": "udp",
"trans_id": 41180,
"rtt": 0.001528024673461914,
"query": "newyork.dmevals.local",
"qclass": 1,
"qclass_name": "C_INTERNET",
"qtype": 1,
Sigma Rule
Streaming Data
1
2
3
4
5

19. Detection Results / Dynamic Output Topic
19
'^(?<timestamp>w{3}sd{2}sd{2}:d{2}:d{2})s(?< hostname>[^s]+)
s%ASA-d-(?< messageID>[^:]+):s(?< action>[^s]+)s(?< protocol>[^s]+)
ssrcsinside:(?< src>[0-9.]+)/(?< srcport>[0-9]+)sdstsoutside:(?< d
est>[0-9.]+)/(?< destport>[0-9]+)'
Sigma Rule
firewalls topic
adds the record data and rule
metadata
if the rule contains a regex with
mapping, adds the mapped
fields
adds any custom fields
dynamic output topic
1
2
3
4

20. Aggregate Topology
20
iterate through each rule for the
processor (product/service
filtering)
validates the streaming data
against the DSL rule
groups by the key
sliding window from rule
counts number of instances
within the window
filters based on aggregation and
operation in rule
dynamic output topic
{
"@stream": "dns",
"@system":
"bobs.bigwheel.local",
"@proc": "zeek",
"ts": 1588205199.82437,
"uid": "Cvf4XX17hSAgXDdGEd",
"id_orig_h": "10.0.1.6",
"id_orig_p": 54243,
"id_resp_h": "10.0.0.4",
"id_resp_p": 53,
"proto": "udp",
"trans_id": 41180,
"rtt": 0.001528024673461914,
"query":
"newyork.dmevals.local",
"qclass": 1,
"qclass_name": "C_INTERNET",
"qtype": 1,
Sigma Rule
Streaming Data
1
3
5
6
7
4
2

21. Confluent Sigma Demo
Sigma Stream Processors
Sigma Streams UI
Sigma Rule Editor
sigma rules topic
DNS
dns
detections
topic
dns topic
rule parsing,
filtering,
aggregation,
windowing
sigma
rules
cache
CONN
DHCP
HTTP
SSL
x509
Zeek Data

22. Demo

23. Considerations for DSL Stream Processing
● Consider how to handle newly arriving rules
○ Apply only to unprocessed messages
○ What about previous data?
■ Spawn separate app or topology
■ When “caught up” merge into single
● Versioned state stores now allow new strategy
○ Only join rules available for corresponding record time
○ For records only join against rules that existed at record time
■ Upcoming KIP-960, KIP-968, KIP-969 for our customized joins
23
https://www.confluent.io/blog/introducing-versioned-state-store-in-kafka-streams/

24. Considerations Continued
● Consider how routines may interact with each other
○ Sigma doesn’t support notion of priority or dependence
● For a DSL in which rules interact (IFTTT)
○ Use Processor API to map to an appropriate DAG
24

25. Whats Next?
“Roadmap” https://github.com/confluentinc/confluent-sigma/projects/1
● Refactoring framework to make DSL pluggable (on going)
○ Implement a builder class: DSL routines -> Kafka Streams topology
○ Re-use all the scaffolding we have built
○ LLM stream processing
● Optimizations
○ Aggregate topology optimizations (bin based on time and key)
○ More graceful management of aggregate topology changes
■ Self coordination
○ General profiling and optimization
● Switch to GlobalKTable from KCache 25