1. Angelo CORSARO, Ph.D.
Chief Technology Officer
OMG DDS Sig Co-Chair
PrismTech
angelo.corsaro@prismtech.com!
海
賊
天
使

2. Defining "
Stream Processing

3. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Stream Processing
¨ Stream Processing can be seen as an architectural style
for building systems that operate over continuous
(theoretically infinite) streams of data
¨ Stream Processing is often reified under one of its many
declinations, such as:
¨ Reactive Systems
¨ Signal Processing Systems
¨ Functional Stream Programming
¨ Data Flow Systems
[1/3]

4. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Stream Processing
¨ Stream Processing Architectures are very natural for
modeling systems needing to react to streams of
data produced by the external world, such as the
data produced by sensors, a camera or even the
data produced by the stock exchange.
¨ Stream Processing Systems usually operate in real-
time over streams and generate in turns other
streams of data providing information on what is
happening or suggesting actions to perform, such
as by stock X, raise alarm Y, or detected spatial
violation, etc.
[2/3]

5. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Stream Processing
¨ Stream Processing Systems are
typically modeled as collection of
modules communicating via
typed data channels
¨ Modules usually play one of the
following roles:
¨ Sources: Injecting data into the System
¨ Filters/Actors: Performing some
computation over sources
¨ Sinks: Consuming the data produced
by the system
[3/3]
Filter
Filter
Filter
source
Sink
Channel

6. Implementing"
Stream Processing

7. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Stream Processing Alternatives
¨ Different possibilities exist to
implement channels as well s
filters
¨ DDS is a very good fit for
channels due to its strong typing
as well as performance and
scalability
¨ Filter can be implemented by
custom business logic or by
leveraging CEP
Filter
Filter
Filter
DDS Writer
DDS Reader
DDS Topic
CEP | Custom

8. Stream Processing "
with DDS

9. Data Distribution in a
Nutshell

10. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Data Distribution in a Nutshell
¨ Data distribution is about
making application defined
data available where needed
and when needed, while
preserving and end-to-end
type contract
¨ Data is a first class concept, it
can be Created, Read,
Updated, and eventually
Disposed (CRUD)
¨ The last value (or last N-values)
of a Data is always available to
applications
Id Temp Scale
101 25 C
202 78 F
303 299 K
DW
DW
DW
DR
DR
DR
struct TempSensor {!
long Id; !
float temp;!
float hum;!
}!
#pragma keylist TempSensor id!
Fully Distributed
Global Data SpaceDW: DataWriter
DR: DataReader

11. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
DDS Topics
¨ A Topic defines the subject of
publications and subscriptions
¨ A Topic has associated a user
defined type and QoS
¨ The Topic name, type and
QoS have a well defined role
in matching subscriptions
¨ Topics can be discovered or
locally defined
Topic
Name
QoSType
DURABILITY,
DEADLINE,
PRIORITY,
…
[1/2]
“org.opensplice.demo.TTempSensor”
struct TempSensor {!
long Id; !
float temp;!
float hum;!
}!
#pragma keylist TempSensor id!

12. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
DDS Topics
¨ DDS Topic types can have
associated keys
¨ Each unique key-value
identify a Topic Instance –
a specific stream of values
[2/2]
Topic
Name
QoSType
“org.opensplice.demo.TTempSensor”
DURABILITY,
DEADLINE,
PRIORITY,
…
struct TempSensor {!
long Id; !
float temp;!
float hum;!
}!
#pragma keylist TempSensor id!

13. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Anatomy of a DDS Application
Domain
Reader/Writers
User Defined for Types
Session
val dp = DomainParticipant(0) !
// Create a Publisher / Subscriber!
Publisher p = dp.create_publisher();!
Subscriber s = dp.create_subscriber();!
// Create a Topic!
Topic<TempSensor> t = !
dp.create_topic<TempSensor>(“com.myco.TSTopic”)!
Domain
Participant
Publisher
DataWrter
Topic Subscriber
DataReader
// Create a DataWriter/DataWriter!
DataWriter<TempSensor> dw = pub.create_datawriter(t);!
DataReader<TempSensor> dr = sub.create_datareader(t);!
Gives access to a
DDS Domain

14. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Anatomy of a DDS Application
Domain
Reader/Writers
User Defined for Types
Session
val dp = DomainParticipant(0) !
// Create a Publisher / Subscriber!
val pub = Publisher(dp)!
val sub = Subscriber(dp)!
// Create a Topic!
val topic = Topic[TempSensor](dp, !
“org.opensplice.demo.TTempSensor”)!
Domain
Participant
Publisher
DataWrter
Topic Subscriber
DataReader
// Create a DataWriter/DataWriter!
DataWriter<TempSensor> dw = pub.create_datawriter(t);!
DataReader<TempSensor> dr = sub.create_datareader(t);!
Pub/Sub
Abstractions

15. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Anatomy of a DDS Application
Domain
Reader/Writers for User Defined for Types
Session
Domain
Participant
Publisher
DataWrter
Topic Subscriber
DataReader
// Create a DataWriter/DataWriter!
val writer = DataWriter[TempSensor](pub, topic)!
val reader = DataReader[TempSensor](sub, topic) !
!
// Write data!
val t = new TempSensor ts(101, 25, 40)!
writer write ts;!
Reader/Writer for
application
defined Topic
Types
// Create a Publisher / Subscriber!
val pub = Publisher(dp)!
val sub = Subscriber(dp)!
// Create a Topic!
val topic = Topic[TempSensor](dp, !
“org.opensplice.demo.TTempSensor”)!
val dp = DomainParticipant(0) !

16. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Domain & Partitions
Topic Instances/Samples
Domain (e.g. Domain 123)
Partition (e.g. Partition “Telemetry”)
Domain
Participant
Topic
Publisher
DataWrter
Subscriber
DataReader

17. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Stream Processing in DDS
¨ DDS naturally supports stream processing through three
main concepts:
¨ Topics
¨ Data Writers
¨ Data Readers
¨ In addition DDS’ dynamic discovery, high availability and
high performance make it very easy to architect, deploy
and upgrade stream processing systems

18. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
DDS Topics as Streams
¨ In Stream Processing,
Streams are typed data
channel
¨ A DDS Topic provide an
elegant way of defining the
Streams that make up a
Streaming System while
capturing their type, as well
as their Non-Functional
Constraints
Filter
Filter
Filter
DDS Topic

19. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
DDS Data Writers as Sources
¨ DDS Data Writers
make ideal sources for
Stream Processing
systems due to their:
¨ Dynamic Discovery
¨ Durability
¨ Fault-Tolerance
Filter
Filter
Filter
DDS Writer

20. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
DDS Data Readers as Sinks
¨ DDS Data Readers
make ideal sinks for
Stream Processing
systems due to their:
¨ Dynamic Discovery
¨ Durability
Filter
Filter
Filter
DDS Reader

21. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
DDS & Filters
¨ Filters/Actors perform
transformations from one or
more input streams to one or
more output streams
¨ The combination of DDS
ContentFilteredTopics with a
powerful language like Scala
can make it relatively easy to
build custom filters
¨ Let’s explore this path…
DDS+Application Logic
Filter
Filter
Filter

22. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Content Filtered Topics
¨ Content Filtered Topics
provide a way of defining a
local projection of the
stream of data associated
with a given topic
¨ Filters are expressed as the
“WHERE” clause of an SQL
statement
¨ Filters can operate on any
attribute of the type
associated with the topic
Example:
// Create a Topic (on default domain)!
val topic = Topic[TempSensor](“TTempSensor”)!
val ftopic = !
ContentFilteredTopic[TempSensor](“CFTempSensor”,!
topic,!
filter,!
params)!
!
// - filter is a WHERE-like clause, such as: !
// “temp > 20 AND hum > 50”!
// “temp > %0”!
// “temp > hum”!
// “temp BETWEEN (%0 AND %1)!
//!
// - params is the list of parameters to pass to the !
// filter expression – if any!

23. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Filter Expression Syntax
¨ DDS Filters are condition
over a topic type attributes
¨ Temporal properties or
causality cannot be
captured via DDS filter
expression

24. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
History
¨ DDS provides a way of
controlling data
windows through the
History QoS
The window keeps
the last n data samples
Data older than “n samples ago”
get’s out the window
pastfuture
now

25. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
[Putting it All Together]"
TempSensor Moving Average
object MovingAverageFilter {!
def main(args: Array[String]) {!
if (args.length < 2) {!
println("USAGE:ntMovingAverageFilter <window> <filter-expression>")!
}!
!
val topic = Topic[TempSensor]("TTempSensor")!
val ftopic = ContentFilteredTopic[TempSensor]("CFTempSensor",topic, args(1))!
!
val rqos = DataReaderQos() <= KeepLastHistory(args(0).toInt)!
val reader = DataReader[TempSensor](ftopic, rqos)!
!
reader.reactions += {!
case e: DataAvailable[TempSensor] => {!
var average: Float = 0!
val window = e.reader.history!
window foreach (average += _.temp)!
average = average / window.length!
println("+--------------------------------------------------------")!
println("Moving Average: " + average)!
}!
}!
}!

26. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Key Points So Far
¨ DDS provides some event processing capabilities that
facilitate the development of Stream Processing Filters
¨ Higher Level programming languages like Scala can make it
very easy to concisely express complex filters
¨ Note: Scala targets the JVM as well as the .Net CLR

27. Stream Processing "
with DDS & CEP

28. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
CEP In Brief
¨ Complex Event Processing (CEP)
engines provide a declarative
way of transforming a set of input
streams into one or more output
streams
¨ The declarative language
provided by CEP is usually based
on some extension of SQL to
include time, pattern matching
and causality
- select *
from TempSensor(temp < 30)
- select avg(temp)
from TempSensor.win:time(1 sec)
CEP
[1/2]

29. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
CEP In Brief
¨ CEP emerged from Stream
Processing as a way to
facilitate the development of
“Filters/Actors” and make it
“declarative”
¨ In line with the Stream
Processing principles, CEP
operate on typed data streams
- select *
from TempSensor(temp < 30)
- select avg(temp)
from TempSensor.win:time(1 sec)
CEP
[1/2]
Filter
Filter
Filter

30. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
DDS + CEP = Stream Processing
¨ DDS provides a powerful Stream
abstraction that is:
¨ Type Safe
¨ High Performance
¨ Highly Available
¨ Decoupled in Time/Space
¨ Dynamically Discoverable
¨ CEP provide a powerful abstraction for
processing streams
¨ The combination of DDS + CEP is not
only natural but a perfect fit for building
high performance, highly available
Stream Processing Systems
CEP
CEP
CEP
DDS Writer
DDS Reader
DDS Topic

31. DDS + CEP in Action

32. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Esper CEP
¨ Esper is an Open Source, pure Java, CEP engine widely used
in a wide variety of applications ranging from the Capital
Market to Defense and Aerospace
¨ http://www.espertech.com
¨ Esper Rules are expressed EPL (Esper Processing
Language) which can be seen as an extension of SQL
with support for time, causality and pattern matching

33. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
OpenSplice + Esper
¨ Esper Provides an EsperIO framework for plugging-in
new stream transports
¨ Plugging OpenSplice into Esper is trivial even w/o
relying on the EsperIO framework
¨ Let’s have a look…

34. Demo Application

35. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
iShapes Application
¨ To explore play with OpenSplice
and Esper, we’ll use the simd-cxx
ishapes application
¨ Three Topics
¨ Circle, Square, Triangle
¨ One Type:
struct ShapeType {!
string color;!
long x;!
long y;!
long shapesize;!
};!
#pragma keylist Shapetype color!
Spotted shapes represent subscriptions
Pierced shapes represent publications

36. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Esper Setup
Step 1: Register Topic Types
val config = new Configuration!
val ddsConf = new ConfigurationEventTypeLegacy!
!
ddsConf.setAccessorStyle(ConfigurationEventTypeLegacy.AccessorStyle.PUBLIC)!
!
config.addEventType("ShapeType", !
classOf[org.opensplice.demo.ShapeType].getName, !
ddsConf)!
!
val cep: EPServiceProvider = !
EPServiceProviderManager.getDefaultProvider(config)!

37. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Esper Setup
Step 2: Register a Listener for receiving Esper Events
val listener = new UpdateListener {!
def update(ne: Array[EventBean], oe: Array[EventBean]) {!
ne foreach(e => {!
"// Handle the event!
})!
}!
}!

38. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Esper Setup
Step 3: Hook-up DDS to Esper
reader.reactions += {!
case e: DataAvailable[ShapeType] => {!
(e.reader read) foreach(runtime sendEvent _)!
!
}!
}!

39. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
iShapes FrameRate
¨ Let’s suppose that we wanted to keep under
control the iShapes Frame rate for ech given color
¨ In Esper this can be achieved with the following
expression:
insert into ShapesxSec !
select color, count(*) as cnt !
from ShapeType.win:time_batch(1 second) !
group by color!

40. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
iShapes Center of Mass
¨ Suppose that we wanted to compute the center of
mass of all the shapes currently displayed over the
last second
¨ The Esper expression for this would be:
select ShapeFactory.createShape(color, cast(avg(x),int), cast(avg
(y),int), shapesize) as NewShape !
from ShapeType.win:time(10 sec)!

41. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
¥ Fastest growing JVM Language
¥ Open Source
¥ www.scala-lang.org
References
OpenSplice DDS
¥ #1 OMG DDS Implementation
¥ Open Source
¥ www.opensplice.org
¥ Scala API for OpenSplice DDS
¥ Open Source
¥ code.google.com/p/escalier
¥ #1 Java-Based CEP Engine
¥ Open Source
¥ www.espertech.com

42. Summing Up

43. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Concluding Remarks
¨ DDS provides a very good
foundation to build high
performance, highly available and
scalable streaming systems
¨ DDS + CEP are the most natural,
effective and efficient way of
building next generation Stream
Processing Systems!

44. Copyright
2010,
PrismTech
–
All
Rights
Reserved.
http://www.opensplice.com/
http://www.opensplice.org/
emailto:opensplicedds@prismtech.com
http://www.youtube.com/OpenSpliceTube http://opensplice.blogspot.com
http://bit.ly/1Sreg
http://www.slideshare.net/angelo.corsaro
http://twitter.com/acorsaro/
D e l i v e r i n g P e r f o r m a n c e , O p e n n e s s , a n d F r e e d o m
OpenSplice DDS