Walk through Streaming Technologies: EPL Riccardo Tommasini - DEIB Politecnico di Milano

1. BigData
Semantic Approach to
Big Data and Event Processing
Walk through Streaming
Technologies: EPL
Riccardo Tommasini
PhD Student Politecnico di Milano
riccardo.tommasini@polimi.it

2. BigData
Who I am
2
Master Degree in C.S. @ Politecnico Of Milano
M.D. Thesis on Stream Reasoning
I’ll start my Phd in November 2k15

3. BigData
Agenda
3
✓
• IntroducAon
• Event Processing Languages Overview
• EPL Running Example:
– Event CreaAon
– DSMS Features
– CEP Features
• Java Demo (if we have Ame)
• QuesAons

4. BigData
Intro: Esper Architecture
Source Link

5. BigData
Esper Features: Efficient Event Processing
• Expressive Event Query Language :
• Continuous queries
• filtering, aggregation, joins of streams
• Comprehensive pattern detection
• Pull and Push
• High performances, low latency

6. BigData
Esper Features: Extensible Middleware
• RunAme statement management
• API or configuraAon driven
• Plug-in SDK for funcAons, aggregaAons, views and
paSern detecAon extensions
• Adapters: CSV, JMS in/out, API, DB, Socket, HTTP
• Data distribuAon service:
• data push management
• service layer

7. BigData
Esper Features: Rich Web-Based User Interface
• Eventlet technology:
• interactive continuous displays
• CEP Engine management
• EPL Statements design
• Drill-down and script browsing integration

8. BigData
Esper Features: HA enabled (EsperHA)
• Per-statement configuration
• Transient combinable with fully resilient
behaviour
• Hot standby API
• Hot backup
• Highly optimised data storage technology
• Engine state RDBMS storage option

9. BigData9
Development
• EPL Statements have SQL-like syntax
• java POJO or .Net object for real-time
customised actionable intelligence
Deployment
• Standalone deployment
• Embedded (into existing middleware
systems)
Development and Deployment

10. BigData
Agenda
10
✓
• IntroducAon ✓
• Event Processing Languages Overview
• EPL Running Example:
– Event CreaAon
– DSMS Features
– CEP Features
• Java Demo
• QuesAons

11. BigData1
1
EPL Statements are
registered into Esper
and continuously
executed as the live
data stream are
pushed through
Event Streams/Complex Event Processing: EPL
Event Processing Language (EPL)
Continuous
Query
Answering
Complex
Pattern Matching
(causality
relationship)

12. BigData1
2
• Push Based
• Data tuple from sensors,
trace files etc.
• Push Based
• Continuously executed
against incoming events
• Receive data tuples from
queries
• Push data tuple to other
queries
• Receive processed tuples
EPL
Query
Subscriber
Source
Listener
EPL ConAnuous Query Processing Model

13. BigData1
3
• Sources act as input
• EPL Queries Integrate
sources
• Listener propagates query
results
• Subscribers act as output
EPL Query Model can be represented as a graph
Graph nodes are functional
components, manually
connect by the data stream.
EPL ConAnuous Query Processing Model

14. BigData
Agenda
14
✓
• IntroducAon ✓
• Event Processing Languages Overview✓
• EPL Running Example:
– Event CreaAon
– DSMS Features
– CEP Features
• Java Demo
• QuesAons

15. BigData1
5
We starts with the demo time
https://github.com/riccardotommasini/esper-tutorial
http://esper-epl-tryout.appspot.com/epltryout/mainform.html
ONLINE VERSION
Java Code Data for the running example
DEMO

16. BigData16
Count the number of fires detected using a set of smoke and
temperature sensors in the last 10 minutes
Request
Events
CondiAon
• Smoke Event: String sensor, boolean state
• Temperature Event: String sensor, double temperature
• Fire Event: String sensor, boolean smoke, double temperature
Fire: The same sensor detects smoke followed by temperature >
50°C
Running Example

17. BigData1
7
Running Example Graph

18. BigData18
EPL create schema clause
create schema schema_name [as]
(property_name property_type
[,property_name property_type [,…])
[inherits inherited_event_type
[,inherited_event_type][,...]]
EPL: Event DeclaraAon

19. BigData19
create schema SmokeSensorEvent(
sensor string,
smoke boolean
);
create schema TemperatureSensorEvent(
sensor string,
temperature double
);
create schema FireEvent(
sensor string,
smoke boolean,
temperature double
);
EPL: Running Example: Events Types

20. BigData
SmokeSensorEvent={sensor='S1', smoke=false}
TemperatureSensorEvent={sensor='S1', temperature=30}
t=t.plus(1 seconds)
SmokeSensorEvent={sensor='S1', smoke=true}
TemperatureSensorEvent={sensor='S1', temperature=40}
t=t.plus(1 seconds)
SmokeSensorEvent={sensor='S1', smoke=true}
TemperatureSensorEvent={sensor='S1', temperature=55}
t=t.plus(1 seconds)
SmokeSensorEvent={sensor='S1', smoke=true}
TemperatureSensorEvent={sensor='S1', temperature=55}
t=t.plus(1 seconds)
SmokeSensorEvent={sensor='S1', smoke=false}
TemperatureSensorEvent={sensor='S1', temperature=55}
20Stream & Complex Event Processing - Introduction
Advancing time manually
is only required by the
online interface, normally
time pass by on its
own :-)
EPL: Running Example: Events Types

21. BigData
Agenda
21
✓
• IntroducAon ✓
• Event Processing Languages Overview✓
• EPL Running Example:✓
– Event CreaAon✓
– DSMS Features
– CEP Features
• Java Demo
• QuesAons

22. BigData22
[insert into insert_into_def]
select select_list
from stream_def [as name] [, stream_def [as name]]
[,...]
[where search_conditions]
[group by grouping_expression_list]
[having grouping_search_conditions]
[output output_specification]
[order by order_by_expression_list]
[limit num_rows]
• Similar to SQL (select,
where)
• views are used inset of tables
EPL: Query RegistraAon

23. BigData23
@Name(‘Q0’) select *
from TemperatureSensorEvent
where temperature > 50;
@Name(‘Q1’) select avg(temperature)
from TemperatureSensorEvent;
Stream Lookup
Aggregate
EPL Queries: Examples

24. BigData24
• Similar to SQL tables
• define data available for querying and filtering
• sorting, aggregation, grouping operation are
possible
• usually represented as windows over the
streams of events
EPL: Views

25. BigData25
Type Syntax Description
Logical Sliding win:time(time period) Sliding time window extending the
specified time interval into the past
Logical Tumbling win:time_batch(time
period)
Tumbling window that batches
events and releases them every
specified time interval, with flow
control options
Physical Sliding win:length(size) Sliding length window extending
the specified number of events into
the past
Physical Tumbling win:length_batch(size) Tumbling window that batches
events and releases them when a
given minimum number is has been
collected
EPL: Statements and Windows

26. BigData26
EPL: Logical Sliding Window

27. BigData
At: 2015-01-07 14:00:03.000
Insert
Q2-output={avg(temperature)=45.0}
At: 2015-01-07 14:00:04.000
Insert
Q2-output={avg(temperature)=50.0}
At: 2015-01-07 14:00:04.000
Insert
Q2-output={avg(temperature)=51.25}
27
Query Q2
Trace
@Name(‘Q2’) select avg(temperature)
from TemperatureSensorEvent.win:time(4 sec);
Running Example: Logical Sliding Window
At: 2015-01-07 14:00:00.000
Insert
Q2-output={avg(temperature)=30.0}
At: 2015-01-07 14:00:01.000
Insert
Q2-output={avg(temperature)=35.0}
At: 2015-01-07 14:00:02.000
Insert
Q2-output={avg(temperature)=41.66...}
In Esper, logical sliding
windows report as soon as a
new event arrives and an old
one expires

28. BigData28
EPL: Logical Tumbling Window

29. BigData29
Query Q3
Trace
@Name(‘Q3’) select avg(temperature)
from TemperatureSensorEvent.win:time_batch(4 sec);
In Esper, logical tumbling
windows report only when the
window closes
Running Example: Logical Tumbling Window
At: 2015-01-07 14:00:04.000
Insert
Q3-output={avg(temperature)=45.0}

30. BigData30
EPL: Physical Sliding Window

31. BigData31
select avg(temperature)
from TemperatureSensorEvent.win:length(5);
Running Example: Physical Sliding Window
Query Q4
Trace
At: 2015-01-07 14:00:00.000
Insert
Q4-output={avg(temperature)=30.0}
At: 2015-01-07 14:00:01.000
Insert
Q4-output={avg(temperature)=35.0}
At: 2015-01-07 14:00:02.000
Insert
Q4-output={avg(temperature)=41.6...}
At: 2015-01-07 14:00:03.000
Insert
Q4-output={avg(temperature)=45.0}
At: 2015-01-07 14:00:04.000
Insert
Q4-output={avg(temperature)=47.0}
In Esper, physical sliding
windows report as soon as a
new event arrives

32. BigData32
Query Q5
Trace
@Name(‘Q5’) select avg(temperature)
from TemperatureSensorEvent.win:length_batch(5);
In Esper, logical tumbling
windows report only when the
window closes
EPL: Physical Tumbling Window
At: 2015-01-07 14:00:04.000
Insert
Q5-
output={avg(temperature)=47.0}

33. BigData3
3
[output [[all|first|last|snapshot|] every
output_rate [seconds | events ]]
• The output clause is
optional
• It can control the event output
rate
• It can suppress output events
EPL: Controlling ReporAng

34. BigData34
Query Q6: Logical Windows Output Clause
Trace
@Name(’Q6') select avg(temperature)
from TemperatureSensorEvent.win:time(4 sec)
output snapshot every 2 sec;
Running Example: Controlling ReporAng
At: 2015-01-07 14:00:02.000
Insert
Q6-output={avg(temperature)=35.0}
At: 2015-01-07 14:00:04.000
Insert
Q6-output={avg(temperature)=45.0}

35. BigData35
Query Q7: Physical Windows Output Clause
Trace
@Name('Q7') select avg(temperature)
from TemperatureSensorEvent.win:length(4)
output snapshot every 2 events;
Running Example: Controlling ReporAng
At: 2015-01-07 14:00:01.000
Insert
Q7-output={avg(temperature)=35.0}
At: 2015-01-07 14:00:03.000
Insert
Q7-output={avg(temperature)=45.0}

36. BigData36
• Looking for a pattern of events over
the incoming data streams
• pattern can be temporal as well as
physical
• it’s implemented via finite states
machine
EPL: Event PaSern Matching

37. BigData
Agenda
37
✓
• IntroducAon ✓
• Event Processing Languages Overview✓
• EPL Running Example:✓✓
– Event CreaAon✓
– DSMS Features✓
– CEP Features
• Java Demo
• QuesAons

38. BigData38
EPL: PaSern Operators
• Logical operators: and, or, not
• Temporal operators that operate on event order:
-> (followed-by)
• Operators that control paSern finder creaAon and
terminaAon: every, every-dis9nct, [num] and un9l
• sub-expression lifeAme guards: 9mer:within,
9mer:withinmax and while-expression

39. BigData39
@Name('Q8') select a.sensor
from pattern [every (a = SmokeSensorEvent(smoke=true) ->
TemperatureSensorEvent(temperature > 50, sensor=a.sensor)
where timer:within(2 sec))];
Running Example: within PaSern Operator
Query
Trace
At: 2015-01-07 14:00:02.000
Insert
Q8-output={a.sensor='S1'}
At: 2015-01-07 14:00:03.000
Insert
Q8-output={a.sensor='S1'}

40. BigData
• Default: every operator the paSern sub-
expression stops when the paSern sub-expression
evaluates to true or false
• Every Clause: every Ame a paSern sub-expression
within an every operator turns true the engine
starts a new acAve sub-expression looking for
more event(s) or Aming condiAons that match the
paSern sub-expression.
40Stream & Complex Event Processing - Introduction
EPL: “every” PaSern Operator
Precedence MaSers!

41. BigData
Simple Example: every (A -> B)
• PaSern
– every ( A -> B )
• Events
– A1 B1 C1 B2 A2 D1 A3 B3 E1 A4 F1 B4
• Results
– Detect an A event followed by a B event. At the Ame
when B occurs the paSern matches, then the paSern
matcher restarts and looks for the next A event.
1. Matches on B1 for combinaAon {A1 , B1}
2. Matches on B3 for combinaAon {A2 , B3}
3. Matches on B4 for combinaAon {A4 , B4}
41Stream & Complex Event Processing - Introduction

42. BigData
• PaSern
– every A -> B
• Events
– A1 B1 C1 B2 A2 D1 A3 B3 E1 A4 F1 B4
• Results
– The paSern fires for every A event followed by a B
event.
1. Matches on B1 for combinaAon {A1 , B1}
2. Matches on B3 for combinaAon {A2 , B3} and {A3 , B3}
3. Matches on B4 for combinaAon {A4 , B4}
42Stream & Complex Event Processing - Introduction
Simple Example: every A -> B

43. BigData
• PaSern
– A -> every B
• Events
– A1 B1 C1 B2 A2 D1 A3 B3 E1 A4 F1 B4
• Results
– The paSern fires for an A event followed by every B
event.
1. Matches on B1 for combinaAon {A1 , B1}
2. Matches on B2 for combinaAon {A1 , B2}
3. Matches on B3 for combinaAon {A1 , B3}
4. Matches on B4 for combinaAon {A1 , B4}
43Stream & Complex Event Processing - Introduction
Simple Example: A -> every B

44. BigData
• PaSern
– every A -> every B
• Events
– A1 B1 C1 B2 A2 D1 A3 B3 E1 A4 F1 B4
• Results
– The paSern fires for every A event followed by every B
event.
1. Matches on B1 for combinaAon {A1 , B1}
2. Matches on B2 for combinaAon {A1 , B2}
3. Matches on B3 for combinaAon {A1 , B3}, {A2 , B3} and {A3 , B3}
4. Matches on B4 for combinaAon {A1 , B4}, {A2 , B4}, {A3 , B4}
and {A4 , B4}
44Stream & Complex Event Processing - Introduction
Simple Example: every A -> every B

45. BigData
EPL: Guards PaSern Operators
Claim: the data window onto a paSern stream
does not serve to limit paSern sub-expression
lifeAme.
• The 9mer:within construct and the and not
constructs can be used to end acAve sub-
expressions.
45Stream & Complex Event Processing - Introduction

46. BigData
• Events
– A1 A2 B1
• PaSern
– every A -> B
• Results
– {A1 , B1} and {A2 , B1}
• Events
– A1 A2 B1
• PaSern
– every A -> (B and not A)
• Results
– {A2 , B1}
– The and not operators
cause the sub-
expression looking for
{A1, B?} to end when A2
arrives.
46Stream & Complex Event Processing - Introduction
Guards Example 1

47. BigData
• Events
– A1 received at to+ 1 sec
– A2 received at to+ 3 sec
– B1 received at to+ 4 sec
• PaSern
– every A -> B
• Results
– {A1 , B1} and {A2 , B1}
• Events
– A1 received at to+ 1 sec
– A2 received at to+ 2 sec
– B1 received at to+ 3 sec
• PaSern
– every A -> (B where Amer:within(2 sec) )
• Results
– {A2 , B1}
– The where 9mer:within operators cause
the sub-expression looking for {A1, B?}
to end aker 2 seconds.
47Stream & Complex Event Processing - Introduction
Guards Example 2

48. BigData
Forwards events to other streams for
further downstream processing: Acts as
a listener
48Stream & Complex Event Processing - Introduction
EPL: “Insert Into” Clause

49. BigData49Stream & Complex Event Processing - Introduction
Running Example: “Insert Into” Clause
@Name(‘Final’) insert into FireComplexEvent
select a.sensor as sensor, a.smoke as smoke, b.temperature as temperature
from pattern [every ( a = SmokeSensorEvent(smoke=true) -> b =
TemperatureSensorEvent(temperature>5, sensor=a.sensor) where timer:within(2 sec) ) ] ;
Final Query
Trace
At: 2015-01-07 14:00:01.000
Insert
FireComplexEvent={sensor='S1',
smoke=true, temperature=40.0}
At: 2015-01-07 14:00:02.000
Insert
FireComplexEvent={sensor='S1',
smoke=true, temperature=55.0}
At: 2015-01-07 14:00:03.000
Insert
FireComplexEvent={sensor='S1',
smoke=true, temperature=55.0}

50. BigData50
Downstream Query
Trace
Running Example Downstream Query
select count(*)
from FireComplexEvent.win:time(10 sec)
At: 2015-01-07 14:00:01.000
Insert
Downstream-output={count(*)=1}
At: 2015-01-07 14:00:02.000
Insert
Downstream-output={count(*)=2}
At: 2015-01-07 14:00:03.000
Insert
Downstream-output={count(*)=3}

51. BigData
Agenda
51
✓
• IntroducAon ✓
• Event Processing Languages Overview✓
• EPL Running Example:✓✓✓
– Event CreaAon✓
– DSMS Features✓
– CEP Features✓
• Java Demo
• QuesAons

52. BigData52
Equivalent methods alternative to EPL:
• Java POJO
• Maps
• XML
One More Thing: Event DeclaraAon
AlternaAves

53. BigData53
package us.wsu.knoesis.tutorial.events;
public class FireEvent {
private String sensor;
private boolean smoke;
}• Getter and Setter naming
convention are relevant to access
parameter form EPL
• the construct must be defined as
well
• toString is used to represent
display the event, consider
redefinition
Event DeclaraAon Java POJO

54. BigData
Agenda
54
✓
• IntroducAon ✓
• Event Processing Languages Overview✓
• EPL Running Example:✓✓✓
– Event CreaAon✓
– DSMS Features✓
– CEP Features✓
• Java Demo
• QuesAons

55. BigData5
5
Thank
You!
End!

56. BigData5
6
RiccardoTommasini+
@rictomm
tomma156
riccardo@knoesis.orgRiccardo Tommasini
riccardotommasini
Contact

57. BigData57
Resources

58. BigData58
Streamreasoning.org
StreamReasoning@GitHub
RDF Stream Processors
PhD CEP Course @Polimi
Stream Reasoning Tutorial
Resources

59. BigData
Ques9ons?
59
???