Robust Stream Processing with Apache Flink

•

5 likes•2,320 views

The document discusses Apache Flink, an open source platform for distributed stream and batch data processing. It describes how Flink allows for stateful stream processing, windowed computations over streams, and robust handling of time, failures, planned downtimes, and reprocessing. The presentation provides an overview of these concepts and includes a code example and details on Flink's distributed and parallel deployment architecture.

Data & Analytics

Robust
Stream Processing
with
Apache Flink
Jamie Grier
@jamiegrier
jamie@data-artisans.com

Who am I?
• Director of Applications Engineering at data
Artisans
• Previously working on streaming computation at
Twitter, Gnip and Boulder Imaging
• Involved in various kinds of stream processing for
about a decade
• High-speed video, social media streaming, general
frameworks for stream processing

Overview
• What is Apache Flink?
• What is Stateful Stream Processing?
• Windowed computation over streams
• Robust Time Handling (Event Time vs Processing Time)
• Robust Failure Handling
• Robust Planned Downtime Handling
• Robust Reprocessing

What is
Apache Flink?
Apache Flink is an open source platform for distributed
stream and batch data processing.

Stream Processing
Your
Code
Data Stream Data Stream

Stateful
Stream Processing
Your
Code
Data Stream Data Stream
State

More Complex
Example
Kafka
Files
Rabb
itMQ
Filter
Map
Join /
Sum
Influx
DB
C*

Distributed and Parallel
Deployment
Kafka
Files
Rabb
it
MQ
Filter
Parse
Join /
Sum
Influx
DB
C*

Robust Stream Processing
with Apache Flink

Windowing in Processing
Time
0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9
Processing Time
Event Time

Windowing in Event
Time
0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9
Event Time

We’re Hiring!
http://data-artisans.com/careers

Flink Forward 2016, Berlin
Submission deadline: June 30, 2016
Early bird deadline: July 15, 2016
www.flink-forward.org

What's hot

Big Data WarsawMaximilian Michels

Streaming in the Wild with Apache FlinkKostas Tzoumas

Ted Dunning-Faster and Furiouser- Flink DriftFlink Forward

Flink 1.0-slidesJamie Grier

data Artisans Product AnnouncementFlink Forward

Robust Stream Processing With Apache FlinkJamie Grier

Stream Processing with Apache FlinkC4Media

Flink. Pure StreamingIndizen Technologies

Extending the Yahoo Streaming Benchmark + MapR BenchmarksJamie Grier

The Evolution of (Open Source) Data ProcessingAljoscha Krettek

Moon soo Lee – Data Science Lifecycle with Apache Flink and Apache ZeppelinFlink Forward

Introduction to Stream Processing with Apache Flink (2019-11-02 Bengaluru Mee...Timo Walther

Apache Flink 101 - the rise of stream processing and beyondBowen Li

A look at Flink 1.2Stefan Richter

Aljoscha Krettek - Apache Flink for IoT: How Event-Time Processing Enables Ea...Ververica

Abstractions for managed stream processing platform (Arya Ketan - Flipkart)KafkaZone

Bay Area Apache Flink Meetup Community Update August 2015Henry Saputra

The Past, Present, and Future of Apache Flink®Aljoscha Krettek

Building a real time Tweet map with Flink in six weeksMatthias Kricke

Apache Flink and what it is used forAljoscha Krettek

What's hot (20)

Big Data Warsaw

Streaming in the Wild with Apache Flink

Ted Dunning-Faster and Furiouser- Flink Drift

Flink 1.0-slides

data Artisans Product Announcement

Robust Stream Processing With Apache Flink

Stream Processing with Apache Flink

Flink. Pure Streaming

Extending the Yahoo Streaming Benchmark + MapR Benchmarks

The Evolution of (Open Source) Data Processing

Moon soo Lee – Data Science Lifecycle with Apache Flink and Apache Zeppelin

Introduction to Stream Processing with Apache Flink (2019-11-02 Bengaluru Mee...

Apache Flink 101 - the rise of stream processing and beyond

A look at Flink 1.2

Aljoscha Krettek - Apache Flink for IoT: How Event-Time Processing Enables Ea...

Abstractions for managed stream processing platform (Arya Ketan - Flipkart)

Bay Area Apache Flink Meetup Community Update August 2015

The Past, Present, and Future of Apache Flink®

Building a real time Tweet map with Flink in six weeks

Apache Flink and what it is used for

Similar to Robust Stream Processing with Apache Flink

Robust Stream Processing with Apache FlinkJamie Grier

Robust stream processing with Apache FlinkAljoscha Krettek

Trend Micro Big Data Platform and Apache BigtopEvans Ye

Big Data Analytics Platforms by KTH and RISE SICSBig Data Value Association

Flink in actionArtem Semenenko

Near real-time anomaly detection at Lyftmarkgrover

A Practical Guide to Selecting a Stream Processing Technology confluent

Cloud lunch and learn real-time streaming in azureTimothy Spann

Data Stream Processing with Apache FlinkFabian Hueske

Scylla Summit 2022: Learning Rust the Hard Way for a Production Kafka+ScyllaD...ScyllaDB

Streaming in the Wild with Apache FlinkDataWorks Summit/Hadoop Summit

Stinger.Next by Alan Gates of HortonworksData Con LA

Hail hydrate! from stream to lake using open sourceTimothy Spann

Berlin Apache Flink Meetup May 2015, Community UpdateRobert Metzger

The Purpose of Cloud-technology for Video ProductionMaarten Verwaest

Flink forward-2017-netflix keystones-paasMonal Daxini

Reintroducing the Stream Processor: A universal tool for continuous data anal...Paris Carbone

Getting started with apache flink streaming apiPreetdeep Kumar

An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.Data Con LA

Stream Processing with Apache Flink (Flink.tw Meetup 2016/07/19)Apache Flink Taiwan User Group

Similar to Robust Stream Processing with Apache Flink (20)

Robust Stream Processing with Apache Flink

Robust stream processing with Apache Flink

Trend Micro Big Data Platform and Apache Bigtop

Big Data Analytics Platforms by KTH and RISE SICS

Flink in action

Near real-time anomaly detection at Lyft

A Practical Guide to Selecting a Stream Processing Technology

Cloud lunch and learn real-time streaming in azure

Data Stream Processing with Apache Flink

Scylla Summit 2022: Learning Rust the Hard Way for a Production Kafka+ScyllaD...

Streaming in the Wild with Apache Flink

Stinger.Next by Alan Gates of Hortonworks

Hail hydrate! from stream to lake using open source

Berlin Apache Flink Meetup May 2015, Community Update

The Purpose of Cloud-technology for Video Production

Flink forward-2017-netflix keystones-paas

Reintroducing the Stream Processor: A universal tool for continuous data anal...

Getting started with apache flink streaming api

An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.

Stream Processing with Apache Flink (Flink.tw Meetup 2016/07/19)

Recently uploaded

SMOTE and K-Fold Cross Validation-Presentation.pptxHaritikaChhatwal1

IBEF report on the Insurance market in IndiaManalVerma4

Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...Thomas Poetter

English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdfblazblazml

Insurance Churn Prediction Data Analysis ProjectBoston Institute of Analytics

Learn How Data Science Changes Our WorldEduminds Learning

Semantic Shed - Squashing and Squeezing.pptxMike Bennett

Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...Boston Institute of Analytics

What To Do For World Nature Conservation Day by Slidesgo.pptxSimranPal17

Data Analysis Project: Stroke PredictionBoston Institute of Analytics

Data Factory in Microsoft Fabric (MsBIP #82)Cathrine Wilhelmsen

modul pembelajaran robotic Workshop _ by Slidesgo.pptxaleedritatuxx

FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024Susanna-Assunta Sansone

Rithik Kumar Singh codealpha pythohn.pdfrahulyadav957181

Cyber awareness ppt on the recorded dataTecnoIncentive

Networking Case Study prepared by teacher.pptxHimangsuNath

6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...Dr Arash Najmaei ( Phd., MBA, BSc)

why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...Jack Cole

Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Boston Institute of Analytics

Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBoston Institute of Analytics

Recently uploaded (20)

SMOTE and K-Fold Cross Validation-Presentation.pptx

IBEF report on the Insurance market in India

Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...

English-8-Q4-W3-Synthesizing-Essential-Information-From-Various-Sources-1.pdf

Insurance Churn Prediction Data Analysis Project

Learn How Data Science Changes Our World

Semantic Shed - Squashing and Squeezing.pptx

Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...

What To Do For World Nature Conservation Day by Slidesgo.pptx

Data Analysis Project: Stroke Prediction

Data Factory in Microsoft Fabric (MsBIP #82)

modul pembelajaran robotic Workshop _ by Slidesgo.pptx

FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024

Rithik Kumar Singh codealpha pythohn.pdf

Cyber awareness ppt on the recorded data

Networking Case Study prepared by teacher.pptx

6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...

why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...

Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...

Bank Loan Approval Analysis: A Comprehensive Data Analysis Project

Robust Stream Processing with Apache Flink

1. Robust Stream Processing with Apache Flink Jamie Grier @jamiegrier jamie@data-artisans.com

2. Who am I? • Director of Applications Engineering at data Artisans • Previously working on streaming computation at Twitter, Gnip and Boulder Imaging • Involved in various kinds of stream processing for about a decade • High-speed video, social media streaming, general frameworks for stream processing

3. Overview • What is Apache Flink? • What is Stateful Stream Processing? • Windowed computation over streams • Robust Time Handling (Event Time vs Processing Time) • Robust Failure Handling • Robust Planned Downtime Handling • Robust Reprocessing

4. What is Apache Flink? Apache Flink is an open source platform for distributed stream and batch data processing.

5. What is Apache Flink?

6. Stream Processing Your Code Data Stream Data Stream

7. Stateful Stream Processing Your Code Data Stream Data Stream State

8. More Complex Example Kafka Files Rabb itMQ Filter Map Join / Sum Influx DB C*

9. Distributed and Parallel Deployment Kafka Files Rabb it MQ Filter Parse Join / Sum Influx DB C*

10. Robust Stream Processing with Apache Flink

11. Code Example!

12. Windowing

13. Processing Time vs Event Time

14. Windowing in Processing Time 0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9 Processing Time Event Time

15. Windowing in Event Time 0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9 Event Time