SlideShare a Scribd company logo

Sweet Streams (Are made of this)

•Download as PPTX, PDF•
•
Corneil du Plessis
Corneil du Plessis

Spring Cloud Data Flow - a Gentle introductions

Software
1 of 37
Version 1.0 October 2022 Sweet Streams (Are made of this) Spring Cloud Data Flow - A gentle overview Copyright © 2022 VMware, Inc. or its affiliates.
Cover w/ Image Agenda ● What is Spring Cloud Data Flow? ● Spring Batch ● Spring Task ● Spring Cloud Functions ● Spring Cloud Stream Applications ● Interview with a User ● Q+A
What is Spring Cloud Data Flow? ● Microservice based Enterprise Application Integration. ○ Batch Jobs ○ Tasks ○ Streams
What is Spring Cloud Data Flow? ● Orchestration and Deployment ○ Tanzu Application Service / Cloud Foundry ○ Kubernetes ○ Local
What is Spring Cloud Data Flow? ● Observability ○ Micrometer ○ Prometheus ○ WaveFront ○ InfluxDB ○ More…
Stream DSL file | splitter | csv-to-json: transform > :input-stream http > :input-stream :input-stream > filter | aggregator > :output-stream :output-stream > jdbc :output-stream > json-to-ws-notification: transform | websocket Legend Stream App Topic App Label
Stream DSL
Stream DSL
Stream Definitions
Stream Definitions
Stream Definitions
Batch/Task Workflows ● DSL can define complex Batch and or Task application topologies in addition to a single Batch or Task application ● Each box is a Spring Batch or Task application. ● Application Flow can split and join, based on optional conditional expressions
How did we build Spring Cloud Data Flow? Spring Cloud Data Flow Spring Boot Spring Cloud Spring Batch Spring Integration Spring AMQP Spring Kafka Spring Security Spring Data
Spring Cloud Data Flow Architecture ● Data Flow UI, REST Client or Shell ● Data Flow Server ● Skipper Server ● Messaging Middleware
Observability Streams
Observability Tasks
Pre-packaged Stream Applications aggregator analytics bridge cassandra cdc-debezium dataflow-tasklauncher elasticsearch file filter ftp groovy header-enricher http http-request image- recognition jdbc jms load-generator log mail mongodb mqtt object-detection pgcopy rabbit redis router rsocket s3 script semantic-segmentation sftp splitter syslog tcp throughput time transform twitter-message twitter-search twitter-stream twitter-trend twitter-update wavefront websocket zeromq
Pre-packaged Stream Applications aggregator dataflow-tasklauncher filter http jdbc splitter transform your-application
Introducing: A user.
Data ingest Data comes in via file or web service Once in the DB the batch process can consume Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum
Batch process
Batch process
Technology
Technology
Design ● How do you eat an elephant? ● Enable more real-time processing ● Improve the feedback loop for failures ● Use Java ● Functional programming paradigm ● How do you eat an elephant?
Design Spring Cloud Data Flow
Stream processing System is built to be event driven Files are broken into individual events File and web service events are processed by streaming applications All streaming apps are custom. (Didn’t use any off the shelf applications) Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum
Stream processing Failures can be handled on individual events Streaming apps are designed to be idempotent Load can be spread out much better Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Error handler
Batch processing System is built to be event driven Files are broken into individual events Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Spring Batch
Idempotent processing ● At some point you must write to the DB (sink) ● Processes are designed to be idempotent ● The sink that persists is designed to cater for items being replayed
Developer experience ● Everything is a Spring Boot application (Our Devs love this) ● Changes can be tested locally ● Custom templates ● Dependency management (Solution BOM) ● Easy to understand applications that are small and do one thing ● Unit and integration testing is easy ● Containers Paving the road to production
Deploying Data Flow
Deployment Data Flow deployment ● Deployed to Kubernetes (using Bitnami Helm Charts) ● Message broker is Apache Kafka ● Customized to use non-OSS database ● JDK17 Stream and Batch applications ● GitOps ○ Applications are defined in git ○ Stream definition (DSL) and deployment info is in git ○ Deployments via Dataflow REST API
Management and Observability Monitoring ● Application metrics (uptime, resource utilization) ● Monitor topic lag ● Logs exported via sidecar application Scaling ● Make sure JVM properties are set (Heap memory, CPU count, GC, etc) ● Single instance of most apps ● No dynamic scaling yet
Questions?
© 2020 Spring. A VMware-backed project. Resources ● dataflow.spring.io ● github.com/spring-cloud/spring-cloud-dataflow ● dataflow.spring.io/docs/applications/pre-packaged/ ● via.vmw.com/sweet-streams
Thank you Contact me at pcorneil@vmware.com @corneil © 2020 Spring. A VMware-backed project.

Recommended

Dataflow.pptx
Dataflow.pptxDataflow.pptx
Dataflow.pptxSadeka Islam
 
Building cloud native data microservice
Building cloud native data microserviceBuilding cloud native data microservice
Building cloud native data microserviceNilanjan Roy
 
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...Brittany Ingram
 
AWS Lambda and Serverless framework: lessons learned while building a serverl...
AWS Lambda and Serverless framework: lessons learned while building a serverl...AWS Lambda and Serverless framework: lessons learned while building a serverl...
AWS Lambda and Serverless framework: lessons learned while building a serverl...Luciano Mammino
 
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...Kaxil Naik
 
My past-3 yeas-developer-journey-at-linkedin-by-iantsai
My past-3 yeas-developer-journey-at-linkedin-by-iantsaiMy past-3 yeas-developer-journey-at-linkedin-by-iantsai
My past-3 yeas-developer-journey-at-linkedin-by-iantsaiKim Kao
 
Delivering Cloud Native Batch Solutions - Dodd Pfeffer
Delivering Cloud Native Batch Solutions - Dodd PfefferDelivering Cloud Native Batch Solutions - Dodd Pfeffer
Delivering Cloud Native Batch Solutions - Dodd PfefferVMware Tanzu
 
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...Gary Arora
 

Recommended

Dataflow.pptx
Dataflow.pptxDataflow.pptx
Dataflow.pptxSadeka Islam
 
Building cloud native data microservice
Building cloud native data microserviceBuilding cloud native data microservice
Building cloud native data microserviceNilanjan Roy
 
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...How Docker Accelerates Continuous Development at ironSource: Containers #101 ...
How Docker Accelerates Continuous Development at ironSource: Containers #101 ...Brittany Ingram
 
AWS Lambda and Serverless framework: lessons learned while building a serverl...
AWS Lambda and Serverless framework: lessons learned while building a serverl...AWS Lambda and Serverless framework: lessons learned while building a serverl...
AWS Lambda and Serverless framework: lessons learned while building a serverl...Luciano Mammino
 
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...
Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...Kaxil Naik
 
My past-3 yeas-developer-journey-at-linkedin-by-iantsai
My past-3 yeas-developer-journey-at-linkedin-by-iantsaiMy past-3 yeas-developer-journey-at-linkedin-by-iantsai
My past-3 yeas-developer-journey-at-linkedin-by-iantsaiKim Kao
 
Delivering Cloud Native Batch Solutions - Dodd Pfeffer
Delivering Cloud Native Batch Solutions - Dodd PfefferDelivering Cloud Native Batch Solutions - Dodd Pfeffer
Delivering Cloud Native Batch Solutions - Dodd PfefferVMware Tanzu
 
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...Gary Arora
 
Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”Amazon Web Services
 
Game Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid MeetupGame Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid MeetupJelena Zanko
 
Story of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streamingStory of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streaminglohitvijayarenu
 
Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017Steef-Jan Wiggers
 
Ultimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on KubernetesUltimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on Kuberneteskloia
 
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS SummitDiscover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS SummitAmazon Web Services
 
Cloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive ApplicationsCloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive ApplicationsVMware Tanzu
 
Stream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksStream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksGuido Schmutz
 
Evolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at PinterestEvolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at PinterestHostedbyConfluent
 
[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQL[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQLWSO2
 
TenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingTenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingChen-en Lu
 
Deep dive into serverless on Google Cloud
Deep dive into serverless on Google CloudDeep dive into serverless on Google Cloud
Deep dive into serverless on Google CloudBret McGowen - NYC Google Developer Advocate
 
Supporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce HelixSupporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce HelixPerforce
 
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...MongoDB
 
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud PlatformonMárton Kodok
 
Cloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastCloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastDatabricks
 
Dynomite @ RedisConf 2017
Dynomite @ RedisConf 2017Dynomite @ RedisConf 2017
Dynomite @ RedisConf 2017Ioannis Papapanagiotou
 
Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2kartraj
 
Challenges In Modern Application
Challenges In Modern ApplicationChallenges In Modern Application
Challenges In Modern ApplicationRahul Kumar Gupta
 
Webinar september 2013
Webinar september 2013Webinar september 2013
Webinar september 2013Marc Gille
 
Cloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A WorkshopCloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A WorkshopCorneil du Plessis
 
QueryDSL - Lightning Talk
QueryDSL - Lightning TalkQueryDSL - Lightning Talk
QueryDSL - Lightning TalkCorneil du Plessis
 

More Related Content

Similar to Sweet Streams (Are made of this)

Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”Amazon Web Services
 
Game Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid MeetupGame Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid MeetupJelena Zanko
 
Story of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streamingStory of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streaminglohitvijayarenu
 
Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017Steef-Jan Wiggers
 
Ultimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on KubernetesUltimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on Kuberneteskloia
 
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS SummitDiscover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS SummitAmazon Web Services
 
Cloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive ApplicationsCloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive ApplicationsVMware Tanzu
 
Stream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksStream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksGuido Schmutz
 
Evolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at PinterestEvolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at PinterestHostedbyConfluent
 
[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQL[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQLWSO2
 
TenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingTenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingChen-en Lu
 
Supporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce HelixSupporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce HelixPerforce
 
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...MongoDB
 
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud PlatformonMárton Kodok
 
Cloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastCloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastDatabricks
 
Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2kartraj
 
Challenges In Modern Application
Challenges In Modern ApplicationChallenges In Modern Application
Challenges In Modern ApplicationRahul Kumar Gupta
 
Webinar september 2013
Webinar september 2013Webinar september 2013
Webinar september 2013Marc Gille
 

Similar to Sweet Streams (Are made of this) (20)

Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”Building Analytic Apps for SaaS: “Analytics as a Service”
Building Analytic Apps for SaaS: “Analytics as a Service”
 
Game Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid MeetupGame Analytics at London Apache Druid Meetup
Game Analytics at London Apache Druid Meetup
 
Story of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streamingStory of migrating event pipeline from batch to streaming
Story of migrating event pipeline from batch to streaming
 
Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017Web app job and functions - TUGAIT 2017
Web app job and functions - TUGAIT 2017
 
Ultimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on KubernetesUltimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on Kubernetes
 
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS SummitDiscover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
Discover MongoDB Atlas and MongoDB Stitch - DEM02-S - Mexico City AWS Summit
 
Cloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive ApplicationsCloud-Native Patterns for Data-Intensive Applications
Cloud-Native Patterns for Data-Intensive Applications
 
Stream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksStream Processing – Concepts and Frameworks
Stream Processing – Concepts and Frameworks
 
Evolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at PinterestEvolution of Real-time User Engagement Event Consumption at Pinterest
Evolution of Real-time User Engagement Event Consumption at Pinterest
 
[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQL[WSO2Con EU 2018] The Rise of Streaming SQL
[WSO2Con EU 2018] The Rise of Streaming SQL
 
TenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience SharingTenMax Data Pipeline Experience Sharing
TenMax Data Pipeline Experience Sharing
 
Deep dive into serverless on Google Cloud
Deep dive into serverless on Google CloudDeep dive into serverless on Google Cloud
Deep dive into serverless on Google Cloud
 
Supporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce HelixSupporting Digital Media Workflows in the Cloud with Perforce Helix
Supporting Digital Media Workflows in the Cloud with Perforce Helix
 
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
 
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
 
Cloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastCloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and Fast
 
Dynomite @ RedisConf 2017
Dynomite @ RedisConf 2017Dynomite @ RedisConf 2017
Dynomite @ RedisConf 2017
 
Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2Serverlessusecase workshop feb3_v2
Serverlessusecase workshop feb3_v2
 
Challenges In Modern Application
Challenges In Modern ApplicationChallenges In Modern Application
Challenges In Modern Application
 
Webinar september 2013
Webinar september 2013Webinar september 2013
Webinar september 2013
 

More from Corneil du Plessis

Cloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A WorkshopCloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A WorkshopCorneil du Plessis
 
Enhancements in Java 9 Streams
Enhancements in Java 9 StreamsEnhancements in Java 9 Streams
Enhancements in Java 9 StreamsCorneil du Plessis
 
Performance Comparison JVM Languages
Performance Comparison JVM LanguagesPerformance Comparison JVM Languages
Performance Comparison JVM LanguagesCorneil du Plessis
 
Microservices Patterns and Anti-Patterns
Microservices Patterns and Anti-PatternsMicroservices Patterns and Anti-Patterns
Microservices Patterns and Anti-PatternsCorneil du Plessis
 
Consume Spring Data Rest with Angularjs
Consume Spring Data Rest with AngularjsConsume Spring Data Rest with Angularjs
Consume Spring Data Rest with AngularjsCorneil du Plessis
 
Gradle: The Build System you have been waiting for!
Gradle: The Build System you have been waiting for!Gradle: The Build System you have been waiting for!
Gradle: The Build System you have been waiting for!Corneil du Plessis
 
Polyglot persistence with Spring Data
Polyglot persistence with Spring DataPolyglot persistence with Spring Data
Polyglot persistence with Spring DataCorneil du Plessis
 
Gradle: The Build system you have been waiting for
Gradle: The Build system you have been waiting forGradle: The Build system you have been waiting for
Gradle: The Build system you have been waiting forCorneil du Plessis
 
Dependency Injection in Spring in 10min
Dependency Injection in Spring in 10minDependency Injection in Spring in 10min
Dependency Injection in Spring in 10minCorneil du Plessis
 

More from Corneil du Plessis (15)

Cloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A WorkshopCloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
Cloud Native Applications for Cloud Foundry using Spring Cloud : A Workshop
 
QueryDSL - Lightning Talk
QueryDSL - Lightning TalkQueryDSL - Lightning Talk
QueryDSL - Lightning Talk
 
Enhancements in Java 9 Streams
Enhancements in Java 9 StreamsEnhancements in Java 9 Streams
Enhancements in Java 9 Streams
 
Reactive Spring 5
Reactive Spring 5Reactive Spring 5
Reactive Spring 5
 
Empathic API-Design
Empathic API-DesignEmpathic API-Design
Empathic API-Design
 
Performance Comparison JVM Languages
Performance Comparison JVM LanguagesPerformance Comparison JVM Languages
Performance Comparison JVM Languages
 
Microservices Patterns and Anti-Patterns
Microservices Patterns and Anti-PatternsMicroservices Patterns and Anti-Patterns
Microservices Patterns and Anti-Patterns
 
Consume Spring Data Rest with Angularjs
Consume Spring Data Rest with AngularjsConsume Spring Data Rest with Angularjs
Consume Spring Data Rest with Angularjs
 
The Evolution of Java
The Evolution of JavaThe Evolution of Java
The Evolution of Java
 
Gradle: The Build System you have been waiting for!
Gradle: The Build System you have been waiting for!Gradle: The Build System you have been waiting for!
Gradle: The Build System you have been waiting for!
 
Polyglot persistence with Spring Data
Polyglot persistence with Spring DataPolyglot persistence with Spring Data
Polyglot persistence with Spring Data
 
Data repositories
Data repositoriesData repositories
Data repositories
 
Gradle: The Build system you have been waiting for
Gradle: The Build system you have been waiting forGradle: The Build system you have been waiting for
Gradle: The Build system you have been waiting for
 
Dependency Injection in Spring in 10min
Dependency Injection in Spring in 10minDependency Injection in Spring in 10min
Dependency Injection in Spring in 10min
 
Spring Data in 10 minutes
Spring Data in 10 minutesSpring Data in 10 minutes
Spring Data in 10 minutes
 

Recently uploaded

Professional Resume Template for Software Developers
Professional Resume Template for Software DevelopersProfessional Resume Template for Software Developers
Professional Resume Template for Software DevelopersVinodh Ram
 
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)jennyeacort
 
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...gurkirankumar98700
 
Folding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a seriesFolding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a seriesPhilip Schwarz
 
chapter--4-software-project-planning.ppt
chapter--4-software-project-planning.pptchapter--4-software-project-planning.ppt
chapter--4-software-project-planning.pptkotipi9215
 
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...OnePlan Solutions
 
Implementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with AzureImplementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with AzureDinusha Kumarasiri
 
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样umasea
 
Building Real-Time Data Pipelines: Stream & Batch Processing workshop Slide
Building Real-Time Data Pipelines: Stream & Batch Processing workshop SlideBuilding Real-Time Data Pipelines: Stream & Batch Processing workshop Slide
Building Real-Time Data Pipelines: Stream & Batch Processing workshop SlideChristina Lin
 
What are the key points to focus on before starting to learn ETL Development....
What are the key points to focus on before starting to learn ETL Development....What are the key points to focus on before starting to learn ETL Development....
What are the key points to focus on before starting to learn ETL Development....kzayra69
 
Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)OPEN KNOWLEDGE GmbH
 
Unveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesUnveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesŁukasz Chruściel
 
Salesforce Certified Field Service Consultant
Salesforce Certified Field Service ConsultantSalesforce Certified Field Service Consultant
Salesforce Certified Field Service ConsultantAxelRicardoTrocheRiq
 
MYjobs Presentation Django-based project
MYjobs Presentation Django-based projectMYjobs Presentation Django-based project
MYjobs Presentation Django-based projectAnoyGreter
 
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdfGOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdfAlina Yurenko
 
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanySuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanyChristoph Pohl
 
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024StefanoLambiase
 
React Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief UtamaReact Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief UtamaHanief Utama
 
What is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need ItWhat is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need ItWave PLM
 

Recently uploaded (20)

Professional Resume Template for Software Developers
Professional Resume Template for Software DevelopersProfessional Resume Template for Software Developers
Professional Resume Template for Software Developers
 
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
 
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽‍❤️‍🧑🏻 89...
 
Folding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a seriesFolding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a series
 
chapter--4-software-project-planning.ppt
chapter--4-software-project-planning.pptchapter--4-software-project-planning.ppt
chapter--4-software-project-planning.ppt
 
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
 
Implementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with AzureImplementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with Azure
 
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
 
Building Real-Time Data Pipelines: Stream & Batch Processing workshop Slide
Building Real-Time Data Pipelines: Stream & Batch Processing workshop SlideBuilding Real-Time Data Pipelines: Stream & Batch Processing workshop Slide
Building Real-Time Data Pipelines: Stream & Batch Processing workshop Slide
 
What are the key points to focus on before starting to learn ETL Development....
What are the key points to focus on before starting to learn ETL Development....What are the key points to focus on before starting to learn ETL Development....
What are the key points to focus on before starting to learn ETL Development....
 
Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)
 
Unveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesUnveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New Features
 
Salesforce Certified Field Service Consultant
Salesforce Certified Field Service ConsultantSalesforce Certified Field Service Consultant
Salesforce Certified Field Service Consultant
 
MYjobs Presentation Django-based project
MYjobs Presentation Django-based projectMYjobs Presentation Django-based project
MYjobs Presentation Django-based project
 
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdfGOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
 
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanySuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
 
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
 
React Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief UtamaReact Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief Utama
 
What is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need ItWhat is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need It
 
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort ServiceHot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
 

Sweet Streams (Are made of this)

  • 1. Version 1.0 October 2022 Sweet Streams (Are made of this) Spring Cloud Data Flow - A gentle overview Copyright © 2022 VMware, Inc. or its affiliates.
  • 2. Cover w/ Image Agenda ● What is Spring Cloud Data Flow? ● Spring Batch ● Spring Task ● Spring Cloud Functions ● Spring Cloud Stream Applications ● Interview with a User ● Q+A
  • 3. What is Spring Cloud Data Flow? ● Microservice based Enterprise Application Integration. ○ Batch Jobs ○ Tasks ○ Streams
  • 4. What is Spring Cloud Data Flow? ● Orchestration and Deployment ○ Tanzu Application Service / Cloud Foundry ○ Kubernetes ○ Local
  • 5. What is Spring Cloud Data Flow? ● Observability ○ Micrometer ○ Prometheus ○ WaveFront ○ InfluxDB ○ More…
  • 6. Stream DSL file | splitter | csv-to-json: transform > :input-stream http > :input-stream :input-stream > filter | aggregator > :output-stream :output-stream > jdbc :output-stream > json-to-ws-notification: transform | websocket Legend Stream App Topic App Label
  • 12. Batch/Task Workflows ● DSL can define complex Batch and or Task application topologies in addition to a single Batch or Task application ● Each box is a Spring Batch or Task application. ● Application Flow can split and join, based on optional conditional expressions
  • 13. How did we build Spring Cloud Data Flow? Spring Cloud Data Flow Spring Boot Spring Cloud Spring Batch Spring Integration Spring AMQP Spring Kafka Spring Security Spring Data
  • 14. Spring Cloud Data Flow Architecture ● Data Flow UI, REST Client or Shell ● Data Flow Server ● Skipper Server ● Messaging Middleware
  • 17. Pre-packaged Stream Applications aggregator analytics bridge cassandra cdc-debezium dataflow-tasklauncher elasticsearch file filter ftp groovy header-enricher http http-request image- recognition jdbc jms load-generator log mail mongodb mqtt object-detection pgcopy rabbit redis router rsocket s3 script semantic-segmentation sftp splitter syslog tcp throughput time transform twitter-message twitter-search twitter-stream twitter-trend twitter-update wavefront websocket zeromq
  • 18. Pre-packaged Stream Applications aggregator dataflow-tasklauncher filter http jdbc splitter transform your-application
  • 20. Data ingest Data comes in via file or web service Once in the DB the batch process can consume Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum
  • 25. Design ● How do you eat an elephant? ● Enable more real-time processing ● Improve the feedback loop for failures ● Use Java ● Functional programming paradigm ● How do you eat an elephant?
  • 27. Stream processing System is built to be event driven Files are broken into individual events File and web service events are processed by streaming applications All streaming apps are custom. (Didn’t use any off the shelf applications) Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum
  • 28. Stream processing Failures can be handled on individual events Streaming apps are designed to be idempotent Load can be spread out much better Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Error handler
  • 29. Batch processing System is built to be event driven Files are broken into individual events Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Lorem ipsum Spring Batch
  • 30. Idempotent processing ● At some point you must write to the DB (sink) ● Processes are designed to be idempotent ● The sink that persists is designed to cater for items being replayed
  • 31. Developer experience ● Everything is a Spring Boot application (Our Devs love this) ● Changes can be tested locally ● Custom templates ● Dependency management (Solution BOM) ● Easy to understand applications that are small and do one thing ● Unit and integration testing is easy ● Containers Paving the road to production
  • 33. Deployment Data Flow deployment ● Deployed to Kubernetes (using Bitnami Helm Charts) ● Message broker is Apache Kafka ● Customized to use non-OSS database ● JDK17 Stream and Batch applications ● GitOps ○ Applications are defined in git ○ Stream definition (DSL) and deployment info is in git ○ Deployments via Dataflow REST API
  • 34. Management and Observability Monitoring ● Application metrics (uptime, resource utilization) ● Monitor topic lag ● Logs exported via sidecar application Scaling ● Make sure JVM properties are set (Heap memory, CPU count, GC, etc) ● Single instance of most apps ● No dynamic scaling yet
  • 36. © 2020 Spring. A VMware-backed project. Resources ● dataflow.spring.io ● github.com/spring-cloud/spring-cloud-dataflow ● dataflow.spring.io/docs/applications/pre-packaged/ ● via.vmw.com/sweet-streams
  • 37. Thank you Contact me at pcorneil@vmware.com @corneil © 2020 Spring. A VMware-backed project.

Editor's Notes

  1. Have you heard of Enterprise Integration Patterns? SCDF provides for the deployment and orchestration of batches, tasks and streams. These are deployed as microservice based Spring Boot Applications into Cloud Foundry / Tanzu Application Service, Kubernetes or as processes on a single machine.
  2. These are deployed as microservice based Spring Boot Applications into Cloud Foundry / Tanzu Application Service, Kubernetes or as processes on a single machine. The platform manages the availability and scalability
  3. Micrometer should be thought of as SLF4J for metrics.
  4. Spring Cloud Data Flow use Spring Boot, Spring Security, Spring Data and Spring Cloud Deployer modules.
  5. I would like to introduce you to one of the rarest sightings in Open source. Vincent Chegwidden works at Discovery using Spring Cloud Data Flow.
  6. Q: How did you get to use Spring Cloud Data Flow?
  7. Q: So I suppose you use or create Spring Cloud Stream applications…
  8. Q: Do you use any Spring Batch functionality?
  9. Q: Do any of the processors have side-effects?
  10. Q: What does the development cycle look like for your custom stream apps?
  11. Q: How do you deploy and manage SCDF and your solution?
  12. Future: Boot 3 support for Stream apps, Task and Batch apps and observability improvements. Data Flow interacting with multiple deployment environments. Environments may be connected…. Polyglot apps, Web Assembly, IoT, Edge.