From Data to Services at the Speed of Business: Applying cloud-native paradigm to combine fast data analytics with microservices architecture for hybrid workloads.
6. From Data to Services at the Speed of Business
Analytics
Machine Learning
Agility
Autonomy
Low total cost of ownership
Real-time
Just-in-time decisions
Time-sensitive data
New business models
Fast innovation
Value-driven alignment
7. Ingest
Data
The analytics and data-driven “value chain”
The key to becoming insight-driven is to optimize the analytics value chain
Analytics Action Performance
Perform Trigger Improve
r
Strategic thinking & High Value Questions
Design and Implementation
8. $13.01 forevery$1
a company spends on analytics, it
gets back spend on data
management and analytics
Source: MIT Sloan, NucleusResearch
Value of leveraging analytics and data-driven decisions
74%of firms say they want to be data-
driven, but only 23%are successful
Source: Forbes: Actionable Insight: Missing Link between Data and Value
2x [companies are twice] likely to
outperform their peers if they use
advanced analytics
Source: MIT Sloan
9. From Data to Services at the Speed of Business
Slow innovation path Lack of
realtime
view
Misalignment
between
analytics and
business
decisions
Top 3 Challenges
10. Businesses want real-time
view into critical data
Analytics infrastructures are
focused on data accumulation and
retrospective analysis
Paradox#1:Rear-viewmirrorarchitectures
11. Analytics-driven means:
experiment, fail fast, recover
fast and learn rapidly
Data science is 80% data
preparation and 20% analytics
Paradox#2:80/20data-to-analyticsratio
12. Analytics need tight
alignment with business
decisions
Most efforts are focused on
managing technology platforms
and data lake governance
Paradox#3:SOAdéjàvu
13. From Data to Services at the Speed of Business
Distributed Analytics
(Apache Spark)
Microservices-driven
Architecture
In-Memory Data and
Business Logic
Processing
Cloud-Native
and Agile
Infrasructure
Microservice-driven and In-Memory Computing
Approach?
15. Spark + In-Memory Data Grid
Large-scale distributed
analytics framework
Unified, scale-out, low-latency data store
Transactional capabilities:
ACID, Event-Driven, Rich Data
modeling
Microservices
16. 16
Elastic Scale-out In-Memory Storage
(Shared-nothing, Linear scalability, Elastic capacity)
Low latency and high throughput
(co-located ops, event-driven, fast indexing)
High availability and Resiliency
(auto-healing, multi-data center replication, fault tolerance)
Rich API and Query Language
(SQL, Spring, Java, .NET, C++)
GigaSpaces XAP In-Memory Data Grid
27. Unified Hybrid Transactional/Analytics Architecture
node 1
Spark master
Grid
master
node 2
Spark worker
Grid
Partition
node 3
Spark worker
Grid
Partition
Lightweight
workers,
small JVMs
Large JVMs,
Fast
indexing
• Push-down predicates (ultra-low latency processing,
30x performance improvement)
• Stateful data-360 sharing across analytics jobs
• Data-locality for high throughput
• Five 9s High Availability
28. Decoupled Hybrid Architecture
In-Memory Data Grid
Realtime Replication
• Scoring models
• Trigger actions
• Events
Transactions Analytics
• Useful when analytics are
mostly batch or long-
running queries.
• Analytics grid can be used
for frequent model training
(CPU intensive), without
impacting transactional
apps
• Flexibility in write-heavy
(transactions) and read-
heavy (analytics)
independent scaling Application
developers
Data Scientists &
Analysts
29. From Data to Services at the Speed of Business
Putting it all together
(GigaSpaces InsightEdge Platform)
30. GigaSpaces In-Memory Computing Platform
Core In-Memory Data Grid
Event
Processing
Data Models
(Spatial, POJO, JSON)
RPC &
Map/Reduce
Web
Containers
Storage-Class Memory
(3DXPoint)
Cluster Management and Service Discovery
Microservices (REST)
Event
Processing
Streaming
Machine
Learning
Spark SQL
Analytics & Big Data
SearchSQL/JDBC
Search and Query
RESTOrchestration
ManagementandMonitoring
SecurityandAuditing
WebMobile Devices
On-Premises Cloud Hybrid
Transactional Apps and
Customer Experience
Message Brokers and Data
Science Workbench
Business Intelligence and
Text Mining
31. Why In-Memory Data Grid?
SQL-99, Polyglot
Data & Search
Multi-Tiered Data
Storage
Cloud-Nativeand Horizontally
Scalable
• RAM
• SSD/Flash
• Storage-Class Memory
(3DXPoint)
• SQL ‘99
• Graph
• JSON
• POJO
• GeoSpatial
• Full Text
Distributed In-Grid Analytics
• SQL
• Streaming
• Machine Learning
• Graph Processing
• Deep Learning
• Textmining
• Geospatial
• In-Memory Event-Driven
Processing
• Distributed Tasks and Compute
Grid
• Real-time Web Services
• In-Memory Aggregations
Advanced In-Grid Transactions and Analytics
Processing Across Center, Edge and Cloud
32. GigaSpaces
Hadoop
Embracing an open source analytics ecosystem
Pick your own fast data architecture (lambda, kappa) and co-locate transaction processing
Kafka
Spark
Simplified Lambda Architecture
(Realtime + Historical)
33. Case Study: Magic Software
IoT Hub + Predictive Analytics (Automotive Telematics)
Challenge:
• Implement predictive analytics and anomaly detection
• Expand insight context through customer/data-360
integration
• Trigger transactional workflows based on prediction criteria
Solution:
• Simplified HTAP with Streaming data pipeline (3 tiers)
• IoT streaming analytics with 9s high availability
“GigaSpaces enables our
customers to simplify and
accelerate telemetry
ingestion, to gain full
business value from IoT
adoption.”
Yuval Lavi, VP of Innovation
Magic Software
http://www.magicsoftware.com
34. Key Takeaways
By the end of this presentation, you hopefully understood that:
➔ Microservices architectures provides data science agility!
Capturing business value from real-time insight can be accomplished by
embracing an agile open source tool chain coupled with in-memory data
grids.
➔ Insight platforms scale at the speed of business
Embracing insight platforms enables an agile path towards real-time digital
transformation and the data-driven enterprise.
➔ Try it all out – It’s open source!
http://insightedge.io / http://gigaspaces.com
http://github.com/InsightEdge
http://insightedge.slack.com
hello@insightedge.io
Book a demo:
35. Webinars
UPCOMING WEBINARS
Wed, Aug 23 | 2:00 PM EST
Machine Learning and Insight Platforms
RECORDED WEBINAR
From Data to Services at the Speed of
Business
Powering IoT Integration Solutions:
Turning Data from Sensors into Real-time
Actionable Insights
Real-time Microservices: From Zero to
Production in Under 3 Months
Visit our site to sign up:
www.gigaspaces.com/webinars
Gigaspaces is a software company focused on in-memory computing, fast data analytics, and insight platforms.
We were established in the early 2000s. We were one of the first to digitize wall street with real-time trade reconciliation, market data systems...etc.
We serve a portfolio of about 300 customers, 50 of which are fortune 500, and we're also an enabling technology to many OEM installations (mostly in Telco) with a deployment footprint in the thousands.
Our flagship product is XAP, and in-memory data grid. But our insight platform, InsightEdge, extends quite beyond that.
We have fused Spark with In-Memory Data Grid
In terms of companies that do business with GigaSpaces, you'll notice the who's who of the global 2000. And while they are diverse in their verticals, they are trying to......
They are trying to get from data to services....
So what does that mean?
So what does that typically look like?
To set the context...
......because the economic value of insight-driven transformation is undeniable
Recent research shows really interesting numbers for what you might call insight-driven businesses
From an ROI perspective, firms are seeing a %1300 ROI
The majority of those who haven’t become fully insight-driven, about 74%, already have plans for introducing analytics at every corner for their business
This is mainly due to the recognition that, having analytics, not only as means of differentiation, but as a fast innovation engine, to be twice as innovate and ourperform their peers.
So how come only 23% are successful??
We can look at some challenges....
So we see generally three paradoxes...
The first one is what I would call the "rear-view mirror architecture paradox"
The second is the 80/20 paradox. Or we can call it the "lean engineering" paradox.
Finally, what we would call the "SOA" deja vu paradox
For those of us who can remember. If we take a trip down the memory lane to 2008-2009, remembering the days of when SOA had big promises of transforming the enterprise by ALIGNING BUSINESS WITH ITA THROUGH SERVICES ORIENTATION. History is repeating itself.
So can we take different approaches?
For those not familiar, in-memory computing means using RAM as the primary storage medium for business and analytics. There by eliminating any form of Disk I/O or network I/O latency, therefore operating at millisecond latencies at very high throughput.
So while traditionally, we were
This approach consists of two components
all common API
tap into other data stores on demand
Data science is in high demand, but short supply – so the ability to leverage the know how, capability, and production readiness eliminates a lot of pain points.
Goal is to provide a unified environment where application developers and data scientists can collaborate.
Data science by itself is an iterative activity which requires a lot of trial and error