This document discusses the rise of the Hadoop ecosystem. It outlines how the ecosystem has expanded from the original Hadoop components of HDFS for storage and MapReduce for distributed computation. New frameworks have emerged that allow for real-time queries, updates, and machine learning on big data. These include Spark, Storm, Drill, and streaming engines. The ecosystem is now a complex network of interoperable tools for storage, computation, analytics and machine learning on large datasets.

1. The Rise
of the
Hadoop
Ecosystem

2. Florian Douetteau
CEO Dataiku
DATA PREPARATION
MODELING
STATISTICS
VISUALIZATION
ALL-IN-ONE
DATA SCIENCE STUDIO

3. DRIVERS FOR
THE NEW “REAL-TIME“
HADOOP ECOSYSTEM
KEY TOOLS AND
FRAMEWORKS
TO BE AWARE OF

4. RAM - CPU - DISK

5. 2000 2013
1000$ / GB
6$ / GB
$10 / GB
$0.06 / GB
memory
divided by 150
disk cost
divided by 250
MAP
REDUCE
times
HACK
REDUCE
times

7. WHOLE DATA
REFINED DATA

8. NEEDLE IN HAYSTACK ?

9. REFINE BEFORE
USE

10. Web Site
– $1B revenue per year
– 10 Millions Unique Visitor per month
– 100.Millions orders / actions / per day
10TB
RAW DATA
1TB
REFINE DATA

11. FITS IN MEMORY
1TB

12. • GOOGLE
• 1 Circle
OPEN SOURCE
– YAHOO – IBM –
LINKEDIN - FACEBOOK
• 2 Circle
– STANDFORD
BERKELEY
– STARTUPS

13. 64m$
6.75m$
14m$
2m$
40m$
20m$
20.5m$
19m$
4m$
100m$
1.8m$
17m$
11m$
7.75m$
1.7m$
2013
2012
2011
2010
2009
$1B per year
Invested
in Big Data
TECH
223m$
301m$

15. HDFS
MAP
REDUCE
1. Safe Large Storage (HDFS)
2. Distributed computation
paradigm (Map Reduce)
3. Resilient long job
1. Disk-CPU locality aware
resource allocation
HADOOP =

16. LOVELY
TANGLED
TOGETHER

18. HDFS
YARN
map
reduce
provider
1
Other cluster
provider
…
THE NEW ECOSYSTEM

19. REALLY FASTER ?

20. REAL-TIME
QUERIES
REAL-TIME
UPDATES
FAST
MACHINE LEARNING

21. REAL-TIME
QUERIES
REAL-TIME
UPDATES
FAST
MACHINE LEARNING

22. DEVELOPPER
CAN
WAIT

23. BUSINESS
WON’T
WAIT

24. Not All Queries
are born
equals

25. MPP Database like performance
for Hadoop
- Created in 2012 by Cloudera
- x100 performance over Hive
(for certain queries)

26. Extensible architecture
for SQL Querying
• Started in 2013
• Apache Incubated Project
• Lucidworks
• Mapr
• ElasticSearch
• …
• Alpha Status
• Open architecture for supporting
SQL like queries to various data
sources:
• Cassandra
• MongoDB
• HDFS
• HBase
Apache DRILL

27. REAL-TIME
QUERIES
REAL-TIME
UPDATES
FAST
MACHINE LEARNING

29. Update the Model Once per week
using the whole history
Apply the model for each user
using the very last events
Real-Time
Navigation
Real-Time
Recommendation

30. STORM Reliable Distributed
Real-Time Computations
- Connect to a variety of data
sources (HDFS, RabbitMQ, JMS etc..)
- Run Computation in java (native) or
python, ruby, perl …
- Guarantees that events are taken
processed
- Distributes workload

31. Write Map-Reduce like program
and executing either in
• Batch
• Real-Time
• Hybrid Batch / Real-Time
• Open Sourced By Twitter in 2013
• Built on top of Storm (and Cascading)
• Program in Scala

32. REAL-TIME
QUERIES
REAL-TIME
UPDATES
FAST
MACHINE LEARNING

33. GOOD PUPILS ITERATE

34. ……..
……..
Stochastic Gradient Descent : ITERATE
K-Means : ITERATE
Pages Rank: ITERATE
……..

35. “Graph” Analytics in Memory
• Created at Carnegie-Mellon in 2009
• Generic Graph Traversal framework
• Packaged Machine Learning
- Recommender Systems
- Graph Analytics
- Clustering
• Easy Python Integration

36. In-Memory Distribution
Prediction Engine
Machine Learning
- Classification
- Regression
- Clustering
- R/Python easy
integration

37. Real-Time Resilient
Distributed Memory
Framework
• Abstraction with any
DAG operation on
data:
- Filter
- Map
- Reduce
- Cache

38. SHARK
MLBASE
STREAMING
Real-Time Queries
Real-Time Updates
In-Memory Learning
SPARK

39. HDFS
YARN
map reduce SPARK
GRAPHLAB
H2OSTREAMING
MLBASE
SHARK
PIG
HIVE
CASCADING
STORM
DRILL
otherstorage
IMPALA

40. dataiku.com
DATAIKU STAND A4
DEMO
DATA SCIENCE STUDIO
Questions now
or later
florian.douetteau@dataiku.com