BI congres 2016-2: Diving into weblog data with SAS on Hadoop - Lisa Truyers - Keyrus

BIG DATA ANALYTICS
BUSINESS INTELLIGENCE
INFORMATION MANAGEMENT
PERFORMANCE MANAGEMENT

© Copyright 2015 – Keyrus 2
DIVING INTO WEBLOG DATA WITH SAS ON
HADOOP
Lisa Truyers, Data Scientist Consultant at Keyrus
March 24, 2016
Logo

Project summary
WHO HAS EVER TRIED TO OPEN A 1 GB FILE ON A COMPUTER?

What is Hadoop?
Project summary
Components of the Hadoop-SAS framework
Setup to load data
Benchmarks
Lessons learned
AGENDA

PROS
 Open-source software framework
 Storage and large-scale data processing
 Easy and economic scaling
 Both structured and unstructured data
 Low-cost commodity hardware
 Starts multiple copies of the same task for
the same block of data
What is Hadoop?
51% OF COMPANIES THINKS ABOUT INTEGRATING
HADOOP IN THEIR COMPANY BY 2016
Philip Russom, TDWI Best Practices Report= Integrating Hadoop into Business

CONS
 Management and high-availability
capabilities are just starting to emerge
 Data security is fragmented
 MapReduce is very batch-oriented
 No easy-to-use, full-feature tools for data
integration, data cleansing, governance
and metadata
 Lacking skilled professionals
What is Hadoop?
MANAGE THE DATA AND USE ANALYTICS TO QUICKLY
IDENTIFY PREVIOUSLY UNKNOWN INSIGHTS: ACCESS
THE DIFFERENT TOOLS OF SAS

WHAT ARE COMPANIES DOING WITH HADOOP?
The percentages mentioned here cover the whole world, not only Europe.
What is Hadoop?
What? Percentage
Data warehouse extensions 46 %
Data exploration and discovery 46 %
Data staging for data warehousing and data integration 39 %
Data lake 39 %
Queryable archive for non-traditional data 36 %
Computational platform and sandbox for advanced analytics 33 %

WHY IS HADOOP (NOT) IMPORTANT?
“Cost savings. Linear scalability. Evaluate ‘the hype’ practically. Complement BI.”
BI architect, telecom, Europe
“Reduces cost of data. New ability to query big data sets. Supply chain improvements. Predictive
analytics.”
Vice president, food and beverage, Asia
“Our existing infrastructure cannot handle the tenfold increase in data volumes.”
Data strategy manager, hospitality, US
“It’s important to realize the potential of big data and to explore new business opportunities.”
Data specialist, consulting, Asia
What is Hadoop?

What is Hadoop?
Project summary
Setup to load data
Benchmarks
Lessons learned
AGENDA

INTRODUCTION
Project summary
1. Discover web traffic data
• Discover web traffic data
• Sheer volume of data makes it impossible to analyse at the moment
• Prove the added value of a combined Hadoop – SAS environment
2. Lead generation
• More business oriented: scoring a neural network model takes one hour on daily basis
• Reducing this time

Project summary
What is Hadoop?
Setup to load data
Benchmarks
Lessons learned
AGENDA

HADOOP COMPONENTS
HBASE PIG HIVE &
HCATALOG
MAP REDUCE
HDFS
AMBARI
OOZIE
FLUME
SQOOP
NFS
WebHDFS
YARN
Base SAS & SAS/ACCESS® to Hadoop™
SAS Metadata
SAS Metadata
SAS IMSTAT for
Hadoop
SAS® Visual Analytics &
Statistics
SAS® LASR™ Analytic
Server
SAS® High-
Performance
Analytic Procedures
SAS® Enterprise
Guide®

HADOOP COMPONENTS
HBASE PIG HIVE &
HCATALOG
MAP REDUCE
HDFS
AMBARI
OOZIE
FLUME
SQOOP
NFS
WebHDFS
YARN
SAS Metadata
SAS Metadata
SAS IMSTAT for
Hadoop
Statistics
Server
SAS® High-
Performance
Analytic Procedures
SAS® Enterprise
Guide®

SAS COMPONENTS
HBASE PIG HIVE &
HCATALOG
MAP REDUCE
HDFS
AMBARI
OOZIE
FLUME
SQOOP
NFS
WebHDFS
YARN
SAS Metadata
Server
SAS® High-
Performance
Analytic Procedures
SAS Metadata
SAS IMSTAT for
Hadoop
Statistics
SAS® Enterprise
Guide®

Project summary
What is Hadoop?
Setup to load data
Benchmarks
Lessons learned
AGENDA

FULL PROCESS
Setup to load data
Day
A Partitioned, non-parsed for day-files
C Partitioned, parsed for day-files
Hour
B Partitioned, non-parsed for hour-files
D Partitioned, parsed for hour-files

Setup to load data

PROCESS C
Setup to load data
Delete HIVE
Table
Transfer to
Hadoop
Parse data Merge Loop

Project summary
What is Hadoop?
SAS-tools used in this project
Setup to load data
Benchmarks
Lessons learned
AGENDA

HADOOP COMPARED TO SERVER
Server
 Query test one day: 35 seconds
 Parsing data on one day: 15 minutes
 Parsing of one week: 4hours 30 minutes
Benchmarks
Hadoop
 Query test on one day: 35 seconds
 Parsing data on one day: 15 minutes
 Parsing of one week: 53 minutes
MORE TIME NEEDED FOR EXTRA BENCHMARKS

Project summary
What is Hadoop?
SAS-tools used in this project
Setup to load data
Benchmarks
Lessons learned
AGENDA

Teamwork is key
• Set-up Hadoop cluster with
Hadoop-experts
• Install SAS with experts from
the company
SAS ON HADOOP
 In SAS, take your time to set the correct
variable length
 Choose the strength of the cluster
rationally
 Create Benchmarks on both environments
(server VS Hadoop) early on so a good
comparison can be done and the correct
decision can be taken
 Data must be large enough on Hadoop to
see a difference
Lessons learned

THANK YOU FOR YOUR ATTENTION
To contact us
www.keyrus.com
contact@keyrus.com

BI congres 2016-2: Diving into weblog data with SAS on Hadoop - Lisa Truyers - Keyrus

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to BI congres 2016-2: Diving into weblog data with SAS on Hadoop - Lisa Truyers - Keyrus

Similar to BI congres 2016-2: Diving into weblog data with SAS on Hadoop - Lisa Truyers - Keyrus (20)

Recently uploaded

Recently uploaded (20)

BI congres 2016-2: Diving into weblog data with SAS on Hadoop - Lisa Truyers - Keyrus