SlideShare a Scribd company logo
1 of 19
Download to read offline
Big Data and Business Intelligence
      Must Converge

    Tony Baer

    tony.baer@ovum.com

    March 6, 2013




1                               © Copyright Ovum. All rights reserved. Ovum is a subsidiary of Informa plc.
Agenda



        Challenges traditional data stewardship practice

        Privacy – is all the world a stage?

        Limits to data lifecycle?

        Data quality: the big, the bad, the ugly – and it all might be good!




2                                                         © Copyright Ovum. All rights reserved. Ovum is an Informa business.
Data stewardship challenges –
    What’s old is new

    Remember?

       Back to undifferentiated ‘gobblobs’ of data

       Programmatic access reigns

       File systems, not (always) tables              10.102.8.152 - - [05/Nov/2003:00:19:54 -0500] "GET
                                                       /inventory/index.jsp HTTP/1.1" 200 4028
                                                       "http://www.mycompany.com/index.jsp" "Mozilla/4.08 [en]
                                                       (Win98; I ;Nav)"

       Batch is back                                  192.168.114.201, -, 03/20/01, 7:55:20, W3SVC2, SALES1,
                                                       172.21.13.45, 4502, 163, 3223, 200, 0, GET,/DeptLogo.gif,
                                                       -, 172.16.255.255, anonymous, 03/20/01, 23:58:11,
                                                       MSFTPSVC, SALES1, 172.16.255.255, 60, 275, 0, 0,

    But…                                                           if index(tempvalue,'?') then
                                                                   tempvalue=scan(tempvalue,1,'?');
                                                                   else if index(tempvalue,'&')>1 then
                                                                   tempvalue=scan(tempvalue,1,'&');

       Volume, variety, velocity, and where’s the
        value??

       Just because you can, should you?

3                                                     © Copyright Ovum. All rights reserved. Ovum is an Informa business.
Data stewardship questions for Big Data


       Can we, should we “control” this data?

       Are there limits to how much we should know?

       Can we just keep piling up data forever?

       Can we cleanse terabytes of data?

       Do we still need “good” data?




4                                                      © Copyright Ovum. All rights reserved. Ovum is an Informa business.
Agenda

        Challenges traditional data stewardship practice

        Privacy – is all the world a stage?

        Limits to data lifecycle?

        Data quality: the big, the bad, the ugly – and it all might be good!




5                                                         © Copyright Ovum. All rights reserved. Ovum is an Informa business.
Privacy –
    the more things change…

    “You have zero privacy
    anyway…. Get over it”
        -- Scott McNealy, 1999




                                 Facebook does not actually
                                 delete images… but instead
                                 merely removes the links – a fix
                                 “is in sight”
                                                         -- ZDNet, 2/6/12

                                 Facebook agrees to 20 years of
                                 federal privacy audits
                                                          -- NY Times, 11/29/11



6                                  © Copyright Ovum. All rights reserved. Ovum is an Informa business.
What privacy?



    Florida made $63m last
    year by selling DMV
    information (name, date
    of birth, type of vehicle
    driven) to companies like
    LexusNexus & Shadow
    Soft.

    -- Terence Craig   & Mary Ludloff
    Privacy and Big Data
    (O’Reilly Media, 2011)




7                                       © Copyright Ovum. All rights reserved. Ovum is an Informa business.
Big Data privacy 101 –
    Don’t be creepy

       Governance problem first,          How Companies Learn Your
        technology second                         Secrets

       Understand the relationship
        with your customers & business
        partners

       Keep communications in
        context

       Don’t catch your customers by      “My daughter got this in the mail!” he
        surprise                           said. “She’s still in high school, and
                                           you’re sending her coupons for baby
                                           clothes and cribs? Are you trying to
       The law still trying to catch up   encourage her to get pregnant?”
                                                           -- NY Times 2/16/12

8                                                   © Copyright Ovum. All rights reserved. Ovum is an Informa business.
Agenda

        Challenges traditional data stewardship practice

        Privacy – is all the world a stage?

        Limits to data lifecycle?

        Data quality: the big, the bad, the ugly – and it all might be good!




9                                                         © Copyright Ovum. All rights reserved. Ovum is an Informa business.
Data lifecycle –
     How long can this go on?

        Google, Yahoo, Facebook, etc.
         don’t deprecate web data

        Hadoop designed for
         economical scale-out

        Moore’s Law, declining cost of
         storage

        Is Hadoop Archive the answer?

        Is Hadoop the new tape?




Management & skills will be the limit     Aerial view of Quincy, WA data ctrs


10                                                               © Copyright Ovum. All rights reserved. Ovum is an Informa business.
Agenda

         Challenges traditional data stewardship practice

         Privacy – is all the world a stage?

         Limits to data lifecycle?

         Data quality: the big, the bad, the ugly – and it all might be
          good!




11                                                       © Copyright Ovum. All rights reserved. Ovum is an Informa business.
Data Quality & Hadoop –
     Big Quality Questions

        Can we cleanse terabytes of data?

        Do we still need “good” data?

        Are there new approaches to cleansing Big Data?




12                                                    © Copyright Ovum. All rights reserved. Ovum is an Informa business.
Framing the issue

          “Garbage in, garbage out,’ but DW forced the
           issue

          Traditional approaches
                 Profiling, cleansing, MDM

          DW vs. Hadoop data quality challenges
                 Known data sets & known criteria vs. vaguely known
                 Bounded vs. less bounded tasks

          Limitations of MapReduce*
                 Cleansing & transformation within a single Map
                  operation;
                 Profiling & matching of unstructured data
                 Matching of data in operations without inter-process
                  communications

         *Source: David Loshin, "Hadoop and Data Quality, Data Integration, Data Analysis" at
         http://www.dataroundtable.com/?p=8841


13                                                                                    © Copyright Ovum. All rights reserved. Ovum is an Informa business.
Is data quality necessary for Hadoop?


        The App
           How mission-critical?
           Regulatory compliance impacts?
           What degree of business impact?

        The Data
           The 4V’s (volume, variety, velocity,
            value) determine what approaches
            to quality are feasible




14                                                 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
Examples


        Web ad placement optimization

        Counter-party risk management
         for capital markets

        Customer sentiment analysis

        Managing smart utility grids or
         urban infrastructure




15                                         © Copyright Ovum. All rights reserved. Ovum is an Informa business.
Bad data may be good


        Sensory data
           Outlier or drift?
           Time to recalibrate devices?
           Time to perform preventive
            maintenance?
           Are new/unaccounted environmental
            factors skewing readings?

        Human-readable data
           Flawed concept of reality?
           Flawed assumptions on data meaning?
           Changes producing ‘new norm’


16                                                © Copyright Ovum. All rights reserved. Ovum is an Informa business.
Big Data quality in Hadoop –
     Emergent approaches

        Crowdsourcing data –
           Collect data far & wide from as many diverse sources as possible. Torrents of data
            overcome the noise.
           Comparative trend analysis of incoming streams to dynamically ID the norm or
            sweet spot of “good” data
        Apply data science to “correct the dots”
           Don’t go record by record. Statistically analyze the data set in aggregate.
           Iteratively analyze & re-analyze nature of data, keep analyzing outliers
           Apply off-the-wall approaches
        Enterprise Architectural approach
           Semantic (domain) model-driven
           Apply cleansing logic at run time
           Critical for sensitive, regulatory-driven apps



17                                                                   © Copyright Ovum. All rights reserved. Ovum is an Informa business.
Summary


        Challenges traditional data stewardship practice
            Combination of old & new
        Privacy – is all the world a stage?
            Best practices, legal requirements still in flux
            Don’t be creepy!
        Limits to data lifecycle?
            Few enterprises are Google or Facebook
            Ability to manage large infrastructure will be major limit
        Data quality
            Strategy depends on type of app & data set(s)
            A spectrum of approaches -- from none to classic ETL to aggregate statistical
            No single silver bullet



18                                                                        © Copyright Ovum. All rights reserved. Ovum is an Informa business.
Disclaimer


     All Rights Reserved.

     No part of this publication may be reproduced, stored in a retrieval system or
     transmitted in any form by any means, electronic, mechanical, photocopying,
     recording or otherwise, without the prior permission of the publisher, Ovum
     (an Informa business).

     The facts of this report are believed to be correct at the time of publication but
     cannot be guaranteed. Please note that the findings, conclusions and
     recommendations that Ovum delivers will be based on information gathered in
     good faith from both primary and secondary sources, whose accuracy we are not
     always in a position to guarantee. As such Ovum can accept no liability whatever
     for actions taken based on any information that may subsequently prove to be
     incorrect.




19                                                             © Copyright Ovum. All rights reserved. Ovum is an Informa business.

More Related Content

What's hot

Defensible rim disposal leads to effective discovery responses - 2011.08.09
Defensible rim disposal leads to effective discovery responses - 2011.08.09Defensible rim disposal leads to effective discovery responses - 2011.08.09
Defensible rim disposal leads to effective discovery responses - 2011.08.09Alfresco Software
 
03 2010 Online Buyer 101 Webinar
03 2010 Online Buyer 101 Webinar03 2010 Online Buyer 101 Webinar
03 2010 Online Buyer 101 WebinarBob Chaput
 
Information Management in the Age of Big Data
Information Management in the Age of Big DataInformation Management in the Age of Big Data
Information Management in the Age of Big Databigdatasyd
 
What does it take to engage employees and customers?
What does it take to engage employees and customers?What does it take to engage employees and customers?
What does it take to engage employees and customers?John Mancini
 
Information Management on Mobile Steroids
Information Management on Mobile SteroidsInformation Management on Mobile Steroids
Information Management on Mobile SteroidsJohn Mancini
 

What's hot (7)

Defensible rim disposal leads to effective discovery responses - 2011.08.09
Defensible rim disposal leads to effective discovery responses - 2011.08.09Defensible rim disposal leads to effective discovery responses - 2011.08.09
Defensible rim disposal leads to effective discovery responses - 2011.08.09
 
Horse meat or beef? (3) D Murphy, National Grid, 21/3/13
Horse meat or beef? (3) D Murphy, National Grid, 21/3/13Horse meat or beef? (3) D Murphy, National Grid, 21/3/13
Horse meat or beef? (3) D Murphy, National Grid, 21/3/13
 
03 2010 Online Buyer 101 Webinar
03 2010 Online Buyer 101 Webinar03 2010 Online Buyer 101 Webinar
03 2010 Online Buyer 101 Webinar
 
Information Management in the Age of Big Data
Information Management in the Age of Big DataInformation Management in the Age of Big Data
Information Management in the Age of Big Data
 
What does it take to engage employees and customers?
What does it take to engage employees and customers?What does it take to engage employees and customers?
What does it take to engage employees and customers?
 
Information Management on Mobile Steroids
Information Management on Mobile SteroidsInformation Management on Mobile Steroids
Information Management on Mobile Steroids
 
Information Overload Phenomena
Information Overload PhenomenaInformation Overload Phenomena
Information Overload Phenomena
 

Viewers also liked

Ovum Trends to Watch 2014: Enterprise Collaboration and The Workplace of the ...
Ovum Trends to Watch 2014: Enterprise Collaboration and The Workplace of the ...Ovum Trends to Watch 2014: Enterprise Collaboration and The Workplace of the ...
Ovum Trends to Watch 2014: Enterprise Collaboration and The Workplace of the ...Richard Edwards
 
Global Mega Trends – Driving Enterprise Mobility
Global Mega Trends – Driving Enterprise MobilityGlobal Mega Trends – Driving Enterprise Mobility
Global Mega Trends – Driving Enterprise MobilityMovate
 
Ovum Telecom Advisory Overview General
Ovum Telecom Advisory Overview GeneralOvum Telecom Advisory Overview General
Ovum Telecom Advisory Overview GeneralChris Upjohn
 
Best of dynami in Atlanta (What's New!)
Best of dynami in Atlanta (What's New!)Best of dynami in Atlanta (What's New!)
Best of dynami in Atlanta (What's New!)Kfjones
 
Jumping into TADHack and Telecom App Development - NGSP 2015
Jumping into TADHack and Telecom App Development - NGSP 2015Jumping into TADHack and Telecom App Development - NGSP 2015
Jumping into TADHack and Telecom App Development - NGSP 2015Alan Quayle
 
Totango and OVUM present: Beyond CRM
Totango and OVUM present: Beyond CRMTotango and OVUM present: Beyond CRM
Totango and OVUM present: Beyond CRMTotango
 
The new digital workspace: An opportunity not to be squandered
The new digital workspace: An opportunity not to be squanderedThe new digital workspace: An opportunity not to be squandered
The new digital workspace: An opportunity not to be squanderedRichard Edwards
 
Next Generation Service Platform Summary 2015
Next Generation Service Platform Summary 2015Next Generation Service Platform Summary 2015
Next Generation Service Platform Summary 2015Alan Quayle
 
The Latest Shifts in B2B Buyer Behavior: New Trends and Real-World Insights f...
The Latest Shifts in B2B Buyer Behavior: New Trends and Real-World Insights f...The Latest Shifts in B2B Buyer Behavior: New Trends and Real-World Insights f...
The Latest Shifts in B2B Buyer Behavior: New Trends and Real-World Insights f...Demandbase
 
Applying eTOM (enhanced Telecom Operations Map) Framework to Non-Telecommunic...
Applying eTOM (enhanced Telecom Operations Map) Framework to Non-Telecommunic...Applying eTOM (enhanced Telecom Operations Map) Framework to Non-Telecommunic...
Applying eTOM (enhanced Telecom Operations Map) Framework to Non-Telecommunic...Alan McSweeney
 

Viewers also liked (12)

Ovum Trends to Watch 2014: Enterprise Collaboration and The Workplace of the ...
Ovum Trends to Watch 2014: Enterprise Collaboration and The Workplace of the ...Ovum Trends to Watch 2014: Enterprise Collaboration and The Workplace of the ...
Ovum Trends to Watch 2014: Enterprise Collaboration and The Workplace of the ...
 
Global Mega Trends – Driving Enterprise Mobility
Global Mega Trends – Driving Enterprise MobilityGlobal Mega Trends – Driving Enterprise Mobility
Global Mega Trends – Driving Enterprise Mobility
 
Ovum Telecom Advisory Overview General
Ovum Telecom Advisory Overview GeneralOvum Telecom Advisory Overview General
Ovum Telecom Advisory Overview General
 
Best of dynami in Atlanta (What's New!)
Best of dynami in Atlanta (What's New!)Best of dynami in Atlanta (What's New!)
Best of dynami in Atlanta (What's New!)
 
Quiénes somos
Quiénes somosQuiénes somos
Quiénes somos
 
Jumping into TADHack and Telecom App Development - NGSP 2015
Jumping into TADHack and Telecom App Development - NGSP 2015Jumping into TADHack and Telecom App Development - NGSP 2015
Jumping into TADHack and Telecom App Development - NGSP 2015
 
Totango and OVUM present: Beyond CRM
Totango and OVUM present: Beyond CRMTotango and OVUM present: Beyond CRM
Totango and OVUM present: Beyond CRM
 
The new digital workspace: An opportunity not to be squandered
The new digital workspace: An opportunity not to be squanderedThe new digital workspace: An opportunity not to be squandered
The new digital workspace: An opportunity not to be squandered
 
Next Generation Service Platform Summary 2015
Next Generation Service Platform Summary 2015Next Generation Service Platform Summary 2015
Next Generation Service Platform Summary 2015
 
The Latest Shifts in B2B Buyer Behavior: New Trends and Real-World Insights f...
The Latest Shifts in B2B Buyer Behavior: New Trends and Real-World Insights f...The Latest Shifts in B2B Buyer Behavior: New Trends and Real-World Insights f...
The Latest Shifts in B2B Buyer Behavior: New Trends and Real-World Insights f...
 
Applying eTOM (enhanced Telecom Operations Map) Framework to Non-Telecommunic...
Applying eTOM (enhanced Telecom Operations Map) Framework to Non-Telecommunic...Applying eTOM (enhanced Telecom Operations Map) Framework to Non-Telecommunic...
Applying eTOM (enhanced Telecom Operations Map) Framework to Non-Telecommunic...
 
Technology Vision 2017 - Overview
Technology Vision 2017 - OverviewTechnology Vision 2017 - Overview
Technology Vision 2017 - Overview
 

Similar to TDWI NYC Chapter - Tony Baer Ovum on Big data, Data quality, and BI Convergence

Making Big Data a First Class citizen in the enterprise
Making Big Data a First Class citizen in the enterpriseMaking Big Data a First Class citizen in the enterprise
Making Big Data a First Class citizen in the enterpriseTony Baer
 
An Overview of BigData
An Overview of BigDataAn Overview of BigData
An Overview of BigDataValarmathi V
 
Big data and the data quality imperative
Big data and the data quality imperativeBig data and the data quality imperative
Big data and the data quality imperativeTrillium Software
 
Action from Insight - Joining the 2 Percent Who are Getting Big Data Right
Action from Insight - Joining the 2 Percent Who are Getting Big Data RightAction from Insight - Joining the 2 Percent Who are Getting Big Data Right
Action from Insight - Joining the 2 Percent Who are Getting Big Data RightStampedeCon
 
Ibm 1129-the big data zoo
Ibm 1129-the big data zooIbm 1129-the big data zoo
Ibm 1129-the big data zooAccenture
 
Ibm 1129-the big data zoo
Ibm 1129-the big data zooIbm 1129-the big data zoo
Ibm 1129-the big data zooAccenture
 
The REAL Impact of Big Data on Privacy
The REAL Impact of Big Data on PrivacyThe REAL Impact of Big Data on Privacy
The REAL Impact of Big Data on PrivacyClaudiu Popa
 
Cloud migration risk
Cloud migration riskCloud migration risk
Cloud migration riskEdgevalue
 
The Bigger They Are The Harder They Fall
The Bigger They Are The Harder They FallThe Bigger They Are The Harder They Fall
The Bigger They Are The Harder They FallTrillium Software
 
Veritas corporate brochure emea
Veritas corporate brochure emeaVeritas corporate brochure emea
Veritas corporate brochure emeaHayatollah Ayoubi
 
Big Dat The Good And The Ugly
Big Dat The Good And The UglyBig Dat The Good And The Ugly
Big Dat The Good And The UglyCarla Potier
 
DAMA Webinar: What Does "Manage Data Assets" Really Mean?
DAMA Webinar: What Does "Manage Data Assets" Really Mean?DAMA Webinar: What Does "Manage Data Assets" Really Mean?
DAMA Webinar: What Does "Manage Data Assets" Really Mean?DATAVERSITY
 
Big data introduction
Big data introductionBig data introduction
Big data introductionChirag Ahuja
 
Less is More: Behind the Data at Risk I/O
Less is More: Behind the Data at Risk I/OLess is More: Behind the Data at Risk I/O
Less is More: Behind the Data at Risk I/OMichael Roytman
 
From Near to Maturity - Presentation to European Data Forum
From Near to Maturity - Presentation to European Data ForumFrom Near to Maturity - Presentation to European Data Forum
From Near to Maturity - Presentation to European Data ForumCastlebridge Associates
 
Big data security
Big data securityBig data security
Big data securityCloudBees
 
Fontys Eric van Tol
Fontys Eric van TolFontys Eric van Tol
Fontys Eric van TolTalentEvent
 

Similar to TDWI NYC Chapter - Tony Baer Ovum on Big data, Data quality, and BI Convergence (20)

Making Big Data a First Class citizen in the enterprise
Making Big Data a First Class citizen in the enterpriseMaking Big Data a First Class citizen in the enterprise
Making Big Data a First Class citizen in the enterprise
 
An Overview of BigData
An Overview of BigDataAn Overview of BigData
An Overview of BigData
 
Big data and the data quality imperative
Big data and the data quality imperativeBig data and the data quality imperative
Big data and the data quality imperative
 
Action from Insight - Joining the 2 Percent Who are Getting Big Data Right
Action from Insight - Joining the 2 Percent Who are Getting Big Data RightAction from Insight - Joining the 2 Percent Who are Getting Big Data Right
Action from Insight - Joining the 2 Percent Who are Getting Big Data Right
 
Ibm 1129-the big data zoo
Ibm 1129-the big data zooIbm 1129-the big data zoo
Ibm 1129-the big data zoo
 
Ibm 1129-the big data zoo
Ibm 1129-the big data zooIbm 1129-the big data zoo
Ibm 1129-the big data zoo
 
The value of our data
The value of our dataThe value of our data
The value of our data
 
The REAL Impact of Big Data on Privacy
The REAL Impact of Big Data on PrivacyThe REAL Impact of Big Data on Privacy
The REAL Impact of Big Data on Privacy
 
Cloud migration risk
Cloud migration riskCloud migration risk
Cloud migration risk
 
The Bigger They Are The Harder They Fall
The Bigger They Are The Harder They FallThe Bigger They Are The Harder They Fall
The Bigger They Are The Harder They Fall
 
Veritas corporate brochure emea
Veritas corporate brochure emeaVeritas corporate brochure emea
Veritas corporate brochure emea
 
Big Dat The Good And The Ugly
Big Dat The Good And The UglyBig Dat The Good And The Ugly
Big Dat The Good And The Ugly
 
DAMA Webinar: What Does "Manage Data Assets" Really Mean?
DAMA Webinar: What Does "Manage Data Assets" Really Mean?DAMA Webinar: What Does "Manage Data Assets" Really Mean?
DAMA Webinar: What Does "Manage Data Assets" Really Mean?
 
Big data introduction
Big data introductionBig data introduction
Big data introduction
 
Less is More: Behind the Data at Risk I/O
Less is More: Behind the Data at Risk I/OLess is More: Behind the Data at Risk I/O
Less is More: Behind the Data at Risk I/O
 
From Near to Maturity - Presentation to European Data Forum
From Near to Maturity - Presentation to European Data ForumFrom Near to Maturity - Presentation to European Data Forum
From Near to Maturity - Presentation to European Data Forum
 
Big data security
Big data securityBig data security
Big data security
 
Fontys Eric van Tol
Fontys Eric van TolFontys Eric van Tol
Fontys Eric van Tol
 
A data powered future
A data powered futureA data powered future
A data powered future
 
Data mining with big data
Data mining with big dataData mining with big data
Data mining with big data
 

More from Fitzgerald Analytics, Inc.

Profiting from customer profitability + big data fitzgerald analytics
Profiting from customer profitability + big data fitzgerald analyticsProfiting from customer profitability + big data fitzgerald analytics
Profiting from customer profitability + big data fitzgerald analyticsFitzgerald Analytics, Inc.
 
2013 12-05 data-driven innovation - fitzgerald analytics workshop at gilbane ...
2013 12-05 data-driven innovation - fitzgerald analytics workshop at gilbane ...2013 12-05 data-driven innovation - fitzgerald analytics workshop at gilbane ...
2013 12-05 data-driven innovation - fitzgerald analytics workshop at gilbane ...Fitzgerald Analytics, Inc.
 
Analytics in Financial Services - Behavioral Finance Event - Data Visualizati...
Analytics in Financial Services - Behavioral Finance Event - Data Visualizati...Analytics in Financial Services - Behavioral Finance Event - Data Visualizati...
Analytics in Financial Services - Behavioral Finance Event - Data Visualizati...Fitzgerald Analytics, Inc.
 
Analytics in financial services prez behavioral finance + data visualizatio...
Analytics in financial services prez   behavioral finance + data visualizatio...Analytics in financial services prez   behavioral finance + data visualizatio...
Analytics in financial services prez behavioral finance + data visualizatio...Fitzgerald Analytics, Inc.
 
Jaime Fitzgerald on Data-Driven Customer Experience in Financial Services and...
Jaime Fitzgerald on Data-Driven Customer Experience in Financial Services and...Jaime Fitzgerald on Data-Driven Customer Experience in Financial Services and...
Jaime Fitzgerald on Data-Driven Customer Experience in Financial Services and...Fitzgerald Analytics, Inc.
 
Data Discovery for Big Big Insights - Tableau Webinar Slides
Data Discovery for Big Big Insights - Tableau Webinar SlidesData Discovery for Big Big Insights - Tableau Webinar Slides
Data Discovery for Big Big Insights - Tableau Webinar SlidesFitzgerald Analytics, Inc.
 
Data visualization trends in Business Intelligence: Allison Sapka at Analytic...
Data visualization trends in Business Intelligence: Allison Sapka at Analytic...Data visualization trends in Business Intelligence: Allison Sapka at Analytic...
Data visualization trends in Business Intelligence: Allison Sapka at Analytic...Fitzgerald Analytics, Inc.
 
Governing the Data to Dollars Value Chain™ - Sept 2012 NYC Data Governance Co...
Governing the Data to Dollars Value Chain™ - Sept 2012 NYC Data Governance Co...Governing the Data to Dollars Value Chain™ - Sept 2012 NYC Data Governance Co...
Governing the Data to Dollars Value Chain™ - Sept 2012 NYC Data Governance Co...Fitzgerald Analytics, Inc.
 
From Big Legacy Data to Insight: Lessons Learned Creating New Value from a Bi...
From Big Legacy Data to Insight: Lessons Learned Creating New Value from a Bi...From Big Legacy Data to Insight: Lessons Learned Creating New Value from a Bi...
From Big Legacy Data to Insight: Lessons Learned Creating New Value from a Bi...Fitzgerald Analytics, Inc.
 
Data to Dollars™ - Practical Analytics in the Big Data Era Jaime Fitzgerald A...
Data to Dollars™ - Practical Analytics in the Big Data Era Jaime Fitzgerald A...Data to Dollars™ - Practical Analytics in the Big Data Era Jaime Fitzgerald A...
Data to Dollars™ - Practical Analytics in the Big Data Era Jaime Fitzgerald A...Fitzgerald Analytics, Inc.
 
Big Data Meets Customer Profitability Analytics
Big Data Meets Customer Profitability AnalyticsBig Data Meets Customer Profitability Analytics
Big Data Meets Customer Profitability AnalyticsFitzgerald Analytics, Inc.
 
Keynote on Financial Services Analytics - Presented aug 2011
Keynote on Financial Services Analytics - Presented aug 2011Keynote on Financial Services Analytics - Presented aug 2011
Keynote on Financial Services Analytics - Presented aug 2011Fitzgerald Analytics, Inc.
 
New insights from big legacy data at bundle (Presented at Text Analytics Worl...
New insights from big legacy data at bundle (Presented at Text Analytics Worl...New insights from big legacy data at bundle (Presented at Text Analytics Worl...
New insights from big legacy data at bundle (Presented at Text Analytics Worl...Fitzgerald Analytics, Inc.
 
Knowledge management for analytic teams jaime fitzgerald and alex hasha - p...
Knowledge management for analytic teams   jaime fitzgerald and alex hasha - p...Knowledge management for analytic teams   jaime fitzgerald and alex hasha - p...
Knowledge management for analytic teams jaime fitzgerald and alex hasha - p...Fitzgerald Analytics, Inc.
 
Analytics in Financial Services: Keynote Presentation for TDWI and NY Tech Co...
Analytics in Financial Services: Keynote Presentation for TDWI and NY Tech Co...Analytics in Financial Services: Keynote Presentation for TDWI and NY Tech Co...
Analytics in Financial Services: Keynote Presentation for TDWI and NY Tech Co...Fitzgerald Analytics, Inc.
 
Jaime Fitzgerald: A Master Data Management Road-Trip - Presented Enterprise D...
Jaime Fitzgerald: A Master Data Management Road-Trip - Presented Enterprise D...Jaime Fitzgerald: A Master Data Management Road-Trip - Presented Enterprise D...
Jaime Fitzgerald: A Master Data Management Road-Trip - Presented Enterprise D...Fitzgerald Analytics, Inc.
 

More from Fitzgerald Analytics, Inc. (18)

Profiting from customer profitability + big data fitzgerald analytics
Profiting from customer profitability + big data fitzgerald analyticsProfiting from customer profitability + big data fitzgerald analytics
Profiting from customer profitability + big data fitzgerald analytics
 
2013 12-05 data-driven innovation - fitzgerald analytics workshop at gilbane ...
2013 12-05 data-driven innovation - fitzgerald analytics workshop at gilbane ...2013 12-05 data-driven innovation - fitzgerald analytics workshop at gilbane ...
2013 12-05 data-driven innovation - fitzgerald analytics workshop at gilbane ...
 
Analytics in Financial Services - Behavioral Finance Event - Data Visualizati...
Analytics in Financial Services - Behavioral Finance Event - Data Visualizati...Analytics in Financial Services - Behavioral Finance Event - Data Visualizati...
Analytics in Financial Services - Behavioral Finance Event - Data Visualizati...
 
Analytics in financial services prez behavioral finance + data visualizatio...
Analytics in financial services prez   behavioral finance + data visualizatio...Analytics in financial services prez   behavioral finance + data visualizatio...
Analytics in financial services prez behavioral finance + data visualizatio...
 
Jaime Fitzgerald on Data-Driven Customer Experience in Financial Services and...
Jaime Fitzgerald on Data-Driven Customer Experience in Financial Services and...Jaime Fitzgerald on Data-Driven Customer Experience in Financial Services and...
Jaime Fitzgerald on Data-Driven Customer Experience in Financial Services and...
 
Data Discovery for Big Big Insights - Tableau Webinar Slides
Data Discovery for Big Big Insights - Tableau Webinar SlidesData Discovery for Big Big Insights - Tableau Webinar Slides
Data Discovery for Big Big Insights - Tableau Webinar Slides
 
Text graph-visualization redux
Text graph-visualization reduxText graph-visualization redux
Text graph-visualization redux
 
Data visualization trends in Business Intelligence: Allison Sapka at Analytic...
Data visualization trends in Business Intelligence: Allison Sapka at Analytic...Data visualization trends in Business Intelligence: Allison Sapka at Analytic...
Data visualization trends in Business Intelligence: Allison Sapka at Analytic...
 
Governing the Data to Dollars Value Chain™ - Sept 2012 NYC Data Governance Co...
Governing the Data to Dollars Value Chain™ - Sept 2012 NYC Data Governance Co...Governing the Data to Dollars Value Chain™ - Sept 2012 NYC Data Governance Co...
Governing the Data to Dollars Value Chain™ - Sept 2012 NYC Data Governance Co...
 
From Big Legacy Data to Insight: Lessons Learned Creating New Value from a Bi...
From Big Legacy Data to Insight: Lessons Learned Creating New Value from a Bi...From Big Legacy Data to Insight: Lessons Learned Creating New Value from a Bi...
From Big Legacy Data to Insight: Lessons Learned Creating New Value from a Bi...
 
Data to Dollars™ - Practical Analytics in the Big Data Era Jaime Fitzgerald A...
Data to Dollars™ - Practical Analytics in the Big Data Era Jaime Fitzgerald A...Data to Dollars™ - Practical Analytics in the Big Data Era Jaime Fitzgerald A...
Data to Dollars™ - Practical Analytics in the Big Data Era Jaime Fitzgerald A...
 
Big Data Meets Customer Profitability Analytics
Big Data Meets Customer Profitability AnalyticsBig Data Meets Customer Profitability Analytics
Big Data Meets Customer Profitability Analytics
 
Keynote on Financial Services Analytics - Presented aug 2011
Keynote on Financial Services Analytics - Presented aug 2011Keynote on Financial Services Analytics - Presented aug 2011
Keynote on Financial Services Analytics - Presented aug 2011
 
New insights from big legacy data at bundle (Presented at Text Analytics Worl...
New insights from big legacy data at bundle (Presented at Text Analytics Worl...New insights from big legacy data at bundle (Presented at Text Analytics Worl...
New insights from big legacy data at bundle (Presented at Text Analytics Worl...
 
Knowledge management for analytic teams jaime fitzgerald and alex hasha - p...
Knowledge management for analytic teams   jaime fitzgerald and alex hasha - p...Knowledge management for analytic teams   jaime fitzgerald and alex hasha - p...
Knowledge management for analytic teams jaime fitzgerald and alex hasha - p...
 
Analytics in Financial Services: Keynote Presentation for TDWI and NY Tech Co...
Analytics in Financial Services: Keynote Presentation for TDWI and NY Tech Co...Analytics in Financial Services: Keynote Presentation for TDWI and NY Tech Co...
Analytics in Financial Services: Keynote Presentation for TDWI and NY Tech Co...
 
Fitzgerald Analytics 1-Page Overview
Fitzgerald Analytics 1-Page OverviewFitzgerald Analytics 1-Page Overview
Fitzgerald Analytics 1-Page Overview
 
Jaime Fitzgerald: A Master Data Management Road-Trip - Presented Enterprise D...
Jaime Fitzgerald: A Master Data Management Road-Trip - Presented Enterprise D...Jaime Fitzgerald: A Master Data Management Road-Trip - Presented Enterprise D...
Jaime Fitzgerald: A Master Data Management Road-Trip - Presented Enterprise D...
 

Recently uploaded

MoneyBridge Pitch Deck - Investor Presentation
MoneyBridge Pitch Deck - Investor PresentationMoneyBridge Pitch Deck - Investor Presentation
MoneyBridge Pitch Deck - Investor Presentationbaron83
 
Introduction to The overview of GAAP LO 1-5.pptx
Introduction to The overview of GAAP LO 1-5.pptxIntroduction to The overview of GAAP LO 1-5.pptx
Introduction to The overview of GAAP LO 1-5.pptxJemalSeid25
 
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...IMARC Group
 
Building Your Personal Brand on LinkedIn - Expert Planet- 2024
 Building Your Personal Brand on LinkedIn - Expert Planet-  2024 Building Your Personal Brand on LinkedIn - Expert Planet-  2024
Building Your Personal Brand on LinkedIn - Expert Planet- 2024Stephan Koning
 
Lecture_6.pptx English speaking easyb to
Lecture_6.pptx English speaking easyb toLecture_6.pptx English speaking easyb to
Lecture_6.pptx English speaking easyb toumarfarooquejamali32
 
Upgrade Your Banking Experience with Advanced Core Banking Applications
Upgrade Your Banking Experience with Advanced Core Banking ApplicationsUpgrade Your Banking Experience with Advanced Core Banking Applications
Upgrade Your Banking Experience with Advanced Core Banking ApplicationsIntellect Design Arena Ltd
 
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...AustraliaChapterIIBA
 
A flour, rice and Suji company in Jhang.
A flour, rice and Suji company in Jhang.A flour, rice and Suji company in Jhang.
A flour, rice and Suji company in Jhang.mcshagufta46
 
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)tazeenaila12
 
Michael Vidyakin: Introduction to PMO (UA)
Michael Vidyakin: Introduction to PMO (UA)Michael Vidyakin: Introduction to PMO (UA)
Michael Vidyakin: Introduction to PMO (UA)Lviv Startup Club
 
Trauma Training Service for First Responders
Trauma Training Service for First RespondersTrauma Training Service for First Responders
Trauma Training Service for First RespondersBPOQe
 
Developing Coaching Skills: Mine, Yours, Ours
Developing Coaching Skills: Mine, Yours, OursDeveloping Coaching Skills: Mine, Yours, Ours
Developing Coaching Skills: Mine, Yours, OursKaiNexus
 
NewBase 25 March 2024 Energy News issue - 1710 by Khaled Al Awadi_compress...
NewBase  25 March  2024  Energy News issue - 1710 by Khaled Al Awadi_compress...NewBase  25 March  2024  Energy News issue - 1710 by Khaled Al Awadi_compress...
NewBase 25 March 2024 Energy News issue - 1710 by Khaled Al Awadi_compress...Khaled Al Awadi
 
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISINGUNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISINGlokeshwarmaha
 
Personal Brand Exploration Presentation Eric Bonilla
Personal Brand Exploration Presentation Eric BonillaPersonal Brand Exploration Presentation Eric Bonilla
Personal Brand Exploration Presentation Eric BonillaEricBonilla13
 
Entrepreneurship & organisations: influences and organizations
Entrepreneurship & organisations: influences and organizationsEntrepreneurship & organisations: influences and organizations
Entrepreneurship & organisations: influences and organizationsP&CO
 
Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access
 
Fabric RFID Wristbands in Ireland for Events and Festivals
Fabric RFID Wristbands in Ireland for Events and FestivalsFabric RFID Wristbands in Ireland for Events and Festivals
Fabric RFID Wristbands in Ireland for Events and FestivalsWristbands Ireland
 
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...Brian Solis
 
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdfGraham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdfAnhNguyen97152
 

Recently uploaded (20)

MoneyBridge Pitch Deck - Investor Presentation
MoneyBridge Pitch Deck - Investor PresentationMoneyBridge Pitch Deck - Investor Presentation
MoneyBridge Pitch Deck - Investor Presentation
 
Introduction to The overview of GAAP LO 1-5.pptx
Introduction to The overview of GAAP LO 1-5.pptxIntroduction to The overview of GAAP LO 1-5.pptx
Introduction to The overview of GAAP LO 1-5.pptx
 
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
 
Building Your Personal Brand on LinkedIn - Expert Planet- 2024
 Building Your Personal Brand on LinkedIn - Expert Planet-  2024 Building Your Personal Brand on LinkedIn - Expert Planet-  2024
Building Your Personal Brand on LinkedIn - Expert Planet- 2024
 
Lecture_6.pptx English speaking easyb to
Lecture_6.pptx English speaking easyb toLecture_6.pptx English speaking easyb to
Lecture_6.pptx English speaking easyb to
 
Upgrade Your Banking Experience with Advanced Core Banking Applications
Upgrade Your Banking Experience with Advanced Core Banking ApplicationsUpgrade Your Banking Experience with Advanced Core Banking Applications
Upgrade Your Banking Experience with Advanced Core Banking Applications
 
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
 
A flour, rice and Suji company in Jhang.
A flour, rice and Suji company in Jhang.A flour, rice and Suji company in Jhang.
A flour, rice and Suji company in Jhang.
 
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
 
Michael Vidyakin: Introduction to PMO (UA)
Michael Vidyakin: Introduction to PMO (UA)Michael Vidyakin: Introduction to PMO (UA)
Michael Vidyakin: Introduction to PMO (UA)
 
Trauma Training Service for First Responders
Trauma Training Service for First RespondersTrauma Training Service for First Responders
Trauma Training Service for First Responders
 
Developing Coaching Skills: Mine, Yours, Ours
Developing Coaching Skills: Mine, Yours, OursDeveloping Coaching Skills: Mine, Yours, Ours
Developing Coaching Skills: Mine, Yours, Ours
 
NewBase 25 March 2024 Energy News issue - 1710 by Khaled Al Awadi_compress...
NewBase  25 March  2024  Energy News issue - 1710 by Khaled Al Awadi_compress...NewBase  25 March  2024  Energy News issue - 1710 by Khaled Al Awadi_compress...
NewBase 25 March 2024 Energy News issue - 1710 by Khaled Al Awadi_compress...
 
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISINGUNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
 
Personal Brand Exploration Presentation Eric Bonilla
Personal Brand Exploration Presentation Eric BonillaPersonal Brand Exploration Presentation Eric Bonilla
Personal Brand Exploration Presentation Eric Bonilla
 
Entrepreneurship & organisations: influences and organizations
Entrepreneurship & organisations: influences and organizationsEntrepreneurship & organisations: influences and organizations
Entrepreneurship & organisations: influences and organizations
 
Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024
 
Fabric RFID Wristbands in Ireland for Events and Festivals
Fabric RFID Wristbands in Ireland for Events and FestivalsFabric RFID Wristbands in Ireland for Events and Festivals
Fabric RFID Wristbands in Ireland for Events and Festivals
 
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
 
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdfGraham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdf
 

TDWI NYC Chapter - Tony Baer Ovum on Big data, Data quality, and BI Convergence

  • 1. Big Data and Business Intelligence Must Converge Tony Baer tony.baer@ovum.com March 6, 2013 1 © Copyright Ovum. All rights reserved. Ovum is a subsidiary of Informa plc.
  • 2. Agenda  Challenges traditional data stewardship practice  Privacy – is all the world a stage?  Limits to data lifecycle?  Data quality: the big, the bad, the ugly – and it all might be good! 2 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 3. Data stewardship challenges – What’s old is new Remember?  Back to undifferentiated ‘gobblobs’ of data  Programmatic access reigns  File systems, not (always) tables 10.102.8.152 - - [05/Nov/2003:00:19:54 -0500] "GET /inventory/index.jsp HTTP/1.1" 200 4028 "http://www.mycompany.com/index.jsp" "Mozilla/4.08 [en] (Win98; I ;Nav)"  Batch is back 192.168.114.201, -, 03/20/01, 7:55:20, W3SVC2, SALES1, 172.21.13.45, 4502, 163, 3223, 200, 0, GET,/DeptLogo.gif, -, 172.16.255.255, anonymous, 03/20/01, 23:58:11, MSFTPSVC, SALES1, 172.16.255.255, 60, 275, 0, 0, But… if index(tempvalue,'?') then tempvalue=scan(tempvalue,1,'?'); else if index(tempvalue,'&')>1 then tempvalue=scan(tempvalue,1,'&');  Volume, variety, velocity, and where’s the value??  Just because you can, should you? 3 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 4. Data stewardship questions for Big Data  Can we, should we “control” this data?  Are there limits to how much we should know?  Can we just keep piling up data forever?  Can we cleanse terabytes of data?  Do we still need “good” data? 4 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 5. Agenda  Challenges traditional data stewardship practice  Privacy – is all the world a stage?  Limits to data lifecycle?  Data quality: the big, the bad, the ugly – and it all might be good! 5 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 6. Privacy – the more things change… “You have zero privacy anyway…. Get over it” -- Scott McNealy, 1999 Facebook does not actually delete images… but instead merely removes the links – a fix “is in sight” -- ZDNet, 2/6/12 Facebook agrees to 20 years of federal privacy audits -- NY Times, 11/29/11 6 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 7. What privacy? Florida made $63m last year by selling DMV information (name, date of birth, type of vehicle driven) to companies like LexusNexus & Shadow Soft. -- Terence Craig & Mary Ludloff Privacy and Big Data (O’Reilly Media, 2011) 7 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 8. Big Data privacy 101 – Don’t be creepy  Governance problem first, How Companies Learn Your technology second Secrets  Understand the relationship with your customers & business partners  Keep communications in context  Don’t catch your customers by “My daughter got this in the mail!” he surprise said. “She’s still in high school, and you’re sending her coupons for baby clothes and cribs? Are you trying to  The law still trying to catch up encourage her to get pregnant?” -- NY Times 2/16/12 8 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 9. Agenda  Challenges traditional data stewardship practice  Privacy – is all the world a stage?  Limits to data lifecycle?  Data quality: the big, the bad, the ugly – and it all might be good! 9 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 10. Data lifecycle – How long can this go on?  Google, Yahoo, Facebook, etc. don’t deprecate web data  Hadoop designed for economical scale-out  Moore’s Law, declining cost of storage  Is Hadoop Archive the answer?  Is Hadoop the new tape? Management & skills will be the limit Aerial view of Quincy, WA data ctrs 10 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 11. Agenda  Challenges traditional data stewardship practice  Privacy – is all the world a stage?  Limits to data lifecycle?  Data quality: the big, the bad, the ugly – and it all might be good! 11 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 12. Data Quality & Hadoop – Big Quality Questions  Can we cleanse terabytes of data?  Do we still need “good” data?  Are there new approaches to cleansing Big Data? 12 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 13. Framing the issue  “Garbage in, garbage out,’ but DW forced the issue  Traditional approaches  Profiling, cleansing, MDM  DW vs. Hadoop data quality challenges  Known data sets & known criteria vs. vaguely known  Bounded vs. less bounded tasks  Limitations of MapReduce*  Cleansing & transformation within a single Map operation;  Profiling & matching of unstructured data  Matching of data in operations without inter-process communications *Source: David Loshin, "Hadoop and Data Quality, Data Integration, Data Analysis" at http://www.dataroundtable.com/?p=8841 13 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 14. Is data quality necessary for Hadoop?  The App  How mission-critical?  Regulatory compliance impacts?  What degree of business impact?  The Data  The 4V’s (volume, variety, velocity, value) determine what approaches to quality are feasible 14 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 15. Examples  Web ad placement optimization  Counter-party risk management for capital markets  Customer sentiment analysis  Managing smart utility grids or urban infrastructure 15 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 16. Bad data may be good  Sensory data  Outlier or drift?  Time to recalibrate devices?  Time to perform preventive maintenance?  Are new/unaccounted environmental factors skewing readings?  Human-readable data  Flawed concept of reality?  Flawed assumptions on data meaning?  Changes producing ‘new norm’ 16 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 17. Big Data quality in Hadoop – Emergent approaches  Crowdsourcing data –  Collect data far & wide from as many diverse sources as possible. Torrents of data overcome the noise.  Comparative trend analysis of incoming streams to dynamically ID the norm or sweet spot of “good” data  Apply data science to “correct the dots”  Don’t go record by record. Statistically analyze the data set in aggregate.  Iteratively analyze & re-analyze nature of data, keep analyzing outliers  Apply off-the-wall approaches  Enterprise Architectural approach  Semantic (domain) model-driven  Apply cleansing logic at run time  Critical for sensitive, regulatory-driven apps 17 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 18. Summary  Challenges traditional data stewardship practice  Combination of old & new  Privacy – is all the world a stage?  Best practices, legal requirements still in flux  Don’t be creepy!  Limits to data lifecycle?  Few enterprises are Google or Facebook  Ability to manage large infrastructure will be major limit  Data quality  Strategy depends on type of app & data set(s)  A spectrum of approaches -- from none to classic ETL to aggregate statistical  No single silver bullet 18 © Copyright Ovum. All rights reserved. Ovum is an Informa business.
  • 19. Disclaimer All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior permission of the publisher, Ovum (an Informa business). The facts of this report are believed to be correct at the time of publication but cannot be guaranteed. Please note that the findings, conclusions and recommendations that Ovum delivers will be based on information gathered in good faith from both primary and secondary sources, whose accuracy we are not always in a position to guarantee. As such Ovum can accept no liability whatever for actions taken based on any information that may subsequently prove to be incorrect. 19 © Copyright Ovum. All rights reserved. Ovum is an Informa business.