SlideShare a Scribd company logo
1 of 34
Download to read offline
High Velocity Intelligence
    Application Monitoring with Splunk
          SNAP Interactive, Inc.


                Presented by:


            Nicholas DiSanto
         Architecture Team Lead
Company Overview
 • SNAP Interactive, Inc.
 • www.AreYouInterested.com
 • Believes it is one of the largest social discovery platforms on the web (based on
   monthly active users)

 • More than 5 million monthly active users
 • Over 1 billion total pieces of structured data from its users
 • Synced to millions of Facebook profiles
 • Receives over 1,000 real-time updates per minute on like actions from Facebook
                                                                              www.snap-interactive.com
About Nick
   • Developing on LAMP stack at tech startups for 10 years
   • Leading a team of core engineers
   • Passionate about experimentation & data driven iteration
   • Striving to eliminate all technical blockers to speed and innovation
   • @NicholasDiSanto

                                                                    www.snap-interactive.com
Summary
  • We use splunk for many, many things!
  • Today, I will share some of our more interesting applications
     • How we get data into splunk
     • What we do with that data
        • Various types of monitoring
        • Extensive user behavior analysis
                                                                    www.snap-interactive.com
What We Give Splunk
   • Custom application logs
   • Structured, minified, event data
   • De-normalized user demographics
   • Application profiling data
   • Error logs

                                        www.snap-interactive.com
Sending Splunk Data
   • Centralize logging functions and:
      • Format arbitrary structured data into splunk extractable
       field/value pairs: field=”value”

     • Normalize and minify field names
     • Detect user_id and augment logs
     • Optionally log a percent of events
     • Target different log files (error, info, analytics)
                                                                   www.snap-interactive.com
User Demographics
   • Our analytics log contains application events, triggered by real
      users

   • We augment these event logs with useful demographic data to
      classify the events
  ✦   Gender
  ✦   Seeking gender
  ✦   Country
  ✦   Ethnicity
  ✦   Date of birth                                                 www.snap-interactive.com
Demographic Power
   • By augmenting event logs with user demographics we can
     perform powerful and detailed analysis of user behavior

   • Target analysis at countries, genders, or age ranges
   • Classify events by days since: registration, login, email open, etc.
   • ...and much more


                                                                       www.snap-interactive.com
Performance Metrics
   • We time key algorithms in our application, and log:
      • server name
      • query name
      • time spent working
   • This lets us graph the average, min and max times of these
    algorithms per server

   • We also dark launch features, benchmarking performance prior
    to official launch.
                                                                  www.snap-interactive.com
Performance
   • Average query time for key algorithms by server




                                                       www.snap-interactive.com
What Splunk Gives Us
   • Monitoring - to measure application health
   • Analysis - to drive future product decisions
   • AB test evaluation - to validate hypotheses
   • Detection - to find patterns & classify users


                                                     www.snap-interactive.com
Monitoring
   • With continuous deployment, detailed monitoring is absolutely
    essential.

   • Each deploy we watch changes in:
      • Realtime classified error graphs
      • Core event stat graphs
   • We also monitor email deliverability, revenue, and performance
    (although not every deploy)

                                                                 www.snap-interactive.com
Error & Event Monitoring
   • Alert us immediately after deploy if something has gone wrong
   • Use realtime background searches
   • Single dashboard with multiple graphs and tables
   • We are exploring realtime sms alerts to the ‘developer on call’
         • Use historial data to identify min/max expected thresholds
          (weighted averages: same time of day, same day of week)

        • Detect consistent deviations and alert
                                                                  www.snap-interactive.com
Error Monitoring
   • Count of all errors : past 30 seconds




                                             www.snap-interactive.com
Error Monitoring
   • All errors: past 5 minutes w/deploys




                                            www.snap-interactive.com
Error Monitoring
   • Rolled up errors: past 5 minutes




                                        www.snap-interactive.com
Error Monitoring
   • Rolled up filtered errors: past 5 minutes




                                                 www.snap-interactive.com
Error Monitoring
   • Rolled up JS errors: past 3 hours




                                         www.snap-interactive.com
Event Monitoring
   • We monitor ~20 event stats, in realtime, each deploy




                                                            www.snap-interactive.com
Event Monitoring
   • Overview and detail views, powered by a single realtime
    background search




                                                               www.snap-interactive.com
Monitor Email & Performance
   • Email
      • Deliverability is essential to business
      • Need to maximize engagement
   • Performance
      • What async jobs may be contributing to high DB load?
      • What performance are end users experiencing?
      • Are particular servers overloaded?
                                                               www.snap-interactive.com
Email Deliverability
   • Overview of key metrics




                               www.snap-interactive.com
Email Monitoring
   • Inserts into email scheduled send queue




                                               www.snap-interactive.com
Performance
   • Asynchronous process timers
   • Can correlate spikes with site issues




                                             www.snap-interactive.com
Analysis
   • We heavily leverage summary indexing for performance gains
   • Daily rollups are grouped judiciously, giving us fast, flexible,
     analysis over long periods

   • We summarize: revenue, email deliverability, core KPI, and
     general stats data

   • Custom dashboards facilitate easy searching
   • Lots of ad hoc searching by product team
                                                                        www.snap-interactive.com
Email Analysis
   • Sends opens, clicks, bounces & FBL rates by email type




                                                              www.snap-interactive.com
Email Analysis
   • Monitor changes in open & click rates by email, ISP, country, etc.




                                                                    www.snap-interactive.com
Email Analysis
   • Analysis dashboard   *Sample data




                                     www.snap-interactive.com
Core KPI Dashboard
   • Powerful targeted cohort analysis   *Sample data




                                                        www.snap-interactive.com
AB Test Results
   • We are constantly running a variety of AB experiments on our
    live users

   • We divide our user population in to nine 10% segments and ten
    1% segments

   • Each segment can be targeted with an experiment
   • All event logs are annotated with the appropriate AB
    experiment name

   • This allows us to measure behavior changes between
    experiment and control groups                               www.snap-interactive.com
Easy AB Analysis
   • All event logs contain the an AB field
   • This identifies the experiment group of the user at that point in
     time

   • Fully integrated into core analysis dashboards
   • Ad hoc analysis becomes simple
     `my_search` (AB=my_test OR AB=ctrl)
     | `my_reporting_command` by AB

                                                                   www.snap-interactive.com
AB Dashboards
   • key metrics: experiment vs control group




                                                www.snap-interactive.com
Latest Splunking - Detection
   • The way our users interact with one another is insightful
   • We can use this data to classify users:
      • Identifying “attractive” users
      • Identifying spammers & scammers
   • We test hypotheses with ad hoc searches
   • Find reliable patterns then setup scheduled searches that
     interface with MySQL

   • This data then feeds into our application in various ways   www.snap-interactive.com
Contact Us
   • SNAP Interactive, Inc.
     www.snap-interactive.com


   • Nicholas DiSanto
     Architecture Team Lead
     ndisanto@snap-interactive.com
     301-BIG-TREE
     @NicholasDiSanto


   • Lindsay Bubbico                 www.snap-interactive.com

More Related Content

Viewers also liked

Tyler Garmin- Kansas City
Tyler Garmin- Kansas CityTyler Garmin- Kansas City
Tyler Garmin- Kansas CitySplunk
 
Splunk Conf2010: Corporate Express presents Splunk with SAP
Splunk Conf2010: Corporate Express presents Splunk with SAPSplunk Conf2010: Corporate Express presents Splunk with SAP
Splunk Conf2010: Corporate Express presents Splunk with SAPSplunk
 
SplunkLive New York 2011: DealerTrack
SplunkLive New York 2011: DealerTrackSplunkLive New York 2011: DealerTrack
SplunkLive New York 2011: DealerTrackSplunk
 
Massimiliano Raks, Naples University on SPECS: Secure provisioning of cloud s...
Massimiliano Raks, Naples University on SPECS: Secure provisioning of cloud s...Massimiliano Raks, Naples University on SPECS: Secure provisioning of cloud s...
Massimiliano Raks, Naples University on SPECS: Secure provisioning of cloud s...SLA-Ready Network
 
SplunkLive! Washington DC May 2013 - Splunk App for VMware
SplunkLive! Washington DC May 2013 - Splunk App for VMwareSplunkLive! Washington DC May 2013 - Splunk App for VMware
SplunkLive! Washington DC May 2013 - Splunk App for VMwareSplunk
 
SplunkLive! Customer Presentation - Satcom Direct
SplunkLive! Customer Presentation - Satcom DirectSplunkLive! Customer Presentation - Satcom Direct
SplunkLive! Customer Presentation - Satcom DirectSplunk
 

Viewers also liked (6)

Tyler Garmin- Kansas City
Tyler Garmin- Kansas CityTyler Garmin- Kansas City
Tyler Garmin- Kansas City
 
Splunk Conf2010: Corporate Express presents Splunk with SAP
Splunk Conf2010: Corporate Express presents Splunk with SAPSplunk Conf2010: Corporate Express presents Splunk with SAP
Splunk Conf2010: Corporate Express presents Splunk with SAP
 
SplunkLive New York 2011: DealerTrack
SplunkLive New York 2011: DealerTrackSplunkLive New York 2011: DealerTrack
SplunkLive New York 2011: DealerTrack
 
Massimiliano Raks, Naples University on SPECS: Secure provisioning of cloud s...
Massimiliano Raks, Naples University on SPECS: Secure provisioning of cloud s...Massimiliano Raks, Naples University on SPECS: Secure provisioning of cloud s...
Massimiliano Raks, Naples University on SPECS: Secure provisioning of cloud s...
 
SplunkLive! Washington DC May 2013 - Splunk App for VMware
SplunkLive! Washington DC May 2013 - Splunk App for VMwareSplunkLive! Washington DC May 2013 - Splunk App for VMware
SplunkLive! Washington DC May 2013 - Splunk App for VMware
 
SplunkLive! Customer Presentation - Satcom Direct
SplunkLive! Customer Presentation - Satcom DirectSplunkLive! Customer Presentation - Satcom Direct
SplunkLive! Customer Presentation - Satcom Direct
 

Similar to SplunkLive! New York Dec 2012 - SNAP Interactive

Primo Reporting: Using 3rd Party Software to Create Primo Reports & Analyze P...
Primo Reporting: Using 3rd Party Software to Create Primo Reports & Analyze P...Primo Reporting: Using 3rd Party Software to Create Primo Reports & Analyze P...
Primo Reporting: Using 3rd Party Software to Create Primo Reports & Analyze P...Alison Hitchens
 
Splunk for Security: Background & Customer Case Study
Splunk for Security: Background & Customer Case StudySplunk for Security: Background & Customer Case Study
Splunk for Security: Background & Customer Case StudyAndrew Gerber
 
Building an Open Source AppSec Pipeline - 2015 Texas Linux Fest
Building an Open Source AppSec Pipeline - 2015 Texas Linux FestBuilding an Open Source AppSec Pipeline - 2015 Texas Linux Fest
Building an Open Source AppSec Pipeline - 2015 Texas Linux FestMatt Tesauro
 
Customer Presentation
Customer PresentationCustomer Presentation
Customer PresentationSplunk
 
ATAGTR2017 Unified APM: The new age performance monitoring for production sys...
ATAGTR2017 Unified APM: The new age performance monitoring for production sys...ATAGTR2017 Unified APM: The new age performance monitoring for production sys...
ATAGTR2017 Unified APM: The new age performance monitoring for production sys...Agile Testing Alliance
 
Engage 2013 - Leveraging Ad Hoc Analysis
Engage 2013 - Leveraging Ad Hoc AnalysisEngage 2013 - Leveraging Ad Hoc Analysis
Engage 2013 - Leveraging Ad Hoc AnalysisWebtrends
 
Rakuten’s Journey with Splunk - Evolution of Splunk as a Service
Rakuten’s Journey with Splunk - Evolution of Splunk as a ServiceRakuten’s Journey with Splunk - Evolution of Splunk as a Service
Rakuten’s Journey with Splunk - Evolution of Splunk as a ServiceRakuten Group, Inc.
 
Elevate your Splunk Deployment by Better Understanding your Value Breakfast S...
Elevate your Splunk Deployment by Better Understanding your Value Breakfast S...Elevate your Splunk Deployment by Better Understanding your Value Breakfast S...
Elevate your Splunk Deployment by Better Understanding your Value Breakfast S...Splunk
 
New Splunk Management Solutions Update: Splunk MINT and Splunk App for Stream
New Splunk Management Solutions Update: Splunk MINT and Splunk App for Stream New Splunk Management Solutions Update: Splunk MINT and Splunk App for Stream
New Splunk Management Solutions Update: Splunk MINT and Splunk App for Stream Splunk
 
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...Data Con LA
 
Adventures in Observability - Clickhouse and Instana
Adventures in Observability - Clickhouse and InstanaAdventures in Observability - Clickhouse and Instana
Adventures in Observability - Clickhouse and InstanaMarcel Birkner
 
Adventures in Observability: How in-house ClickHouse deployment enabled Inst...
 Adventures in Observability: How in-house ClickHouse deployment enabled Inst... Adventures in Observability: How in-house ClickHouse deployment enabled Inst...
Adventures in Observability: How in-house ClickHouse deployment enabled Inst...Altinity Ltd
 
Splunk MINT and Stream Breakout
Splunk MINT and Stream BreakoutSplunk MINT and Stream Breakout
Splunk MINT and Stream BreakoutSplunk
 
Zero to ten million daily users in four weeks: sustainable speed is king
Zero to ten million daily users in four weeks: sustainable speed is kingZero to ten million daily users in four weeks: sustainable speed is king
Zero to ten million daily users in four weeks: sustainable speed is kingplumbee
 
Real User Experience Insight External
Real User Experience Insight ExternalReal User Experience Insight External
Real User Experience Insight Externaloracleonthebrain
 
Snowplow: open source game analytics powered by AWS
Snowplow: open source game analytics powered by AWSSnowplow: open source game analytics powered by AWS
Snowplow: open source game analytics powered by AWSGiuseppe Gaviani
 
Splunk in Rakuten: Splunk as a Service for all
Splunk in Rakuten: Splunk as a Service for allSplunk in Rakuten: Splunk as a Service for all
Splunk in Rakuten: Splunk as a Service for allTimur Bagirov
 
How Gousto is moving to just-in-time personalization with Snowplow
How Gousto is moving to just-in-time personalization with SnowplowHow Gousto is moving to just-in-time personalization with Snowplow
How Gousto is moving to just-in-time personalization with SnowplowGiuseppe Gaviani
 
[2C6]Everyplay_Big_Data
[2C6]Everyplay_Big_Data[2C6]Everyplay_Big_Data
[2C6]Everyplay_Big_DataNAVER D2
 
Machine Learning + Analytics in Splunk
Machine Learning + Analytics in Splunk Machine Learning + Analytics in Splunk
Machine Learning + Analytics in Splunk Splunk
 

Similar to SplunkLive! New York Dec 2012 - SNAP Interactive (20)

Primo Reporting: Using 3rd Party Software to Create Primo Reports & Analyze P...
Primo Reporting: Using 3rd Party Software to Create Primo Reports & Analyze P...Primo Reporting: Using 3rd Party Software to Create Primo Reports & Analyze P...
Primo Reporting: Using 3rd Party Software to Create Primo Reports & Analyze P...
 
Splunk for Security: Background & Customer Case Study
Splunk for Security: Background & Customer Case StudySplunk for Security: Background & Customer Case Study
Splunk for Security: Background & Customer Case Study
 
Building an Open Source AppSec Pipeline - 2015 Texas Linux Fest
Building an Open Source AppSec Pipeline - 2015 Texas Linux FestBuilding an Open Source AppSec Pipeline - 2015 Texas Linux Fest
Building an Open Source AppSec Pipeline - 2015 Texas Linux Fest
 
Customer Presentation
Customer PresentationCustomer Presentation
Customer Presentation
 
ATAGTR2017 Unified APM: The new age performance monitoring for production sys...
ATAGTR2017 Unified APM: The new age performance monitoring for production sys...ATAGTR2017 Unified APM: The new age performance monitoring for production sys...
ATAGTR2017 Unified APM: The new age performance monitoring for production sys...
 
Engage 2013 - Leveraging Ad Hoc Analysis
Engage 2013 - Leveraging Ad Hoc AnalysisEngage 2013 - Leveraging Ad Hoc Analysis
Engage 2013 - Leveraging Ad Hoc Analysis
 
Rakuten’s Journey with Splunk - Evolution of Splunk as a Service
Rakuten’s Journey with Splunk - Evolution of Splunk as a ServiceRakuten’s Journey with Splunk - Evolution of Splunk as a Service
Rakuten’s Journey with Splunk - Evolution of Splunk as a Service
 
Elevate your Splunk Deployment by Better Understanding your Value Breakfast S...
Elevate your Splunk Deployment by Better Understanding your Value Breakfast S...Elevate your Splunk Deployment by Better Understanding your Value Breakfast S...
Elevate your Splunk Deployment by Better Understanding your Value Breakfast S...
 
New Splunk Management Solutions Update: Splunk MINT and Splunk App for Stream
New Splunk Management Solutions Update: Splunk MINT and Splunk App for Stream New Splunk Management Solutions Update: Splunk MINT and Splunk App for Stream
New Splunk Management Solutions Update: Splunk MINT and Splunk App for Stream
 
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
 
Adventures in Observability - Clickhouse and Instana
Adventures in Observability - Clickhouse and InstanaAdventures in Observability - Clickhouse and Instana
Adventures in Observability - Clickhouse and Instana
 
Adventures in Observability: How in-house ClickHouse deployment enabled Inst...
 Adventures in Observability: How in-house ClickHouse deployment enabled Inst... Adventures in Observability: How in-house ClickHouse deployment enabled Inst...
Adventures in Observability: How in-house ClickHouse deployment enabled Inst...
 
Splunk MINT and Stream Breakout
Splunk MINT and Stream BreakoutSplunk MINT and Stream Breakout
Splunk MINT and Stream Breakout
 
Zero to ten million daily users in four weeks: sustainable speed is king
Zero to ten million daily users in four weeks: sustainable speed is kingZero to ten million daily users in four weeks: sustainable speed is king
Zero to ten million daily users in four weeks: sustainable speed is king
 
Real User Experience Insight External
Real User Experience Insight ExternalReal User Experience Insight External
Real User Experience Insight External
 
Snowplow: open source game analytics powered by AWS
Snowplow: open source game analytics powered by AWSSnowplow: open source game analytics powered by AWS
Snowplow: open source game analytics powered by AWS
 
Splunk in Rakuten: Splunk as a Service for all
Splunk in Rakuten: Splunk as a Service for allSplunk in Rakuten: Splunk as a Service for all
Splunk in Rakuten: Splunk as a Service for all
 
How Gousto is moving to just-in-time personalization with Snowplow
How Gousto is moving to just-in-time personalization with SnowplowHow Gousto is moving to just-in-time personalization with Snowplow
How Gousto is moving to just-in-time personalization with Snowplow
 
[2C6]Everyplay_Big_Data
[2C6]Everyplay_Big_Data[2C6]Everyplay_Big_Data
[2C6]Everyplay_Big_Data
 
Machine Learning + Analytics in Splunk
Machine Learning + Analytics in Splunk Machine Learning + Analytics in Splunk
Machine Learning + Analytics in Splunk
 

More from Splunk

.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routineSplunk
 
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTVSplunk
 
.conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica).conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica)Splunk
 
.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank International.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank InternationalSplunk
 
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett .conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett Splunk
 
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär).conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)Splunk
 
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu....conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...Splunk
 
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever....conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...Splunk
 
.conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex).conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex)Splunk
 
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)Splunk
 
Splunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11ySplunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11ySplunk
 
Splunk x Freenet - .conf Go Köln
Splunk x Freenet - .conf Go KölnSplunk x Freenet - .conf Go Köln
Splunk x Freenet - .conf Go KölnSplunk
 
Splunk Security Session - .conf Go Köln
Splunk Security Session - .conf Go KölnSplunk Security Session - .conf Go Köln
Splunk Security Session - .conf Go KölnSplunk
 
Data foundations building success, at city scale – Imperial College London
 Data foundations building success, at city scale – Imperial College London Data foundations building success, at city scale – Imperial College London
Data foundations building success, at city scale – Imperial College LondonSplunk
 
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...Splunk
 
SOC, Amore Mio! | Security Webinar
SOC, Amore Mio! | Security WebinarSOC, Amore Mio! | Security Webinar
SOC, Amore Mio! | Security WebinarSplunk
 
.conf Go 2022 - Observability Session
.conf Go 2022 - Observability Session.conf Go 2022 - Observability Session
.conf Go 2022 - Observability SessionSplunk
 
.conf Go Zurich 2022 - Keynote
.conf Go Zurich 2022 - Keynote.conf Go Zurich 2022 - Keynote
.conf Go Zurich 2022 - KeynoteSplunk
 
.conf Go Zurich 2022 - Platform Session
.conf Go Zurich 2022 - Platform Session.conf Go Zurich 2022 - Platform Session
.conf Go Zurich 2022 - Platform SessionSplunk
 
.conf Go Zurich 2022 - Security Session
.conf Go Zurich 2022 - Security Session.conf Go Zurich 2022 - Security Session
.conf Go Zurich 2022 - Security SessionSplunk
 

More from Splunk (20)

.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine
 
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
 
.conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica).conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica)
 
.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank International.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank International
 
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett .conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
 
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär).conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
 
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu....conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
 
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever....conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
 
.conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex).conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex)
 
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
 
Splunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11ySplunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11y
 
Splunk x Freenet - .conf Go Köln
Splunk x Freenet - .conf Go KölnSplunk x Freenet - .conf Go Köln
Splunk x Freenet - .conf Go Köln
 
Splunk Security Session - .conf Go Köln
Splunk Security Session - .conf Go KölnSplunk Security Session - .conf Go Köln
Splunk Security Session - .conf Go Köln
 
Data foundations building success, at city scale – Imperial College London
 Data foundations building success, at city scale – Imperial College London Data foundations building success, at city scale – Imperial College London
Data foundations building success, at city scale – Imperial College London
 
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
 
SOC, Amore Mio! | Security Webinar
SOC, Amore Mio! | Security WebinarSOC, Amore Mio! | Security Webinar
SOC, Amore Mio! | Security Webinar
 
.conf Go 2022 - Observability Session
.conf Go 2022 - Observability Session.conf Go 2022 - Observability Session
.conf Go 2022 - Observability Session
 
.conf Go Zurich 2022 - Keynote
.conf Go Zurich 2022 - Keynote.conf Go Zurich 2022 - Keynote
.conf Go Zurich 2022 - Keynote
 
.conf Go Zurich 2022 - Platform Session
.conf Go Zurich 2022 - Platform Session.conf Go Zurich 2022 - Platform Session
.conf Go Zurich 2022 - Platform Session
 
.conf Go Zurich 2022 - Security Session
.conf Go Zurich 2022 - Security Session.conf Go Zurich 2022 - Security Session
.conf Go Zurich 2022 - Security Session
 

Recently uploaded

ISONIKE Ltd Accreditation for the Conformity Assessment and Certification of ...
ISONIKE Ltd Accreditation for the Conformity Assessment and Certification of ...ISONIKE Ltd Accreditation for the Conformity Assessment and Certification of ...
ISONIKE Ltd Accreditation for the Conformity Assessment and Certification of ...ISONIKELtd
 
Introduction to The overview of GAAP LO 1-5.pptx
Introduction to The overview of GAAP LO 1-5.pptxIntroduction to The overview of GAAP LO 1-5.pptx
Introduction to The overview of GAAP LO 1-5.pptxJemalSeid25
 
Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access
 
Building Your Personal Brand on LinkedIn - Expert Planet- 2024
 Building Your Personal Brand on LinkedIn - Expert Planet-  2024 Building Your Personal Brand on LinkedIn - Expert Planet-  2024
Building Your Personal Brand on LinkedIn - Expert Planet- 2024Stephan Koning
 
MC Heights construction company in Jhang
MC Heights construction company in JhangMC Heights construction company in Jhang
MC Heights construction company in Jhangmcgroupjeya
 
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for BusinessQ2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for BusinessAPCO
 
7movierulz.uk
7movierulz.uk7movierulz.uk
7movierulz.ukaroemirsr
 
Personal Brand Exploration Presentation Eric Bonilla
Personal Brand Exploration Presentation Eric BonillaPersonal Brand Exploration Presentation Eric Bonilla
Personal Brand Exploration Presentation Eric BonillaEricBonilla13
 
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...Brian Solis
 
Fabric RFID Wristbands in Ireland for Events and Festivals
Fabric RFID Wristbands in Ireland for Events and FestivalsFabric RFID Wristbands in Ireland for Events and Festivals
Fabric RFID Wristbands in Ireland for Events and FestivalsWristbands Ireland
 
Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access
 
Project Brief & Information Architecture Report
Project Brief & Information Architecture ReportProject Brief & Information Architecture Report
Project Brief & Information Architecture Reportamberjiles31
 
Amazon ppt.pptx Amazon about the company
Amazon ppt.pptx Amazon about the companyAmazon ppt.pptx Amazon about the company
Amazon ppt.pptx Amazon about the companyfashionfound007
 
To Create Your Own Wig Online To Create Your Own Wig Online
To Create Your Own Wig Online  To Create Your Own Wig OnlineTo Create Your Own Wig Online  To Create Your Own Wig Online
To Create Your Own Wig Online To Create Your Own Wig Onlinelng ths
 
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdfGraham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdfAnhNguyen97152
 
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)tazeenaila12
 
Cracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptxCracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptxWorkforce Group
 
Team B Mind Map for Organizational Chg..
Team B Mind Map for Organizational Chg..Team B Mind Map for Organizational Chg..
Team B Mind Map for Organizational Chg..dlewis191
 
AMAZON SELLER VIRTUAL ASSISTANT PRODUCT RESEARCH .pdf
AMAZON SELLER VIRTUAL ASSISTANT PRODUCT RESEARCH .pdfAMAZON SELLER VIRTUAL ASSISTANT PRODUCT RESEARCH .pdf
AMAZON SELLER VIRTUAL ASSISTANT PRODUCT RESEARCH .pdfJohnCarloValencia4
 
MoneyBridge Pitch Deck - Investor Presentation
MoneyBridge Pitch Deck - Investor PresentationMoneyBridge Pitch Deck - Investor Presentation
MoneyBridge Pitch Deck - Investor Presentationbaron83
 

Recently uploaded (20)

ISONIKE Ltd Accreditation for the Conformity Assessment and Certification of ...
ISONIKE Ltd Accreditation for the Conformity Assessment and Certification of ...ISONIKE Ltd Accreditation for the Conformity Assessment and Certification of ...
ISONIKE Ltd Accreditation for the Conformity Assessment and Certification of ...
 
Introduction to The overview of GAAP LO 1-5.pptx
Introduction to The overview of GAAP LO 1-5.pptxIntroduction to The overview of GAAP LO 1-5.pptx
Introduction to The overview of GAAP LO 1-5.pptx
 
Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024
 
Building Your Personal Brand on LinkedIn - Expert Planet- 2024
 Building Your Personal Brand on LinkedIn - Expert Planet-  2024 Building Your Personal Brand on LinkedIn - Expert Planet-  2024
Building Your Personal Brand on LinkedIn - Expert Planet- 2024
 
MC Heights construction company in Jhang
MC Heights construction company in JhangMC Heights construction company in Jhang
MC Heights construction company in Jhang
 
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for BusinessQ2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
 
7movierulz.uk
7movierulz.uk7movierulz.uk
7movierulz.uk
 
Personal Brand Exploration Presentation Eric Bonilla
Personal Brand Exploration Presentation Eric BonillaPersonal Brand Exploration Presentation Eric Bonilla
Personal Brand Exploration Presentation Eric Bonilla
 
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
 
Fabric RFID Wristbands in Ireland for Events and Festivals
Fabric RFID Wristbands in Ireland for Events and FestivalsFabric RFID Wristbands in Ireland for Events and Festivals
Fabric RFID Wristbands in Ireland for Events and Festivals
 
Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024
 
Project Brief & Information Architecture Report
Project Brief & Information Architecture ReportProject Brief & Information Architecture Report
Project Brief & Information Architecture Report
 
Amazon ppt.pptx Amazon about the company
Amazon ppt.pptx Amazon about the companyAmazon ppt.pptx Amazon about the company
Amazon ppt.pptx Amazon about the company
 
To Create Your Own Wig Online To Create Your Own Wig Online
To Create Your Own Wig Online  To Create Your Own Wig OnlineTo Create Your Own Wig Online  To Create Your Own Wig Online
To Create Your Own Wig Online To Create Your Own Wig Online
 
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdfGraham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdf
 
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
 
Cracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptxCracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptx
 
Team B Mind Map for Organizational Chg..
Team B Mind Map for Organizational Chg..Team B Mind Map for Organizational Chg..
Team B Mind Map for Organizational Chg..
 
AMAZON SELLER VIRTUAL ASSISTANT PRODUCT RESEARCH .pdf
AMAZON SELLER VIRTUAL ASSISTANT PRODUCT RESEARCH .pdfAMAZON SELLER VIRTUAL ASSISTANT PRODUCT RESEARCH .pdf
AMAZON SELLER VIRTUAL ASSISTANT PRODUCT RESEARCH .pdf
 
MoneyBridge Pitch Deck - Investor Presentation
MoneyBridge Pitch Deck - Investor PresentationMoneyBridge Pitch Deck - Investor Presentation
MoneyBridge Pitch Deck - Investor Presentation
 

SplunkLive! New York Dec 2012 - SNAP Interactive

  • 1. High Velocity Intelligence Application Monitoring with Splunk SNAP Interactive, Inc. Presented by: Nicholas DiSanto Architecture Team Lead
  • 2. Company Overview • SNAP Interactive, Inc. • www.AreYouInterested.com • Believes it is one of the largest social discovery platforms on the web (based on monthly active users) • More than 5 million monthly active users • Over 1 billion total pieces of structured data from its users • Synced to millions of Facebook profiles • Receives over 1,000 real-time updates per minute on like actions from Facebook www.snap-interactive.com
  • 3. About Nick • Developing on LAMP stack at tech startups for 10 years • Leading a team of core engineers • Passionate about experimentation & data driven iteration • Striving to eliminate all technical blockers to speed and innovation • @NicholasDiSanto www.snap-interactive.com
  • 4. Summary • We use splunk for many, many things! • Today, I will share some of our more interesting applications • How we get data into splunk • What we do with that data • Various types of monitoring • Extensive user behavior analysis www.snap-interactive.com
  • 5. What We Give Splunk • Custom application logs • Structured, minified, event data • De-normalized user demographics • Application profiling data • Error logs www.snap-interactive.com
  • 6. Sending Splunk Data • Centralize logging functions and: • Format arbitrary structured data into splunk extractable field/value pairs: field=”value” • Normalize and minify field names • Detect user_id and augment logs • Optionally log a percent of events • Target different log files (error, info, analytics) www.snap-interactive.com
  • 7. User Demographics • Our analytics log contains application events, triggered by real users • We augment these event logs with useful demographic data to classify the events ✦ Gender ✦ Seeking gender ✦ Country ✦ Ethnicity ✦ Date of birth www.snap-interactive.com
  • 8. Demographic Power • By augmenting event logs with user demographics we can perform powerful and detailed analysis of user behavior • Target analysis at countries, genders, or age ranges • Classify events by days since: registration, login, email open, etc. • ...and much more www.snap-interactive.com
  • 9. Performance Metrics • We time key algorithms in our application, and log: • server name • query name • time spent working • This lets us graph the average, min and max times of these algorithms per server • We also dark launch features, benchmarking performance prior to official launch. www.snap-interactive.com
  • 10. Performance • Average query time for key algorithms by server www.snap-interactive.com
  • 11. What Splunk Gives Us • Monitoring - to measure application health • Analysis - to drive future product decisions • AB test evaluation - to validate hypotheses • Detection - to find patterns & classify users www.snap-interactive.com
  • 12. Monitoring • With continuous deployment, detailed monitoring is absolutely essential. • Each deploy we watch changes in: • Realtime classified error graphs • Core event stat graphs • We also monitor email deliverability, revenue, and performance (although not every deploy) www.snap-interactive.com
  • 13. Error & Event Monitoring • Alert us immediately after deploy if something has gone wrong • Use realtime background searches • Single dashboard with multiple graphs and tables • We are exploring realtime sms alerts to the ‘developer on call’ • Use historial data to identify min/max expected thresholds (weighted averages: same time of day, same day of week) • Detect consistent deviations and alert www.snap-interactive.com
  • 14. Error Monitoring • Count of all errors : past 30 seconds www.snap-interactive.com
  • 15. Error Monitoring • All errors: past 5 minutes w/deploys www.snap-interactive.com
  • 16. Error Monitoring • Rolled up errors: past 5 minutes www.snap-interactive.com
  • 17. Error Monitoring • Rolled up filtered errors: past 5 minutes www.snap-interactive.com
  • 18. Error Monitoring • Rolled up JS errors: past 3 hours www.snap-interactive.com
  • 19. Event Monitoring • We monitor ~20 event stats, in realtime, each deploy www.snap-interactive.com
  • 20. Event Monitoring • Overview and detail views, powered by a single realtime background search www.snap-interactive.com
  • 21. Monitor Email & Performance • Email • Deliverability is essential to business • Need to maximize engagement • Performance • What async jobs may be contributing to high DB load? • What performance are end users experiencing? • Are particular servers overloaded? www.snap-interactive.com
  • 22. Email Deliverability • Overview of key metrics www.snap-interactive.com
  • 23. Email Monitoring • Inserts into email scheduled send queue www.snap-interactive.com
  • 24. Performance • Asynchronous process timers • Can correlate spikes with site issues www.snap-interactive.com
  • 25. Analysis • We heavily leverage summary indexing for performance gains • Daily rollups are grouped judiciously, giving us fast, flexible, analysis over long periods • We summarize: revenue, email deliverability, core KPI, and general stats data • Custom dashboards facilitate easy searching • Lots of ad hoc searching by product team www.snap-interactive.com
  • 26. Email Analysis • Sends opens, clicks, bounces & FBL rates by email type www.snap-interactive.com
  • 27. Email Analysis • Monitor changes in open & click rates by email, ISP, country, etc. www.snap-interactive.com
  • 28. Email Analysis • Analysis dashboard *Sample data www.snap-interactive.com
  • 29. Core KPI Dashboard • Powerful targeted cohort analysis *Sample data www.snap-interactive.com
  • 30. AB Test Results • We are constantly running a variety of AB experiments on our live users • We divide our user population in to nine 10% segments and ten 1% segments • Each segment can be targeted with an experiment • All event logs are annotated with the appropriate AB experiment name • This allows us to measure behavior changes between experiment and control groups www.snap-interactive.com
  • 31. Easy AB Analysis • All event logs contain the an AB field • This identifies the experiment group of the user at that point in time • Fully integrated into core analysis dashboards • Ad hoc analysis becomes simple `my_search` (AB=my_test OR AB=ctrl) | `my_reporting_command` by AB www.snap-interactive.com
  • 32. AB Dashboards • key metrics: experiment vs control group www.snap-interactive.com
  • 33. Latest Splunking - Detection • The way our users interact with one another is insightful • We can use this data to classify users: • Identifying “attractive” users • Identifying spammers & scammers • We test hypotheses with ad hoc searches • Find reliable patterns then setup scheduled searches that interface with MySQL • This data then feeds into our application in various ways www.snap-interactive.com
  • 34. Contact Us • SNAP Interactive, Inc. www.snap-interactive.com • Nicholas DiSanto Architecture Team Lead ndisanto@snap-interactive.com 301-BIG-TREE @NicholasDiSanto • Lindsay Bubbico www.snap-interactive.com

Editor's Notes

  1. snap is located down the street, our flagship product is areyouinterested.com one of the largest social discovery sites on the web with more than 5 M MAU. publicly traded company, most revenue from subs
  2. lead the architecture team, we build the core infrastructure of the site, explore new technologies and work closely with ops. This includes delivering analytics platforms for produce and monitoring/debugging tools for developers.
  3. we use splunk for a lot of stuff; today i only have time to highlight a few of the more interesting things. specifically i’ll explain what we give splunk, and how; the monitoring capabilities that it gives us and most interesting the analytics we can now perform.
  4. so what do we index? most importantly our custom application log which contains structured... data ;we also profile parts of our application and log that. and of course we index our error_logs
  5. when writing logs from our application we use a centralized logging class that gives us some simple but highly beneficial functionality.
  6. anytime the structured data contains a uid, we integrate key information about that user. considered lookup tables but this is much more performant.
  7. this information gives us tremendous power. we can...
  8. as i mentioned, we also log performance metrics, we simply wrap parts of our code with timers, and log the time spent working
  9. so that’s pretty much what we give splunk, so what does it give us?
  10. With regards to monitoring, we are a continuous deployment shop; deploying site changes 15-50 times a day.
  11. our goal with monitoring is to know immediately if we break something. for performance we use realtime background searches that each power multiple dashboard views
  12. we also have a frontend error monitoring dashboard
  13. we also monitor email and performance very closely
  14. in an attempt to maximize engagement we recently started scheduling all of our email sends. now in addition to monitoring sends we need to monitor the health of the schedule queue
  15. when a server gets hot, one of the first things we try to do is correlate
  16. make data driven product decisions. product team doing lots of analysis, summary indexing for performance. recently upgraded to 5.0 smooth 6min downtime; will continue to use summary indexing.
  17. these mysterious unlabled lines are an example of how we can monitor open and click rates for our emails. targeting specific templates, countries, genders, ISPs, or outbound IP addresses.
  18. this is one of the most interesting dashboards we have. This allows us to perform cohort analysis on our users, tracking a group that joined on a particular day and measure their lifetime behavior