Nadia, a marketing director, wants to know if a new video ad campaign will be effective. She discusses impact tracking and key performance indicators with the speaker before launching the campaign. The speaker recommends taking a Bayesian approach when evaluating marketing campaigns, even when A/B testing is not possible. This involves using tools like CausalImpact to build a model based on historical data, simulate what would have occurred without the campaign, and compare it to actual post-campaign data to estimate the campaign's impact. The speaker provides a workflow example using Bayesian time series modeling and CausalImpact to help Nadia evaluate her new ad spend policy for a website, without running a controlled experiment.

1. 2019
2019
EARL London | 10-12 September, 2019
“Hard Talks”: Talking Bayes to Business
A marketing spend use-case
Yizhar (Izzy) Toren

2. 2019
About me
! Adopted R at 2003 & introduced it to my university
! Bayesian by belief - Frequentist by practice
! I call myself a “Data Scientist” because I know math, stats & just
enough programming to be dangerous
! Currently focused on forecasting & causality (for elasticity,
optimisation, etc.) and NLP for recommendations & search
Find me on @BigEndianB Linkedin github.com/ytoren

3. 2019
Agenda
! Motivation: Is my ad campaign working?
! Why Bayes?
! Use Case: Measuring impact without a test
○ Toolkits: CausalImpact
○ A workflow example

4. 2019
Meet Nadia
Nadia is a marketing director.
Nadia is smart.
She wants to know if a new video ad campaign will
be effective.
She talks to you about impact, tracking & KPIs before
planning the campaign.
BE LIKE NADIA
🙋

5. 2019
💁
Meet Nadia
Nadia is a marketing director.
Nadia is smart responsible.
She wants to know if a new video ad campaign will
be effective.
She talks to you about impact, tracking & KPIs before
planning releasing the campaign.
BE LIKE NADIA, but be better next time

6. 2019
Meet Nadia
Nadia is a marketing director.
Nadia is smart responsible.
She wants to know if a new video ad campaign will
be effective.
She talks to you about impact, tracking & KPIs before
planning after releasing the campaign.
BE LIKE NADIA, but be better next time
🙎

7. 2019
In a perfect the real world
! We have a model of population & causality
(e.g. more ads ➡ more signups)
! We have well defined KPIs (daily signups) and
understanding of effect size
! Sufficient volume for significance & power
! Sufficient velocity for timely answer
! Good randomisation & user tracking infra for
A/B tests
💁
harder than
you’d think

8. 2019
Is it working?
Good news! We pass
the IOTT (Intra-Ocular
Trauma Test)
Test group
after
before
95% CI: [102.2,130.9]
P-value < 2e-15

9. 2019
Is it still working?
Life is noisy and complicated ➡ Let’s test!
- Nadia: “Can we say the ad campaign worked?”
- You: “Well… we saw X% increase daily visits, with p < 0.005”
- Nadia: “So… 99.5% its working?”
- You: “ , also not necessarily”
And without a testing? 😳
Test group

10. 2019
So Why Bayes?
! Get the answers you want (p-values = hard talk)
! A healthy conversation with stakeholders (priors)
! Problems first not solutions backwards
! Sometimes you just can’t test
! Because you have nifty tools in *
* And some of them only in R, so you now have a great excuse to introduce R into your org toolset!

11. 2019
Use Case
! On 09/12 we implemented a new ad-spend policy on a
content website (“ThyPipe”), where we can’t run tests
! Nadia asks: “Is the new policy better than the old one?”
! Translation: what would have happened if we did not
change anything? (a.k.a “The Counterfactual”)

12. 2019
Data
! User signups from different sources (ours is source1)
! Queries from a search engine (rhymes with “Doodle”)
! Calendar: Holidays, seasonality, feature releases, etc.

13. 2019
Possible Approaches
! A/B testing
! Compare “Before” / “After” ❌
! Difference in differences 🔎🦄
! Multivariate regression + 🔎📦
time series + GLM + …
! Bayesian Time Series ✔
We “got lucky”: CausalImpact 🎯

14. 2019
CausalImpact
Step 1: Fit the best model you can
on “before” data (out-of-the box or
customized via BSTS)
Step 2: Compare actual “after” data
to the model simulation of the
same period: This is the effect!
“We see 4% increase but it’s probably
noise: 45% chance we’re below the min
3% uplift required”
Before After

15. 2019
Workflow example
Model
(BSTS)
Simulations
Actual
DB signals
(historical data, regressors)
Calendar
Git signals
Manual Signals
CausalImpact
Work inProgress!

16. 2019
Summary
! A/B testing is great, when testing is feasible and the
answers are meaningful
! If you can’t test - simulate!
! Think problem first, not solution backwards
! Priors are an opportunity to engage with stakeholders
! Use powerful tools, but with care 🕸
! More details on github.com/ytoren/presentation-bsts

17. 2019
Questions?

18. 2019
Thank you!
We’re Hiring!
Find me on @BigEndianB Linkedin github.com/ytoren

19. 2019
Appendix

20. 2019
The answers you want
P(“it works”) P(data|“it works”)
P(data)
P(“it works”|data) =
The answer
Nadia wants
Prior Likelihood
Might be Hard
to Compute
p-value = P(data|”it’s not working”)
You are already having a “hard talk”...

21. 2019
Priors: The “Hard Talk”
• Choice of priors: subjective 🙀, but there are guidelines
• New discussions with stakeholders:
• Internally: “if you had to guess”, surveys, games, ...
• Externally: industry benchmarks
• Some obvious defaults (mean=0, “natural” limits, ...)
• Defaults from your tools (when in doubt - )
Your new job: Translate business insights into a distribution

22. 2019
Time to
Solve
Problem
Scope
Time to
Solve
Problem
Scope
Thinking & Framing
Tools
Scope
Tool
Scope
Frequentist: “Solution Backwards” Bayesian: “Problem First”
! Frequentist: phrase the problem to fit the tools
! Bayes: find a model that fits the problem (but in a finite time…)
Solutions
Solutions

23. 2019
Some More Toolkits
! Stan
○ Fully flexible & powerful
○ New syntax
○ Cross platform
○ Multiple R wrappers (BRMS, stanARM, …)
! Prophet:
○ Stan wrapper
○ R & python bindings
○ (log) additive Gaussian only
○ Time series trend oriented
! BSTS:
○ R library (with R syntax)
○ (log) additive Gaussian / Poisson(sometimes)
○ Regression/causality oriented
○ CausalImpact is a wrapper
Easy Hard
Flexible
Specific
BSTS