A presentation from 2008 to a private security conference in the UK.

1. Phil Huggins
Private Conference Spring 2008

2.  Risk
 Uncertainty
 Cardinals versus Ordinals
 Probability Estimation Accuracy
 Suppressing Uncertainty
 Foxes and Hedgehogs
 Uncertainty Calibration

3.  A state of uncertainty
 Set of possible ?negative? outcomes
 A set of possibilities with quantified
probabilities and consequences

4.  Lack of complete certainty
 More than one possibility
 True outcome not known
 A set of probabilities assigned to a set of
possibilities

5.  Cardinal -> Size
 Cardinal -> A number or %
 Ordinal -> Position in Set
 Ordinal -> High/Medium/Low or 1/2/3/4/5
 Ordinals often have different meanings
depending on the person

6.  An estimate of reality
 The usefulness of an estimate has two
measures
 Calibration
▪ How close to reality is the estimate
 Discrimination
▪ The range of the estimate

7.  We’re the ‘experts’
 Client doesn’t want doubt
 Client pushes the risk of the decision onto
you
 Hitting targets for risk reduction focuses on
goals not outcomes

8. ‘The fox knows many things but the
hedgehog knows one big thing’ – Isiah Berlin
Foxes Hedgehogs
• Probability is an ad-hoc • Probability is a deductive
estimation exercise
• Low confidence in their • Highly confident in their
estimates estimates
• Tolerant of uncertainty • Intolerant of uncertainty

9.  Foxes are generally better calibrated and
discriminated
 Less ‘foxy’ foxes are slightly worse at
discrimination
 Best overall at making short term predictions
in field of expertise
 Hedgehogs worst overall when making long
term predictions in field of expertise
 Hedgehogs win less but win big

10.  90% Confidence Interval (CI)
 ‘You are 90% certain the probability of occurrence is
between 60% and 50%’
 95% sure below the upper bound
 95% sure above the lower bound
 Ask yourself 20 general knowledge questions asking for
numerical answers
 Assign lower and upper bounds to your 90% CI of being correct
 Ask yourself 20 true/false questions
 Set a value to how certain you are 50%, 60%, 70%, 80% 90%, 100%

11.  General Knowledge
 You expected to get 9 out of 10 right (90% CI)
 What did you get?
 You probably got less than that right
 True / False
 Convert % to decimal (50% = 0.5)
 Add them up
 Compare to number of questions you got right
 If number of correct answers was 2.5 or more less
then you are overconfident

12.  Take multiple tests in succession
 Equivalence Bets (next slide)
 Pros and Cons
 Consider two pros why you should be confident
and two cons why you could be wrong
 Avoid Anchoring
 95% certain below upper bound
 95% certain above lower bound

13.  Two choices:
1. Win £1000 if answer falls within your upper and lower
bounds.
2. Spin a dial split into 90% and 10% slices. You win £1000 if
it finishes on the 90% slice
 Most people (80%) prefer to spin the dial
 Ask yourself which you would prefer
 Preferring 1 is over confident
 Preferring 2 is under confident
 Just pretending to bet money makes you better

14.  After 3 tests 70% of people are perfectly
calibrated
 After 3 tests 20% of people are significantly
improved
 10% of people never really get it
 Testing has shown that most people relied on
to make estimates are in the first two groups
already

15. Thanks for listening
http://blog.blackswansecurity.com