1000 grandy using our laptop

Pragmatic Analytics
Mike Grandy
AVP – Data Science, MetLife
October 31, 2017

Pragmatists Demand Results
“The overriding goal of analytics in financial
services companies is to generate business
value.”

Why Do Analytics Projects Fail?
Three reasons why analytics projects can
fail to achieve expected business value
Ill-Defined Opportunity
Human Bias
Methodological Overreach

Reason 1:
Ill-Defined Opportunity
Problem:Addressing the wrong question
New Data:
Should we buy?
New Tech:
Should we use?
Pain Point:
How to fix?
KPI:
Can you predict?
Solution: Recast as a modified decision hypothesis
Model Outcome Decision Metric Benefit

Well-Defined Opportunity
• Data needed
• Target
• Potential predictors
• Model DB structure
• Tool/algorithm
How to build the model
• Value metric
• Method
• Test vs. control
• Pre vs. post
How to track the value
Framing the opportunity well leads you to…

Reason 2:
Human Bias
Confirmation Bias
Decisions skewed by
pre-existing beliefs
Post-hoc Storytelling
Hindsight cements
the interpretation
Vested Interests
Weak, overly simple and
intuitive models don’t impress

Countering Human Bias
Self-awareness
Collaboration
w/SME
Open-
mindedness
to others’ ideas
Skepticism
of own ideas
Tolerance of
ambiguity Traits to
Cultivate
Reduce dependence on
human judgment
(more exhaustive search)
Recommended Reading:
The Signal and the Noise by Nate Silver
Superforecasting by D. Gardner & P.Tetlock
You Are Not So Smart by David McRaney
The Undoing Project by Michael Lewis
Moneyball by Michael Lewis

Reason 3:
Methodological Overreach
When we become overly aggressive in our
modeling methodology without proper
safeguards against models not generalizing
to new data

Thought Experiment #1
 Imagine you are forced to build a model
that will perform just as well on new data
 What would I want?
◦ Large holdout dataset to validate
◦ Easy modeling problem
 Many, many observations
 Predictable target
 Enough predictors logically linked to target
 A simple, safe model

Model Risk Grid
No
validation
data
Large, pure
holdout
dataset
Robustness of validation
Modeling
Degree of
Difficulty
EASY:
Many cases
Predictable target
Lots of attributes
HARD:
Low volume
Noisy target
Limited attributes
1
Limited
validation
data
Increasing
Risk
Of
Not
Generalizing
2

Highest lift from worst possible data
Cases
Attributes
Tactics
High cardinality attributes
Augment attributes
More Attributes
Exhaustive search
A t t r i b u t e s
A
t
t
r
i
b
u
t
e
s
Create new attributes from
existing attributes:
Transformations
Calculations of attribute pairs
(difference, ratio, etc.)

Highest lift from worst possible data Tactics
High cardinality attributes
Augment attributes
Exhaustive search
Sample/resample cases
Utilize interactions
Case Attr 1 Attr 2 Attr 3 Attr 4 Attr 5 … Attr n
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
1
2
4
5
9
12
13
15
16
20
1
2
4
5
9
12
13
15
16
20
1
2
4
5
9
12
13
15
16
20
2
15
12
9
15
1
16
2
5
4
End Result:
Highly complex descriptive model

How Good DoesYour Model Need
to Be?
 “Why Most Published Research Findings
Are False,” by John Ioannidis
◦ Root cause: lack of robust validation
 Considering how robust my validation can
be, and how difficult the modeling
problem, how aggressive/safe should I be?
◦ Sometimes a safe model is not enough
◦ Often a safe but upgradeable model is ideal
◦ “Best” model doesn’t always mean “most lift”

Validate,Validate,Validate
 Use the largest validation dataset you can
◦ Pure holdout, recent data
 If your algorithm is “self-validating”
◦ Validate it yourself anyway
◦ Or, understand how well it prevents overfit
 See Dr. John Elder’s “Target Shuffling” concept
 Track the business value, post-
deployment…no excuses

Conclusion
Pragmatists don’t take chances on business value
 Define your analytics opportunities well, upfront
 Guard against human bias
 Avoid unnecessary risks from methodological overreach
 Validate as thoroughly as possible
 Track the business value post-deployment
Act as though your professional reputation depends on
your ability to generate real business value

1000 grandy using our laptop

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to 1000 grandy using our laptop

Similar to 1000 grandy using our laptop (20)

More from Rising Media, Inc.

More from Rising Media, Inc. (20)

Recently uploaded

Recently uploaded (20)

1000 grandy using our laptop