7 Steps for Data-Driven Decision Making

7 Steps for Data-Driven
Decision-Making
Sacha Litman, Managing Director, Measuring
Success
José Fernández, Director of GuideStar Exchange

Questions? info@measuring-success.com #7steps

Agenda

5 minutes Opening poll and introduction
5 minutes Benefits to using data
15 minutes Illustrative Case study
5 minutes Q&A and concurrent poll
25 minutes 7 Steps to data-driven decisions
5 minutes Take-away principles & talking points
10 minutes Q&A and concurrent closing poll
1 minute Conclusion


A. Benefits


Benefits of data-driven decision making
to Customers
Success matters because your mission is important!
If you don’t measure and use data, you are not
achieving the full potential of your mission.


Benefits to Management & Staff
Focus limited time and energies on key activities that are
real issues and have impact on our mission

Set goals for staff and hold them accountable

“Cannot manage what you do not measure” – Peter Drucker

“What gets measured gets done”

“High performing organizations use data analytics roughly
three times as extensively as the lower performers in their
field” – Competing on Analytics, HBS Press


Benefits to Boards and Donors
Boards Donors
Focus their energies away Ability to show “return on
from emotions & anecdotes investment”

Decisions not made by loudest Attract large gifts and grants
or wealthiest voice by confidence engendered by
data
“Analytical projects aimed at
improved outcomes had a Data analytics was one of the
median ROI of 55% for CRM top three areas of increased
and 139% for financial investment by companies
management” during the recession.
– Competing on Analytics, HBS
Press


Popular Press Examples
Professional sports: Oakland A’s (Moneyball)
Maximize team wins while minimizing payroll
By hiring players as indicated by data, not scouts’ gut

Health Care: Intermountain Health Care (NY Times Magazine Nov 9, 2009)
Improve patient outcomes (survival, longevity, quality of life) while
minimizing cost
By selecting treatment/procedure supported by evidence not Doctor’s
gut instinct or unneccessary procedures

Technology: Google
Increase advertising revenues ($21 billion) by targeting ads to right
customers
By mining data on all users of free Google consumer software


B. Quick Case Study


1. Chapter-based
community centers
Experiencing some with member drops, financial insolvency.
Challenge to business model & brand consistency.

Association HQ

Local Affiliates or Regions
(N=150)

Individual Members / Users
(N=1000 to 10,000 per affiliate)


2. Association sought “early warning
system” & understanding of what led to
successful outcomes


3. Build measurement and dashboard
system as pilot experiment

Engaged 6 willing chapters
Built:
Customer survey
Employee survey
Financial analysis tool
Member participation tool


4. Focused on rankings and metrics that
were statistically valid

Regression analysis ties activities to outcomes
Comparison
Against peer chapters in other geographies
Against local competition from other organizations
Against own prior measures (longitudinal)
Within demographic segments of member base
(see following slides)


4a. Identify activities associated with outcomes
(multiple regression analysis)
Budget Management versus Value for Membership Dollar
Correlation
Value for Membership Dollar
Average Score 1-5 Scale

Perceived Budget Management & Transparency
(1= strongly disagree, 5 = strongly agree)


5. Like doctor, ran diagnostics on each
chapter every 2 years
Rolled out pilot to all chapters;
readiness factors to participate
Mutually agreed on a plan for
improvement
Some consulting support

Select Measures from Customer Rank Score Priority Goals & Strategy
Survey (of 15) % Str
Agree

Membership Value for the Dollar 7 25% Medium Focus on
quality, budget
perceptions
Professionals welcoming 2 35% Low
Budget Perceived as well 14 16% High Double scores in
managed 2 years.


Data Creates Organizational
Alignment & Intentionality

2. Consumer Latent Needs
(Purpose-Driven)
1. Consumer
3. Institutional
Expressed
Strategic Vision
Wants

6. Contribution to 4. Financial
People Pipeline Sustainability
(Intake, Attrition) 5. Resource (Surplus/Deficit)
Intensiveness
(Mgmt Time, Sqft,
Foot Traffic)


6. Turnaround: focus lead to improvement from
low to average in 2 years; now aiming for top
Personal Conversations with Customers
Monthly or more often

27%


Data encourages prioritization:
80% of board & mgmt team hypotheses about what we
anecdotally “believe” is a problem is not supported by data!


7. Chapter ruled by anecdotes: 80% of
assumptions were not supported by data
Assumed they were strongest with eldest & wealthiest portions of
participant base
Likelihood to recommend to a friend
% Strongly Agree

25-34 35-44 45-54 55-64 65-74 Over 75

50% 43% 50% 48% 45% 40%

Less than $100,000- $200,000- $300,000- $400,000- $500,000 and
$100,000 $199,999 $299,999 $399,999 $499,999 over

40% 42% 67% 63% 42% 46%

Shock, challenged the data, acceptance
Management team focused energies on improvement with eldest
and wealthiest. 2 years later, scores significantly higher there.


8. Rising Tide Lifts All Boats: top, average,
and low performers all improved
Surplus Margin for Early Childhood Program
Surplus as % of Expenses, after allocating all overhead

X X


9. New policy: value, not price
Highest Willingness
Assumed that key driver of To Pay (center)
Demand Function
member retention was price
C. Perceived
Analysis shows not price, but Quality of Services Least Ability to Affect
perceived quality and value-
for-dollar
Result: association stopped
encouraging price B. Commitment A. Financial
subsidization, encouraged To Issue Ability
perceived quality improvement
Perceived value and value for
dollar are tracked carefully and
promoted system wide


10. Outcome improvement: participation
Chapters that embraced this approach outperformed others
significantly in participant enrollment and financial sustainability,
despite the recession
Participant Retention Rate:
91% to 96%
New Participant Rate:
5% to 10%

Net Participation:
96% to 106%
Financial sustainability increased: coverage ratio (% expenses
from fees & membership) grew from 74% to 80%


C. 7 Steps to Data-Driven
Decision Making


Data driven decision making is parallel to what
we learned in middle school science class
Identify issue
State hypothesis: “I
believe…”
Perceived mechanism/
cause
Design experiment
Examine data
Confirm or reject
hypothesis


7 Stages of Data-Driven Decision Making

1.
Framing
the 2.
Problem Hypothesis
Develop-
ment 3. Data
Collection
4. Data
Analysis 5.
Inter-
pretation 6.
Decision
Making 7.
Commun-
ication


2. Hypothesis Development
Brainstorm hypotheses. Many will turn out to be
incorrect, but that’s ok so long as you can
articulate a plausible mechanism.
Remember, our consistent experience with organizations is
that over 80% of their initial hypotheses as to what is
driving a problem are not supported by the data!
During brainstorm define:
Hypothesis (may need to revise several times)
Mechanism (cause & effect)
How could measure it


2. Hypothesis Development:
Gann Academy
Initial Hypothesis: unevenness in students’ experience
getting their individual learning needs met by the school
Plausible Mechanism: additional academic support
(learning center).
Major differentiator in students’ learning experience (for those
in the same classrooms)
Final Hypothesis: students who received additional
academic support from learning center felt their
individual learning needs were better met by the school.


2. Hypothesis Development:
Gann Academy
How measure?
Parent survey
Question on individual learning needs met by school
Cut data by whether student used learning center
t Scale 10a. Teachers
Understand
Needs & Employ
Effective Learning Learning
Techniques Center Yes Center No

Strongly Disagree 3%
Disagree 9%
Neither Agree Nor Disagree 16% Count 84 240
Agree 44%
Average 3.5 4.0
Strongly Agree 28%

Average 3.8


3. Data Collection
Do… Don’t…
Prioritize hypotheses Just go after what’s
Work backwards: figure out easiest
how will use results of data Overtax your resources –
first before collect it data collection can be very
Consider several methods of time intensive
data collection Go after more data than
Save time and money by you can act upon –
considering simple, pre- Garbage In Garbage Out
built tools allowing some (GIGO)
customization (instead of
building own or buying
highest end product you
won’t use)


3. Data Collection
Tool Pro Con
Survey Data collection phase is To get quality results and high
automated and not resource response rates, need to put lots
intensive of time into asking the right
questions, making it look
Obtain perceptions of quality professional, and assuring
confidentiality
Financial analysis Data your organization already Requires reorganizing and “fine
has slicing” the financials, so need a
strong CFO
Tracking system Integrated into daily efforts Takes a lot of discipline and
(CRM) time to build data
Self-reported from Already have the database Time consuming if database
existing databases does not contain the right
information may end up with
lots of estimates


3. Data Collection : Gann Academy
Good: financial modeling Bad: admission tracking
Clarified board’s needs up Purchased high end
front: 5 year projections, database system
key levers of sustainability Not trained properly
Used already-built but Orphan Excel spreadsheets
simple model (in Excel) Never clarified what
instead of building own management needed for
Worked hard to gather decision making, so
consistent financial data “GIGO”
(counting $ in development
and CFO’s office)
“What if” scenario building
enabled board buy-in


3. Data Collection : Gann Academy
Work backwards. Decide how you will use the data first, as this
will dictate whether and how to ask for the data.


5. Interpretation: draw correct conclusions from
data. Danger in “seeing what want to believe”
You don’t need to be a statistician to correctly interpret data
However, do need to understand basic concepts
Mean
Standard deviation
Counts (sample size)
Benchmarking
Frequently, someone will “find” something in the data that is not
valid but it supports their personal views


5. (Mis)Interpretation: Gann Academy
Relative Strength of Gann Academy’s
Board member who was Attention to Individual Needs vs.
convinced the most severe Most Attractive Alternative School
(higher score is better)
threat on enrollment was from
charter and public schools, not Gann vs. Gann vs. Gann vs.
area private schools Charter
Hypothesis: public schools are or General
Private
better at supporting individual magnet public
Schools
public schools
student needs and thus pose the
schools
greatest risk of attrition for
Gann Academy families.
Count 50 4 51
Data source: teacher attention
Average 3.8 3.4 3.6
to individual needs cut by most
attractive alternative option


5. Interpretation: Benchmarking
Relative to What?
Contextualization is critical:
Peer group (similar organizations
nationally/internationally)
Local competition*
Own longitudinal history
Across demographic groups*

*Depicted on a prior slide


5. Interpretation: Benchmarking

Relative to Peer Group
of other similar high
schools around the
country
Note if had used
primary schools (K-8) as
peer group


7. Communication
Sharing with board, participants, donors, employees
Fear: sharing data will undermine management’s authority
Reality: sharing data engenders trust
Diffuse concerns through transparency
Even if current scores are not as high as would like, donors value a
roadmap toward and metrics for success


7 Stages of Data-Driven Decision Making

1.
Framing
the
2.
Problem
Hypothesis
Develop-
ment 3. Data
Collection
4. Data
Analysis 5.
Inter-
pretation 6.
Decision
Making 7.
Commun-
ication


Common Non-Profit Applications
of Data-driven Decision Making
Outcome measurement
Identifying activities that give greatest “bang for buck” in limited
resource environment
Financial sustainability
Setting program fees or membership dues levels
Identifying most susceptible demographic groups
Donor prioritization using predictive models
Board reporting
Program evaluation


If your organization makes data-driven
decisions you will be able to…
Focus limited energies and resources on only the 20% of hypotheses that
are supported by the data
Motivate staff and volunteers because they know they are investing in
what works
Align efforts of staff, volunteers, board, donors, and customers
Monitor progress toward goals
Have a dashboard for your performance
Make strategic board decisions more efficiently and gain more buy-in
Distinguish yourself against and outperform your competition by
developing a core competence few non-profits have
Observe improvements in key areas such as: financial sustainability,
donations, participation, mission impact


7 Steps for Data-Driven Decision Making

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to 7 Steps for Data-Driven Decision Making

Similar to 7 Steps for Data-Driven Decision Making (20)

More from GuideStar

More from GuideStar (20)

Recently uploaded

Recently uploaded (20)

7 Steps for Data-Driven Decision Making