10/31/2016
PREDICTING LIKE A PRO
“All models are wrong, but some are useful”
Frank D. Cohen
Director of Analytics
Doctors Management
[email protected]
Frank D. Cohen does not have a financial conflicts to report at this time.
LEARNING OBJECTIVES
• Examine how predicting and estimating are powerful partners
• Compare four techniques to become a better predictor
• Predict outcomes to improve business outcomes
2
1
10/31/2016
THE FOX AND THE HEDGEHOG
"The fox knows many things, but the hedgehog knows one big thing.“
[Archilocus, circa 1500 A.D.]
A fox and a hedgehog were strolling through a country path. Periodically,
they were threatened by hungry wolves. The fox —being blessed with
smarts, speed and agility — would lead packs of wolves on a wild chase
through the fields, up and down trees, and over hill and dale. Eventually
the fox would return to the path, breathless but having lost the wolves,
and continue walking. The hedgehog, being endowed with a coat of
spikes, simply hunkered down on its haunches when menaced by the
wolves and fended them off without moving. When they gave up, he
would return to his stroll unperturbed.
4
2
10/31/2016
THE HEDGEHOG
• View the world through the lens of a single defining
idea
• If the only tool you have is a hammer, then pretty soon,
everything starts to look like a nail
• “Hedgehogs have the keen ability to focus and drive
along a single path” [Isaiah Berlin]
• Those who built the good-to-great companies were, to
one degree or another, hedgehogs. They used their
hedgehog nature to drive toward what we came to call
a Hedgehog Concept for their companies. [Jim Collins,
“Good to Great”]
5
THE FOX
• Draws on a wide variety of experiences and for whom
the world cannot be boiled down to a single idea
• “Foxes are complex thinkers who account for a variety
of circumstances and experiences” [Isaiah Berlin]
• “Those who led the comparison companies tended to be
foxes, never gaining the clarifying advantage of a
Hedgehog Concept, being instead scattered, diffused,
and inconsistent.” [Jim Collins, “Good to Great”]
6
3
10/31/2016
ARE YOU A FOX OR A HEDGEHOG?
• Hedgehogs tend to have a focused worldview, an
ideological leaning, strong convictions; foxes are
more cautious, more centrist, more likely to
adjust their views, more pragmatic, more prone
to self-doubt, more inclined to see complexity
and nuance. And it turns out that while foxes
don’t give great sound-bites, they are far more
likely to get things right. [Nicholas Kristof,
Columnist for the New York Times]
7
PREDICTION VS. ESTIMATION
• Estimation is a rough calculation of the value,
number, quantity, or extent of something based
on existing data
• Predicting is, more or less, estimating some
value in the future without having the
information necessary to accurately estimate
• Which is harder; predicting or estimating?
4
10/31/2016
ARE YOU A GOOD PREDICTOR?
• The phenomenon that one’s incompetence in a
particular area also renders the individual
incapable of detecting his or her own
incompetence, resulting in a false sense of
confidence.
• The same failings that make them incompetent
also make them unable to see it.
[The Dunning-Kruger Effect]
HISTORICAL PREDICTIVE METHODS
• Oracle of Delphi
• Astrology
• Tarot cards
• Secret codes in sacred
texts
• Ouija board
• Paul the Octopus
• Tea leaves
5
10/31/2016
US OF PA IN MANAGEMENT
Predicting is hard, especially predicting the future
[Niels Bohr]
CHAOS AND THE PREDICTION HORIZON
• When a perturbation goes unchecked,
the system goes to chaos
• Ashby’s law of requisite variety
• By reducing uncertainty, we can
extend the prediction horizon
• The goal, then, is not to eliminate
chaos but rather to move the
prediction horizon to the right
6
10/31/2016
PREDICTION METHODS
• Linear decomposition
• Prediction by Analogy
• Extrapolation method
• Trend analysis / Time-series analysis
• Scenario method
• The Fermi Method
• Diversity Prediction Theorem
LINEAR DECOMPOSITION
• Linear decomposition relies upon the ability to
deconstruct (or decompose) an aggregate whole into its
component parts
– The value of the parts together equal (or approximate) the value
of the whole
– We have the ability to measure (or estimate) the value of the
parts
• What will it cost to open a new medical practice
• How much will it cost to bring on a new physician
• How much productivity loss can I expect when
implementing a new EHR?
7
10/31/2016
LINEAR DECOMPOSITION – PROS AND
CONS
• Pros
– We often know all of the parts that make up a whole
– There are many resources to research the parts’ values
– Quite accurate at predicting as long as the model is
linear
• Cons
– Sometimes, it is difficult to know the value of each of
the parts
– The value of the parts does not always equal the value
of the whole
– Not all interactions are linear
PREDICTION BY ANALOGY
• Estimating the value of a given entity
based on similar values in a different
entity
• What is the RVU value of a procedure that
does not have an official RVU assigned?
• How much time will it take to perform a
new procedure?
• "Used with care, an analogy is a form of
scientific model that can be used to
analyze and explain the behavior of other
phenomena.“
–
[Future Ready: How to Master Business Forecasting. John Wiley & Sons.
p. 287. ISBN 978-0-470-74705-]
8
10/31/2016
PREDICTION BY ANALOGY – PROS AND
CONS
• Pros
– Different people see relationships differently (different boxes)
– Analogous measurements are effective at setting upper and lower
boundaries
– Known information is known; it can be verified
• Cons
– Focus only on the similarities
– Sometimes, we try to fit an event into an analogy when it doesn’t
fit
– We oftentimes accept assumptions without testing or validating
EXTRAPOLATION METHOD
• Extrapolation is an estimation of a value
based on extending a known sequence of
values or facts beyond the area that is
certainly known.
• Requires that the known sequence (sample)
is representative of the universe to which
the prediction will fall
• What is the predicted damage estimate for a
CMS audit?
• How much time and effort can I expect my
providers to report?
• What should be my bonus pool for next
year?
9
10/31/2016
EXTRAPOLATION METHOD – PROS AND
CONS
• Pros
– Based on scientific principles and therefore, is replicable
– Accuracy of an extrapolated estimate is known (can be
calculated)
– Widely accepted by statistical community and courts
• Cons
– Sampling is never absolutely precise (range of values v. an
absolute value)
– Sampling has some probability of missing critical elements
– Sampling itself has many potential pitfalls
TREND ANALYSIS
• Trend analysis is the process of comparing data over time to identify results or
trends and to predict events and values beyond the end of the time sequence
• How many work RVUs will be reported next year?
• How long will it take a new physician to cover their expenses?
• What will our compliance risk look like left unchecked?
• What will our cost-per-RVU be over the next 24 months?
10
10/31/2016
TREND ANALYSIS – PROS AND COS
• Pros
– Widespread application responsible for many software solutions
– Based on historical data, which can be validated and verified
– Statistical modeling creates a range of estimates, allowing for
error
• Cons
– Historical data are not always indicative of future events
– It is difficult to predict ‘turning points’ in a time-series data set
– There is a predictive horizon that is not always known a prior
SCENARIO METHOD
• The process of
– visualizing what future conditions or events are probable
– What their consequences or effects would be like and
– How to respond to and/or benefit from them
• Must be physically, socially and politically plausible
• What EHR system fits my practice based on 3 future scenarios?
• What will be the financial impact if ACA is replaced?
11
10/31/2016
SCENARIO METHOD – PROS AND CONS
• Pros
– Doesn’t necessarily tell you what you should or should not do
– Produces up to many different potential outcomes and events
– Very useful in situations with high stakes and high uncertainty
• Cons
– Very broad with regard to chronological, geographical and
thematic scope
– The problems of complexity can obfuscate obvious observations
– Uncertainty and impact can be highly subjective (who chooses?)
THE FERMI METHOD (DIMENSIONAL
ANALYSIS)
• A Fermi problem is a multi-step problem that can be solved in a
variety of ways, and whose solution requires the estimation of
key pieces of information
– Break the problem into parts and then multiply the parts together
• How many piano tuners in Chicago?
• How many golf balls would fit into a minivan?
• How many physicians are needed in a given community?
• Estimate the number of people that will visit ERs this year?
• How much wood could a wood chuck chuck if a wood chuck
could chuck wood?
12
10/31/2016
HOW MANY PIANO TUNERS IN LOS ANGELES?
• 3.2 million households
• 20% have pianos (640,000)
• 10% tune regularly (64,000)
• Tune on average once a year (64,0000
tunings)
• 2 hours to tune on average (128,000
hours)
• 2080 work hours per year (61 work
years)
• 61 Piano tuners in Los Angeles
THE FERMI METHOD – PROS AND CONS
• Pros
– Often, a quick and simple initial method to underlie a formal
approach
– Helps to identify where errors may lie within a complex
calculation
– Normally, the tools and information we need are readily available
• Cons
– Most often, answers are not overly accurate
– Many assumptions must be made, which can quickly multiply
error
– Can become a lazy alternative to what should be more rigorous
effort
13
10/31/2016
DIVERSITY PREDICTION THEOREM
Error held as truth has much the effect of truth. In politics and
religion this fact upsets many confident predictions.
[George Iles]
DIVERSITY PREDICTION THEOREM
• A diverse crowd will always be more accurate than its
average member
– A crowd’s squared error equals the average individual squared
error minus the diversity of the predictions
• The greater the diversity, the better the outcome,
specifically:
– Problem solving
– Prediction and projection
– Preference aggregation
14
10/31/2016
THE JELLY BEAN EXAMPLE
29
ARE YOU A FOX OR A HEDGEHOG?
• Hedgehogs tend to have a focused worldview, an
ideological leaning, strong convictions; foxes are
more cautious, more centrist, more likely to
adjust their views, more pragmatic, more prone
to self-doubt, more inclined to see complexity
and nuance. And it turns out that while foxes
don’t give great sound-bites, they are far more
likely to get things right. [Nicholas Kristof, Columnist
for the New York Times]
30
15
10/31/2016
LET FRANK KNOW WHAT YOU THOUGHT!
Fill out the speaker evaluation
emailed to you at the end of each
day or immediately through the
MGMA16 mobile app.
FOR MORE INFORMATION
• Director of Analytics
• Doctors Management
• [email protected]
• 727.442.9117
16