CalibratedBayes,andInferential
ParadigmforOf7icialStatisticsintheEra
ofBigData
RodLittle
Overview
•  Design-basedversusmodel-basedsurvey
inference
•  CalibratedBayes
•  SomethoughtsonBayesandadaptive
design
Ross-Royall Symposium talk
2
Survey estimation
•  Design-basedinference:populationvaluesare
7ixed,inferenceisbasedonprobability
distributionofsampleselection.Obviouslythis
assumesthatwehaveaprobabilitysample(or
“quasi-randomization”,wherewepretendthat
wehaveone)
•  Model-basedinference:surveyvariablesare
assumedtocomefromastatisticalmodel
•  Probabilitysamplingisnotthebasisfor
inference,butisusefulformakingthesample
selectionignorable.(seee.g.Gelmanetal.,
2003;Little2004)
Ross-Royall Symposium talk
3
Design vs model-based survey inference
•  Twomainvariantsofmodel-basedinference:
–  Superpopulationmodels:Frequentistinference
basedonrepeatedsamplesfroma
“superpopulation”model(Royall)
–  Bayes:addpriordistributionforparameters;
inferenceabout7initepopulationquantitiesor
parametersbasedonposteriordistribution
•  Afascinatingpartofthemoregeneraldebate
aboutfrequentistversusBayesianinferencein
statisticsatlarge:
–  Design-basedinferenceisinherentlyfrequentist
–  Purestformofmodel-basedinferenceisBayes
Ross-Royall Symposium talk
4
Limitations of design-based approach
•  Inferenceisbasedonprobabilitysampling,buttrue
probabilitysamplesareharderandhardertocome
by:
–  Noncontact,nonresponseisincreasing
–  Face-to-faceinterviewsincreasinglyexpensive
–  Can’tdo“bigdata”(e.g.internet,administrativedata)
fromthedesign-basedperspective
•  Theoryisbasicallyasymptotic--limitedtoolsfor
smallsamples,e.g.smallareaestimation
Ross-Royall Symposium talk
5
Design-Based Approach Has Implicit Models
•  Althoughnotexplicitlymodel-based,modelsare
neededtomotivatethechoiceofestimator
–  E.g.theHorvitz-Thompson(HT)estimatorassumesan
yi / π i
implicitHTmodelthatare“exchangeable”(iid
conditionalonparameters)
–  Ifimplicitmodelsareunreasonable,thentheresulting
inferencescanbeverypoorinmoderatesamples(Basu’s
elephantbeinganextremecase)
•  Modelsarisemoreexplicitlyinthe“modelassisted”paradigm(GREG)
Ross-Royall Symposium talk
6
“Quasi”design-based inference
•  Keyfeatureofdesign-basedapproachisweights,
inverselyproportionaltoprobofinclusion
•  Weightsforselection,nonresponse,poststrati7ication
•  Modelingtheinclusionpropensities,usingfrequentist
orBayesianmethods,leadstoweightsthatareless
variable,potentiallyincreasingprecision
•  Inferenceremainsessentiallydesign-based–inmy
view;afullBayesiananalysisinvolvesmodelsforthe
surveyvariables
•  Needtermstocodifythisdistinction:maybeweight
modelingandpredictionmodeling
Ross-Royall Symposium talk
7
Model-based approaches
•  Inmodel-based,ormodel-dependent,approaches,
modelsarethebasisfortheentireinference:estimator,
standarderror,intervalestimation
•  Twomainvariants:
–  Superpopulationmodeling
–  Bayesian(fullprobability)modeling
•  Commonthemeistopredictnon-sampledand
nonrespondingportionofthepopulation,conditionalon
thesampleandmodel
•  Superpopulationmodelsaresuper,butBayesisbetter!
Ross-Royall Symposium talk
8
Parametric models
Usuallypriordistributionisspeci7iedviaparametricmodels:
p(Y | Z ) = ∫ p(Y | Z ,θ ) p(θ | Z )dθ
p(Y | Z ,θ ) = parametric model, as in superpopulation approach
p(θ | Z ) = prior distribution for θ
Inference about θ is then obtained from its posterior
distribution, computed via Bayes’ Theorem:
p(θ | Yinc , Z ) =∝ p(θ | Z ) × L(θ | Yinc , Z )
L(θ | Yinc , Z ) = Likelihood function
That is: Posterior = Prior x Likelihood…
Posterior for θ leads to inference about population
quantities by posterior predictive distribution
Ross-Royall Symposium talk
9
The model-based perspective- pros
•  Flexible,uni7iedapproachforallsurveyproblems
–  Modelsfornonresponse,responseandmatchingerrors,
smallareamodels,combiningdatasources,bigdata
–  Causalinferencerequiresmodels
•  Bayesianapproachisnotasymptotic,providesbetter
small-sampleinferences
•  Probabilitysamplingisjusti7iedasmakingsampling
mechanismignorable,improvingrobustness
–  Rubin’stheoryonignorableselection/nonresponseisthe
rightframeworkforassessingnon-probabilitysamples
Ross-Royall Symposium talk
10
The model-based perspective- cons
•  Explicitdependenceonthechoiceofmodel,which
hassubjectiveelements(butassumptionsare
explicit)
•  Badmodelsprovidebadanswers–justi7iable
concernsabouttheeffectofmodelmisspeci7ication
•  Modelsareneededforallsurveyvariables–need
tounderstandthedata,andpotentialformore
complexcomputations
•  Infrastructure:needpersonneltrainedinstatistical
modeling
Ross-Royall Symposium talk
11
The current “status quo” -- designmodel compromise
•  Design-basedforlargesamples,descriptivestatistics
–  Butmaybemodelassisted,e.g.regressioncalibration:
N
N
i =1
i =1
TˆGREG = ∑ yˆi + ∑ I i ( yi − yˆi ) / π i , yˆi = model prediction
–  modelestimatesadjustedtoprotectagainstmisspeci7ication,
(e.g.Särndal,SwenssonandWretman1992).
•  Model-basedforsmallareaestimation,
nonresponse,timeseries,…
•  Attemptstocapitalizeonbestfeaturesofboth
paradigms…but…attheexpenseof“inferential
schizophrenia”(Little2012)?
Ross-Royall Symposium talk
12
Example: when is an area “small”?
n
o
m
e
t
e
r
Design-based inference
-----------------------------------
n0 = “Point of
inferential
schizophrenia”
Model-based inference
How do I choose n0?
If n0 = 35, should my entire statistical philosophy
and inference be different when n=34 and n=36?
n=36, CI: [
n=34, CI: [
] (wider since based on direct estimate)
] (narrower since based on model)
Ross-Royall Symposium talk
13
Multilevel (hierarchical Bayes) models
n
o
m
e
t
e
r
µ%a = wa yπ a + (1 − wa )µˆ a
Model estimate
Direct estimate
1
wa 0
Sample size n
Bayesian multilevel model estimates borrow
strength increasingly from model as n decreases
Ross-Royall Symposium talk
14
Calibrated Bayes
•  FrequentistsshouldbeBayesian
–  Bayesisoptimalunderacorrectlyspeci7iedmodel
•  Bayesiansshouldbefrequentist
–  Weneverknowthemodel(andallmodelsarewrong)
–  Inferencesshouldberobusttomisspeci7ication,havegood
repeatedsamplingcharacteristics
•  CalibratedBayes(Box1980,Rubin1984,Little2006,2012,
2013)
–  InferencebasedonaBayesianmodel
–  Modelchosentoyieldinferencesthatarewell-calibratedin
afrequentistsense
–  Aimforposteriorcredibilityintervalsthathave
(approximately)nominalfrequentistcoverage
Ross-Royall Symposium talk
15
Calibrated Bayes models for surveys should
incorporate sample design features
•  The“Calibrated”partofCalibratedBayesimplies:
•  Generallyweakpriorsthataredominatedbythe
likelihood(“objectiveBayes”)
•  Modelsthatincorporatesamplingdesignfeatures:
–  Capturedesignweightsandstratifyingvariablesas
covariatesinthepredictionmodel(e.g.Gelman2007)
–  Clusteringviahierarchicalrandomeffectsmodels
Ross-Royall Symposium talk
16
Full model for Y and I
p(Y , I | Z ,θ , φ ) = p ( Y | Z , θ ) p ( I | Y , Z , φ )
Model for
Population
Model for
Inclusion
•  Fullposteriordistributionofparameters(hard):
p(θ ,φ | Yobs , Z , I ) ∝ p(θ ,φ | Z ) L(θ ,φ | Yobs , Z , I )
•  Posteriordistributionignoringtheinclusionmechanism
(easier):
p(θ | Yobs , Z ) ∝ p(θ | Z ) L(θ | Yobs , Z )
•  Whenthefullposteriorreducestothissimplerposterior,
theinclusionmechanismiscalledignorableforBayesian
inference(Rubin1976)
Ross-Royall Symposium talk
17
Conditions when inclusion mechanism
can be ignored
•  Twogeneralandsimplesuf7icientconditionsforignoringthe
data-collectionmechanismare:
Inclusion at Random (IAR):
p( I | Y , Z ,φ ) = p( I | Y , Z , φ ) for all Y .
obs
Bayesian Distinctness:
p(θ , φ | Z ) = p(θ | Z ) p(φ | Z )
•  Ignorabilityisspeci7ictothesurveyvariableY,unlike
probabilitysampling,whichguaranteesignorabilityforany
outcome
•  Inadaptivedesign,canincludeparadataorsurveydata
Yobs
fromearlierwaves
Ross-Royall Symposium talk
18
Bayes and responsive design
•  PredictiveBayesmodelinghasmorepotential
forgainsinef7iciencythanBayesianweight
modeling
–  Needtomodelsurveyvariables!
–  Speci7ically,modelrelationshipofsurveyvariables
withweights(ascovariates)
Ross-Royall Symposium talk
19
Example: subsampling callbacks
•  ElliottandLittle(2000JASA)assessedsubsampling
callbacksforNationalComorbidityStudy(NCS)
•  “Ouranalysissuggeststhatrandomlydroppinga
subsetoflatecallbackswillsaveresourceswhenever
(a)thepercall-backorperinterviewcostisincreasing,
or(b)theprobabil-ityofasuccessfulinterviewattempt
isdecreasing…Ingeneral,itappearsthatsurveyswith
constantormodestlyincreasingcallbackcosts,suchas
the1991NCS,yieldtrivialsavings,whereassurveys
thatchangemodefrompostaltotelephoneorface-tofaceinterview,suchastheU.S.CensusBureau'sACS,
yieldsubstantialsavings.”
Ross-Royall Symposium talk
20
Example: subsampling callbacks
•  “…ourapproachyieldsconservativeestimatesof
ef7iciencygainsfromsubsampling,inthesensethat
calculationshaveassumeddesign-basedinference
forpopulationmeans,withweightsincludedto
compensatefordifferentialprobabilitiesof
selection.Ifmodelingassumptionsaremadeaboutthe
distributionsofoutcomesacrosscallbackstrata,
thendifferentsubsamplingschemesmightbeoptimal”
ElliottandLittle(2000)
Ross-Royall Symposium talk
21
Example: weighting for nonresponse
corr 2 ( X , Y )
Low
High
bias ---,var --- bias ---, var ⇓
corr ( X , R)
High bias ---, var ↑ bias ⇓, var ⇓
2
Low
Too often weighting adjustments put us
here … Modeling of relationship
between weights and the outcomes is
needed to get us out of this square!
Ross-Royall Symposium talk
We need good
predictors of Y –
but we focus on
predictors of R…
22
Example: Penalized Spline of
Propensity Prediction (PSPP)
•  PSPP (Little & An 2004, Zhang & Little 2009, 2011).
•  Regressionimputationthatis
–  Non-parametric (spline) on the propensity to respond
–  Parametriconothercovariates
•  Exploits the key property of the propensity score that
conditional on the propensity score and assuming
missing at random, missingness of Y does not depend
on other covariates
•  This property leads to a model-based version of double
robustness (as in GREG).
•  Does very well in simulation studies
Ross-Royall Symposium talk
23
Penalized Spline of Propensity model
Estimate: Y*=logit (Pr(R=1|X1,…,Xp ))
Impute using the regression model:
(Y | Y * , X 1 ,..., X p ; β ) ~
N ( s(Y * ) + g (Y * , X 2 ,..., X p ; β ), σ 2 )
§ Nonparametric part
§ Needs to be correctly
specified
§ We choose penalized spline
§ Parametric part
§ Misspecification does not
lead to bias
§ Increases precision
§ X1 excluded to prevent multicollinearity
Ross-Royall Symposium talk
24
Missing Not at Random Models
•  Dif7icultproblem,sinceinformationto7it
non-MARislimitedandhighlydependenton
assumptions
•  Sensitivityanalysisispreferredapproach–
thoughthisformofanalysisisnotappealing
toconsumersofstatistics,whowantclear
answers
Ross-Royall Symposium talk
25
An MNAR model: Proxy Pattern-Mixture Analysis
xi = x( zi ) = best predictor of yi given covariates zi
(estimated on respondents, and scaled to same variance as yi )
[ yi , xi | ri = r ] ~ G ( µ ( r ) , Σ( r ) )
Pr(ri = 1| xi , yi ) = g ( yi* (λ ) ) , yi* (λ ) = xi + λ yi
MAR: λ = 0, MNAR: λ ≠ 0 (Andridge and Little 2011)
[yi indep ri | yi* (λ )], which identifies the model for given λ
g () is arbitrary, unspecified
Sensitivity analysis for different choices of λ (e.g. 0,1,∞)
If xi is a noisy measure of yi , it may be plausible to assume λ = ∞
leading to method for adjustment for predictors with measurement
error (West and Little, Applied Statistics 2013)
Ross-Royall Symposium talk
26
Indices of potential absolute bias
(PAB) for a mean
λ = ∞ leads to following measures of bias for mean of Y :
•  LetbetheestimatedcorrelationbetweenXandY,
ρˆ > 0
basedonthesampledata.
x
•  LetdenotethesamplemeanofXfromthe
administrativedataandbethemeansofXandY
xR , y R
fromtherespondents.
•  De7inetheunadjustedpotentialabsolutebias(PABU)as
PABU = x − xR / ρˆ
•  De7inetheadjustedpotentialabsolutebias(PABA)as
PABA = x − xR (1 − ρˆ 2 ) / ρˆ
Ross-Royall Symposium talk
27
Bayes and responsive design
•  Developpriorsbasedonprevioussurveys
–  Design-basedapproachignores(ortreats
informally)informationfromprevioussurveys
–  Bayescanusepriorsurveysas“meta-data”to
informdecisionsforcurrentsurvey
–  Priorscanaccommodatedown-weightingof
previoussurveyinformation:e.g.“power”priors
(ChenandIbrahim2000StatScience)
–  Bayesianpowercalculations–neglectedtopic,
particularlyinsamplesurveycontext
Ross-Royall Symposium talk
28
Bayesian updating
•  Bayesruleisnatural…thetheorem…forsequential
decision-making:
Dk = data at stage k
p(θ | D0 , D1 ,...., Dk ) ∝ p(θ | D0 , D1 ,..., Dk −1 ) L(θ | Dk )
•  Selectionisignorableforlikelihoodinference,ifdesign
atanystagedependsondatabeforethatstage
•  Basisforsequentialtreatmentallocationinclinical
trials–whichmodelstheoutcomes!
•  Relationshipbetweenoutcomesandpropensity(e.g.
PSPP)canbemodeledandupdatedfrompriorstages
Ross-Royall Symposium talk
29
Conclusion
•  IviewBayesianmodelingasanatural
frameworkfordevelopingresponsivedesign
andanalysis
•  Nofreelunch:modelsmakeassumptions
•  Butassumptionsareexplicitandcanbe
evaluatedandcriticized.
Ross-Royall Symposium talk
30
References 1
Box,G.E.P.(1980),SamplingandBayesinferenceinscienti7ic
modelingandrobustness(withdiscussion),JRSSA,143,383-430.
Joyce,P.M.,Malec,D.,Little,R.J.,Gilary,A.,Navarro,A.andAsiala,
M.E.(2014).StatisticalModelingMethodologyfortheVotingRights
ActSection203LanguageAssistanceDeterminations.JASA,109,
36-47.
Gelman,A.(2007).Struggleswithsurveyweightingandregression
modeling.Statist.Sci.,22,2,153-164(withdiscussionand
rejoinder).
Gelman,A.,Carlin,J.B.,Stern,H.S.andRubin,D.B.(2003),Bayesian
DataAnalysis,2nd.edition.NewYork:CRCPress.
Godambe,V.P.(1955).Auni7iedtheoryofsamplingfrom7inite
populations.JRSSB,17,269-278.
Horvitz,D.G.&Thompson,D.J.(1952).Ageneralizationofsampling
withoutreplacementfroma7initeuniverse.JASA,47,663-685.
Little,R.J.A.(2004).ToModelorNottoModel?CompetingModesof
InferenceforFinitePopulationSampling.JASA,99,546-556.
Ross-Royall Symposium talk
31
References 2
Little,R.J.A.(2006).CalibratedBayes:ABayes/frequentistroadmap.
Am.Statist.,60,3,213-223
_____(2012).CalibratedBayes:analternativeinferentialparadigmfor
of7icialstatistics(withdiscussionandrejoinder).JOS,28,3,309-372.
_____(2013).SurveySampling:PastControversies,CurrentOrthodoxies,
andFutureParadigms.InPast,PresentandFutureofStatisticalScience,
COPSS50thAnniversaryVolume,X.Lin,D.L.Banks,C.Genest,G.
Molenberghs,D.W.Scott,andJ.-L.Wang,eds.CRCPress.
Rubin,DB(1984),Bayesianlyjusti7iableandrelevantfrequency
calculationsfortheappliedstatistician,AnnalsStatist.12,1151-1172.
Särndal,C.-E.,Swensson,B.&Wretman,J.H.(1992),ModelAssisted
SurveySampling,SpringerVerlag:NewYork.
Zheng,H.&Little,R.J.(2005).Inferenceforthepopulationtotalfrom
probability-proportional-to-sizesamplesbasedonpredictionsfroma
penalizedsplinenonparametricmodel.JOS,21,1-20.
Ross-Royall Symposium talk
32