MaximumLikelihoodApproachfor
SymmetricDistributionProperty
Estimation
Jayadev Acharya,Hirakendu Das,Alon Orlitsky,Ananda Suresh
Cornell,Yahoo,UCSD,Google
Propertyestimation
• 𝒑:unknown discretedistributionover𝒌 elements
• 𝒇(𝒑):apropertyof𝒑
• 𝜺:accuracyparameter,𝜹: errorprobability
• Given:accesstoindependentsamplesfrom𝒑
• Goal:estimate𝒇(𝒑) to±𝜺 withprobability > 𝟏 − 𝜹
Usually,𝜹 constant,say0.1
Focusofthetalk:large𝒌
Samplecomplexity
𝑋. , 𝑋0 , … , 𝑋2 :independentsamplesfrom𝑝
𝑓5(𝑋.2):estimateof𝑓(𝑝)
Samplecomplexityof𝑓5 (𝑋.2):
𝑆 𝑓5, 𝜀, 𝑘, 𝛿 = min{𝑛: ∀𝑝, Pr |𝑓5(𝑋.2 ) − 𝑓(𝑝)| ≥ 𝜀 ≤ 𝛿}
Samplecomplexityof𝑓:
𝑆 𝑓, 𝜀, 𝑘, 𝛿 = min 𝑆 𝑓5 , 𝜀, 𝑘, 𝛿
G5
Symmetricproperties
Permutingsymbollabelsdoesnotchange𝑓(𝑝)
Examples:
• 𝐻 𝑝 ≜ − ∑K 𝑝K ⋅ log 𝑝K
• 𝑆 𝑝 ≜ ∑K 1QRST
Renyi entropy,distancetouniformity,unseensymbols,
divergences,etc
Sequencemaximumlikelihood
𝑁K : #times𝑥 appearsin𝑋.2 (multiplicity)
𝑝W :empiricaldistribution
𝑝KW =
𝑁K
𝑛
𝑓 W 𝑋.2 = 𝑓(𝑝W )
Empiricalestimatorsaremaximumlikelihoodestimators
𝑝W = arg max 𝑝(𝑋.2 )
Q
Callthissequencemaximumlikelihood (SML)
Empiricalentropyestimation
Empiricalestimator:
𝐻W
𝑁K
𝑛
=𝐻
= Z log .
𝑛
𝑁K
K
𝑘
W
𝑆 𝐻 , 𝜀, 𝑘, 0.1 = Θ
𝜀
𝑋.2
𝑝W
Variouscorrectionsproposed:
Miller-Maddow,Jackknifedestimator,Coverageadjusted,…
Samplecomplexity:Ω(𝑘)
𝐻, 𝜀, 𝑘, 0.1 = 𝑜(𝑘) (existential)
Note:For𝜀 ≪ 1/𝑘,empiricalestimatorsarethebest
[Paninski’03]:𝑆
Entropyestimation
[ValiantValiant’11a]: ConstructiveproofsbasedonLP:
d
𝑆 𝐻, 𝜀, 𝑘, 0.1 = Θc efg d
• [YuWang’14, HanJiaoVenkatWeissman’14, ValiantValiant11b]:
Simplifiedalgorithms,andexactrates:
𝑆 𝐻, 𝜀, 𝑘, 0.1 = Θ
d
cefg d
Supportcoverage
Expectednumberofsymbolswhen𝑝 issampled𝑚 times
𝑆i 𝑝 = Z(1 − 1 − 𝑝K
i)
K
Goal:Estimate𝑆i 𝑝 to±(𝜀 ⋅ 𝑚)
[OrlitskySureshWu’16, ZouValiantetal’16]:
𝑛=Θ
i
.
log
efg i
c
samplesfor𝛿 = 0.1
Knownresultssummary
Manysymmetricproperties:entropy,supportsize,distance
touniform,supportcoverage
• Differentestimatorforeachproperty
• Sophisticatedresultsfromapproximationtheory
Mainresult
Simple,MLbasedplug-inmethodthatissample-optimal
forentropy,support-coverage,distancetouniform,
supportsize.
Profiles
Profileofasequenceisthemultiset ofmultiplicities:
Φ 𝑋.2 = {𝑁K }
𝑋.2 = 1𝐻, 2𝑇 , 𝑜𝑟𝑋.2 = 2𝐻, 1𝑇 ,Φ 𝑋.2 = {1,2}
Symmetricpropertiesdependonmultiset ofprobabilities
Coinsw/bias0.4,and0.6havesamesymmetricproperty
Optimalestimatorshavesame outputfor sequenceswith
sameprofile.
Profilesaresufficientstatistic
Profilemaximumlikelihood [OSVZ’04]
Probabilityofaprofile:
𝑝 Φ(𝑋.2) =
Z
𝑝(𝑌.2)
opq :r opq sr tpq
Maximizetheprofileprobability:
uvw
𝑝r
= arg max 𝑝(Φ 𝑋.2 )
Q
𝑋.2 = 1𝐻, 2𝑇 :
SML:(2/3, 1/3)
PML:(1/2, 1/2)
PMLforsymmetricproperties
Toestimateasymmetric𝑓(𝑝):
• Find𝑝uvw Φ(𝑋.2 )
• Output𝑓(𝑝uvw )
Advantages:
• Notuningparameters
• Notfunctionspecific
Mainresult
PMLissample-optimal forentropy,supportcoverage,
distancetouniformity,andsupportsize.
Ingredients
GuaranteeforPML.
If𝑛 = 𝑆 𝑓, 𝜀, 𝑘, 𝛿 ,then𝑆 𝑓
If 𝒏 = 𝑺 𝒇, 𝜺, 𝒌, 𝒆}𝟑
𝒏
#profilesoflength𝑛 <
𝑝uvw
, 2𝜀, 𝑘, 𝛿
Wy q
⋅ .T ≤𝑛
,then 𝑺 𝒇 𝒑𝑷𝑴𝑳 , 𝟐𝜺, 𝒌, 𝟎. 𝟏 ≤ 𝒏
Wy q
.T
Ingredients
𝑛 = 𝑆 𝑓, 𝜀, 𝛿 ,achievedbyanestimator𝑓5 (Φ(𝑋 2 )):
𝑝:underlyingdistribution.
• ProfilesΦ 𝑋 2 suchthat𝑝 Φ 𝑋 2
> 𝛿,
𝑝uvw Φ ≥ 𝑝 Φ > 𝛿
uvw
𝑝r
Φ ≥𝑝 Φ >𝛿
uvw
uvw
𝑓 𝑝r
− 𝑓 𝑝 ≤ 𝑓 𝑝r
− 𝑓5 Φ + 𝑓5 Φ − 𝑓 𝑝
• Profileswith𝑝 Φ 𝑋 2
< 2𝜀
< 𝛿,
𝑝(𝑝 Φ 𝑋 2 < 𝛿 < 𝛿 ⋅ #𝑝𝑟𝑜𝑓𝑖𝑙𝑒𝑠𝑜𝑓𝑙𝑒𝑛𝑔𝑡ℎ𝑛
Ingredients
Bettererrorprobabilityguarantees.
Recall:
𝑘
𝑆 𝐻, 𝜀, 𝑘, 0.1 = Θ
𝜀 ⋅ log 𝑘
StrongererrorguaranteesusingMcDiarmid’s inequality:
𝑆
Ž.•
}2
𝐻, 𝜀, 𝑘, 𝑒
=Θ
d
c⋅efg d
Withtwicethesampleserrordropsexponentially
Similarresultsforotherproperties
Mainresult
PMLissample-optimal forentropy,unseen,distanceto
uniformity,andsupportsize.
EvenapproximatePML isoptimalforthese.
Algorithms
• EMalgorithm[Orlitsky etal]
• ApproximatePMLviaBethePermanents[Vontobel]
• ExtensionsofMarkovChains[VatedkaVontobel]
PolynomialtimealgorithmsforapproximatingPML
Summary
• Symmetricpropertyestimation
• PMLplug-inapproach
• Universal,simpletostate
• Independentofparticularproperties
• Directions:
• Efficientalgorithms– forapproximatePML
• Reliesheavilyonexistenceofotherestimators
InFisher’swords…
Ofcoursenobodyhasbeenabletoprovethatmaximum
likelihoodestimatesarebestunderallcircumstances.
Maximumlikelihoodestimatescomputedwithallthe
informationavailablemayturnouttobeinconsistent.
Throwingawayasubstantialpartoftheinformationmay
renderthemconsistent.
R.A.Fisher
References
1. Acharya,J.,Das,H.,Orlitsky,A.,&Suresh,A.T.(2016).AUnifiedMaximumLikelihoodApproachforOptimalDistributionPropertyEstimation.arXiv preprint
arXiv:1611.02960.
2. Paninski,L.(2003).Estimationofentropyandmutualinformation.Neuralcomputation,15(6),1191-1253.
3. Wu,Y.,&Yang,P.(2016).Minimax ratesofentropyestimationonlargealphabetsviabestpolynomialapproximation.IEEETransactionsonInformationTheory,
62(6),3702-3720.
4. Orlitsky,A.,Suresh,A.T.,&Wu,Y.(2016).Optimalpredictionofthenumberofunseenspecies.ProceedingsoftheNationalAcademyofSciences,201607774.
5. Orlitsky,A.,Santhanam,N.P.,Viswanathan,K.,&Zhang,J.(2004,July).Onmodelingprofilesinsteadofvalues.InProceedingsofthe20thconferenceon
Uncertaintyinartificialintelligence (pp.426-435).AUAIPress.
6. Valiant,G.,&Valiant,P.(2011,June).Estimatingtheunseen:ann/log(n)-sampleestimatorforentropyandsupportsize,shownoptimalvianewCLTs.In
Proceedingsoftheforty-thirdannualACMsymposiumonTheoryofcomputing (pp.685-694).ACM.
7. Jiao,J.,Venkat,K.,Han,Y.,&Weissman,T.(2015).Minimax estimationoffunctionals ofdiscretedistributions. IEEETransactionsonInformationTheory, 61(5),
2835-2885.
8. Vatedka,S.,&Vontobel,P.O.(2016,July).Patternmaximumlikelihoodestimationoffinite-statediscrete-timeMarkovchains.InInformationTheory(ISIT),2016
IEEEInternationalSymposiumon (pp.2094-2098).IEEE.
9. Vontobel,P.O.(2014,February).TheBetheandSinkhorn approximationsofthepatternmaximumlikelihoodestimateandtheirconnectionstotheValiantValiantestimate.InInformationTheoryandApplicationsWorkshop(ITA),2014 (pp.1-10).IEEE.
10. Valiant,G.,&Valiant,P.(2011,October).Thepoweroflinearestimators.InFoundationsofComputerScience(FOCS),2011IEEE52ndAnnualSymposiumon (pp.
403-412).IEEE.
11. Orlitsky,A.,Sajama,S.,Santhanam,N.P.,Viswanathan,K.,&Zhang,J.(2004).Algorithmsformodelingdistributionsoverlargealphabets.InInformationTheory,
2004.ISIT2004.Proceedings.InternationalSymposiumon (pp.304-304).IEEE.