De pe n d . Mode l . 2014 ; 2 : 73 – 8 7 Research Article Fod é To un ka ra ∗ a n d Lo u i s - Pa u l R i vest Open Access Some New Random E f ect Models for Correlated B i nary Responses Abstract : Exchangeable copulas are used to model an extra - binomial variation in Bernoulli experiments with f a variable number of trials . Maximum likelihood inference procedures for the intra - cluster correlation are constructed for several copula families . The selection of a particular model is carried out using the Akaike information criterion ( AIC ) . Pro fi le likelihood con fi dence inte r − v als for the intra - cluster correlation are con structed and their performance are assessed in a simulation experiment . The sensitivi t − y of the inference to the speci fi cation of the copula family is also investigated through simulations . Numerical examples are pre sented . Ke y − w ords : Multivariate exchangeable copulas , Exchangeable binary data , Pro fi le inte v − r al , Maximum likeli hood MSC : Prima r − y62 H 10 , seconda y − r62 H 12 DOI 10 . 2 478 / d emo - 2014 - 0 6 Re ce ive d May 1 2 , 2014 ; a cce pted Octo be r 16 , 2014 one.tf I ntroduction Data in the form of clustered bina r − y responses arise in many fi elds of study . For example , in ophthalmolo y − g the two eyes of a patient are a cluster . In toxicological studies [ 8 ] , a litter is a cluster of newborn animals . In education , the school and the classroom are clusters of students . Units within a cluster are more likely to be similar than units from di ff erent clusters and the study variables often exhibit an intra - cluster correlation . Standard s tatistical methods that ignore such a correlation provide a poor fi t of data and can lead to erro neous inferences . Such an extra - binomial variation can be accommodated by specifying a random cluster e ff ect : the responses in a cluster are conditionally independent given the cluster e ff ect . A common model is the beta binomial , see [ 23 ] , where the conditional probability of success in a cluster has a beta distribu tion . This model has been criticized for its lack of fl exibility when the cluster sizes are variable [ 6 ] . Other random e ff ects models for clustered data include the logistic - normal - binomial model described by [ 24 ] , and the probit normal - binomial , see [ 15 ] . Several measures of association for bina r − y traits have been proposed in the literature . These include the Kappa coe ff i cient [ 5 , Chap . 8 ] , the odds ratio [ 3 , 1 1 ] . This work focusses on the estimation of the ” uncor r−e c e−t d ” intra - cluster correlation ( ICC ) . The ICC is a measure of the similitude of observations taken in the same cluster . It can be interpreted as an index of familial aggregation and as measure of inter - rater agreement , depending on the situation . When planning a cluster sampling or a cluster randomization trial , the clusters are the statis tical units and the precision of the results depend on the ICC . Nowadays cluster randomized trials are widely used to assess various modes of community inte r − v entions . They involve assigning randomly clusters of per ∗ Corresponding Author : Fodé Toun k − ara: Department of Mathematics and Statistics , Université Laval quotedblbase145 av . de la Médecine , Québec ( Québec ) G 1 V 0 A 6 Canada , E - mail : fode . tounkara .1@ ulaval . ca Louis - Paul Rivest : Department of Mathematics and Statistics , Université Laval quotedblbase1045 av . de la Médecine , Québec ( Québec ) G 1 V 0 A 6 Canada , E - mail : Louis - Paul . Rivest copyright214 @ mat . ulaval . ca é To u n kara et a l . Fo d , l i cen see D e G ruyter Open . Th i s wor k i s l i cen sed u n d er th e Creative Commons Attri b uti o n - N o n Commercial - N o De r ivs 3 . 0 Li c en se . 74 bar.two Fod é To u n ka ra a n d Lo u i s - Pa u l Rivest sons , often belonging to the same neighborhoodf or group , to the arms of a trial . In such situations the ICC plays a crucial role in the sample size determination , as it measures the homogeneity within clusters [ 5 ] . ICC estimates are now published in the epidemiological literature for various t − y pes of inte v − r ention , see [ 16 ] for a recent example . It is therefore important to have precise ICC estimators with reliable methods for constructing con fi dence inte r − v als . The estimation of the ICC for binary data has been considered by many authors . [ 17 ] compare more than 20 estimators of the ICC , derived using either maximum likelihood or a modi fi ed method of moments , in a Monte Carlo study . They conclude that the estimator of [ 7 ] can be used as an omnibus estimator as it gen - erally has good sampling properties . Recently , the construction of con fi dence inte r − v als for the ICC has been investigated by [ 4 , 18 , 25 ] . The fi rst modeled the extra binomial variation using [ 12 ] generalized binomial dis - tribution , [ 4 ] use a z - transform and a distribution - free large sample variance to construct an ICC con fi dence inte r − v al while [ 18 ] assumes a beta - binomial distribution . [ 18 ] also compared six methods for constructing con fi dence intervals for the ICC and found that , for beta - binomial data , the best procedure was a pro fi le like - lihood con fi dence interval ( PLCI ) . Note that inference about the ICC using a Bayesian framework have also been investigated , see [ 1 , 21 , 22 ] . This work investigates the construction of PLCI for the ICC using a new class of probability models for exchangeable clustered bina r − y data , associated with exchangeable Archimedean copulas [ 13 , chap . 2 ] . This family contains several speci fi cations for the extra binomial variation . The procedure investigated in this work consists in ( i ) selecting a model for the random cluster e ff ect and ( ii ) constructing a pro fi le con fi dence inte r − v al for the ICC using the model selected at step 1 . Section 2 introduces a class of model for an extra binomial variation associated with multivariate Archimedean copulas . Graphical models ’ comparisons are provided . Section 3 discusses model selection , the calculation of maximum likelihood estimates for the ICC , and the construction of pro fi le likelihood con fi dence inte r − v als . These new statistical methods are investigated through simulations in Section 4 . Two numerical examples are used to illustrate the proposed approach in Section 5 . two.tf With i n c luster dependency modeling fo r b i nary exchangeable data two.tf .one.tf Model formulation Consider a random sample of K clusters of size ni , i = one.tf, two.tf, ..., K . Let Yij , j = one.tf, two.tf, ..., ni be a set of ni exchangeable Bernoulli random variables in cluster i , where Yij = one.tf is a success and 0 is a failure . These random variables are distributed and possibly dependent . The total number of successes in the Pnidentically i ith cluster is Yi = j=one.tf Yij . A standard model for the within cluster dependency assumes that the probability of success pi is random and varies from one cluster to the next . The models considered here associate to cluster i a positive random variable θi whose Laplace transform , ψα( t ) = E (e−tθi ), depends on a positive dependency parameter α is a such way that P (θi = one.tf ) = one.tf when α = zero.tf. Let π ∈ (zero.tf, one.tf ) stands for the marginal probability of success ; the proposed model is −one.tf (π) pi = e−θi ψα i = one.tf, ..., K , ( 2 . 1 ) −one.tf (π) denotes the function inverse of the Laplace transform ψα(·) evaluated at π. Under model ( 2 . 1 ) , where ψα pi ∈ (zero.tf, one.tf ) and the marginal probability of success is E parenlef t−pi ) = π. This model has a clear separation between π and the random variable for the dependency , θi . Indeed , ψα−one.tf (π) is a mere scale parameter for the distribution of − log p i . Model ( 2 . 1 ) assumes exchangeability within each cluster . This assumption could be questionable for familial data where the mother - child association could be stronger than that between t − wo siblings . Table 1 gives the inverse of the Laplace transforms ψα−one.tf ( t ) and the densities fα (θ) of the random e ff ects θi for four mod els commonly used in the copula literature . For Gumbel ( G ) and Clayton ( C ) families , fα is a density de fi ned on R+ while , for the models of Frank ( F ) and Joe ( J ), fα is a probability mass function de fi ned on the positive Some N ew Ra n d om E ff e ct Mode l s fo r C o r re l a te d B i n a ry Res po n s e s bar.two 75 Table 1 : I nve rs e of La p l a ce t ra n sfo rm s , t h e d e n s i t i e s of t h e ra n d om e ff e cts θi , a n d moments fo r fo u r co pu l a fa m i l i e s , C l ayto n ( C ) , Fra n k ( F ) , G um be l ( G ) , J oe ( J ) . Fam i l i es C ψα−one.tf ( t ) fα (θ) λk (t−α −αt− −one.tf )/α θ one.tf /α−one.tf exp(−θ/α) αone.tf /α Γ(one.tf /α) one.tf }− α {kπ −α −k+− απ k FG {−e−α log(t)−}α+one.tf P osone.tf itiveStable integers N + . For the models in ( 2 . 1 ) , the k - moments E parenlef t − pki ) have the following form , λk = ψα{ k ψα−one.tf (π)}; explicit expressions for these moments are given in Table 1 . It follows that , given π and the association parameter α, the moment based de fi nition of the intra - cluster correlation ( ICC )ρ under models ( 2 . 1 ) is given by ρ= Eparenlef t − ptwo.tf ) − {Eparenlef t − pi)}two.tf ψα(two.tf ψα−one.tf (π)) − π two.tf i = Ep − parenlef ti){one.tf − Ep − parenlef ti)} π(one.tf − π) . (2.2) The ICC is the correlation between di ff erent Bernoulli variables , Yis and Yi` , of cluster i . It is an increasing function of α, with ρ = zero.tf when α = zero.tf for all models in Table 1 . Further interpretations of this coe ff i cient can be found in [ 5 , Chap . 8 ] . For Gumbel ’ s model , the random e ff ect θi has a positive stable distribution with parameter one.tf /(α + one.tf ) and fα (θ) does not have a closed form expression . For this model , the moments λk are equal to those for the q model of [ 8 ] . This provides an additional motivation for Kuk ’ s proposal as it can be derived using stable random e ff ects in ( 2 . 1 ) . Consider now Clayton ’ s family . When α = one.tf, θi has a negative exponential distribution one.tf , one.tf ). Therefore Clayton ’ s model with parameter 1 , and pi has a beta distribution with parameters ( ψ−one.tf (π) α with α = one.tf givesintegervaluedthe randomβ−binomial variabledistribution de fi ned on withparameters F rank0 s line − minus − N + .F or ,θi model,θi isan one.tf ). F orF rank0 s andJoe0 s parenlef tψα one.tf π, one.tf for Joe ’ s family it is the Sibuya distribution . The alogarithmic distribution ,while (copula later distribution is hea v − y tailed as no moment exists . See [ 13 ] for more information about these distributions . It might seem unrealistic to have latent cluster e ff ects distributed according to a positive stable law or to a discrete distribution . These assumptions are made to get , within ( 2 . 1 ) , tractable models with probabili t − y mass function ( pmf ) having closed form expressions . Swapping the successes and the failures , ( 2 . 1 ) gives new models for an extra binomial variation . The probability of success is then expressed as −one.tf (one.tf −π) pi = one.tf − e−θi ψα i = one.tf, ..., K . ( 2 . 3 ) This duality was noted by [ 8 ] whose p model can be expressed as ( 2 . 3 ) where θi has a positive stable dis - tribution . In the sequel , models obtained with ( 2 . 3 ) will be called dmodel since , as will be seen in the next section , they arise when the joint distribution of the Bernoulli variables Yij is expressed using a copula . In this paper , Kuk ’ s p model is labeled by the dGumbel model . For ( 2 . 3 ) , the intra cluster correlation is ρ = {ψα(two.tf ψα−one.tf (one.tf − π)) − (one.tf − π)two.tf }/{π(one.tf − π)}. Two approaches are available to calculate the pmf of Yi . The fi rst method evaluates alternating sums directly using the moments λk given in Table 1 : ni P (Yi = k ) = E {pki (one.tf − p i )ni −k } (2.4) k = ni k j P (−j=zero.tf ni k one.tf ) − ni − k j λk+j . (2.5) For large clusters , say larger than 30 , the evaluation of the alternating sums ( 2 . 5 ) is numerically unstable and yields negative probabilities . The second method to calculate the pmf of Yi is to evaluate ( 2 . 4 ) directly by integrating over the random e ff ects distribution . For the random e ff ects having gamma distributions of k one.tf π− e 76 bar.two Fod é To u n ka ra a n d Lo u i s - Pa u l Rivest Clayton ’ s family , we tried using the Gauss - Laguerre quadrature method . It gave poor approximations for large values of the dependence parameter α. Integrating over the random e ff ect is possible for the models of Joe and Frank as they have discrete distributions . For these models , PN − one.tf(π)) ni}f (θ)] [one.tf i − θ=one.tf {one.tf (one.tf − e−θ ψα − α P N θ=one.tf e−θψ −one.tf α (π) k{ one.tf − ) P (Yi = k) ≈ braceex − bracelef tmid − braceex − bracelef tbt (kn (2.6) where N is a positive integer large enough for these approximations to be accurate . The values N = one.tf zero.tf zero.tf and N = f our.tf zero.tf zero.tf yielded nearly identical results in the small cluster simulations that are presented in Section 4 . We used N = f our.tf zero.tf zero.tf in the large cluster simulations presented in the next section . Note that binomial mixture models constructed in terms of Laplace transforms can be found in [ 2 , 8 ] . The innovative aspect of our paper is to parameterize these models in terms of a marginal probability π and the ICC ρ and to investigate this large class of models in a systematic way . two.tf .two.tf A copula formulation of the model This section derives an alternative formulation for ( 2 . 1 ) , in terms of the joint su r −v ival distribution of the Bernoulli random variables Yij , j = one.tf, two.tf, ..., ni for the units within a cluster . Dropping subscript i , let {yj } be a sequence of n zeros and ones . Under ( 2 . 1 ) the j oint su r − v ival distribution of the Yj ’ s within a cluster is P (Yone.tf ≥ y one.tf, ..., Yn ≥ y n ) = Cα,n {F̄ ( y one.tf ), ..., F̄ ( y n ) } , ( 2 . 7 ) where F̄ ( y ) is the marginal su v − r ival function or Yj , F̄ ( y ) = P ( Y ≥ y ) = π y for y = zero.tf, one.tf, and Cα,nP(uone.tf , ..., un ) = (uj )} α{ is,f or uj ∈(zero.tf , one.tf )j ψG nj = one.tf ψα−one.tf thevariables see [13 . = 2]one.tf, .f or.., n, moreanndimensionaldiscussion. Note , Chap , Archimedean the that the inverse copulaψ−one.tf α(·)of f orψα isa dependencydecreasing function de fi ned on (zero.tf, one.tf ) such that ψα−one.tf (one.tf ) = zero.tf. Under (2.7), λk = P (Yone.tf ≥ one.tf, ..., Yk ≥ one.tf ) = Cα,k {π, ..., π} = ψα{kψα−one.tf (π)}. Equation ( 2 . 7 ) is proved by noticing that , assuming ( 2 . 1 ) , P (Yone.tf ≥ y one.tf, ..., Yn ≥ y n ) = E { P (Yone.tf ≥ y one.tf, ..., Yn ≥ y X = E[{exp{−θ yj ψα−one.tf (π)}] X = ψα{ yj ψα−one.tf (π)} X = ψα[ ψα−one.tf {F̄ (yj )}] n | θ)} since , by construction , yj ψα−one.tf (π) = ψα−one.tf {F̄ (yj )}. The value α = zero.tf gives the independence copula , Czero.tf,n (uone.tf , ..., un ) = uone.tf × ... × un . As α goes to ∞ the copulas for all the models of Table 1 converge to what is known as the Fréchet upper bound , X lim ψα{ ψα−one.tf (uj )} = min uj . α→∞ j This corresponds to the perfect correlation where all the Bernoulli variables within a cluster takes the same value , e . g . P (Yone.tf = ... = Yn ) = one.tf. An attractive feature of all models in Table 1 is that (π, ρ) has the full range ]zero.tf, one.tf [×]zero.tf, one.tf [, as π ∈ (zero.tf, one.tf ) and α ∈ (zero.tf, ∞). Moreover , ρ is a monotone function of α satis f − y ing the following properties α −→ ∞ ⇐⇒ ρ −→ one.tf andα = zero.tf ⇐⇒ ρ = zero.tf. There is a copula behind most models for clustered binary data . Consider for instance the generalized bi - nomial distribution , see [ 12 ] . Under this model the joint su r − v ival distribution within a cluster can be expressed as ( 2 . 7 ) with n Cα (uone.tf , ..., un ) = (one.tf − α) Y uj + α min uj , α ∈ (zero.tf, one.tf ). j = one.tf Some N ew Ra n d om E ff e ct Mode l s fo r Co r re l ate d B i n a ry Res po n s e s bar.two 77 This expression highlights that the generalized binomial distribution is a discrete mixture model featuring t − wo latent classes with respective probabilities one.tf − α and α. The clusters in the fi rst class are made of indepen - dent units while in the second class all the units of a cluster have the same Y values . Such a model is rather unlikely in practice and one would expect ( 2 . 1 ) to represent better the within cluster association . Despite its unrealistic description of the dependency , the generalized binomial has been widely used to investigate statistical procedures for cluster binary data , see [ 17 , 25 ] . Finally note that the estimation of the intra cluster correlation for discrete data using copulas was investi - gated by [ 19 ] in a di ff erent context . They were concerned with split - cluster designs , where individuals within a cluster are randomized to one of the t − wo arms of a trial . They use bivariate copulas to model the association be t − ween the mean responses for the treated and for the untreated individuals in a cluster . two.tf .three.tf Graph i ca l comparisons A total of 8 Archimedean copulas models are available to account for an extra binomial variation . They were compared using exploratory analysis techniques such as correspondence analysis . The fi ndings of these com - parisons are now brie fl y presented . First the beta - binomial , the Clayton and the d - Clayton models are , for all practical purposes , equal . This is illustrated in ( a ) , ( b ) , and ( c ) of Figure 1 where the pmf for these three mod - els obtained when n = one.tf f ive.tf, π = zero.tf.one.tf, zero.tf.two.tf, zero.tf.three.tf and ρ = zero.tf.one.tf are presented . Similar results are found for ρ = zero.tf.three.tf ; they are presented in the supplementa r − y material section . This means that when the beta - binomial is a can - didate model for a data set , there is no need to include the Clayton and dClayton model as they provide fi ts that are ve r − y similar . Graphs ( d ) , ( e ) and ( f ) of Figure 1 compare the pmf for Joe ( J ) , dJoe ( dJ ) , Gumbel ( G ) and dGumbel ( dG ) models . The dJoe and dGumbel pmfs have nearly identical unimodal shapes while of the Gumbel and the Joe pmfs are similar with a bimodal structure . The pmfs of Figure 1 have a wide range of values for Pr (Yi = zero.tf ), when π = zero.tf.two.tf it goes from 0 . 5 to 0 . 25 . Additional comparisons , with ρ = zero.tf.three.tf are presented in the supple - menta r − y material section . In all comparisons , the Joe and the dJoe pmf presented the most extreme shapes , while the beta - binomial pmf has an average shape , between these two extremes . Therefore , in the small clus - ter simulations presented in the next section , only these three models are considered as being representative of all the models of Table 1 . Figure 2 presents some pmf comparisons for clusters of size n = f our.tf zero.tf, with ρ = zero.tf.zero.tf f ive.tf and π = zero.tf.one.tf, zero.tf.two.tf, zero.tf.three.tf. Such small values of ρ are typical in communi t − y intervention studies , see ( Légaré et al . , 2011 ) . The models presented are the beta - binomial ( BB ) , Frank ( F ) , dFrank ( dF ) , Joe ( J ) and dJoe ( dJ ) . Figures 2 shows that the pmf for all the models are di ff erent , except possibly for the dFrank and dJoe models , whose shape are ve r − y similar . The pmf shows a unimodal structure for the models of dFrank and dJoe , and a bimodal structure under F ’ s and J ’ s models . Joe ’ s pmf assigns a large probability to zero as compared with the others models and the beta binomial pmf has an average shape between that of the Joe and the dJoe models . Graphs for ρ = zero.tf.one.tf are presented in the supplementa r − y material section . three.tf Maximum l i kelih ood i n fe ren ce fo r I CC parenleft − rho) This section shows how to calculate maximum likelihood estimates for ρ and π using data {(ni , y i) : i = one.tf, ..., K } . It also discusses the construction of pro fi le likelihood con fi dence inte r − v als for ρ. Inference on ρ is carried out in a frequentist rather than a Bayesian set - up as this is commonly used in research on clustered randomized trials ( Eldridge & Ker r − y, 2012, p . 193 ) . 78 bar.two Fod é To u n ka ra a n d Lo u i s - Pa u l Rive st (b) (a) (d) (c) (f ) (e) Figu re 1 : A co mpa r i s o n o f t h e p ro ba b i l i ty mass fu n ct i o n fo r c l u ste r s i ze 15 u n d e r s ome s ma l l c l u ste rs mode l s , fo r ρ = zero.tf.one.tf a n d π = zero.tf.one.tf, zero.tf.two.tf, zero.tf.three.tf Some N ew Ra n d om E ff e ct Mode l s fo r Co r re l ated B i n a ry Re s po n s e s bar.two (b) 79 (a) (c) Figu re 2 : A com pa r i s o n of t h e p ro ba b i l i ty mass fu n ct i o n fo r c l u ste rs of s i z e 40 u n d e r t h e be ta - b i n o m i a l ( B B ) , J oe ( J ) , dJ oe ( dJ ) , Fra n k ( F ) a n d d - Fra n k ( d I G ) , fo r ρ = zero.tf.zero.tf f ive.tf a n d π = zero.tf.one.tf ( a ), zero.tf.two.tf ( b ), zero.tf.three.tf ( b ) three.tf .one.tf Maximun Li keli hood Estimates The parameters of interest are ρ and π, thus α needs to be expressed in terms of these two parameters . The only model for which equation ( 2 . 2 ) leads to an explicit expression for α is Gumbel ’ s families for which α= log(two.tf ) log[one.tf + log{ρ(one.tf −π)+π} ] log π − one.tf. For all the other models of Table 1 , solving ( 2 . 2 ) numerically is needed to calculate α; this can be done using the R function uniroot . The log - likelihood is Qi=one.tf ni ni −yi log L (π, ρ) = log { K }}. (3.1) E {pyi i (one.tf − p i ) yi 80 bar.two Fod é To u n ka ra a n d Lo u i s - Pa u l Rivest It can be evaluated using one of the t − wo methods presented in Section 2 . 1 . The maximum likelihood estimates are the values π̂ and ρ̂ that maximize the log - likelihood de fi ned in ( 3 . 1 ) . They were calculated using R . First an R function evaluating ( 3 . 1 ) in terms of ρ and π was written where α was evaluated in terms of ρ and π by solving ( 2 . 2 ) using uniroot . The search inte r − v al for α was set to (one.tf zero.tf −f ive.tf , one.tf zero.tf one.tf zero.tf ); for some copulas ( 2 . 2 ) cannot be evaluated for α values as large as one.tf zero.tf one.tf zero.tf and this generates a uniroot warning . This had no impact on the result since uniroot automatically lowers the upper bound of the search interval when this happens . Then the values of π and ρ that maximizedP the likelihood were obtained using the function optim of R . We used as starting values , the sample mean P π̂zero.tf = y i/ ni and ρ̂ FC , the Fleiss Cuzick moment based estimate of ρ de fi ned in the next section . The log - likelihood is maximized in the square [one.tf e−f ive.tf , .nine.tf nine.tf nine.tf nine.tf nine.tf ]×[one.tf e−f ive.tf , .nine.tf nine.tf nine.tf nine.tf nine.tf ]. The Fisher information matrix was then estimated as minus the Hessian of the likelihood function ; this yields standard errors , s . e . , for the two estimates . Negative estimates of ρ are not possible . Negative estimates of ρ have little practical interest , indeed [ 5 , Chap . 8 ] recommend setting them to zero and proceeding as the data with cluster was independent . three.tf .two.tf Model se lection criteria A model ’ s fi t can be assessed using a deviance , equal to the likelihood ratio statistics comparing the copula model to that of a saturated model where each cluster has t − y of success pi , its own fi xed nprobabili i K P )yiyi (ni − y i )ni −yi /nni i}− log L (π̂, ρ̂)], (3.2) Dev = two.tf [ log i = one.tf yi Note however that this deviance does not have an asymptotic χtwo.tf K−two.tf distribution when the copula model fi ts well as the null copula model is not a special case of the saturated model with cluster speci fi c p i when K is fi xed . To select a speci fi c model to work with , we used the Akaike Information Criterion ( AIC ) , de fi ned by AIC = −two.tf log L (π̂, ρ̂) + two.tf m , ( 3 . 3 ) where m , the number of parameters , is equal to 2 for all the copula models considered in this work . The preferred model is the one with the minimum AIC value . We did not use the model averaging approach of [ 14 ] as this does not permit the calculation of pro fi le con fi dence inte r − v als . three.tf .three.tf Pro fi le Li ke l i hood Con fi dence I nte r − v a l ( PLCI ) Wald con fi dence intervals for ρ, with a one.tf zero.tf zero.tf (one.tf −τ ) con fi dence level , are given by ρ̂± zone.tf −τ /two.tf s . e .(ρ̂), where zone.tf −τ /two.tf is a normal quantile . This procedure may perform poorly for small to moderate sample s izes when estimating ρ, see Donner and Eliasziw ( 1992 ) . This section reviews the construction of PLCI for ρ when using copula models associated to ( 2 . 1 ) . Such inte r − v als are generally considered to be more accurate than Wald con fi dence inte r−v als . The pro fi le likelihood for ρ is given by LP rho−parenlef t) = maxπ L (π, ρ), ρ ∈ (one.tf e−f ive.tf , .nine.tf nine.tf nin We used the R function opt imize for its evaluation ; it involved a uniroot function to evaluate α as a function of ρ and π. It may happen that LP (ρ) cannot be numerically evaluated for large values of ρ; to calculate the pro fi le con fi dence inte r − v al it is important to identi y − f a value ρ M such that LP parenlef t − rho) can be evaluated for ρ ∈ (one.tf e−f ive.tf , ρ M ) . The one.tf zero.tf zero.tf (one.tf − τ )% PLCI for ρ is de fi ned as the set one.tf two.tf ˆ χone.tf (one.tf − τ )}, (3.4) CI = {ρ : log LP parenlef t − rho) > log L rho − parenlef t, π̂) − two.tf where χtwo.tf one.tf (one.tf − τ ) is the one.tf − τ quantile of a chi - squared distribution with one degree of freedom . The set CI ˆ is typically an inte r − v al and is denoted rho − parenlef t L , ρ̂U.) We computed the w − to limits by fi nding two solutions to the ˆ t − rho π̂) − one.tf χtwo.tf (one.tf − τ ), one in the interval (zero.tf, ρ̂) and the other equation log LP (ρ) = log L parenlef , two.tf one.tf in the interval (ρ̂, ρM,) Some N ew Ra n d om E ff e ct Mode l s fo r Co r re l ated B i n a ry Re s po n s e s bar.two 81 ˆ t − rho π̂) − one.tf χtwo.tf (one.tf − τ ) < log LP (one.tf e−f ive.tf ), using the R function uniroot . When log L parenlef , two.tf one.tf the inte r − v al lower bound was set at ρ̂ L = zero.tf. If ρ M is not speci fi ed correctly then log LP rho − parenlef tM ) cannot be evaluated ; in such cases uniroot returns ρ̂ as the value of the con fi dence inte v − r al upper bound . This is not a legitimate upper bound and samples for which ρ̂ U = ρ̂ were rejected in the simulations ; such unacceptable upper bounds occurred only in the large ˆ t − rho L , ρ̂ U ) the R function plkhc i could also be used . cluster simulations . For a quick evaluation of parenlef When the true value of ρ is close to zero.tf, the lower bound ρ̂ L is likely to be 0 and in this case the true coverage of the con fi dence might be slightly above its nominal level . PLCI for ρ, constructed using the beta - binomial distribution , were investigated by [ 18 ] . He concluded through Monte Carlo s imulations that the pro fi le con fi dence inte r − v al performs well over a wide range of pa - rameter values . A goal of this work is to investigate whether the beta - binomial pro fi le con fi dence inte r − v al is an omnibus procedure applicable in all circumstances or whether the fi t of the beta - binomial model should fi rst be ascertained . four.tf S im ulation Study The purpose of this simulation study is to assess whether the AIC criterion ( 3 . 3 ) of Section 3 . 2 provides a reliable model selection criterion . Its goal is also to assess the performance of the pro fi le likelihood con fi dence intervals for ρ constructed for the models of Table 1 . Two sets of simulations were carried out . The fi rst featured small clusters , with s izes ranging be t − ween 1 and 15 with a mean of 4 . The second one had large clusters with ni between 27 to 65 with a mean of 38 . Data was generated from the beta - binomial distribution , the Joe , dJoe , frank and dFrank models . For J ’ s and dJ ’ s models , cluster speci fi c θi were generated from the Sibuya distribution while for the Frank and dFrank model , the logarithmic distribution was used . The accuracy of the AIC model selection criterion was measured using the proportions of times that model Mzero.tf is selected when the data was simulated from model Mone.tf : Pone.tf zero.tf zero.tf zero.tf Ii i=one.tf m i nAIC (Mzero.tf , Mone.tf ) = one.tf zero.tf zero.tf zero.tf × one.tf zero.tf zero.tf, where the indicator Ii equal to 1 if model Mzero.tf provides the smallest AIC for sample i and 0 otherwise . The simulations also investigated nine.tf f ive.tf % two - sided con fi dence inte r − v als for ρ. Their performance was measured using the coverage ( cov ) and the average length ( CIL ) de fi ned as Pone.tf zero.tf zero.tf zero.tf Cov = i=one.tf Ai one.tf zero.tf zero.tf zero.tf Pone.tf zero.tf zero.tf zero.tf × one.tf zero.tf zero.tf ; CIL = i=one.tf ) (ρ̂Ui − ρ̂Li one.tf zero.tf zero.tf zero.tf , ˆ t − rhoLi , ρ̂U ) , and 0 othe r − w ise . All programming was done where Ai equal to 1 if the true value ρ is in parenlef i in R . Samples with y i = zero.tf or y i = ni for all clusters were rejected . four.tf .one.tf Small c luster simulations Three models are investigated in the small cluster simulations , beta - binomial , Joe and dJoe . Two values of ρ and π, namely 0 . 1 and 0 . 3 , and three values of K , 100 , 250 , and 500 , were considered . Table 2 summarizes the proportion of times that a model is selected as a function of the model from which the data is simulated . The relatively high values on the diagonals of each quadrant of Table 2 show that the models are identi fi able . This is more pronounced , as either K , π or ρ increases . When a data set is generated from Joe ’ s model , dJoe is nearly never selected and vice - versa . These results are in line with the fi ndings of the graphical comparisons of Figures 1 where these two models were very di ff erent . Table 2 also con fi rms the middle ground nature of the beta - binomial since this model has the smallest number of correct identi fi cations . Table 3 reports results on PLCI that assume that the model is selected correctly . It evaluates PLCI for ρ under the beta - binomial , the Joe and the dJoe models . The PLCI has generally good statistical properties ; its coverage is close to nominal 95% level and its average length is relatively small . As expected , the expected 82 bar.two Fod é To u n ka ra a n d Lo u i s - Pa u l Rivest Table 2 : Sma l l c l u ste r s i mu l ati o n s : Th e p ro po rt i o n of t i mes t h at a mode l i s s e l e cte d by t h e AI C c r i te r i o n a s a fu n ct i o n of t h e mode l u s e d to s i m u l ate t h e data . Table ignored! con fi dence length decreases as the number of clusters increases . The PLCI for Joe ’ s model is always shorter than the other two , with a very good empirical coverage . Thus PLCI constructed using the three models in - vestigated in this section are reliable statistical techniques . Additional simulation results are presented in the Supplementa r − y Material Section . First the coverage and the average length of [ 25 ] ICC con fi dence inte r − v al constructed using the moment estimator of [ 7 ] , P (n yi i − yi)/n i ρ̂F C = one.tf − ( ni − K)π̂(one.tf − π̂) are reported together with the performance of the beta - binomial PLCI when the model is misspeci fi ed , that is when the data comes from the Joe and the dJoe model ( see Web Table 1 in the ” Web Appen d i − x B ” ) The Zou & Donner con fi dence interval is ve r − y conse v − r ative as it is always much wider than the PLCI . These simulations also show that the PLCI for the beta - binomial is sensitive to a model misidenti fi cation . For instance for dJ data , K = two.tf f ive.tf zero.tf, π = zero.tf.one.tf, and ρ = zero.tf.one.tf, the estimated coverage of the beta - binomial PLCI , 76.70%, is much smaller than the nominal value of 95%. However , when data are from J ’ s model this method in general has a tendency to provide the observed coverage probabilities that are slightly larger than the nominal level . Table 3 : Sma l l c l u ste r s i mu l ati o n s : Co n fi d e n c e i n te rva l l e n gth ( C I L ) a n d em p i r i ca l cove ra ge ( COV ) of two - s i d ed PLC I fo r ρ with a n o m i n a l nine.tf f ive.tf % co n fi d e n c e l eve l u n d e r beta - b i n om i a l , J oe , a n d dJ oe mode l s Table ignored! Some N ew Ra n d om E ff e ct Mode l s fo r Co r re l ate d B i n a ry Res po n s e s bar.two 83 four.tf .two.tf Large c luster simulations We ran large cluster simulations with K = 10, 25, and 50 clusters . Two values of ρ, 0.5 and 0 . 1 , and of π, 0.1 and 0 . 3 , were investigated . Besides the beta - binomial model , the models of Frank , dFrank , Joe , and dJoe were considered . Table 4 shows that the AIC has a hard time identifying the models when K = one.tf zero.tf. The proximity , in Figure 2 , of the pmf for Joe ’ s and Frank ’ s model shows up in Table 4 where the AIC has di ff i culties distin - guishing one from the other . This improves as K increases and at K = f ive.tf zero.tf most models are identi fi able . The identi fi ability increases with π. Results in Tables 5 show that , when the model is correctly identi fi ed , the PLCI Table 4 : La rge c l u ste r s i m u la t i o n s : Th e p ro po rt i o n of t i mes t h a t a mode l i s s e l e cte d by t h e A I C c r i te r i o n a s a fu n ct i o n of t h e mode l u s ed to s i m u la te t h e d ata . Table ignored! constructed with the beta - binomial , the Joe , the dJoe , the Frank and the dFrank models have good coverage properties , even with a relatively low ( K = one.tf zero.tf ) number of clusters . As expected , the average con fi dence in - te r − v al length decreases with the number of clusters . In general , the Joe and the Frank PLCI provide smaller expected lengths and an empirical coverage close to the nominal values In Web Table 2 of ” W e − b A ppn−e d i − x B ” we r p − e ort e − d o − c n − fi denc inte v − r a l − e n g h − t ( CIL a n − d m − e pirica c − o ve a − r g ( COV ) of Zou and Donner ’ s interval and of the beta - binomial PLCI , when the data comes from J ’ s , dJ ’ s F ’ s and dF ’ s distributions . We can see that Zou and Donner ’ s method is very conservative . On the other hand , the PLCI based on beta - binomial model provides coverage below the nominal level when the model is misspeci fi ed . Thus using an omnibus PLCI based on the beta - binomial distribution is not a robust statistical procedure . 84 bar.two Fod é To u n ka ra a n d Lo u i s - Pa u l Rivest Table 5 : La rge r c l u ste r s i m u l a t i o n s : Co n te rva l l e n gt h ( C I L ) a n d em p i r i ca l c ove ra ge ( COV ) of two - s i de d PLC I fo r ρ wi th a n o m i n a l nine.tf f ive.tf c o n fi d e n ce l eve l fo r d a ta ge n e ra te d f ro m 5 mode l s fi de n ce i n Table ignored! Examples five.tf In this section , we present two real life examples to illustrate the performance of estimators obtained under ( 2 . 1 ) and ( 2 . 3 ) . five.tf .one.tf A p i lot tria l about shared decision making This example is concerned with the impact of training physicians in shared decision making on their patients ’ involvement in the decision making process , see [ 9 ] for more details . The data in Table 7 presents the num ber of patients y i reporting an active role in the decision about taking an antibiotics treatment for an acute respiratory infection , and the total patient enrollment ni in K = nine.tf community health se r − v ices of the Quebec region . Table 6 : N um be r of pati e n ts re po r t i n g a n a ct ive ro l e i n t h e de c i s i o n ma ki n g p ro ce s s fo r 9 c ommunity h e a lt h se r−v ices yni i line − line three.tf line − line three.tf eight.tf eight.tf three.tf nine.tf line − line one.tf zero.tf line − line f our.tf nine.tf seven.tftwo.tf The ICC ρ enters the sample s ize calculation in these trials . Getting a reliable estimate is needed for plan - ning purposes . Its value is typically small in community intervention trials and the upper limit of a con fi dence inte r − v al is useful to calculate a conservative sample size . The de v − iance of the independence model is one.tf nine.tf.three.tf six.tf six.tf for nine.tf degrees of freedom ; the p - value of the indepen dence test is Pr parenlef t − chitwo.tf %. This suggests that nine.tf > one.tf nine.tf.three.tf six.tf six.tf ) = one.tf.three.tf ρ > zero.tf. This is a large cluster data set and only six sets of estimates are available . The values of ρ̂ reported in Table 7 di ff er little , except for those obtained with the dJ and dF models which are null . The corresponding AIC are larger than that for the independence model , suggesting a poor fi t for these models . The best fi tting model is that of Frank . Thus Frank ’ s model is plausible for this data and the 95% pro fi le con fi dence inte r − v al for ρ of ( 0 . 1 , 0 . 97 ) appears to be reliable . As an application of this result , suppose that one were to design a cluster randomized trial to compare the e ff i ciency of a method for training physicians in share decision making , with respect to no training , on the Some N ew Ra n do m E ff e ct Mode l s fo r Co r re l ated B i n a ry Res po n s e s mato rs fo r a com m u n i ty i n te rve n t i o n t r i a l . bar.two 85 Table 7 : S ix s e t of e s t i Table ignored! proportion of patients engaging in share decision making . The test level is set at 5%, the power at 80% and the t − wo proportions of patients engaging in shared decision making are 50% and 70% for the untreated and the treated communi t − y health services . The unit are patients and suppose that a community health service contributes 10 patients to the trial . How many community health services are needed , for each arm of the trial , to achieve the planned power of 80%? The result critically depends on the value of ρ. Using the standard sample size formula ( [ 4 , section 2 . 2 ] , [ 5 , Chap . 7 ] ) with ρ = zero.tf.zero.tf nine.tf seven.tf, 17 health community services are needed while this number drops to 12 if the calculations are done with the mle ρ̂ = zero.tf.zero.tf three.tf three.tf. five.tf .two.tf Chron i c Obstructive Pu lmonary D i sease ( COPD ) data This data is presented in Example 3 of [ 10 ] , and is considered by [ 20 , 25 ] . In this data the familial aggregation of the COPD is used as a measure of how genetic and environmental factors may contribute to disease etiology . It involves 203 siblings from 100 families with sizes ranging from 1 to 6 . The bina r − y response here indicates whether a given s ibling has impaired pulmonary functions . In the Table 8 , we report estimates of π and ρ, pro fi le con fi dence inte r − v als for ρ, and deviances obtained for all the models considered in this work . Results derived with the nonparametric ( NP ) model of [ 20 ] , with six parameters λk , k = one.tf, ..., six.tf, are also provided . As expected from graphical comparisons , the dG and dJ models have ve r − y similar fi ts . These models provide the best fi ts to the COPD data in terms of the AIC value . Moreover , the likelihood ratio test for comparing the fi t of dGumbel model with that of the nonparametric model has χtwo.tf f our.tf = one.tf.seven.tf f ive.tf p - value = 0.78; this is not signi fi cant and the dGumbel fi ts well . The dGumbel 95% pro fi le con fi dence inte r − v al for ρ is (zero.tf.zero.tf three.tf six.tf, zero.tf.three.tf nine.tf f ive.tf ); it is ve y − r close to the dJoe inte r − v al that has been shown to reliable in the small cluster simulations . Their length are 10% smaller than that of the Zou & Donner interval . As expected , the beta - binomial distribution , the Clayton and dClayton models give similar fi ts . Joe ’ s and Frank ’ s model do not fi t . This agrees with graphical comparisons of Figure 1 where these t − wo pmfs give much larger probabilities to 0 than the other model . Globally all pro fi le likelihood con fi dence intervals are similar in this example , with di ff erences less than 10% in con fi dence inte r − v al length . Given the small family size , the fact that the selection of a particular model has a limited impact on the analysis does not come as a surprise . The statistical methods proposed in this work are useful when dealing with larger clusters . six.tf D iscussion This paper has suggested to model the intra cluster association for bina r−y data using multivariate Archimedean copulas . This family contains awide range of distributional shape for the extra binomial variation . It has been demonstrated through a simulation study that the AIC is a useful criterion for selecting a particular model for the extra binomial variation . An important conclusion of our study is that pro fi le likelihood con fi dence intervals for Archimedean copulas models have good coverage properties , even when the number of clusters is small . The model selection step is important . An omnibus method , such as using the beta - binomial pro fi le 86 bar.two Fod é To u n ka ra a n d Lo u i s - Pa u l Rive st Table 8 : MLEs o f π a n d ρ, a n d two - s i de d nine.tf f ive.tf % c o n fi d e n ce i n te r − v a l fo r ρ u n de r 11 mode l s fo r t h e ( COPD ) d ata . De gre e s of fre ed om ( DF ) , Devia n ce ( Dev ) a n d , AI C c r i te r i o n of mode l s a re a l s o re po rte d . Table ignored! likelihood con fi dence inte r − v al for the ICC regardless of whether this model fi ts well , may lead to con fi dence inte r − v al with poor coverage properties . Acknowledgement : The authors thank the two anon y − m ous referees for their helpful comments and sugges tions . The fi nancial support of the Canada Research Chair on Statistical Sampling and Data Analysis and of the Fonds pour la recherche sur la nature et les technologies of Québec is gratefully acknowledged . References [ 1 ] Ah med , M . a n d S ho u k r i , M . ( 2010 ) . A baye s i a n e st i mato r of t h e i n t ra c l u ste r co r re l a t i o n c oe ff i c i e n t fro m c o r re l a te d b i n a ry re s po n s e s . J . Data Sci . , 8 : 127 – 137 . [ 2 ] Ala n ko , T . a n d D u ff y , J . C . ( 1996 ) . C om po u n d b i n om i a l d i st r i buti o n fo r mode l i n g co n s um pti o n d a ta . The Statistician , 45 : 269 – 28 6 . [ 3 ] An a n t h , C . V . a n d P re i s s e r , J . S . ( 1999 ) . B iva r i ate l ogi s t i c re gre s s i o n : Mode l i n g t h e a s s o c i a t i o n of s ma l l fo r gestatio n a l age b i rt h s i n twi n gestatio n s . Stat . Med . , 18 : 2011 – 2023 . [ 4 ] C h a k ra bo r ty , H . , Moo re , J . , Ca r l o , W . A . , H a rtwe l l , T . D . , a n d Wright , L . L . ( 2009 ) . A s i m u l a t i o n base d te c h n i q u e to e st i mate i n t ra c l u ste r c o r re la t i o n fo r a b i n a ry va r i a b l e . Contemporary Clinical Trials , 30 : 71 – 8 0 . [ 5 ] E l d r i d ge , S . a n d K e r ry , S . ( 2012 ) . A Practical Guide to Cluster Randomised Trials in Health Services Research . Wi l ey , N ew Yo r k . [ 6 ] Fe n g , Z . a n d C r i zz l e , J . E . ( 1992 ) . Co r re l ate d b i n o m i a l va r i ate s : Pro pe rt i e s of e st i mato rs of i n t ra c l a s s co r re l ati o n a n d i ts e ff e ct o n s a m p l e s i z e ca l c u l ati o n . Stat . Med . , 11 : 1600 – 1614 . [ 7 ] Fleiss,J.L.a n d C uz i c k , J . ( 1979 ) . Th e re l i a b i l i ty of d i c h oto mous j u d gments : U n eq u a l n um be rs of j ud ges pe r s u bj e ct . Appl . Psychol . Meas . , 3 : 5 37 – 5 42 . [ 8 ] K u k , A . Y . C . ( 20 4 ) . A ge n e ra l i ze d e st i mati n g eq u a t i o n a p p ro a c h to mode l l i n g fo eta l re s po n s e i n d eve l o pmenta l toxi c i ty st ud i e s when t h e n um be r of i m p l a n ts i s d o s e d e pe n d e n t . J . Roy . Statist . Soc . Ser . C , 5 2 : 5 2 – 61 . [ 9 ] Lé ga ré , F . , La b re cq u e , M . , Le B l a n c , A . , N j oya , M . , La u r i e r , C . , C ô té , L . , G od i n , G . , Th ivi e rge , R . L . , O ’ Co n n o r , A . , a n d S . , S . - J . ( 2011 ) . Tra i n i n g fa m i ly p hys i c i a n s i n s h a re d d e c i s i o n ma ki n g fo r t h e u s e of a n t i b i oti cs fo r a c u te re s p i rato ry i n fe ct i o n s : a p i l o t c l u ste re d ra n d om i zed co n t ro l l ed t r i a l . Health E − xpect., 1 : 96 – 110 . [ 10 ] Li a n g , K . Y . , Qaq i s h , B . , a n d Zege r , S . ( 1992 ) . Mu lt iva r i ate regre s s i o n a n a lys e s fo r ca tego r i ca l data . J . Roy . Statist . Soc . Ser . B . , 5 4 : 3 – 40 . [ 11 ] Li p s i tz , S . R . , La i rd , N . M . , a n d H a r r i n gto n , D . P . ( 1991 ) . Ge n e ra l i z ed e st i mati n g eq u ati o n s fo r c o r re l a te d b i n a ry data : u s i n g t h e od d s rati o a s a measu re of a s s o c i ati o n . Biometrika , 78 : 1 5 3 – 160 . [ 12 ] Madsen , R . W .(1993).Ge n e raline−line l i z ed bline−line i n om i aline−line l d i striline−line bline−line uti o n . Comm . Statist . Theory Methods , 22 : 3065 – 30 86 . [ 13 ] Mai , J . - M . a n d Sch e re r , M . ( 2012 ) . Simulating Copulas ; Stochastic Models , Sampling A lgorithms and Applications . Se r i e s i n Qua n t i tative Fi n a n ce : Volu me 4 . Wo r l d Sc i e n t i fi c Pu b l i s h i n g Com pa n y , N ew Yo r k . [ 14 ] Ni ko l o u l o po u l o s , A . K . a n d Ka r l i s , D . ( 20 8 ) . Mu lt iva r i a te l ogi t co p u l a mode l with a n a pp l i ca t i o n to d e n ta l data . Stat . Med . , 27 : 6393 – 6406 . [ 15 ] Och i , Y . a n d P re n t i c e , R . L . ( 1984 ) . Li ke l i h o od i n fe re n ce i n a co r re l ate d p ro b i t re gre s s i o n mode l . Biometrika , 71 : 5 31 – 5 42 . So me N ew Ra n d om E ff e ct Mode l s fo r Co r re l a te d B i n a ry Res po n s e s bar.two 87 [ 16 ] Pa l s , S . L . , B e aty , B . L . , Pos n e r , S . F . , a n d B u l l , S . ( 2 9 ) . Esti mate s of i n t ra c l a s s c o r re la t i o n fo r va r i a b l e s re l ate d to be h avi o ra l h iv / std p reve n t i o n i n a p re d om i n a n t ly afri ca n ame r i ca n a n d h i s pa n i c s a m p l e o f yo u n g wo me n . Health Education & Behavior , 36 : 18 2 – 194 . [17] Ri d o ut , M . S . , Demétrio , C . G . B . , a n d Fi rt h , D . ( 1999 ) . Esti mati n g i n t ra c l a s s c o r re l a t i o n fo r b i n a ry data . Biometrics , 5 5 : 1 37 – 148 . [ 18 ] Sa h a , K . K . ( 2012 ) . Pro fi l e l i ke l i h ood - ba s ed c o n fi d e n ce i n te rva l of t h e i n t ra c l a s s co r re l ati o n fo r b i n a ry o u tco me d ata s am p l ed fro m c l u ste rs . Stat . Med . , 31 : 3982 – 4002 . [ 19 ] Sh o u k r i , M . M . , K uma r , P . , a n d C o l a k , D . ( 2011 ) . An a lyz i n g d e pe n d e n t p ro po r t i o n s i n c l u s te r ra n d om i ze d t r i a l s : Mode l i n g i n te r - c l u ste r co r re l ati o n vi a co p u l a fu n ct i o n . Comput . Statist . Data Anal . , 5 5 : 1226 – 1235 . [ 20 ] Stefa n e s c u , C . a n d Tu r n bu l l , B . W . ( 2003 ) . L i ke l i h ood i n fe re n c e fo r exch a n gea b l e b i n a ry d ata wi t h va ryi n g c l u s te r s i zes . Biometrics , 59 : 18 – 24 . [ 21 ] Tu r n e r , R . M . , Omar , R . Z . , a n d Th om ps o n , S . G . ( 2001 ) . Baye s i a n meth od s of a n a lys i s fo r c l u ste r ra n d om i ze d t r i a l s with b i n a ry o utco me d a ta . Stat . Med . , 2 0 : 45 3 – 472 . [ 22 ] Tu r n e r , R . M . , Oma r , R . Z . , a n d Th om p s o n , S . G . ( 2006 ) . Co n st ru ct i n g i n te rva l s fo r t h e i n t ra c l u s te r co r re l a ti o n coe ff i c i e n t u s i n g baye s i a n mode l l i n g , a n d a p p l i cati o n i n c l u ste r ra n d om i ze d t r i a l s . Stat . Med . , 2 5 : 1443 – 1456 .
© Copyright 2024 Paperzz