Stefan Seifert & Maria Nieswand
Operational Conditions
in Regulatory Benchmarking –
A Monte-Carlo Simulation
Workshop: Benchmarking of Public Utilities
November 13, 2015, Bremen
Agenda
1
2
3
4
5
6
2
Motivation and Literature
Methodologies
The DGP
Simulation Design and Performance Measures
Initial Results
Conclusion and Outlook
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
1
Motivation
Regulatory Approaches for Electricity DSOs
Source: Agrell & Bogetoft, 2013
3
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
1
Motivation
• Benchmarking widely used in regulation – sectors in which environmental factors play
an important role
• Accuracy of estimates influences revenue caps, industry performance, firm survival,
and ultimately customers via prices
• Methodological advances to account for environmental factors and heterogeneity
• Non-parametric approaches: z-variables in 1-stage DEA (Johnson and Kuosmanen,
2012), conditional DEA (Daraio & Simar, 2005 & 2007), …
• Parametric approaches: Latent Class (Greene, 2002; Orea & Kumbhakar, 2004),
Zero-inefficiency SF (Kumbhakar et al., 2013), …
• Semi-parametric approaches: StoNEzD (Johnson & Kuosmanen, 2011), …
• BUT: Regulatory models typically based on standard DEA or SFA
4
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
1
Motivation
Aim of this study
• Systematical performance evaluation of Latent Class, StoNEzD and conditional DEA in
the presence of environmental factors
• Generalization of results via Monte-Carlo-Simulation
 Guidance for regulators to choose estimators given industry
structure and industry characteristics
Scope of this study
• Consideration of different model set-ups imitating real regulatory data
 Cross section with variation in sample sizes, noise and inefficiency distributions
and in terms of the true underlying technology
 Consideration of different cases of impact of environmental variables
5
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
1
Related Literature
Monte Carlo Simulation Studies
• Basic MC evidence in original research papers
• Andor & Hesse (2014): StoNED vs SFA vs DEA
• Henningsen, Henningsen & Jensen (2014): multi-output SFA
• Krüger (2012): order-m vs order-𝛼 vs DEA
• Badunenko, Henderson & Kumbhakar (2012): KSW bootstrapped DEA vs FLW
• Badunenko & Kumbhakar (forthcoming): persistent and transient ineff. SFA
Few studies focusing on environmental variables
• Cordero, Pedraja & Santin (2009) – z-variables in DEA
• Yu (1998) – z-variables in DEA and SFA
6
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
2
Methodologies
7
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
2
Methodology – Notation
• Production function 𝑦𝑖 = 𝑓 𝑥𝑖 ∗ exp(𝛿𝑧)
• 𝑁 observations, 𝑖 = 1, … , N
1
• Input 𝑥𝑖 , … , 𝑥𝑖
𝑚
∈ ℝ 𝑚 to produce output 𝑦𝑖 ∈ ℝ
• Deviation from the frontier 𝜀 = 𝑣 − 𝑢
• 𝑢~𝑁 + 0, 𝜎𝑢
2
, 𝑣~𝑁 0, 𝜎𝑣
2
• Expected inefficiency 𝜇
• Environmental factors
1
• Vector of environmental factors 𝑧𝑖 , … , 𝑧𝑖
𝛿1, … , 𝛿 𝑞 )
8
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
𝑞
∈ ℝ 𝑞 with impact (𝛿 =
2
Methodology – conditional DEA
• DEA with firm specific reference sets (Daraio & Simar, 2005, 2007)
depending on realization of 𝑧 s.t. 𝜃 𝐷𝐸𝐴 = sup 𝜃 𝐷𝐸𝐴 𝑥, 𝜃 𝐷𝐸𝐴 𝑦 ∈ Ψ 𝑧
• Estimation of the reference set: Kernel estimation
• Frontier reference point is 𝑓 = 𝜃 𝐷𝐸𝐴 𝑦 (output oriented for comparability)
9
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
2
Methodology – Latent Class
• LC SFA tries to account for unobserved factors and heterogeneity in technologies
(Greene, 2002; Orea & Kumbhakar, 2004)
• Consideration of J classes to estimate – class-specific shape of 𝑓(𝛽𝑗 , 𝑥𝑖 )
ln𝑦𝑖 = ln 𝑓(𝛽𝑗 , 𝑥𝑖 ) + 𝑣𝑖 |𝑗 − 𝑢𝑖 |𝑗
• Endogenous selection of class membership: multinomial logit model
𝑃𝑖𝑗 (𝜑𝑗 ) =
10
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
exp(𝜑𝑗 𝑧𝑖 )
𝑗 exp(𝜑𝑗 𝑧𝑖 )
with 𝜑𝐽 = 0
2
Methodology – Latent Class
Estimation: ML or MSL - Likelihood function (𝐿𝐹) as function of
• 𝜏 - parameters of the technology – pre-specified functional form
• 𝜑 - parameters describing class membership
𝑛
ln 𝐿𝐹(𝜏, 𝜑) =
𝑛
ln 𝐿𝐹𝑖 (𝜏, 𝜑) =
𝑖=1
𝐽
ln
𝑖=1
𝐿𝐹𝑖𝑗 𝜏𝑗 ∗ 𝑃𝑖𝑗 (𝜑𝑗 )
𝑗=1
• Posterior class membership probability can be calculated as
𝑃 𝑗𝑖 =
𝐿𝐹𝑖𝑗 𝜏𝑗 ∗ 𝑃𝑖𝑗 (𝜑𝑗 )
𝑗 𝐿𝐹𝑖𝑗
𝜏𝑗 ∗ 𝑃𝑖𝑗 (𝜑𝑗 )
• This class membership probability can then be used to either weight the efficiency
scores – or the frontier reference points
 Weighted frontier reference point: 𝑓𝑖 =
11
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
𝑗𝑃
𝑗 𝑖 ∗ 𝑓𝑖 (𝛽𝑗 , 𝑥𝑖 )
2
Methodology – StoNEzD
StoNEzD for Normal-Half-Normal Noise / Ineff.
1. Stage QP: Estimation of average function 𝑔(𝑥)
• No functional form (but piece-wise linear)
• 𝛿 is common to all firms
2. Stage: Decomposing residuals of first stage
• MM estimator to derive 𝜎𝑢
E u = 𝜇 = 𝜎𝑢 2/𝜋
• Shift of 𝑔(𝑥) by expected value of inefficiency
to derive frontier estimate 𝑓(𝑥)
 Frontier reference point: 𝑓𝑖 = 𝜙𝑖 ∗ exp 𝛿𝑧𝑖 ∗ exp(𝜇)
12
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
2
Methodology – Comparison of cDEA, LC SFA and StoNEzD for production function
cDEA
LC SFA
StoNEzD
Type
Non-parametric
Parametric
Semi-parametric
Error / Inefficiency
Deterministic
Stochastic
Stochastic
Shape
Constrained
Parametrically
constrained
Constrained
Scaling
assumption
Necessary
Possible
Possible
Convexity of T
Yes
No
Yes
Reference set
Observation specific
All observations,
weighted
All observations
Effect of z on
frontier
Observation specific
Grouped, but
observation specific
via weighting
General effect
13
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
3
The DGP
14
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
3
Data Generating Process
• DGPs are created to replicate real world regulatory data
General relationship
𝑦 = 𝑓 𝑥 ∗ exp 𝛿𝑧 ∗ exp(𝑣 − 𝑢)
Sample Size
𝑛𝑠𝑚𝑎𝑙𝑙 = {25, 50,100,150,250}
+ 4% observations twice as large in terms of inputs
 𝑛 = {26,52,104,156,260}
Inputs
15
4 correlated Inputs
𝑥1 … 𝑥4 ~Uni(0,10) for small and 𝑥~Uni(10,20) for large firms
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
3
Data Generating Process
Functional form of 𝑓(𝑥)
• Translog 𝑝0 = 1, 𝑝𝑚 = 0.10, pml = 0.05, pmm = 0.02
Inefficiency and Noise 𝑢~|𝑁 0, 𝜎𝑢
2
|, 𝑣~𝑁 0, 𝜎𝑣
𝜎𝑢 = {0.05,0.1,0.15}
𝜎𝑣 = {0,0.05,0.1}
• Noise-to-Signal:
16
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
2
with
3
Data Generating Process
Environmental Factors 4 different distributions considered,
1 symmetric, 3 skewed, 1 correlated with inputs
17
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
4
Simulation Design and Performance
Measures
18
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
3
Simulation Design
Scenarios
• So far only two different scenarios: Baseline (BL) and High Impact (HI) scenarios
• Only one 𝑧-Variable considered each, variation in impact 𝛿
𝑦 = 𝑓 𝑥 ∗ exp 𝛿𝑧 ∗ exp(𝑣 − 𝑢)
• Each scenario estimated with variation in sample size, 𝜎𝑢 and 𝜎𝑣 , for each estimator
19
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
3
Simulation Design
Implementation
Replications: 𝑅 = 100  100*9*5 = 4500 data sets for 3 estimators
R samples for u and v for each scenario
x,y and z are constant over one scenario
Samples with strong deviations from the DGP are discarded
(correlations in 𝜌𝑥 = ±0.15, wrong skewness in 𝑣 − 𝑢)
StoNEzD
Implemented with Sweet Spot Approach (Lee et al. 2013)
MoM with 𝑀3 set to -0.0001 if wrong skewness occurs
Latent Class
CD estimation
Estimation with 2 - 4 classes, reported is max BIC
5 repetitions with „randomized“ starting values
cDEA
20
Least squares cross validation, Epanechnikov kernel
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
3
Performance measures
Performance Evaluation
Evaluated at frontier reference points corrected for 𝛿𝑧
Performance Measures
Equally weighted deviation in percentage points
Bias > 0 overestimation of the frontier and of inefficiency
Average squared deviation, higher impact of larger
deviation
21
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
5
Initial Results
22
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
5
Initial Results
Generally…
• LC most often outperforms cDEA and StoNEzD
• Distribution of z does not seem to matter concerning bias
• Correlation of z & x has only little effect (BL4 vs. the others)
• Also magnitude of environmental effect seems to play a minor role (HI vs BL)
23
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
5
Initial Results
• LC SFA
• Performs generally well, stable and efficient
• Frontier overestimation tendecies in higher noise cases
• cDEA
• High sensitivity against noise
• Underestimation of frontier in small samples, overestimation in larger samples
• StoNEzD
• General underestimation of the frontier  favorable for firms
• Performs well with low inefficiency and small samples
• But problems with high inefficiency
• … but does not seem to be generally efficient
24
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
6
Outlook
25
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
5
Conclusion and Outlook
• Additional Scenarios
• Scenarios with multiple z variables
• Scenarios with heterogeneity in technologies induced by zs  𝛽 = 𝛽(𝑧)
• Misspecified scenarios?
• Estimation
• Optimization of optimization routines – still failed estimations although the
estimated model is the true underlying model
• Suggestions?
26
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
Vielen Dank für Ihre Aufmerksamkeit.
DIW Berlin — Deutsches Institut
für Wirtschaftsforschung e.V.
Mohrenstraße 58, 10117 Berlin
www.diw.de
Redaktion
5
28
Initial Results
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
5
29
Initial Results
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
5
30
Initial Results
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
5
31
Initial Results
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
5
32
Initial Results
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
5
33
Initial Results
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
5
34
Initial Results
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
5
35
Initial Results
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand
0
References
• Agrell, P. and Bogetoft, P. (2013). Benchmarking and Regulation. CORE Discussion Papers 2013008.
• Andor, M. and Hesse, F. (2014). The stoned age: the departure into a new era of efficiency analysis? a monte carlo comparison of stoned and the
oldies (SFA and DEA). JPA, 41(1):85-109.
• Badunenko, O., Kumbhakar, S. (2015) When, Where and How to Estimate Persistent and Time-Varying Efficiency in Panel Data Models. WP.
• Cordero, J. M., Pedraja, F., and Santin, D. (2009). Alternative approaches to include exogenous variables in DEA measures: A comparison using
Monte carlo. Comput. Oper. Res., 36(10):2699-2706.
• Daraio, C. and Simar, L. (2005). Introducing Environmental Variables in Nonparametric Frontier Models: a Probabilistic Approach. JPA, 24(1):93-121.
• Daraio, C. and Simar, L. (2007). Conditional nonparametric frontier models for convex and nonconvex technologies: a unifying approach. JPA,
28(1):13-32.
• Greene, W. H. (2005). Reconsidering heterogeneity in panel data estimators of the stochastic frontier model. Journal of Econometrics, 126(2):269303.
• Haney, A. B. and Pollitt, M. G. (2009). Efficiency analysis of energy networks: An international survey of regulators. Energy Policy, 37(12):5814- 5830.
• Johnson, A. and Kuosmanen, T. (2011). One-stage estimation of the effects of operational conditions and practices on productive performance:
asymptotically normal and efficient, root-n consistent StoNEzD method. JPA, 36(2):219-230.
• Jondrow, J., Knox Lovell, C. A., Materov, I. S., and Schmidt, P. (1982). On the estimation of technical inefficiency in the stochastic frontier production
function model. Journal of Econometrics, 19(2-3):233-238.
• Krüger, J. J. (2012). A monte carlo study of old and new frontier methods for efficiency measurement. EJOR, 222:137-148.
• Kuosmanen, T. (2012). Stochastic semi-nonparametric frontier estimation of electricity distribution networks: Application of the stoned method in
the Finnish regulatory model. Energy Economics, 34(6):2189-2199.
• Lee, C.-Y., Johnson, A. L., Moreno-Centeno, E., and Kuosmanen, T. (2013). A more efficient algorithm for convex nonparametric least squares. EJOR,
227(2):391-400.
• Orea, L. and Kumbhakar, S. C. (2004). Efficiency measurement using a latent class stochastic frontier model. Empirical Economics, 29(1):169-183.
• Yu, C. (1998). The effects of exogenous variables in efficiency measurement - a monte carlo study. EJOR, 105(3):569-580.
36
Benchmarking of Public Utilities
Stefan Seifert & Maria Nieswand