Portfolio theory in terms of partial
covariance
Alec Schmidt
Kensho Technologies
Financial Engineering, Stevens Institute of Technology
Financial Risk and Engineering, NYU School of Engineering
Contents
• Main concepts
• Why use partial correlations in portfolio theory?
• Pearson’s and partial correlations between the US equity
ETFs
• Performance of the partial covariance based portfolios
1. Main concepts
1.1 Pearson’s correlations
Key assumption of the mean-variance theory (Markowitz 1952):
Portfolio risk can be described with the covariance matrix 𝜎𝑖𝑗 of
asset returns:
σp = 𝑖 𝑗 𝑤𝑖 𝑤𝑗 𝜎𝑖𝑗
(1)
In (1), w𝑖 is weight of asset 𝑖; 𝑖 =1,2, …, N; N is the number of
assets in the portfolio.
(2)
𝑖 𝑤𝑖 = 1
Covariance between two asset returns can be expressed in terms of
their Pearson’s correlation coefficient ρij
𝜎𝑖𝑗 = ρij 𝜎𝑖 𝜎𝑗
(3)
where 𝜎𝑖 2 is variance of return 𝑖.
1. Main concepts
1.1 Pearson’s correlations (continued)
Pearson’s correlation coefficient ρij is widely used for
describing asset price co-movements (Mantegna 1999, Plerou et
al 1999, Cizeau et al 2000, Laloux et al 2000, Forbes & Rigibon
2001, Campbell et al 2008, Podobnik et al 2009, Aste et al
2010, Krishan et al 2010, Pollet & Wilson 2010). Other measures
of asset price co-movements, such as concordance (Cashin et al
1999), conditional correlations (Engle 2002), and
Gerber
statistic (Gerber, Markowitz, & Pujara 2015) have been also
explored.
Main concepts
1.2 Mean-variance portfolio model
Expected portfolio return Rp = 𝑖 𝑤𝑖 E(r𝑖 )
Find 𝑤𝑖 that
a) minimize the risk σp for given Rp
or
b) maximize Sharpe ratio (Rp - Rf)/σp; Rf is a risk-free rate.
Some problems:
1. Estimation errors => Equal weight portfolios (DeMiguel et
al. 2009).
2. Extreme long and short positions => 𝑤𝑖 ≥ 0 (Jacobs et al
2013).
3. Poor diversification (Green & Hollifield 1992) => 𝑤𝑖 ≥ 𝑤0 >
1. Main concepts
1.3 Partial correlations
Pearson’s correlation between two asset prices may be affected by
relationships of these prices with some other, common external
variable.
Partial
correlations
filter
out
such
relationships
(Shapira et al (2009); Kenett et al 2010, 2012, 2014).
Partial correlation coefficient, ρij|k, between variables Xi and Xj
that is conditioned on variable Xk measures correlation between
residuals of linear regressions of Xi on Xk , and Xj on Xk (Johnston
& DiNardo 1999). It can be expressed in terms of the Pearson’s
correlations:
ρ
=
𝜌𝑖𝑗 −𝜌𝑖𝑘 𝜌𝑗𝑘
(4)
2. Why to use partial correlations in
portfolio
theory: a physicist’s view (I)
Let’s talk about like-charge attraction for a moment:
Coulomb’s law: like charges repel each other, right?
Well, not always. Coulomb’s law is valid in a dielectric
medium (vacuum included).
2. Why use partial correlations in portfolio
theory: a physicist’s view (II)
Consider colloid particles in electrolyte:
Under some conditions (e.g. large particles with low charges
and/or multi-valent counter-ions), colloid particles interact
effectively as neutral particles due to complete screening of the
particle charges by electrolyte counterions (Schmidt 1987, 1993,
1999, 2001; Livshits & Schmidt 1988; Outhwaite & Molero 1992;
Hribar & Vlachy 1997).
In order to describe true particle interactions, one should filter
out the collective effects caused by particle interactions with
the medium.
2. Why use partial correlations in portfolio
theory: a physicist’s view (III)
According to CAPM,
E(ri) = rf + βi (E(rM) - rf)
(4)
In (4), rf is the risk-free rate of interest, rM is the market
return, and βi = cov(ri, rM)/σM2. Practitioners treat the CAPM
equation (4) as an empirical regression for estimating asset
excess return, αi , and sensitivity of asset return to market
return, βi (Grinold & Kahn 1999):
ri = αi + βi rM + εi
(5)
Since the total market return contributes to returns of
individual assets, it is natural to expect that the market
direction affect correlations between the asset returns. Hence
the choice of partial correlations conditioned on entire market
(SPY as proxy).
3. Correlations among the US equity ETFs (I)
Pearson’s correlations between returns of the US equity sector ETFs and SPY
ETF
Ticker
Sector
Pearson's correlation with SPY
2007
2008
2009
2010
2011
2012
2013
XHB
Homebuilders
0.68
0.71
0.85
0.84
0.92
0.74
0.75
XLY
Consumer Discrete
0.89
0.87
0.92
0.94
0.96
0.90
0.92
XRT
Retail
0.82
0.77
0.84
0.84
0.90
0.82
0.82
XLP
Consumer Staples
0.79
0.83
0.77
0.87
0.91
0.79
0.79
XLE
Energy
0.80
0.84
0.89
0.92
0.92
0.87
0.84
XLF
Finance
0.89
0.85
0.89
0.92
0.94
0.90
0.93
XLV
Healthcare
0.86
0.84
0.71
0.86
0.94
0.87
0.85
XLI
Industrial
0.90
0.91
0.92
0.96
0.98
0.92
0.92
XLB
Materials
0.87
0.85
0.89
0.90
0.95
0.88
0.86
XLK
Technology
0.86
0.93
0.91
0.94
0.95
0.91
0.83
XLU
Utilities
0.71
0.83
0.71
0.83
0.84
0.53
0.65
RWR
Real Estate
0.77
0.81
0.86
0.86
0.90
0.71
0.68
3. Correlations among the US equity ETFs (II)
Statistics of correlations between returns of the US equity
ETFs
Pearson’s corr.
Partial corr.
Year
p>0.1
corr < 0
p>0.1
corr < 0
2007
0
0
53.0%
21.2%
2008
0
0
30.3%
30.3%
2009
0
0
33.3%
37.9%
2010
0
0
34.8%
39.4%
2011
0
0
37.9%
39.4%
2012
0
0
37.9%
39.4%
2013
0
0
43.9%
34.8%
3. Correlations among the US equity ETFs
(III)
Correlation networks
Pearson distance: dij = 1 – ρij(k)
Pearson’s correlations
correlations
for 2009 – 2013
Partial
2013
4. Partial correlations - based mean variance
portfolio
4.1 General relationships
Portfolio partial covariance
σp2 = 𝑁
𝑖,𝑗=1 𝑤𝑖 𝑤𝑗 𝜎𝑖𝑗|𝑘
Relationship between partial correlations and partial
covariance (Whittaker 1990):
σij|k ≡ cov(Xi, Xj | Xk) = cov(Xi, Xj) - cov(Xi, Xk) cov(Xj, Xk)/var(Xk)
= ρij|k σi|k σj|k
In particular,
σi|k2 = σi2 - σik4/σk2
5. Optimal portfolios formed by major US
equity ETFs (I)
Max Sharpe with wi > 0. Partial correlations yield highly
diversified portfolios
Year
SPY
return
2014
0.136
2013
2012
2011
2010
2009
2008
2007
0.255
0.132
0.009
0.123
0.204
-0.45
0.052
Times included
Corr.
XHB
XLY
XRT
XLP
XLE
XLF
XLV
XLI
XLB
XLK
XLU
RWR
Weights ≥ 1%
Pearson
0.00
0.00
0.00
0.00
0.00
0.00
0.09
0.00
0.00
0.00
0.25
0.66
3
Partial
0.01
0.11
0.00
0.11
0.08
0.15
0.13
0.11
0.05
0.21
0.33
0.00
10
Pearson
0.00
0.21
0.21
0.00
0.00
0.00
0.52
0.06
0.00
0.00
0.00
0.00
4
Partial
0.00
0.11
0.00
0.11
0.10
0.17
0.13
0.11
0.03
0.21
0.04
0.00
9
Pearson
0.55
0.00
0.00
0.00
0.00
0.00
0.45
0.00
0.00
0.00
0.00
0.00
2
Partial
0.00
0.11
0.00
0.13
0.11
0.14
0.13
0.10
0.03
0.22
0.04
0.00
9
Pearson
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
0.00
1
Partial
0.00
0.11
0.28
0.07
0.00
0.00
0.18
0.00
0.00
0.04
0.32
0.00
6
Pearson
0.00
0.00
0.95
0.05
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
2
Partial
0.00
0.08
0.00
0.13
0.13
0.16
0.11
0.11
0.02
0.22
0.04
0.00
9
Pearson
0.00
0.00
0.61
0.00
0.00
0.00
0.00
0.00
0.00
0.39
0.00
0.00
2
Partial
0.01
0.03
0.05
0.11
0.13
0.12
0.17
0.10
0.03
0.20
0.05
0.01
12
Pearson
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1
Partial
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1
Pearson
0.00
0.00
0.00
0.38
0.61
0.00
0.00
0.00
0.00
0.00
0.01
0.00
3
Partial
0.00
0.00
0.00
0.16
0.29
0.00
0.00
0.11
0.11
0.24
0.09
0.00
6
Pearson
1
1
3
3
1
0
3
1
0
1
3
1
Partial
2
6
2
8
6
5
6
6
6
7
7
1
5. Optimal portfolios formed by major US
equity ETFs (2)
Out-of-sample performance of maximum Sharpe portfolios formed by ETFs
Calibration window: 36 months starting in Jan 2007
Rebalancing
Monthly
Yearly
Portfolio
Return
σ
Sharpe
MDD
Return
σ
Sharpe
MDD
EWP
0.108
0.147
1.273
0.033
0.113
0.157
0.860
0.063
Pearson
0.130
0.146
1.302
0.033
0.146
0.160
1.097
0.064
Partial
0.108
0.132
1.337
0.030
0.123
0.142
0.862
0.059
Calibration window: 12 months starting in Jan 2007
Rebalancing
Monthly
Yearly
Portfolio
Return
σ
Sharpe
MDD
Return
σ
Sharpe
MDD
EWP
0.065
0.193
0.989
0.043
0.059
0.210
0.610
0.090
Pearson
0.047
0.170
0.897
0.039
0.062
0.216
0.575
0.095
Partial
0.043
0.166
0.929
0.038
0.027
0.191
0.596
0.086
Returns for the 12-month calibration window are due to the fact that the
bear market of 2008 is included in its performance statistics but is not
included in the performance statistics for the 36-month window.
10/1/2015
7/1/2015
4/1/2015
1/1/2015
10/1/2014
7/1/2014
4/1/2014
1/1/2014
0.80
10/1/2013
0.90
7/1/2013
1.00
4/1/2013
1/1/2013
10/1/2012
7/1/2012
4/1/2012
1/1/2012
10/1/2011
7/1/2011
4/1/2011
1/1/2011
10/1/2010
7/1/2010
4/1/2010
1/1/2010
10/1/2009
7/1/2009
4/1/2009
1/1/2009
10/1/2008
7/1/2008
4/1/2008
1/1/2008
Portfolio weights
5. Optimal portfolios formed by major US
equity ETFs (3)
One-year calibration window for Pearson correlations
1.10
XLP
XLV
XLY
XRT
XLU
XLK
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
1/1/2010
3/1/2010
5/1/2010
7/1/2010
9/1/2010
11/1/2010
1/1/2011
3/1/2011
5/1/2011
7/1/2011
9/1/2011
11/1/2011
1/1/2012
3/1/2012
5/1/2012
7/1/2012
9/1/2012
11/1/2012
1/1/2013
3/1/2013
5/1/2013
7/1/2013
9/1/2013
11/1/2013
1/1/2014
3/1/2014
5/1/2014
7/1/2014
9/1/2014
11/1/2014
1/1/2015
3/1/2015
5/1/2015
7/1/2015
9/1/2015
11/1/2015
Portfolio weights
5. Optimal portfolios formed by major US
equity ETFs (4)
Three-year calibration window for Pearson correlations
1.10
1.00
0.90
XLP
XLV
XLY
XRT
XLU
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
10/1/2015
7/1/2015
4/1/2015
1/1/2015
10/1/2014
7/1/2014
4/1/2014
1/1/2014
10/1/2013
7/1/2013
4/1/2013
1/1/2013
10/1/2012
7/1/2012
0.60
4/1/2012
0.70
1/1/2012
0.80
7/1/2011
0.90
10/1/2011
1.00
4/1/2011
1.10
1/1/2011
10/1/2010
7/1/2010
4/1/2010
1/1/2010
10/1/2009
7/1/2009
4/1/2009
1/1/2009
10/1/2008
7/1/2008
4/1/2008
1/1/2008
Portfolio weights
XLP
XLV
XLY
XLl
XRT
XLE
XLF
XLI
XLU
XLK
SPY price
0.50
SPY price
5. Optimal portfolios formed by major US
equity ETFs (5)
One-year calibration window for partial correlations
250
200
150
0.40
100
0.30
0.20
50
0.10
0.00
0
9/1/2015
0.60
11/1/2015
0.70
7/1/2015
0.80
5/1/2015
0.90
3/1/2015
1.00
1/1/2015
11/1/2014
9/1/2014
7/1/2014
5/1/2014
3/1/2014
1/1/2014
11/1/2013
9/1/2013
7/1/2013
5/1/2013
3/1/2013
1/1/2013
11/1/2012
9/1/2012
7/1/2012
5/1/2012
3/1/2012
1/1/2012
11/1/2011
9/1/2011
7/1/2011
5/1/2011
3/1/2011
1/1/2011
11/1/2010
9/1/2010
7/1/2010
5/1/2010
3/1/2010
1/1/2010
Portfolio weights
5.Optimal portfolios formed by major US
equity ETFs (6)
Three-year calibration window for partial correlations
1.10
XLP
XLV
XLY
XLl
XRT
XLE
XLF
XLI
XLU
XLK
0.50
0.40
0.30
0.20
0.10
0.00
6. Conclusions:
• Partial
correlations
increase
mean-variance
portfolio
diversification
and
yield
almost
constant asset weights outside bear market.
• Partial
correlations-based
portfolio
may
outperform Pearson correlation–based portfolio
when proper model input parameters (calibration
window and rebalancing frequency) are chosen.
Q&A