Computational Techniques for Sparse Matrices
Jamie Glaves
School of Computing, Maths, & Digital Technology
Manchester Metropolitan University
January, 2017
Abstract
Sparse matrices are matrices with only a small percentage of non-zero elements (Tewarson,
1973). In research and scientific applications, especially in computational fluid dynamics and
large systems of chemical kinetics, matrices are often very large and sparse. Therefore, in order
to achieve a computational advantage sparse matrices must be treated differently by adopting
methods where the sparsity feature of the original matrix can be fully exploited.
This study will consider several matrix solution methods for a standard sparse, linear systems, such as the Conjugate gradient method, which can be flexibly adapted to parallel computing. Matrix preconditioning methods, such as decompositions, will also be presented and
compared, supplemented with in-house MATLAB codes, as well as various common reorderings
for sparse matrices, such as the reverse Cuthill-McKee algorithm. The performance will be compared in applications to real problems including fluid flow, with both computational accuracy
and efficiency when considering large sparse systems considered.
Key words: sparse matrices; ordinary differential equations; stiff ordinary differential equations; computational methods
1
Introduction
There is no one universal condition to be satisfied to qualify a matrix as sparse as opposed
to full. However, the importance of examining sparsity is illustrated by James H. Wilkinson’s
definition of a sparse matrix as ”any matrix with enough zeroes that it pays to take advantage
of them” (Björck, 1996).
Problems with sparse behaviour exist in many areas of mathematics and computational science, including quantum physics (Gong, 2016), fluid dynamics and systems of chemical kinetics
(Sandu et al. (1995), Nejad (2005)).
The majority of problems considered in this report consider the general linear system:
Ax = b,
(1)
where x is an unknown vector with coefficient matrix A, and b is the known solutions of the
system (Watkins, 1991). Unless explicitly defined otherwise, all references to A, x, or b in this
paper refer to this generality.
This first two parts of this paper will study general matrix solutions, first direct methods,
and then iterative methods, which are better suited to modern computational problems due to
utilising parallel programming (Golub and Ortega, 1993).
2
Direct Methods and Matrix Decompositions
Direct methods ”form the backbone of a wide range of problems” in Mathematics (Davis,
Rajamanickam, and Sid-Lakhdar, 2016). Gaussian elimination is the most commonly used
example of these for its robustness.
1
Manchester Metropolitan University
Tomorrow’s Mathematician’s Today 2017
However, partial pivoting step that often comes before Gaussian elimination was found
to be ”not necessary in an adaptive solver” of sparse ODE systems, according to Thompson
(1986). The author has verified this by running simulations on MATLAB using 1000x1000
sparse systems.
2.1
Matrix decompositions
Matrix decompositions are also known as factorisations, as they factorise A into matrices in
order to create easier to solve systems of equations.
For example, the LU decomposition is the ”lower upper” decomposition, as it uses the
decomposition:
A = LU,
(2)
(Golub and Van Loan, 2013) where L is a lower triangular matrix and U is an upper triangular
matrix. To solve the system, this is then substituted into Eq. (1), which splits it into two usually
simpler-to-solve systems. For example, for LU factorisation, the systems become:
L(U x) = b →
Ly = b
U x = y.
Here U x is set to be a variable y and solve the original system for y. Then, we solve U x = y
using this y (Duff, Erisman, and Reid, 1986). A similar process can be performed for any
splitting of A, several of which are covered here.
The QR decomposition is a very similar scheme, where A is instead split into:
A = QR,
(3)
where Q is an orthogonal matrix (i.e. satisfies QT Q = I) and R is a upper (or ’Right’) triangular
matrix. (Press et al., 2007). MATLAB codes already exist and are built-in for these two
decomposition methods, called lu and qr respectively.
The Cholesky factorisation of A requires A to be positive definite. For a matrix to be positive
definite, it is required to be square, real, symmetric, and to satisfy:
xT Ax > 0,
for all nonzero x ∈ Rn , where Rn is the set of all non-zero n × 1 arrays (Watkins, 1991).
The decomposition is given as:
A = LL∗ ,
(4)
where L is a lower triangular matrix, as defined when studying the LU decomposition, and L∗
is the conjugate transpose of L.
3
Iterative Methods for Matrix Solutions
Iterative methods differ from direct methods in that whilst direct methods find one solution
to a linear system, iterative methods go from a starting value of x to ”iterate” estimates of
x, converging to a solution if the method is successful. With the development of scientific
computing, these methods have gradually become more preferred to direct methods; especially
when dealing with sparse and large matrices (Saad, 2003).
Common examples of iterative methods include the Jacobi and Gauss-Seidel methods. In
general, these methods include decomposing A and then applying these to a scheme. A is
decomposed such that:
A = D − E − F,
Jamie Glaves
(5)
2
Manchester Metropolitan University
Tomorrow’s Mathematician’s Today 2017
where D is the diagonal of A, −E is its lower triangular matrix part and −F its upper part.
(Saad, 2003).
To converge to the solution faster, the Gauss-Seidel scheme can be applied to the Successive Overrelaxation scheme - commonly abbreviated to the SOR scheme - where a relaxation
parameter is inserted.
The Conjugate Gradient (CG) method is computationally flexible to implement and requires
only matrix-vector multiplications (Helfenstein and Koko, 2012). However, it requires A to be
positive definite.
It is defined in Shewchuk (1994) as:
xi+1 = xi + αi pi ,
(6)
where αi is:
αi =
−(pi , pi )
.
(pi , Api )
Note that (u, v) refers to the inner product of the vectors such that:
(u, v) = uT v.
(Golub and Ortega, 1993).
Listing 1: Algorithm for Conjugate gradient method, from (Golub and Ortega, 1993).
Set x (0) ; set p (0) = r (0) = b - A * x (0) .
Until convergence :
a ( k ) = -( r ( k ) ,r ( k ) ) /( p ( k ) ,A * p ( k ) )
x ( k +1) = x ( k ) - alpha ( k ) * p ( k )
r ( k +1) = r ( k ) + alpha ( k ) * A * p ( k )
beta ( k ) = ( r ( k +1) ,r ( k +1) ) /( r ( k ) ,r ( k ) )
p ( k +1) = r (+1) + beta ( k ) * p ( k )
A comparison between the Gauss-Seidel method - with and without various preconditionings,
which will be studied in the next section - and the conjugate gradient methods was run. This
table is printed in Table 1. It can be seen that for a randomly generated positive definite system,
the conjugate gradient method is far better than the Gauss-Seidel, even with preconditioning
methods applied.
Table 1: Computational time for solving an NxN system (with density 0.1) with iterative methods
N
Gauss-Seidel
100 0.007928804
500 0.502849338
1000 3.219242909
3000 116.8587276
4
Preconditioned
Gauss-Seidel (RCM)
0.007056294
0.405763259
3.281714849
105.7410554
Preconditioned
Conjugate gradient
Gauss-Seidel (MD)
0.006839876
0.002512009
0.303914523
0.002419189
2.618037809
0.004361086
99.96394861
0.044996817
Sparse Matrices
MATLAB is very well-suited for dealing with sparse matrices as there are many in-built functions
to deal with sparse matrices.
The reader should familiarise themselves with the MATLAB command spy, which produces
an image - known as a sparsity pattern - where the non-zero elements of an input matrix are
represented by a symbol (by default, a blue dot).
In MATLAB, the command sparse stores a sparse matrix in terms of its non-zero terms,
instead of dedicating a small amount of memory to each zero element. Naturally, the sparser a
matrix is, the more memory is saved.
Jamie Glaves
3
Manchester Metropolitan University
4.1
Tomorrow’s Mathematician’s Today 2017
Matrix reorderings
Before we begin to apply the methods covered in the previous two sections, it is usually a good
idea and often necessary to perform reorderings on them beforehand. This is to reduce the
phenomenon of ”fill-in” when we perform the computations; i.e. unnecessarily changing zero
elements into non-zero elements, slowing down the computational time for the problem (Ingram,
2006).
For example, in order to let direct methods converge to a solution for a system of equations
more quickly, reorderings can be performed to manipulate a system to our needs. For example,
one goal is to reduce the bandwidth of a sparse matrix. Bandwidth here is defined as a measure
of the deviance of non-zero terms from the diagonal, and for our purposes will be assessed purely
visually.
Note that this extra step does not mean we should consider sparsity a problem with large
systems of equations so much as a natural symptom of the large size of certain systems, and
they do not begin to actually solve the system.
A popular example of a matrix reordering is the reverse Cuthill-McKee algorithm. This
looks at the number of connections for each node in a system and reorders them largely based
on this number of connections. The aim of this algorithm is to reduce the bandwidth of the
matrix.
Similarly, but slightly more simply, the column count reordering counts the number of connections and orders them such that nodes with the highest number of connections are represented
at the end of the matrix (Nejad, 2016). In the example in the next part, note that whilst there
are still terms away from the diagonal, terms are instead collected at the bottom-right of the
matrix.
A similar algorithm that we have studied in the project is the minimum degree algorithm,
where the only key difference is the number of connections to each node is updated at each
step.
5
5.1
Case Studies
Test model
MATLAB can be used to generate large sparse symmetric systems with the only limitations
being computational. For example, these can be used to generate sparsity patters for each of
preconditioning, to see how they become more diagonal or form an ”arrowhead” shape, and
these are in Fig. 1.
5.2
Crank-Nicolson scheme and Brusselator model
The Crank-Nicolson scheme is a scheme used to solve partial differential equations, which has
sparse solutions for large grids (Shiach, 2016). It can be used in solving the advection scheme.
However, because the matrix solved in the Crank-Nicolson scheme is already largely diagonal,
the preconditionings do not affect computational time. Hence, it is important to show discretion
whilst dealing with sparse matrices, and it can be unwise to apply reorderings without an
understanding of what they intent to achieve.
Similarly, the Brusselator system models diffusion in the chemical reaction, and is an example
of a potentially stiff ordinary differential equation that can have sparse properties (MathWorks,
2016). The Brusselator equations are:
u0i = 1 + u2i vi − 4ui + α(N + 1)2 (ui−1 − 2ui + ui+1 )
(7)
vi0
(8)
= 3ui −
u2i vi
2
+ α(N + 1) (vi+1 − 2vi + vi+1 ).
Let’s consider that these are solved in the time interval [0, 10], where α = 1/50, with initial
conditions:
ui (0) = 1 + sin(2πxj ),
vi (0) = 3.
Jamie Glaves
4
Manchester Metropolitan University
Tomorrow’s Mathematician’s Today 2017
(a) Original sparsity pattern
(b) Reverse Cuthill-McKee
(c) Column Count algorithm
(d) Minimum degree algorithm
(e) LU factorisation
(f) QR factorisation
Figure 1: Sparse symmetric matrix with a size of 1000x1000 and a density of 0.003, and how different
preconditionings affect the placement of the elements.
(a) Sparsity pattern used in Crank-Nicolson scheme
(b) Comparison of computational times
Figure 2: Matrix and computational time with preconditioning used in solving the Crank-Nicolson
scheme for t = [010], with 20 nodes.
The more nodes we consider for this time interval, the more sparse the system used to solve this
system becomes. However, the A matrix is even more diagonal than that of the Crank-Nicolson
scheme,
6
Conclusions
Both direct and iterative solution methods have been studied in this paper, both appear to be
robust and effective, although some methods perform better depending on the system.
For example, direct methods perform well when the system is not too large. For a large,
sparse and positive definite system, for example, test results have proven that the Conjugate
gradient method is far more effective than the Gauss-Seidel method. Therefore, further work
that we recommend is to be more able to identify the properties of a sparse system to indicate
Jamie Glaves
5
Manchester Metropolitan University
Tomorrow’s Mathematician’s Today 2017
which methods are best suited to solving it.
With regards to the preconditioning, the author has found that if the coefficient matrix A
is already largely diagonal, it is not worthwhile to applying a reordering beforehand. Both real
applications studied in this paper are like this. However, when dealing with especially large
sparse problems - and, notably, problems historically have grown in magnitude over the years
(Saad, 2003) - it is essential to take advantage of sparsity in a large coefficient matrix itself.
References
Björck, Å. (1996). Numerical Methods for Least Squares Problems. Society for Industrial
and Applied Mathematics.
Davis, T. A., Rajamanickam, S., and Sid-Lakhdar, W. M. (2016). “A survey of direct
methods for sparse linear systems”. In: Acta Numerica 25, pp. 383 –566.
Duff, I. S., Erisman, A. M., and Reid, J. K. (1986). Direct Methods for Sparse Matrices.
New York: Oxford University Press.
Golub, G. and Ortega, J. M. (1993). Scientific Computing: An Introduction With Parallel
Computing. Academic Press.
Golub, G. and Van Loan, C. F. (2013). Matrix Computations. Fourth. John Hopkins.
Gong X.; Chen, Z.; Wu W. (2016). “The application of Cholesky decomposition in valence
bond calculation”. In: Journal of Computational Chemistry 37, pp. 2157 –2162.
Helfenstein, R. and Koko, J. (2012). “Parallel preconditioned conjugate gradient algorithm on GPU”. In: Journal of Computational and Applied Mathematics 236, pp. 3584–
3590.
Ingram, S. (2006). Minimum Degree Reordering Algorithms: A Tutorial. University of
British Columbia. url: https://www.cs.ubc.ca/ ~sfingram/cs517_final.pdf
(visited on 12/13/2016).
MathWorks (2016). Solve Stiff ODEs. url: https://uk.mathworks.com/help/matlab/
math/solve-stiff-odes.html.
Nejad, L. (2005). “A Comparison of Stiff ODE Solvers for Astrochemical Kinetics Problems”. In: Astrophysics and Space Science 299.
— (2016). Computational Methods in Ordinary Differential Equations. Lecture notes.
Press, W.H. et al. (2007). “Section 2.10: QR Decomposition”. In: Numerical Recipes: The
Art of Scientific Computing.
Saad, Y. (2003). Iterative Methods for Sparse Linear Systems.
Sandu, A. et al. (1995). “Efficient implementation of fully implicit methods for atmospheric chemistry kinetics”. In: Reports on Computational Mathematics 79.
Shewchuk, J. R. (1994). “An Introduction to the Conjugate Gradient Method Without
the Agonizing Pain”. In:
Shiach, J. (2016). Numerical Methods for Partial Differential Equations. Lecture notes.
Manchester Metropolitan University.
Tewarson, R. P. (1973). Sparse Matrices. New York: Academic Press.
Thompson, S. (1986). “The effect of partial pivoting in sparse ordinary differential equation solvers”. In: Computers and Mathematics with Applications, pp. 1183 –1191.
Watkins, D. S. (1991). Fundamentals of Matrix Computations. Second. New York: Wiley.
Acknowledgements
I wish to thank Drs. Lida Nejad and Jon Shiach very much for their assistance
and their research insight.
Jamie Glaves
6