CS 584

Assignment
Systems of Linear Equations

A linear equation in n variables has the form
a0x0 + a1x1 + … + an-1xn-1
= b
A set of linear equations is called a system.
 A solution exists for a system iff the solution
satisfies all equations in the system.
 Many scientific and engineering problems
take this form.

Solving Systems of Equations
a0,0x0 + a0,1x1 + … + a0,n-1xn-1
= b0
a1,0x0 + a1,1x1 + … + a1,n-1xn-1
= b1
an-1,0x0 + an-1,1x1 + … + an-1,n-1xn-1 = bn-1

Many such systems are large.
– Thousands of equations and unknowns
Solving Systems of Equations

A linear system of equations can be
represented in matrix form
a0,0
a0,1
… a0,n-1
x0
a1,0
a1,1
… a1,n-1
x1
an-1,0
an-1,1 … an-1,n-1
Ax = b
xn-1
b0
=
b1
bn-1
Solving Systems of Equations
Solving a system of linear equations is done
in two steps:
– Reduce the system to upper-triangular
– Use back-substitution to find solution
 These steps are performed on the system in
matrix form.
– Gaussian Elimination, etc.

Solving Systems of Equations


Reduce the system to upper-triangular form
a0,0
a0,1
… a0,n-1
x0
0
a1,1
… a1,n-1
x1
0
0
… an-1,n-1
xn-1
Use back-substitution
b0
=
b1
bn-1
Reducing the System
Gaussian elimination systematically
eliminates variable x[k] from equations k+1
to n-1.
– Reduces the coefficients to zero
 This is done by subtracting a appropriate
multiple of the kth equation from each of the
equations k+1 to n-1

Procedure GaussianElimination(A, b, y)
for k = 0 to n-1
/* Division Step */
for j = k + 1 to n - 1
A[k,j] = A[k,j] / A[k,k]
y[k] = b[k] / A[k,k]
A[k,k] = 1
/* Elimination Step */
for i = k + 1 to n - 1
for j = k + 1 to n - 1
A[i,j] = A[i,j] - A[i,k] * A[k,j]
b[i] = b[i] - A[i,k] * y[k]
A[i,k] = 0
endfor
endfor
end
Parallelizing Gaussian Elim.
Use domain decomposition
– Rowwise striping
 Division step requires no communication
 Elimination step requires a one-to-all
broadcast for each equation.
 No agglomeration
 Initially map one to to each processor

Communication Analysis
Consider the algorithm step by step
 Division step requires no communication
 Elimination step requires one-to-all bcast
– only bcast to other active processors
– only bcast active elements
 Final computation requires no
communication.

Communication Analysis
One-to-all broadcast
– log2q communications
– q = n - k - 1 active processors
 Message size
– q active processors
– q elements required

T = (ts + twq)log2q
Computation Analysis
Division step
– q divisions
 Elimination step
– q multiplications and subtractions
 Assuming equal time --> 3q operations

Computation Analysis

In each step, the active processor set is
reduced by one resulting in:
CompTime = k =0 n - k - 1
n -1
CompTime = 3n(n - 1) / 2
Can we do better?
Previous version is synchronous and
parallelism is reduced at each step.
 Pipeline the algorithm
 Run the resulting algorithm on a linear array
of processors.
 Communication is nearest-neighbor
 Results in O(n) steps of O(n) operations

Pipelined Gaussian Elim.


Basic assumption: A processor does not need to
wait until all processors have received a value to
proceed.
Algorithm
– If processor p has data for other processors,
send the data to processor p+1
– If processor p can do some computation using
the data it has, do it.
– Otherwise, wait to receive data from processor
p-1
Conclusion
Using a striped partitioning method, it is
natural to pipeline the Gaussian elimination
algorithm to achieve best performance.
 Pipelined algorithms work best on a linear
array of processors.
– Or something that can be linearly mapped
 Would it be better to block partition?
– How would it affect the algorithm?
