Image Re-slicing for Parallel Computing
Mark Sedrak | Supervised by Darren Thompson & Sam Moskwa
13 February 2013 | Big Day In - Summer Vacation Project
IM&T ADVANCED SCIENTIFIC COMPUTING
Project Introduction
• Aim: To improve the existing cluster re-slicing routine in X-TRACT
with Parallel Computing.
• Moore’s Law: Hardware and Data expansion.
• To be covered:
– Image Re-Slicing.
– Parallel Computing and the use of Super Computers.
– My work through-out the project
2 | Image Re-Slicing for Parallel Computing| Mark Sedrak
Image Re-slicing
• What is it?
– Slices from a CT reconstruction
• Synchrotron
• MRI
3 | Image Re-Slicing for Parallel Computing| Mark Sedrak
Reconstructed Image
4 | X-ray imaging tools for HPC clusters and the Cloud | Darren Thompson
Image Re-slicing
• Uses:
– Medical Imaging
– Image reconstruction (Materials, Objects, etc)
• XTRACT
– Software developed by CSIRO
• Data Sizes
N / M*
1k / 720
2k / 1,440
4k / 2,880
8k / 5,760
N2 float
(projection /
slice)
4 MB
16 MB
64 MB
256 MB
5 | Image Re-Slicing for Parallel Computing| Mark Sedrak
NM float
(sinogram)
2.8 MB
11¼ MB
45 MB
180 MB
N2M float
N3 float
(all
(all slices)
sinograms)
2.8 GB
4 GB
22½ GB
32 GB
180 GB
256 GB
1.4 TB
2 TB
Parallel Computing
• Serial vs. Parallel Programming
– Serial: Instructions are executed one-by-one in sequence.
– Parallel: Instructions can be executed simultaneously.
• Splits the work
• Aspects of Parallel Systems
• Communication
– Embarrassingly Parallel, Coarse-Grain Parallel, Fine-grain Parallel
• Memory
– Shared Memory, Distributed Memory
• Problem Definition
– Data Parallel, Task Parallel
6 | Image Re-Slicing for Parallel Computing| Mark Sedrak
Supercomputers and Message Parsing
• Super Computers
– Clusters
• TBI “Mini Cluster”, MASSIVE
• Bragg: Dual 8-Core CPU’s, 128GB RAM, 40Gb/s InfiniBand
• Burnet (Specs): Dual 6-Core CPU’s, 48/96 GB RAM, 40Gb/s InfiniBand
– File Systems
• GPFS, HNAS
• Message Passing Interface (MPI)
– A Framework for sharing information between distributed memory
processes
– Different communication types: 1-1, 1-Many, Many-to-Many
– Synchronous vs. Asynchronous Communication
7 | Image Re-Slicing for Parallel Computing| Mark Sedrak
My Project
• Optimising the re-slicing routine
– Generic and portable
• Three main aspects
– Communication
– Computation/Shuffling
– File I/O (Input/Output)
• Developed Three Main Methods
– Method 1: (Single Mass communication, High Memory)
– Method 2: (Multiple Smaller Communication, High Memory)
– Method 3: (Multiple Smaller Communication, Low Memory)
8 | Image Re-Slicing for Parallel Computing| Mark Sedrak
Results - 2k dataset
400
350
350
300
300
250
250
200
200
150
150
100
100
50
50
0
0
0
10
20
30
40
50
60
70
0
5
10
15
20
Method1 - Bragg
Method 1 - Burnet
400
350
350
300
300
25
30
35
250
250
200
200
150
150
100
100
50
50
0
0
0
5
10
15
20
25
Method 2 - Burnet
9 | Image Re-Slicing for Parallel Computing| Mark Sedrak
30
35
0
5
10
15
Method 2- Bragg
20
25
30
Results - Overview
Image Re-slicing routine - 2k data set
250
•Shows method 1 compared with
M2, on Both Burnt and Bragg
Time(s)
200
•Issues
•Shared users
•Resource Limits
•Bottlenecks (File System)
150
100
50
0
1
2
3
4
5
M1 - Burnet
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20
Nodes
M2 - Burnet
M1 - Bragg
M2 - Bragg
Re-slicing Routine Optimisation Results
9
8
7
6
5
4
3
2
1
0
•Shows the effectiveness of
outputs for the Different Data
Sizes, of M1,2,3 on both
Burnet and Bragg
M1 M2 M3
M1 M2 M3
Bragg
Burnet
1k
M1 M2 M3
M1 M2 M3
Bragg
Burnet
2k
10 | Image Re-Slicing for Parallel Computing| Mark Sedrak
M1 M2 M3
M1 M2 M3
Bragg
Burnet
4k
•4k Data Set, 256 GB
•7-10 min
Summary
• Image Re-Slicing
• Using Parallel Computing to Solve the Data Problem
• File I/O bottleneck, recommend a parallel file system.
11 | Image Re-Slicing for Parallel Computing| Mark Sedrak
Thank you
IM&T ASC
Mark Sedrak
Student
e [email protected]
w www.csiro.au
IM&T ADVANCED SCIENTIFIC COMPUTING