Parallel blood flow simulations
Lampros Mountrakis,
UNIVERSITY OF AMSTERDAM
ELECTRIC ANT LAB B.V.
http://uva.computationalscience.nl/
1
13 December 2015
Super D Event
SURFsara
L.Mountrakis@{uva.nl,gmail.com}
Motivation
- Thrombosis in IA
‣ Intracranial Aneurysms are a disorder in which the
wall of a cerebral artery weakens, causing a localized
dilation (or ballooning) of the blood vessel.
‣ Usually asymptomatic and frequent (1-5% of the adult population)
‣
‣
Quite lethal in case of rupture:
‣ 30-day mortality rate of 45%
‣ 30% of survivors with moderate to severe
disability
Image: Wikipedia
The formation of a thrombus --an aggregation of
platelets and fibrin-- may lower the risk of rupture
‣ The transport of blood cells is important for the
formations of a thrombus in an IA.
Image: THROMBUS
Ref: Cerebral Aneurysms; Jonathan L. et al; N Engl J Med 2006
2
Motivation
- Thrombosis in IA
University of Geneva
3
BLOOD
RBC
Blood is composed of:
• Plasma (55-60%) (also called hematocrit)
• Red blood cells (RBCs, ~40-45%)
• White Blood Cells (WBCs)
• Platelets
WBC
Platelet
RBC
RBC
RBC
Main functions:
‣ Deliver oxygen and transport waste
‣ Protect against infections and
parasites
‣ Heal wounds
4
Platelets
BLOOD
•
Blood is a dense suspension of
deformable RBCs immersed in
plasma
• 40-45% volume fraction 5 million RBCs in 1 mm3 •
Complex behaviour of blood is
highly attributed to RBCs
• Non-newtonian rheology (Shear
thinning fluid)
•
Transport of other cells (WBC,
platelets) and other substances
5
[Aarts1988]
SOME NUMBERS
•
5 million RBCs in 1 mm3,
•
Artery of r=1mm and L=4mm:
• NRBC=62M (V=12.5mm3)
•
Coarse-grained representation 1
of an RBC : 250-500 vertices.
10
•
O(10 )…
•
Realistic geometries? Parallel and scalable blood flow code
Image: Melchionna 2011
ficsion
framework for immersed cell suspensions
1 Pivkin & Karniadakis, PRL 2008
6
ficsion
•
•
Build on top of palabos
A C++ parallel lattice Boltzmann solver:
• open source
• wide range of modules
• parallel: Domain Decomposition
• scalable (proven in haemodynamical
flows)
• has necessary data structures
Not a walk in the park..
• API? Data structures? Control of the communication ?
7
palabos.org
P1
P2
P3
P4
P5
P6
P7
P8
P9
ficsion
Immersed Boundary-Lattice Boltzmann Method (IB-LBM)
Fluid plasma — Newtonian
Lattice Boltzmann Method
as the fluid solver for Plasma
Image:thermopedia
Immersed Boundary Method
Eulerian description
to couple FEM and LBM
Spectrin-link model
Red blood cells
• Surface particles
Finite Element Model
for RBCs and platelets
(connectivity—mesh)
Single cell: stretching and tumbling/tank treading
24
20
DA
18
diameter [µm]
•
•
Stress-free model
Mean model
Experiment
22
16
14
12
10
8
(volume, surface)
6
4
10
8
6
2
0
0
Local quantities
Cell-global quantities DT
0
5
25
10
50
75
100
125
150
175
200
8
ficsion
Modeled as:
LatticeBlock
ParticleField3D
Serial
•
•Fluid
•Surface particles
•Cell global quantities
Fluid
SurfaceParticles
ParticleField3D
• Lagrangian particles class
analogous to a cell-list (fully
parallel)
D-D
Fluid
CellParticles SurfaceParticles
CellParticles
“Field” specific envelopes
Domain 0
Domain 0
Domain 1
ficsion EXAMPLE
10
SCALING PERFORMANCE: WEAK & STRONG
Fully periodic domain with an initial
fluid velocity
• All subdomains have the same
number of fluid nodes and
number of RBCs
➡ “perfectly” load balanced
situation
•
45%
11
Cartesius SURFsara
The NL
WEAK & STRONG SCALING
#procs
Domain
#procs Domain
1
2
4
1
2
4
Strong Scaling
83.7k RBC
662.2k RBC
Weak Scaling
328k RBCs
8k
4k
12
ficsion — CHANNEL-FLOW
45.2k RBC 3.4k Platelets in
Np=1024
Visualisation Paul Melis
13
ficsion — CHANNEL-FLOW
45.2k RBC 3.4k Platelets in
Np=1024
Visualisation Paul Melis
Load imbalance?
13
INTRACRANIAL ANEURYSMS
Two dimensional simulations have
revealed a high hematocrit region close
to the walls of the aneurysm.
This can signify two things:
•
not a physiological case and may
contribute to the wall weakening
and rupture of the aneurysm and..
•
…load balancing issues!
Problems not from idle processes, but
from overloaded!
Simulations in aneurysmal geometries with 2D model
14
15
BLOOD CELL SEPARATION: MICROFLUIDIC DEVICES
•
Separating blood cells is often a
prerequisite in performing diagnostic
operations
Deterministic lateral displacement is a
passive particle separation technique in
which particles encounter an array of
obstacles arranged in a specific geometry
• Computer-aided design is a well grounded
strategy for draft prototyping microfluidic
devices.
•
16
Validated blood models, fully resolving cell
mechanics and blood flow are important in
the prediction of blood cell separation
processes
• Traversing the parameter space and reaching a
set of favourable solutions faster saves time and expenses.
•
Experiment Consortium “RheoCube”
Virtual Rheology Lab for Complex Fluids:
Scientific start-up developing high-fidelity
Methods: LBM coupled to rigid-body solver
HPC: MPI, OpenMP, OpenCL
simulation models of complex fluids and running
virtual rheometry experiments (HPC simulations) to
explain, predict, and help improve materials &
processing.
SURFsara: performance optimization, resources
End-user, manufacturer of
Magneto-rheological fluids
Experiment Goal
Interface
www.RheoCube.com
High-fidelity virtual rheometer framework. A
one-stop-shop simulation tool for studying and
prototyping complex fluids.
High-fidelity rheology with full access to the
material's micro-structure dynamics.
State-of-the-art scientific models. Far beyond
what traditional CFD can do.
Powered by parallel super-computers in the
background. All automatic.
On-demand usage model. For both, frequent and
occasional users.
Dedicated data servers. No sensitive data in the
public cloud.
Infrastructure
CONCLUSIONS
Validation
Collective behaviour: viscosity, cell free layer, margination
Presented ficsion
•
•
Parallel IB-LBM solver based on
palabos
Platelet margination
Viscosity
[LM et al. Interface
Satisfactory weak and strong scaling
for up to 4k and 8k processors
•
.Real sized geometries will be delayed
•
Plenty of room in the micro-scale
CFL thickness
OUTLOOK
validation macroscopic
quantities…
• thrombosis…
• load balancing…
• …realistic geometries!
•
Thrombosis
platelet
constant concentration
in plasma
Together with
experimentalists from
Moscow State
University we are
working a model for
primary hemostasis
and a set of validation
benchmarks to test it.
19
vWF
fibrinogen
nt
e
c
fa
i
Po
R1
build-up of fibrin
R2
r
Su
ICAM
p-selectin
collagen
CEPAC
thrombin build-up
and unlocking
CEPAC
EC
immediate
unlocking/activation
ECM
ACKNOWLEDGEMENTS
Eric Lorenz
Saad Alowayyed
Orestis Malaspinas Bastien Chopard
20
Paul Melis
Alfons Hoekstra
Thank you!
21