Homa Karimabadi 1UCSD, La Jolla, CA
Hybrid Code: H. X. Vu2, Y. A. Omelchenko1 , M. Tatineni3, A. Majumdar3, U. Catalyurek4
Fully Kinetic Simulations:
W. Daughton5, V. Roytershteyn1
Scientific Visualization, Data Mining and Computer Vision:
B. Loring8, T. Sipes2, A. Yilmaz9
MHD Simulations:
J. Dorelli6, J. Raeder7
2SciberQuest, Inc., Del Mar, CA 3SDSC, La Jolla, CA 4Ohio State Univ. 5Los Alamos National Lab. 6NASA Goddard Space Flight Center 7Univ. Of New Hampshire 8Lawrence Berkeley National Lab. 9Ohio State University
SIAM Conference on Parallel Processing for Scientific Computing
February 15-17, 2012
1
¾
Computations:
NSF’s Kraken – G. Brook / Y. Mack
DOE’s Jaguar – J. Wells
NASA’s Pleiades – J. Chang / M. Cary
DOE’s Hopper – F. Verdier
¾
Data Analysis:
Black light – J. Welling
Longhorn
Nautilus / Kraken – RDAV support has been critical
LENS
¾
What is space plasma physics?
plasma = ionized gas examples of plasmas: stars, neon signs, plasma TV
¾
Computational challenge in space sciences
¾
Codes & simulation characteristics
¾
New science enabled through petascale simulations
¾
Remaining bottlenecks
90 million miles or ~ 100 Suns
4
Video: Animation of Coronal Mass Ejection (CME)
A burst of fast material from the sun generates magnetic reconnection events in Earth's magnetic field. This eventually
sends high-speed electrons and protons into Earth's upper atmosphere to form aurorae. Credit: NASA/Goddard Space
Flight Center Conceptual Image Lab
yEarth
is embedded in the Sun’s atmosphere
yEarth’s dipole field shields us for the most part from the
effects of the Sun
y But a process called magnetic reconnection fractures
the Earth’s protective shield, exposing us to the effects
of solar activity
ySpace
weather affects the Earth and its technological systems
- Over $4 billion in satellite losses
- Damage sensitive electronics on orbiting spacecraft
- Cause colorful auroras, often seen in the higher latitudes
- Create blackouts on Earth when they cause surges in power grids
y
Travels at 20km/s to 3200km/s
y
Can take 1‐5 days to reach the Earth
y
During solar maxima can have 3 / day
y
During solar mimina can have 1 every 5 days
y
Release terawatt of power
¾Sun is about 109 times larger than Earth in diameter and is about 330 times
more massive.
¾Distance of Sun from Earth is ~90 million miles ~ 100 Suns.
¾Earth weighs about ~ 1.3x1025 pounds.
8
Simulation Domain
120 x 30 x 30 RE3
Solar Wind Inflow Boundary Condition
1 RE =130 c/ωp
Ω-1=0.5sec
20 RE
1 RE =6 c/ωp
Ω-1=1.5sec
100 RE
9
yMulti-Scale Coupled System
y Spatial scales vary from centimeters to 200 RE (span of 1011 spatial scales!)
y Temporal scales vary from less than milliseconds to days (span of 108 temporal
scales!)
y Kinetic effects have global consequences
Requires Particle Simulations
yMulti-physics
y electron physics: e.g., controls reconnection rate
y ion physics: e.g., dominates formation of boundaries and transport
y global features and dynamics: e.g., magnetotail/energetic particles
y coupling to the ionosphere
¾Requires yottaflops (1024) and beyond
¾Even with infinite computational power, numerical round off
would swamp the results
y MHD (single fluid)
y used extensively
y does not resolve important ion physics
y does not correctly capture the physics of reconnection
y not suitable for studies of boundaries & discontinuities
Petascale computing has enabled:
y Hybrid (fluid electrons, kinetic ions)
y fluid electrons, kinetic ions
y it is the next stage of advance in 3D global simulations
y resolves ion spatial scales (ion inertial length) and ion temporal
scales (gyroperiod)
y Full Particle (kinetic electrons, kinetic ions)
y it is the most complete description
y only 2D is possible
Our largest simulation
involves 15 billion cells
and 3.2 trillion particles
Hybrid Code
►Nonuniform Mesh + Multi‐time zones
►Discrete Event Technique
- robust (stable) and efficient (no idle computation)
- works for arbitrary meshes
- Predict local Δt for each variable f (“state”) based on its estimated
trajectory, f=fE(t) and a given ΔfE (selected based on local CFL/reaction
conditions)
Fully Kinetic Code
► 2D global for capturing the microphysics of reconnection in a global setting
13
•
Ions: kinetic particles
•
Electrons: massless, quasi‐neutral (ene = qini) fluid
•
Electromagnetic fields:
‐ Faraday’s law
(∂ /∂ t) B = ‐c ∇ × E
‐ Ampere’s law
∇ × B = 4 π J /c = 4π qini (vi ‐ve) /c
‐ Electric field from electron momentum equation
E = ‐vi × B /c ‐∇pe /(qini)‐B × (∇ × B) /(4π qini)‐ηJ
14
•
Stretched mesh
•
Temporal zones
15
Omelchenko and Karimabadi, JCP, 2011
16
y
Large I/O (over 200 TB from a single run)
y
Memory limited
y
Expensive checkpointing
y
“Noisy”, 3D multi‐variate data
y
Long integration times
17
Teraflop: Billion Particles
Petaflop: Trillion Particles
Daughton et al., Nature Physics, 2011
19
Daughton et al., Nature Physics, 2011
20
Our largest simulation with over 3 trillion particles
24
2D: Teraflop
3D: Petaflop
25
31
… Petascale computing is leading to breakthrough science in space sciences
… Bottlenecks to progress are:
‐ Visualization of multi‐variate, “noisy”, 3D data
‐ Sharing of data sets
‐ Archiving
‐ I/O
‐ Disk space
‐ Allocation time on supercomputers
‐ Algorithmic issues