Towards Enabling a Richer User Experience for General Relativistic Astrophysicists in Real-Time
Visualization and Interactive Steering
Magda Slawinska, Tanja Bode, Jeremy Logan, Michael Clark, Hongbo Zou, Matthew Kinsey, Matthew Wolf, Pablo Laguna, Scott Klasky
Georgia Institute of Technology, Atlanta, GA, University of Tennessee, Knoxville, TN, Oak Ridge National Laboratory, Oak Ridge, TN
CURRENT I/O AND ANALYSIS ISSUES
PROJECT OVERVIEW
Current I/O: mainly HDF5/ASCII
I Expensive I/O: up to 40% of simulation walltime
I Limiting output variables to a small number of 2D slices
I Simulation reruns if variables were not output
I Proneness to corruption with large files or incomplete runs
I Mostly static, shell-based, post-processing-based data
analysis and visualization
I
The goal is to enrich user’s experience
in numerical relativistic (NR) simulations
by allowing physicists to monitor and
steer NR simulations at run-time.
NR SIMULATIONS WITH MAYA
Maya is a 3D NR code
I Solves Einstein’s equations for black
holes, stars, etc.
I Based on the Einstein Toolkit, Cactus,
and Kranc
I Uses the Carpet AMR module
I
Time, Memory
reports, Ylm
Data Transform
(1)
Input
Stitch Run
(2)
DATA PROCESSING AND ANALYSIS
Wf_amp_phase/wf_
add_amp_phase
(5)
Ej-modesum
gif, mp4,
avi etc.
animation
(8)
(9)
(10)
VisIt
(6)
(7)
Radiusphase-3d
Angularvelocity
Trajectories
Output
AnimateT
rajs
Radiated
Energy
Tracker
Radius
Bremsst
rahlung
Psi4
phase
BBHydro
R Re[Psi4]
Radiated
Angular
Momentum
Tracker
Phase
Total Rest
Mass
rho
Animation
WDBH
Run
Speed
AM Flux
Tracker
Rdot
Outflows
Maximum
Density
Energy
Flux
Psi4
amplitude
Tracker
Omega
Rho
Movie
Max of
Lorentz Factor
BBHydro sum.
BBHStar
…
BBHMHD
OUR APPROACH
Improve I/O performance by incorporating ADIOS
I Customize output data on-the-fly with ADIOS/FlexPath
I Provide advanced steering capabilities
I
MAYA WORKFLOW
Python
script
cron
Maya Simulation
ADIOS Thorn
run
periodically
runview.sh
Stage Area
the data
with basic
processing
runsummary.sh
generate
movie
frames
h5movie.sh
ADIOSBPmovie.
sh
Data Processing Nodes
Control Path
Others
Data Path
Data Processing
User
animation,
poster
frame images
Summary
directory:
contains .png, .
html, movies
etc
VisIt
Stage Area
VisIt client
Control Path
Data Path
Data Processing
Parallel Storage
HPC Simulation
Machine
Visualization
Machine
User’s
Desktop
ADIOS integration
I Carpet I/O thorn + VisIt ADIOS
plugin
Maya workflow expansion
I Cataloging existing workflow
I Expanding bash-based scripts to
deal with the new infrastructure
I Improving query processing via
enabling customization and
increasing performance; more
@FlexQuery poster
I Adding particle support
ACKNOWLEDGMENTS
Dynamic, in-line
visualization
Numerical Relativistic
Simulation
Fast IO
VisIt
all the images of timeseries &
1d data, tarred data, .html
Parallel Storage
Simulation Machine
2d
hdf5
data
Processed Data
Analysis
Run, data stage-in
Website publish
http://numrel.org
Fast IO
plugin
processed data
submit
runview job
CarpetIO
plugin
ADIOSBP
plugin
control
Data Processing
and
Data Visualization
Browse, check, if correct, etc; Stop simulation if
catastrophic
User
run
Data
Visualization
Data Processing
Use FlexPath staging transport
within ADIOS:
I An event-based method with inline
transformations
I Memory-to-memory coupling and
disk-to-memory transformations
I In-system JIT compiler to deploy
binary filters at runtime
I Multi-protocol overlay networking,
including multi-machine
I And more . . . @FlexPath poster
CURRENT PROGRESS
BBHydro sum.
BBH
Physical quantities of interest
I Periodical checkpointing
I Some quantities computed during
simulation
I Post-processing including interactive
analysis, scripted plots, and movies
(4)
AngularTrajectories
velocity files
files
Tracker Output
ps/png
files
Run Summary
I
radius files
GW Extraction
Stitch
timeseries
Data Transform
Ej_from_mppsi4_r
*, Ej_from_Psi4r_r
(3)
Stitch
1dseries
collect
run
Psi4r with amp
and phase, grid,
Tracker etc.
1d data with
timeseries
h5movie.sh
Importing/Exporting
Raw Data
DYNAMIC REMOTE DATA ANALYSIS AND VIZ
Work supported in part by
NSF grants 1205864,
1212433, 0903973,
0941417 and 0955825.
Computations at XSEDE
TG-PHY120016 and the
Cygnus cluster at Georgia
Tech.
{magg, michael.clark, kins, plaguna}@gatech.edu, [email protected], {hongbo,mwolf}@cc.gatech.edu, [email protected], [email protected]