HPC in astrophysical turbulence and magnetic fields
Sharanya Sur (IIA, Bangalore)
Sharanya Sur
Turbulence and magnetic fields
1 / 14
Outline
Today’s Roadmap
I
The magnetic Universe
I
Grand Challenge problems in astrophysical MHD
• Understanding saturation properties of Fluctuation dynamos
• Connection with observables
I
• Probing morphology of structures and role of magnetic fields
Scalability of existing MHD codes, resource requirements
I
Future Outlook
I
Collaborators : Dr. P. Bhat (Princeton University), Prof. A. Shukurov
(Newcastle University), & Prof. K. Subramanian (IUCAA)
Sharanya Sur
Turbulence and magnetic fields
2 / 14
Magnetic Universe
Magnetic Universe
I
Most of astrophysical objects in the Universe host magnetic fields
• Earth : ⇡ 1 G, with irregular reversals over 2 ⇥ 105 yr
• Sun : ⇡ 1
3 G, 11 yr solar cycle
• Galaxies : ⇡ 10µG, ordered on several kpc
galB
• Galaxy clusters : ⇡ µG strengths, on 10 kpc scales
I
Key Questions :
• How do such magnetic fields arise?
I
• What are the agents of magnetic field generation?
Turbulent Dynamo mechanism responsible for field growth and
maintenance
Sharanya Sur
Turbulence and magnetic fields
3 / 14
Magnetic Universe
Astrophysical turbulence
Astrophysical turbulence - sources and significance
I
Turbulence requires a continuous supply of energy
• Instabilities in a flow - Shear instability
shear
• Magneto-rotational instability in accretion disks
• Cosmological structure formation shocks
• Supernovae explosion in the ISM
• From subsonic (in cluster cores) to supersonic (in the ISM)
I
Significance
• Energy transfer from large scales of motion
• Jupiter’s great Red spot
• Augments molecular transport - causing mixing of the fluid
• Large/Small -scale field generation via turbulent dynamo
Sharanya Sur
Turbulence and magnetic fields
4 / 14
Magnetic Universe
Length scales in astrophysics
Length scales - a quick look
I
Di↵erent length scales in astrophysical systems
• L ) the largest scale in the problem (system size)
• l0 ) energy injection scale or forcing scale
• l⌫ ) viscous scale, l⌫ ⇠ Re
3/4
1, l⌫ ⌧ l0
l0 , For Re
• l⌘ ) magnetic di↵usivity scale
• Spitzer values for viscosity and magnetic di↵usivity yield
Pm = Rm/Re ⇠ 10 5 T 4 /n ) Pm
1
• l⌘ ⇠ Pm
1/2
l⌫ ; As Pm
1, l⌘ ⌧ l⌫
• For astrophysical systems : L
I
l0
l⌫
l⌘
For our Galaxy :
• L ⇠ 104 pc, l0 ⇠ 102 pc, Re ⇠ 105 , l⌫ ⇠ 10
• Pm ⇠
1014 ,
Sharanya Sur
l⌘ ⇠
104
2
pc
km =) extremely tiny scale !!
Turbulence and magnetic fields
5 / 14
Magnetic Universe
Need for HPC
Need for large scale HPC
I
Astrophysical system characterized by Pm
I
Turbulence varies from nearly incompressible (in cluster cores) to
highly compressible (in the ISM of galaxies)
I
Critical computational bottlenecks
• Pm
I
I
I
1 with Re, Rm
1 implies k⌘ > k⌫ ) need higher resolution
1
highPm
• Dynamo growth slow in supersonic turbulence; takes longer to
reach steady state ) computations become prohibitively expensive
Research restricted to either Pm = 1 or to about Pm ⇡ 50 with high
Rm but nearly laminar Re
Difficult to make meaningful comparisons with observations
Next generation processors and improved node interconnects crucial
• Also require faster communication network for data transfer
Sharanya Sur
Turbulence and magnetic fields
6 / 14
Magnetic Universe
Dynamo saturation
Understanding Fluctuation dynamo saturation
I
Fluctuation dynamos generic in the ISM
I
Growth time ⇠ 107 yr, much shorter than the galactic/cluster lifetime
Kinematic phase
Saturated phase
I
Crucial to probe the e↵ect of Lorentz forces on dynamo saturation
I
How coherent is the magnetic field in the saturated state?
Sharanya Sur
Turbulence and magnetic fields
7 / 14
Magnetic Universe
Dynamo saturation
Understanding Fluctuation dynamo saturation
I
Connection to observables
like Rotation Measure in galaxies and
R
clusters : RM = K L ne B · dl (Bhat & Subramanian, 2013)
2474 dynamos
P. Bhat and
Subramanian
Fluctuation
and K.
their
RM signatures
2473
There are other cases when the Gaussian PDF does not provide a
good fit to the wings of C(x). Thus, we also calculate for comparison
σ̄RM directly as the standard deviation of the set of RM(xi , yi , t)
(henceforth method II). A third method (method III) of estimating
σ̄RM , which however assumes the statistical isotropy of the random
magnetic field generated by the fluctuation dynamo, is to relate it
to the integral scale of the field. We have using equation 9 of CR09
and equation (3) above
√
√
3 Lint kf
3 Lint
=
,
(5)
σ̄RM =
2
2
2
l
where Lint is the integral scale of the random magnetic field and is
defined by
R
(2 /k)M(k, t) dk
R
.
(6)
Lint (t) =
M(k, t) dk
Note that the integral scale as defined here has the same order of magnitude as the integral scales LL and LN defined, respectively, 6.
using
the longitudinal
transverse
correlation
functions.
Figure
Comparison
of integraland
scale
for runs B,
D, F and G.
The lines
Fortheany
statistically
homogeneous,
invariant
on
upper
half of the plot
correspond toisotropic,
the velocityreflection
integral scales,
LVint ,
and
those
on
the
lower
half
correspond
to
the
magnetic
integral
scales,
&.
and divergence-free vector field, LL = 2LN = (3/8)Lint (MoninLint
The
line styles
matched
with
in Fig.power
1 to bespectra
able toM(k,
compare
the
Yaglom
1975).areThus,
given
thethose
magnetic
t), one
times
at which the
growing
to thethe
corresponding
and hence
normalizedregime
RM,
can calculate
the integral
integralscales
scalestart
Lint (t)
in
the. magnetic
One can field
also growth.
see that for a fixed kf , the magnitude and evoluσ̄RM
0.159 and 0.841,
assuming
values whereand magnetic
essentially
reflect the evolution
of the
integral
scale L7int
tion of σ̄RM
Sharanya
Sura Gaussian PDF. The RMTurbulence
fields
self-similar
fashion, maintaining
the integral
scale.
However,
by./ 14
Figure 5. The time evolution of the normalized RM (σ̄RM ) for the 5123
run I
(F), with RM = Re = 622. The crosses show the result of the direct
calculation by shooting 3N2 LOS through the simulation box. The triangles
show the result of the direct estimate of the standard deviation of RM, and
the stars the result of integrating the energy spectrum (method III).
Crucial to probe degree of coherence and synchrotron polarisation and
emissivity for high Pm with Rm, Re
1 case
Magnetic Universe
Morphology of structures
Morphology of structures in the ISM
I
ISM is both turbulent and magnetized, multiphase environment
deAvillez & Breitschwerdt, A & A, 2005
I
3D nature of these structures still an open question
• Probe this using Minkowski functionals
• Do magnetic fields play a role in regulating the morphology?
Sharanya Sur
Turbulence and magnetic fields
8 / 14
Magnetic Universe
Morphology of structures
Morphology of structures in the ISM
I
Magnetic structures in the kinematic phase ; Filamentarity increases
with Rm
Wilkin, Barenghi & Shukurov PRL, 2007
I
Cross-correlate numerical results with galactic HI observations?
Sharanya Sur
Turbulence and magnetic fields
8 / 14
Computational resources and scalability
Code scaling
A. Timings
Are existing codes scalable?
I
Scaling of parallel MHD codes like FLASH and Pencil
Strong Scaling Test - 5123
10
Time/Step [secs]
Data Points
Ideal Scaling
1
1000
10000
# of Procs
FLASH code
I
Figure 12: Scaling results on three different machines. The thin straight line denotes p
scaling.
Pencil Code
Future developments :
• Fine tune performance with next generation accelerator cards
• Modify codes to run on GPU’s? ) estimates for Speedup etc.
5123 gas + 64×106 particles
Sharanya Sur
Turbulence and magnetic fields
9 / 14
Computational resources and scalability
Code scaling
A. Timings
Are existing codes scalable?
I
Scaling of parallel MHD codes like FLASH and Pencil
Strong Scaling Test - 5123
10
Time/Step [secs]
Data Points
Ideal Scaling
1
1000
10000
# of Procs
FLASH code
I
Figure 12: Scaling results on three different machines. The thin straight line denotes p
scaling.
Pencil Code
Computational resource requirement
• Simulations are core intensive - need large-scale, central facilities
• Minimise data transfer time from remote server to host server
5123 gas + 64×106 particles
Sharanya Sur
Turbulence and magnetic fields
9 / 14
Future Outlook
Future Outlook
I
Theoretical Challenges :
• Crucial to understand Fluctuation dynamo saturation
• Current understanding limited by major computational bottlenecks
• Need highly resolved simulations at Rm, Re
and supersonic turbulence
I
1 for both subsonic
• Holds promise of opening up new avenues
Computational Challenges :
• Existing codes scalable up to 4096 cores, need to be tested on
accelerator nodes
• Code modifications to run on GPU’s to be thought through
• Data transfer rates from remote to host to be improved
• Need large-scale, nationalised supercomputing facilities
Sharanya Sur
Turbulence and magnetic fields
10 / 14
Extra Slides
Magnetic fields in nearby galaxies
I
Magnetic field in galaxies
back
Magnetic fields in M31
Magnetic fields in M51
Sharanya Sur
Turbulence and magnetic fields
11 / 14
Extra Slides
Shear instability
Shear instability
I
Instability develops at the interface of two fluids
Sharanya Sur
Turbulence and magnetic fields
back
12 / 14
Extra Slides
Shear instability
High Pm systems
I Viscous and magnetic dissipation scales in Pm
ALL-SCALE
TURBULENT DYNAMO
hout
ount
otaetc.
the
ally
978;
braall
g of
cale
and
lds.
ause
rge-
1 systems 277
back
Fig. 1.—Sketch of scale ranges and energy spectra in a large-Prm medium.
Figure from : Schekochihin et al., 2004, ApJ
fluctuating components with averages being done over scales
Sharanya Sur
Turbulence and magnetic fields
13 / 14
Extra Slides
Required resources
Computational resource requirement
I
Simulations are core intensive - large scale, central facilities necessary
I
Project timelines - categorized as Immediate and Near Future
I
Required computing resources for Immediate Projects - P1 and P2
• Fluctuation dynamo simulations at 10243 resolution
# of cores
1024
1024
1024
1024
1024
2048
2048
2048
2048
2048
2048
2048
# of hrs
48
48
48
72
48
48
72
48
48
72
96
96
Mach number
0.3
0.3
0.3
0.3
1.0
1.0
1.0
2.0
2.0
2.0
1.0
10.0
Pm
1.0
10.0
20.0
50.0
1.0
10.0
50.0
1.0
10.0
50.0
1.0
1.0
Time (kCPU hrs)
49
49
49
74
49
98
148
98
98
148
197
197
Total simulation time : 1.25 Million CPU hrs
Sharanya Sur
Turbulence and magnetic fields
14 / 14
Extra Slides
Required resources
Computational resource requirement
I
Required computing resources for Near Future Projects - P3
• Probing the role of interstellar turbulence in galaxy outflows
Type of
Run
Resolution
5122
⇥ 1024
5122 ⇥ 1024
5122 ⇥ 1024
5122 ⇥ 1024
5122 ⇥ 1024
5122 ⇥ 1024
5122 ⇥ 1024
SnTurbGrav
# of Sims: 7
StirTurbSnGrav
# of Sims: 12
Grand Total # of Sims: 19
Sharanya Sur
5122 ⇥ 1024
5122 ⇥ 1024
5122 ⇥ 1024
5122 ⇥ 1024
5122 ⇥ 1024
5122 ⇥ 1024
5122 ⇥ 1024
5122 ⇥ 1024
5122 ⇥ 1024
5122 ⇥ 1024
5122 ⇥ 1024
5122 ⇥ 1024
⌃g
[M pc
250
250
150
150
150
50
50
150
150
250
250
150
150
250
250
150
150
150
150
2]
 1
[Myr]
–
–
–
–
–
–
–
6.5
6.5
10
10
10
10
20
20
20
20
30
30
[yr
⌃˙ SN
1 kpc 2 ]
0.002
0.001
0.001
0.0006
0.0003
0.0003
0.0001
0.0027
0.0009
0.003
0.001
0.0018
0.0006
0.0015
0.0005
0.0009
0.0003
0.0006
0.0002
Turbulence and magnetic fields
Time
[kCPU hrs]
200
200
200
200
200
200
200
Total: 1400
200
200
200
200
200
200
200
200
200
200
200
200
Total: 2400
Total: 3.8 million CPU hrs
14 / 14