www.bsc.es
BSC tools hands-on session
Objectives
Copy ~nct00001/tools-material into your ${HOME}
– cp –r ~nct00001/tools-material ${HOME}
Contents of tools-material
– bin/
• Some BSC tools scripts to make your life easier
– apps/
• CGPOP mini-application for the hands-on session
– slides/
• All the slides wrt BSC tools
– packages/
• Paraver packages & tutorials
– doc/
• Extrae user guide
2
Paraver
Install the Paraver binaries in your laptop
– Binaries for linux x86 & x86-64, windows & mac
Configure the Paraver package
– Set-up the tutorials directory
• Go into Help > Tutorials, click on Preferences Window
– Edit Tutorials root to the directory that contains the tutorial
» tools-material/packages/Tutorials
3
Trace generation workflow
Application
Application
Process
Application
Process
Process
Extrae
Extrae
Extrae lib
Instrumentation
MPIT
MPIT
MPIT
Paraver
Dimemas
Clustering
Folding
…
.prv
.pcf
.row
Analysis
4
Obtaining traces (CGPOP)
Build the CGPOP application
– cd ~/tools-material/apps/CGPOP
– ./build
Generate trace-file for the CGPOP application
– cd ~/tools-material/apps/CGPOP/jobscript
– edit job.lsf
• Uncomment module load bsctools & TRACE definition
– Optionally, choose a trace-file name
– submit the job
• bsub < job.lsf
5
Inside job.lsf
#!/bin/bash
#BSUB
#BSUB
#BSUB
#BSUB
#BSUB
#BSUB
#BSUB
#BSUB
-n 48
-oo cgpop_%J.out
-eo cgpop_%J.err
-R"span[ptile=16]"
-x # Exclusive use
-J cgpop
-W 00:10
-U tools
module load netcdf/4.1.3
# To trace, uncomment the two following lines and add TRACE
# in between mpirun and the binary
module load bsctools
export TRACE=~/tools-material/apps/trace.mpi.f.sh
export TRACE_NAME=
time mpirun ${TRACE} ../mpi2s1D/cgpop.linux_icc.180x120
6
Inside trace.*.sh
#!/bin/bash
# Workaround for MN3
export TMPDIR=${TMPDIR}/extrae
mkdir –p ${TMPDIR}
Configure instrumentation
export EXTRAE_CONFIG_FILE=${HOME}/tools-material/extrae/extrae.xml
export LD_PRELOAD=${EXTRAE_HOME}/lib/libmpitrace.so
$@
Execute the binary passed as a parameter
Inject instrumentation code
7
Library alternatives
Library
libseqtrace
libmpitrace[f]1
Serial
MPI
OpenMP



libomptrace

libpttrace
libompitrace[f] 1

libptmpitrace[f] 1

1
pthread


for Fortran codes
8
Extrae XML configuration
<counters enabled="yes">
<cpu enabled="yes" starting-set-distribution="1">
<set enabled="yes" domain="all" changeat-time="100000us“>
PAPI_TOT_INS,PAPI_TOT_CYC,PAPI_L1_DCM,PAPI_L2_DCM,PAPI_L3_TCM,PAPI_FP_INS,PAPI_BR_MSP
</set>
<set enabled="yes" domain="all" changeat-time="100000us">
PAPI_TOT_INS,PAPI_TOT_CYC,PAPI_LD_INS,PAPI_SR_INS,RESOURCE_STALLS,PAPI_BR_UCN,PAPI_BR_CN,PAPI_VEC_
SP
</set>
<set enabled="yes" domain="all" changeat-time="100000us">
PAPI_TOT_INS,PAPI_TOT_CYC,RESOURCE_STALLS:LB,RESOURCE_STALLS:RS,RESOURCE_STALLS:SB,RESOURCE_S
TALLS:ROB,PAPI_VEC_DP
</set>
</cpu>
<network enabled="no" />
<resource-usage enabled="no" />
<memory-usage enabled="no" />
</counters>
GUIA DE USUARIO
explained.xml
Extrae XML configuration
<buffer enabled="yes">
<size enabled="yes">500000</size>
<circular enabled="no" />
</buffer>
<sampling enabled="no" type="default" period="50m" variability="10m" />
<merge enabled="yes"
synchronization="default"
tree-fan-out="16"
max-memory="512"
joint-states="yes"
keep-mpits="yes"
sort-addresses="yes"
overwrite="yes"
>
$TRACE_NAME$
</merge>
Extrae XML configuration
<mpi enabled="yes">
<counters enabled="yes" />
</mpi>
<openmp enabled="yes">
<locks enabled="no" />
<counters enabled="yes" />
</openmp>
<pthread enabled="no">
<locks enabled="no" />
<counters enabled="yes" />
</pthread>
<callers enabled="yes">
<mpi enabled="yes">1-3</mpi>
<sampling enabled="no">1-5</sampling>
</callers>
Using Paraver
Analyse the results with Paraver
Tutorials available
–
–
–
–
–
Tutorial 1: Introductory tutorial on the usage of the Paraver GUI
Tutorial 2: Analysis of HydroC application
Tutorial 3: Introductory tutorial on the usage of Dimemas
Tutorial 4: General Paraver & Dimemas analysis applied to WRF
Tutorial 5: General Paraver & Dimemas analysis methodology
REMEMBER! Configuration files (*.cfg) can be applied to any
trace-file.