FLY vers. 3.1
User Guide
U. Becciani V. Antonuccio-DeLogu and M. Comparato
Email: [email protected], [email protected]
Introduction
FLY is a tree N-body parallel code for three-dimensional selfgravitating collisionless systems evolution. It is based on the
tree Barnes-Hut algorithm and periodical boundary conditions are
implemented by means of the Ewald summation technique.
Each particle evolves following the laws of Newtonian physics, and
the differential equations are integrated using the numerical
Leapfrog integration scheme. All the particles, having the same
mass, must be included in a cubic box having fixed dimensions.
The supported cosmological models are: Standard Cold Dark Matter
(Omega_tot=1), Lambda Models (with Cold Dark Matter), Open Models
(Omega_tot<1).
The code was originally developed on CRAY T3E system using the
logically SHared MEMory access routines (SHMEM). This release uses
the MPI-2 standard.
This is a guide to allow the user to install FLY and create the
executables to run a simulation. For a detailed discussion on the
FLY features, please refer to the FLY Reference Guide, or send an
e-mail message to the authors.
The system evolution, generally starts from a time-step equal to 0
and executes up to a final time-step as indicated in the MAX STEP
parameter in the FLY_3.1/bin/dyn_pars file or at the end of the
programmed output (in redshift values) FLY_3.1/bin/out_32.tab. The
evolution is done by executing several steps (hereafter FLY job):
each of them
evolving the system for several
time-steps, as
indicated in the NUM. STEP parameter in FLY_3.1/bin/stat_pars
file.
The
input file of each FLY job, will be /tmp/FLY/posvel_0
(IBOD_FILE + CURR.STEP).
CURR.STEP (in the FLY_3.1/bin/dyn_pars
file) gives the current time-step starting point of the FLY job.
The FLY job will produce
the intermediate output file
/tmp/FLY/posvel_10 (IBOD_FILE + (CURR.STEP+NUM STEP)), and update
the CURR.STEP to this new value (CURR.STEP=CURR.STEP+NUM STEP),
allowing the exact execution of the next FLY job.
FLY produces also a little header file for each intermediate file,
like /tmp/FLY/posvel_10_head containing the current redshift, the
current time-step and the DT used in the last executed time-step.
This information can be reused to reset the dyn_pars file to restart the simulation from a previous time-step.
NOTE: To run this version of FLY all the working directories must
be shared among all the processors that execute FLY.
1) Download FLY
FLY is freely available at the http://www.ct.astro.it/fly/
address. Download the FLY.tar.gz and copy the FLY.tar.gz file in
a directory
(i.e. your HOME directory) in the system where you
will run. Make you sure that a subdirectory with the same name
FLY_3.1 is not present, to avoid conflicts.
1.1) Unpacking FLY
Give the following commands: gunzip FLY.tar.gz
FLY.tar
The tar command
and
tar -xvf
will create the following directories
./FLY_3.1
(FLY installation directory)
./FLY_3.1/bin
files)
(Executable
./FLY_3.1/src
(Source code and assistant program)
./FLY_3.1/src/tcl
(TCL program: graphic interface)
./FLY_3.1/Testcase
(Testcase files)
./FLY_3.1/Ewald
(Code for Ewald table generation)
programs
and
input
parameters
./FLY_3.1/FLY_ug.doc (This file)
./FLY_3.1/FLY_ref_guide.ps (Reference Guide version 2.1.)
The Reference Guide version 2.1 can be used, with minor changes
reported in this document, to setup all the parameters to run a
simulation.
2) Setting the box for a simulation
Fix the box dimension (BOX SIZE) in arbitrary unit (generally in
Mpc).
2
3) Setting the number of particles of the simulation
Fix the number of particles (N. BODIES) you like to simulate. This
must be a multiple of the processor number you will use.
4) Creating the initial condition file
Create a file containing the initial positions and velocities of
all the particles. FLY does not contain the program for the
initial condition
generation, but it is possible to use any
program the user likes (e.g. COSMICS by E. Bertschinger), provided
the input file as described below. The input file must contain
all the positions and after all the velocities of all the
particles in double C format (8 byte for each component).
FLY uses a C routine to read and write the checkpoint particles
data file. The initial file input must be produced using the C
language, like in the following example.
void write_b_bin(pos,vel,cfilename,nlong)
FILE *f_b_bin;
double *pos,*vel;
signed int *nlong;
char *cfilename;
{
f_b_bin = fopen(cfilename,"r+");
fwrite(pos,sizeof(double),*nlong,f_b_bin);
fwrite(vel,sizeof(double),*nlong,f_b_bin);
fclose(f_b_bin);
}
where in C language, pos and vel must be defined with the
following structure pos(1:nbodies,3) and vel(1:nbodies,3) in order
to maintain the same memory map distribution of the Fortran
language.
Anyhow
the
Fortran
data
format
pos(3,1:nbodies)
vel(3,1:nbodies) must be the input data format of FLY.
and
It can be often convenient to create a
/tmp/FLY directory
(typically a temporary massive area) to put the initial file
posvel_0.
The initial condition file must be produced without control
characters: it must have an unblocked file structure with "stdio"
style buffering, compatible with the C fwrite and fread functions.
3
Otherwise an ASCII input file can be given with the following data
format:
REAL(KIND=8),
DIMENSION(3,1:nbodies) :: pos,vel
....generate the
pos and vel arrays...
OPEN(UNIT=12,FILE='posvel_0',STATUS='NEW')
DO i=1,nbodies
write(12,1000) pos(1:3,i)
ENDDO
DO i=1,nbodies
write(12,1000) vel(1:3,i)
ENDDO
CLOSE(UNIT=12)
1000 FORMAT(3(1X,F20.10))
5) Creating the input parameter files
You can use the graphical interface to create all input parameters
files, if the system you use has the Tcl/Tk installed.
prompt> cd ./FLY_3.1/src/tcl
prompt> wish fly_3.1.tcl
The graphical interface will guide the user to create all the
following files.
Otherwise you can use the line command assistant program, on a
Linux System, to create the input parameters files. The assistant
program can be created, compiling the prepare.F90 file in the
source code directory.
prompt> cd ./FLY_3.1/src
prompt> ./assistant_linux
The assistant program is structured with several sections and,
where applicable, it gives a default value.
The user must fix a Working Directory where all the input
parameter files will be stored and an Executable Directory. The
assistant can not create these directories but the user must
create them before starting the assistant.
4
The following files (5.1 – 5.6) must be created in the working
directory. We will use the assistant program to discuss the
content of each file.
At the beginning, the assistant asks to set the Working Directory
where all the following input parameter files will be created.
This directory must be created. The default value is indicated in
square brackets
Setting the Working Directory
Please set the working directory. The default value is ../bin :
Working Directory [../bin/] :
The user must indicate the absolute path of the Working Directory.
The Testcase directory included in the FLY package, gives an
example of all the input files.
5.1) Creating the ./FLY_3.1/bin/fly_fnames file
This file must have the name above indicated. It contains the
filenames (including the path) of all the files that will be
created. If this file does not exist, the assistant program will
create it, and asks for the filenames: the default filenames will
be as follow. If this file already exists, the assistant will use
this filenames as default values.
Starting generation of ../bin/fly_fnames file...
Do you want to create this file (Y/N) : y
All NOT-ABSOLUTE filename paths will be ../bin/ :
Please give stat_pars filename [stat_pars] :
Please give dyn_pars filename [dyn_pars] :
Please give ew_grid filename [ew_grid] :
Please give ew_tab filename [ew_tab] :
Please give out32.tab filename [out32.tab] :
Please give ql.tab filename [ql.tab] :
At the end the file will be created, containing the filenames that
the user gives.
5
STAT_PARS
DYN_PARS
EW_GRID
EW_TAB
OUT32_TAB
QL_TAB
=stat_pars
=dyn_pars
=ew_grid
=ew_tab
=out32.tab
=ql.tab
In the following we will use the default filenames.
5.2) Creating the stat_pars file
This file has the following structure:
HEADER
=try_sim
NUM. STEP=
10
MAX_TIME = 10800
DELTA T. =
0.00450
DT VAR. =T
OPEN PAR.= 0.80
SOFT PAR.=
0.01000
QUADRUP. =T
OMEGA.CDM=
0.7000
OMEGA.HDM=
0. 0
OMEGA.LAM=
0.3000
HUB_CONST= 0.650000
N. BODIES=
2097152
MASS BODY=
0.00120
IBOD_FILE=/tmp/FLY/posvel_
IBOD_TYPE=B
OBOD_FILE=/tmp/FLY/out_
OBOD_TYPE=B
QLK_FILE =/tmp/FLY/qlk_
This file contains some information on the simulations that must
be never changed during the system evolution. The assistant asks
for the following:
5.2.1)
Identification header of simulation (max 40 char)
:
Type an HEADER string to identify the simulation.
5.2.2) Number of step of each FLY job [10]:
It is the number of time-steps that each
6
FLY job will perform.
5.2.3) Maximum single CPU time (sec) for each run [10800] :
It is the
be lower
submitted
parameter
a single
reference
maximum CPU time allowed for each job. This value must
or equal to the single CPU time as specified for the
job. If this value is greater than 0 the NUM. STEP
is not considered and the number of time-step cycles for
job, is automatically computed by FLY using this
number.
5.2.4) Delta T integrator time-step
:
It is the DELTA T. integrator time-step used in the system
evolution (format F20.15). For cosmological simulations a possible
choice can be found in Efstathiou et al. (ApJS, vol. 57, p. 241,
1985). Give here a safety value.
5.2.5) Use FLY Variable dt ? (T/F) [T] :
FLY also uses an adaptive Delta T integrator determination
automatically computed in dt_comp.F subroutine.
As a default FLY adopt a criterion based on the evaluation of the
maximum acceleration among all the particles:
   soft 
p  

 max i 1, N ai 
where soft, maxi=1,N|ai| are the
Users that want to use this adaptive Delta T integrator must
answer T to this question. If the user answers F to this question,
FLY will use the DELTA T. integrator the user given in the
previous question.
5.2.6)
Opening Angle Parameter
(> 0.1) [0.8]
:
The OPEN PAR (format F4.2) is used to open or close the cells
during the tree walking phase to build the interaction list for
each particle.
5.2.7) Softening length
:
The SOFT PAR value (format F20.15) is crucial in determining the
amount of "hardening" of the gravitational interaction, ultimately
the amount of small-scale substructure which is not destroyed
during the gravitational evolution of the system.
5.2.8) Use quadrupole momentum ? (T/F) [T]
7
:
This is the QUADRUP. value. If QUADRUP.=T (true) the quadrupole
momentum is used in the particle-cell
interaction to compute the force components.
5.2.9) Omega CDM parameter
:
It is the OMEGA.CDM value (format F20.15).
5.2.10) Omega HDM parameter
:
It is the OMEGA.HDM value (format F20.15).
5.2.11) Omega LAM parameter
:
It is the OMEGA.LAM value (format F20.15).
Note: the equations of motion in the present version can be solved
only for globally CLOSED models (OMEGA.CDM+OMEGA.HDM+OMEGA.LAM=1).
5.2.12) Hubble constant [0.65] :
It is the Hubble constant value (format F10.6).
5.2.13) Number of processors of the parallel run [8]:
It must be the number of processor you want to use during the FLY
simulation. This value must be greater than 0. In order to obtain
the best performances we recommend using a number of processors
equal to a power of two.
5.2.14) Number of particles of the simulation:
Give the N. BODIES value.
5.2.15) Mass of each particle:
Give the MASS BODY value (format F20.15).
5.2.16) Root-name of the checkpoint file of FLY (max 256 char)
[/tmp/FLY/posvel_] :
8
This is the root-name of the filename (IBOD_FILE) including the
path of the initial condition (see sect. 4). The root-name must
not contain the final number (like 0) that is automatically
appended by FLY.
5.2.17) Checkpoint file type of FLY:
:
B (binary) or
A (ASCII) [B]
The user must specify B for binary or A for Ascii data format for
the IBOD_FILE data type.
5.2.18) Root-name of the Programmed Output file of FLY (max 256
char) [/tmp/FLY/out_] :
This is the filename (OBOD_FILE) including the path of the files
containing the programmed output position and velocity (max 256
char).
It must not contain the final number
that is
automatically appended by FLY code. These output files are the
scientific data output created by FLY. The redshift value when
these
output
files
will
be
created,
are
listed
in
./FLY_3.1/bin/out32.tab file. The
user must use a different
filenames than the previous one (5.2.16 par.).
5.2.19) Pogrammed Output file type of FLY:
(ASCII) [B] :
B (binary) or
A
The user must specify B for binary or A for Ascii data format for
the OBOD_FILE data type.
5.2.19.1) Binary output data file
The data output is generated in binary format with a C procedure,
as an unblocked file structure with "stdio" style.
void c_write_f32(pos,vel, cfilename, nlong)
double *pos,*vel;
signed int *nlong;
char *cfilename;
FILE *f32;
{
f32 = fopen(cfilename,"wb")
fwrite(pos,sizeof(double),*nlong,f32);
fwrite(vel,sizeof(double),*nlong,f32);
fclose(f32);
}
9
5.2.19.2) Ascii output data file
FLY can generate an ASCII output file
as follows:
REAL(KIND=8), DIMENSION(3,1:nbodies)
CHARACTER(LEN=132)
....generate the
/tmp/FLY/out_10)
:: pos,vel
:: fileout
fileout name as root_name//time_step (i.e.
OPEN(UNIT=32,FILE=fileout,STATUS='NEW')
DO i=1,nbodies
write(32,1000) pos(1:3,i)
ENDDO
DO i=1,nbodies
write(32,1000) vel(1:3,i)
ENDDO
CLOSE(UNIT=32)
1000 FORMAT(3(1X,F20.10))
5.2.20) Root-name of the ASCII quick-look file of FLY (max 256
char) [/tmp/FLY/qlk_] :
Give the root-name of the file containing the initial position and
velocity (max 256 char). The root-name must not contain the final
number that is automatically appended by FLY code (the same
number, a redshift value, of the files in 5.2.18 par.). We must
use a different filename than the previous files (5.2.16
and
5.2.18 par.).
These output files are not more automatically produced by FLY. The
ASCII files can be created using random particles positions and
velocity, using an external program. The randomized table
./FLY_3.1/bin/ql.tab
is
automatically
created
by
the
FLY
assistant.
The redshift values, when these output files will be created, are
listed in ./FLY_3.1/bin/out32.tab file.
During the system simulation an header file (suffix _hea, i.e.
/tmp/FLY/qlk_hea) will be automatically created.
The qlk_hea file has a format like the following:
header_line
10
ASCII
time
262144
17 /gpfs/temp/ube/data262k/qlk_60.0000
10 /gpfs/temp/ube/data262k/qlk_50.0000
68 /gpfs/temp/ube/data262k/qlk_20.0000
The value 262144 is the number of data points included in the
quick-look file, the filename suffix ( _60.0000) is the
corresponding redshift value when the output is produced, and the
number before the filenames (17, 10, 68) is the number of timestep that FLY has run from the previous step. The qlk files must
be produced by an external user program.
5.3) Creating the dyn_pars file
This file contains information on the simulations that can be
changed during the system evolution, and has the following
structure:
CURR.REDS= 50.00
CURR.STEP=
0
MAX STEP= 100
LIV. GROU=
9
BODY GRO.= 16
BAL. PAR = 0.95
GR. PAR = 0.50
SORT_LEV.=
3
BOX SIZE = 50.0000
X MIN VER= 0.000000
Y MIN VER= 0.000000
Z MIN VER= 0.000000
This file is re-read at the begin of each time-step during a FLY
job. This file is automatically updated by FLY at the end of each
FLY job: the CURR. STEP and the CURR.TIME are updated at the final
time-step that the FLY job reaches.
5.3.1)
Redshift initial value :
Give the CURR.TIME: in the present cosmological version this is
the redshift parameter at the current time-step.
5.3.2) Initial time-step value (0 - 999) [0]:
Give the initial time-step value CURR.STEP. We strongly recommend
starting from 0 as initial time step. In this case FLY
automatically checks for all the particle positions and re-arrange
these inside the cubic region.
11
If you give a different time-step value you are responsible to
verify that all the particle positions are inside the box region,
otherwise an error will occur at the run-time.
5.3.3)
Final time-step value (1 - 999) :
Give the final time-step value MAX STEP. When a FLY job reaches
this value the simulation is stopped. FLY stops the simulation
when MAX
STEP is reached or the final redshift listed in
out_32.tab is reached (see 5.4 par.)
5.3.4)
Critical level for grouping value
[9] :
Give the tree level threshold LIV. GROU where the grouping cell
can be started to form, being LIV. GROU=0 the root level. We
recommend using a six level for a 2-million-particles simulation
up to 9 levels for 16-million-particles.
5.3.5)
Particles max value in grouping cells
[16] :
Give the BODY GROU parameter value. It is the maximum number of
particles that a grouping cell can contain. We recommend to use a
value from 8 up to 32.
5.3.6) Load Balance Parameter (0.01 - 1.0) [0.95]
:
Give the BAL. PAR. value. It is used to balance the load in the
processor. This value is re-computed during the run, to allow a
perfectly load balancement. This number is the percentage of the
local particles that must be computed by the local processor. The
remaining portion is computed by the first available processor. We
recommend to use the default value 0.95.
5.3.7) Group Balance Parameter (0.01 - 1.0) [0.50] :
Give the GR. PAR. value. It is used to balance the load of
particles groups, in the processor. This value is re-computed
during the run, to allow a perfectly load balancement. This number
is the percentage of the local group cells that must be computed
by the local processor. The remaining portion is computed by the
first available processor. We recommend to use the default value
0.95.
12
5.3.8)
Size of the box value (BOX SIZE) : 50.0
Give the box dimension BOX SIZE, in arbitrary unit (data format
F9.4) (generally in Mpc).
5.3.9) X-Y-Z coordinates of the position of the lower-left corner
in the box.
Position X :
Position Y :
Position Z :
It gives the coordinates of the
lower-left corner of the cubic
region, in arbitrary units (data format F10.6).
5.3.10) NOTES
The parameters
GROUP FL. and BOX COMP. are for future usages.
The parameter SORT_LEV= L is automatically computed by the
assistant. The FLY_sort utility builds the tree up to this level,
where there are 2**3 x L cells (L=0 being the level of the tree
root). The FLY_sort utility produces a sorted file containing the
fields of position and velocity, so that particles with a near tag
number are also near the physical space. Data organized in this
way are the input of the FLY simulation code. Each processor has
the same number of particles, Nbodies/N_PEs, near the space, the
reserved and/or local memory, being
N_PEs the number of
processors.
5.4) Creating the FLY out32.tab table.
This table contains the redshift values when the user wants to
have an output file for data analysis. It is read at the beginning
of each FLY job that produces the output files and the quick-look
files, with the names listed in the
OBOD_FILE and QLK_FILE
parameters, and a suffix given by the current redshift.
5.4.1) How many FLY programmed output do you need (<= 100) ? :
The assistant asks for the number of programmed output the user
want to have. Give the N number of programmed data output.
5.4.1.1) Please give
the N values of the FLY programmed output.
If N=5 the assistant program asks for 5 time-step numbers:
13
Programmed
Programmed
Programmed
Programmed
Programmed
output
output
output
output
output
number
number
number
number
number
1
2
3
4
5
[
[
[
[
[
0.0]
0.0]
0.0]
0.0]
0.0]
:
:
:
:
:
60.0
50.0
20.0
10.0
0.1
FLY will generate the output files:
/tmp/FLY/out_60.0000,..../tmp/FLY/out_0.1000
and
/tmp/FLY/qlk_60.0000,..../tmp/FLY/qlk_0.1000
(see the 5.2.18 and 5.2.20 paragraphs).
5.5) Creating the FLY quick-look table ql.tab file
The assistant can create the table ql.tab that is a random number
sequence N_ran of values, included in the range from 1 to nbodies.
It will be used to extract N_ran bodies in the simulation to form
a quick-look file.
The ql.tab file must contain not more than 262144 random different
numbers. Extra values are ignored by FLY.
5.6) Creating the Ewald grid tables ew_grid and ew_tab files
These tables are used for the Boundary Periodical conditions.
These are implemented by means of the Ewald summation, without the
use of the Fast Fourier Transform techniques. The README file, in
the Ewald directory, contains short notes for tables generation.
6) The executables FLY and the FLY_sort utility.
The assistant can help the user to create the executable FLY
program. The assistant will create the module file, customized as
the user requirements. It will create the makefiles to create the
executables and will run the make utility.
6.1) Preparing the Module file ./FLY_3.1/src/fly_h.F
The assistant asks for:
6.1.1) Available RAM memory for each processor (MBytes) [1024] :
14
This must be the value (in MBytes) of the total RAM available to
each processor during the run. It is important to give the correct
value of the global RAM of each processor. The more memory the
processor can have, and the more will be the FLY performances.
6.1.2) Maximum Interaction List length [1000] :
This value is used for auxiliary array dimensions. This is the
maximum length for an interaction list formed during the tree walk
procedure. This number depends on the box size, the nbodies, the
opening parameter and the clustered (or uniform) conditions will
evolve. It is very important, for the FLY performance, to give a
good safety value but close to the expected maximum length of the
interaction list.
Fixing the opening parameter = 0.8 in a box size of 50.0 Mpc this
value is approximatively
2-million-particle simulation (uniform)
= 394
16-million-particle simulation (clustered) = 636
In this case 1000 is a good safety value for the required
Interaction List length.
maximum
6.2 mkfl_FLY and mkfl_FLY_sort makefiles and executables.
The assistant can create the makefiles and run the make utility to
create the executables. It will ask for:
6.2.1) Please give the FLY executable filename [FLY] :
The user must indicate the executable filename: the defalut name
is FLY in the executable directory, but the user can specify a
different absolute path. The user can specify in this answer the
Executable Directory (the defaut being ./FLY_3.1./bin), where the
executable file will be created.
6.2.2) Do you want a statistical code performance output? (Not
recommended!) (Y/N) [N] :
The user must indicate if he wants to have a statistical report.
This statistical report concerns ONLY the code performance, and it
must not be used during a production run.
15
6.3) Compiler options.
The user must set the compilers and the Options. The following
options are given:



-DSTA : is used to obtain a statistical report on the number
of remote GET and PUT calls. can be used to test the fly
efficiency and must not be used during the simulation run;
-DSORT : generate the FLY_sort utility.
-DFLASH : generate code for Paramesh interaction
6.4) Compilation
The assistant produces two files mkfl_FLY and mkfl_FLY_sort in the
src directory.
The user can use the make utility to create the executables. The
assistant automatically compile the code. It starts the following:
make
make
make
make
–f
–f
–f
–f
mkfl_FLY clean
mkfl_FLY (this creates the FLY_mpi executable)
mkfl_FLY_sort clean
mkfl_FLY_sort (this creates the FLY_mpi_sort utility)
NOTE: Dependig on the version of mpich2 installed in the system, a
compilation ERROR can occur on the parameter MPI_ADDERSS_KIND. In
this case you must modify the mpif.h file (default location in
/usr/local/include) and set the MPI_ADDRESS_KIND=4 .
7) Use the assistant to create the FLY job script
The user must create the script for a FLY job.
The FLY executable creates a file called ../bin/ckstop when it
regularly stops, and the end of the simulation is not yet reached.
It can be
checked before automatically starting a new FLY job
(see par. 8).
8) Running a simulation with FLY.
The following procedure is recommended to run a simulation using
several FLY jobs:
1)
Create
the
input
file
at
the
time-step
0
/tmp/FLY/posvel_0) and start the FLY_sort utility. It will
create /tmp/FLY/posvel_sort_0 : a new sorted input file.
Replace this new file with /tmp/FLY/posvel_0.
2) Start the first FLY job
16
(i.e.
The user must create a script command
prompt/FLY_3.1/job> bsub
< ./FLY_job.lsf (if you uses lsf)
The script must checks for the file ckstop:
if ( test
then
bsub
-f ckstop )
< ./FLY_job.lsf
else
echo "ckstop: Stop simulation."
fi
If the file ckstop is founded the simulation can be continued up
to the programmed end. The FLY job script submits the new FLY job.
If the file ckstop is not founded the simulation has reached the
programmed end. Otherwise an error may be occurred during the
previous FLY job.
9) Testcase
The user can adopt the testcase reported in the FLY_3.1/Testcase/
directory. Create the FLY executable (as above described) and
change to this subdirectory to run a test simulation with 1024
particles and 10 time-step. Use the following files in the
directory: fly_fnames, stat_pars,
dyn_pars, ew_grid, ew_tab,
ql.dat, out32.tab, the input file 1024_asc_10.
In the following there is a step by step procedure to execute the
testcase.
9.1)Start the assistant
We suppose that FLY is installed in the /home/test directory.
Start the assistant_linux in the src directory and gives
answers, boldface characters, to the following questions.
the
/home/test/FLY_3.1/src> ./assistant_linux
Welcome to FLY assistant!
Answer all the questions following the instructions
show.
………..
17
that FLY
assistant will
9.2) Setting the ../Testcase directory
In this phase we must not create files that are already in the
directory ../Testcase (fly_fnames, stat_pars, dyn_pars, out32.tab,
ql.tab, ew_tab and ew_grid)
GENERAL SETTING
Please set the working directory. The default value is ../bin :
Working Direct [../bin/] : ../Testcase
Confirm this ../Testcase/ Working Directory (Y/N) : Y
Working direct: ../Testcase/
mkdir: cannot create directory `../Testcase/': File exists
Starting generation of ../Testcase/fly_fnames
file...
Do you want to create this file (Y/N) : N
WARNING: The../Testcase/fly_fnames will contain the default names (Y/N)[Y] :
Please accept the default input data.
---------------------------fly_fnames INPUT DATA
---------------------------STAT_PARS =stat_pars
DYN_PARS
=dyn_pars
EW_GRID
=ew_grid
EW_TAB
=ew_tab
OUT32_TAB =out32.tab
QL_TAB
=ql.tab
---------------------------fly_fnames
---------------------------Confirm these parameter values (Y/N) : Y
../Testcase/fly_fnames file...
created!
Starting generation of stat_pars= ../Testcase/stat_pars file
Do you want to create this file (Y/N) : N
WARNING: You must give the stat_pars file at run-time
Starting generation of dyn_pars= ../Testcase/dyn_pars file
Do you want to create this file (Y/N) : N
WARNING: You must give the dyn_pars file at run-time
Starting generation of the FLY programmed red-shift output
../Testcase/out32.tab table
Do you want to create this file (Y/N) : N
WARNING: You must give the out32.tab file at run-time
out32.tab =
Starting generation of FLY quick look table ql.tab = ../Testcase/ql.tab
file
Do you want to create this table? (Y/N) N
Starting
generation of
Ewald grid tables
18
ew_grid and ew_tab
...
Y
Do you want to create these files (Y/N)
:
N
WARNING: You must give the ew_grid and ew_tab files, at run-time
9.3) Creation of fly_h.F90 module
In this phase we must create fly_h.F file to create
executable. We suppose to use two processes to execute
Testcase, and we fix 128 MByte RAM for each process.
Starting generation of the Module file
Do you want to create this file (Y/N)
:
../src/fly_h.F ...
Y
Number of processors of the parallel run :
Please give the number of
the
../Testcase/stat_pars file.
Number of particles: 1024
2
particles as reported in the N. BODIES field
Available RAM memory for each processor (MBytes) [2048]
Maximum Interaction List length [1000] :
the
the
of
128
1000
Please check the input data you given.
---------------------------fly_h.F Module File INPUT DATA
---------------------------Number of processors=
2
Number of Bodies=
1024
RAM per processor (MBytes)=
Maximum Interaction List length=
---------------------------fly_h.F Module File
---------------------------Confirm these parameter values (Y/N)
128
1000
:
Y
The fly_h.F MODULE file ... created!
9.4) Creation of FLY_mpi in the ../Testcase directory
In this phase we must create the makefile mkfl_FLY to create the
executable. We suppose to use Intel Fortran compiler (ifc). If you
use another fortran compiler you must give the right answers. We
answer with return key to accept the default values. At the end
the compilation starts and the FLY_mpi executable is created in
the ../Testcase directory. It is important to read the 6.4
paragraph to check the compilation phase.
19
Starting generation of mkfl_FLY and mkfl_FLY_sort makefiles and
FLY_sort executables
Do you want to create these files (Y/N)
:
FLY and
Y
NOT-ABSOLUTE filename path will be ../Testcase/
Please give the FLY executable filename [../Testcase/FLY_mpi] :
Do you want to have a statistical code performance output? (Not recommended!)
(Y/N) [N] :
Fortran compiler to be adopted [ifc] :
Fortran compiler options [-quiet -O0 -r8 -w95 -Vaxlib -I/usr/local/include] :
C compiler to be adopted [gcc] :
C compiler options [] :
Library path [-L/usr/local/lib -lmpich -L.] :
Please check the input data you given.
---------------------------mkfl_FLY and mkfl_FLY_sort INPUT DATA
---------------------------Executable FLY filename
=../Testcase/FLY_mpi
Executable FLY_sort filenam
=../Testcase/FLY_mpi_sort
Statistical code performances = N
Fortran Compiler
= ifc
Fortran Compiler options
=
-quiet -O0 -r8 -w95 -Vaxlib -I/usr/local/include
C Compiler
= gcc
C Compiler options
=
Library path
= -L/usr/local/lib -lmpich -L.
---------------------------fly_h.F Module File
---------------------------Confirm these parameter values (Y/N)
:
Y
rm -f fly_h.o fly.o open_mpi_win.o close_mpi_win.o null.o error.o sys_init.o
read_params.o read_redsh.o read_b_asc.o read_b_bin.o ini_bal.o init_pars.o
init_ew.o init_pos.o reset_pos.o inpar_dyn.o step.o step_force.o tree_build.o
tree_sort.o ch_all.o ch_dall.o tree_gen.o find_group.o cell_prop.o acc_comp.o
ilist_group.o force_group.o force.o ilist.o upd_pos.o upd_vel.o wr_native.o
write_b_asc.o write_b_bin.o out_32.o leapf_corr.o io.o ch_linee.o dt_comp.o
flashBlockToFlyCell.o findParticlesInCell.o
ifc -quiet -O0 -r8 -w95 -Vaxlib -I/usr/local/include -c fly_h.F90
ifc -quiet -O0 -r8 -w95 -Vaxlib -I/usr/local/include -c fly.F90
ifc -quiet -O0 -r8 -w95 -Vaxlib -I/usr/local/include -c open_mpi_win.F90
………..
ifc -quiet -O0 -r8 -w95 -Vaxlib -I/usr/local/include -c leapf_corr.F90
gcc -c io.c
ifc -quiet -O0 -r8 -w95 -Vaxlib -I/usr/local/include -c ch_linee.F90
………
ifc -o ../Testcase/FLY_mpi fly_h.o fly.o open_mpi_win.o close_mpi_win.o null.o
error.o
sys_init.o
read_params.o
read_redsh.o
read_b_asc.o
read_b_bin.o
ini_bal.o init_pars.o init_ew.o init_pos.o reset_pos.o inpar_dyn.o step.o
step_force.o tree_build.o tree_sort.o ch_all.o ch_dall.o tree_gen.o find_group.o
cell_prop.o acc_comp.o ilist_group.o force_group.o force.o ilist.o upd_pos.o
upd_vel.o wr_native.o write_b_asc.o write_b_bin.o out_32.o leapf_corr.o io.o
20
ch_linee.o
dt_comp.o
flashBlockToFlyCell.o
L/usr/local/lib -lmpich -L.
…….
findParticlesInCell.o
-
To start the run, you must cd in the working directory.
If need, start mpd on the system:
mpdboot -n N -f my_mpd_hostfile
where N is the number of different hosts listed in the file my_mpd_hostfile
and execute the run:
mpiexec -n
2
fly_executable > filename_out_log
or submit it in a queue system
FLY assistant has completed. Have yourself a good FLY .... in the Universe!
9.5) Start mpd daemon
This procedure is to start mpd on a single PC.
If mpd daemon is not already started, gives the command
> mpd &
and the command
> mpdtrace
to verify
that the mpd daemon is up and running. The output
should be the hostname of the machine you are running on
If you want to use more than one PC, you must start mpd on all
systems, giving the command
> mpdboot –n N –f mpd.hosts
where N is the number of hosts and mpd.hosts is a file listing the
hosts. See the MPICH2 documentation for more details
9.6) Start the FLY_mpi executable
Now we must
executable
change
in
the
Testcase
directory
to
start
the
/home/test/FLY_3.1/Testcase> mpiexec –n 2 ./FLY_mpi > FLY_out
The FLY_out file will contain a log file. The run will produce the
following files:
1024_asc_20,
1024_qlk_head.
1024_asc_20_head,
21
1024_out_asc_45.0000,
All the above files can be compared with the corresponding files
having .ref extension, in the Testcase directory. The obtained
result must be similar to the reference files, considering
different arithmetic for different machines.
Note:
1) The run updates the dyn_pars file. If you want to restart the
run to produce the same results, you must copy dyn_pars.ori file
in dyn_pars.
/home/test/FLY_3.1/Testcase> cp dyn_pars.ori dyn_pars
2) If you don’t want to create the executable file FLY_mpi you can
use the FLY_mpi_2pe.ref executable already present in the Testcase
directory. It was created for Linux systems. You can give the
mpiexec command to start the execution:
/home/test/FLY_3.1/Testcase> mpiexec –n 2 ./FLY_mpi_2pe.ref > FLY_out
10) Tcl/Tk Graphic interface
The ./FLY_3.1/src/tcl/ directory contains a FLY_3.1.tcl file. In
all the systems where Tcl/Tk is installed, it can be used to
create all the input parameters file above described.
The user can choose a working directory where all the files will
be stored and an executable directory where the executable FLY
program will be stored.
All the files have the same fields as above described; please
refer to the above paragraph for more details.
To use the graphical interface it is necessary that TCL/TK is
installed in the system. Ask the system administrator for
information
and
details.
Set
the
current
directory
in
./FLY_3.1/src/tcl/ and give the command wish fly_3.1.tcl to start.
All the main windows have a help button, which gives a short
description of all the fields in the window.
The fly_tcl.ini file will be automatically created, and will
contain some default information that can be used for the next
run.
10.1) The main graphical FLY interface
After the setting of the platform the main interface will be shown
(Fig.1). The user can set the Working Directory and the Executable
Directory using the absolute path. This program will create (when
possible) the not existing directories. If the TCL version
installed is greater than 7.4, the Browse buttons are active: the
22
user must select a directory and click the Load button to accept
the directory (Fig. 3).
The user sets the Working Directory and clicks on Load button of
the
fly_fnames DATA FILE section Fig. 1. If the fly_fnames file
exists, the fields will be loaded. The user can click the Default
button to load the default names of the input parameters files.
The setting button on the FLY interface will allow the user to set
all the input parameters above mentioned.
Fig . 1
23
Fig. 2
10.2) Creating the stat_pars file
Select the Define STATIC Parameters from the Setting Menu
(Fig.
1). The window of Fig. 3 will appear. The user can load the STATIC
file specified in Fig. 1 and set or update all fields. To save the
file click the Save button.
Fig. 3
10.3) Creating the dyn_pars file
Select the Define DYNAMIC Parameters from the Setting Menu (Fig.
1). The window of Fig. 4 will appear. The user can load the
DYNAMIC file specified in Fig. 1 and set or update all fields. To
save the file click the Save button.
24
Fig. 4
10.4) Creating the out_32.tab file
Select the Define OUTPUT Redshift from the Setting Menu (Fig. 1).
The window of Fig. 5 will appear. The user can load the out_32.tab
file specified in Fig. 1 and set or update all fields. To save the
file click the Save button. The Next button allows to add new
input fields for the output redshift file
25
Fig. 5
10.5) Creating the ql.tab file
Select the Define ql_tab from the Setting Menu
(Fig. 1). The
window of Fig. 6 will appear. The user must specify the Number of
bodies and the Random numbers as specified in par. 5.5
Fig. 6
26
10.6) Creating the Ewald grid tables ew_grid and ew_tab files
Select the Ewald Generate Menu
(Fig. 1). The window of Fig. 7
will appear. The user must specify the Box Size as specified in
the DYNAMIC file (par. 5.3.8)
Fig. 7
10.7) Preparing the Module file ./FLY_3.1/src/fly_h.F
Select the fly_h.F from the Generate Menu (Fig. 1). The window of
Fig. 8 will appear. The user must specify all the fields as
specified in par. 6.1
Fig. 8
27
10.8) Creating mkfl_FLY and mkfl_FLY_sort makefiles
Select the Makefiles from the Generate Menu (Fig. 1). The window
of Fig. 9 will appear. The user must specify all the fields and
compilers Options.
Fig. 9
10.9) Creating executable FLY and FLY_sort
Select Make from the FLY interface (Fig. 1). The program will ask
to create the FLY and the FLY_sort executables.
10.10) Interactive Run
This button (Fig.1) allows the user only to show an example
command indicated in the fields. The command must be given at the
system prompt.
10.11) Batch Run
This button (Fig.1) allows the user only to show some notes for
job submission.
28
11) Troubleshouting
Using the assistant and the graphical interface, the user sets
parameters of the fly_h.F module and uses it to create the FLY
executable.
But many other parameters must be adjusted by the
user if some errors occur. This section describes the main errors
that can occur and the action that must be taken to avoid the
error. All the following parameters are included in the fly_h.F
module.
After any change in the parameter value, FLY must be re-compiled.
ch_all error 1: Abort. It is not possible to have the minimum RAM-cache size.
Action: there is not enough available memory for this execution. Increase the
available memory if it is possible, or reduce the nb_loc (number of local body)
or nc_loc (number of local cells) parameters value and re-make the executable.
find_group error 1: overflow. Action: increase the maxilf parameter value, the
maximum length of temporary storage to form an Interaction List.
find_group error 2: overflow. Action: there is a serious problem with the memory
boundary. Report the bug to the FLY authors.
force error1: overflow. Action: increase the maxnterm
ilist error 1: overflow. Action: increase the maxnterm parameter value.
ilist error 2: overflow. Action: increase the maxilf parameter value. If the
error will persist, please report the bug to the FLY authors.
ilist_group error 1: overflow. Action: increase the maxnterm parameter value.
ilist_group error 2: overflow. Action: increase the maxnterm parameter value.
ilist_group error 3: overflow. Action: increase the maxilf parameter value. If
the error will persist, please report the bug to the FLY authors.
ilist_group error 4: overflow. Action: increase the maxnterm parameter value.
ilist_group error 5: overflow. Action: increase the maxnterm parameter value.
read_b_asc error1: inconsistent input. Action: the number of bodies nbodies read
from the stat_pars file doesn't match with the nbodsmax parameter. Set nbodsmax
equal to nbodies and re-compile FLY.
read_b_bin error1: inconsistent input. Action: the number of bodies nbodies read
from the stat_pars file doesn't match with the nbodsmax parameter. Set nbodsmax
equal to nbodies and re-compile FLY.
read_params error 1: nbodies greater than nbodsmax. Action: the number nbodies
read from the stat_pars file doesn't match with the nbodsmax parameter. Set
nbodsmax equal to nbodies and re-compile FLY.
tree_gen error 1: Max level reached: # greater then nmax_level. Action: The
number of levels created to build the tree structure, is greater than the
maximum allowable levels (equal to lmax). Increase the value of lmax parameter,
and re-compile FLY.
tree_gen error 2: overflow. Action: The number of cells created to build the
tree structure, is greater than the maximum allowable internal tree cells (equal
to nbodsmax). Increase the value of ncells parameter, and re-compile FLY.
tree_sort error 1: Max level reached: # greater then nmax_level. Action: The
number of levels created to build the tree structure, is greater than the
maximum allowable levels (equal to lmax). Increase the value of lmax parameter,
and re-compile FLY.
tree_sort error 2: overflow. Action: The number of cells created to build the
tree structure, is greater than the maximum allowable internal tree cells (equal
to nbodsmax). Increase the value of ncells parameter, and re-compile FLY.
29
Index
Introduction..................................................... 1
1) Download FLY.................................................. 2
1.1) Unpacking FLY ............................................. 2
2) Setting the box for a simulation.............................. 2
3) Setting the number of particles of the simulation............. 3
4) Creating the initial condition file........................... 3
5) Creating the input parameter files............................ 4
5.1) Creating the ./FLY_3.1/bin/fly_fnames file ................ 5
5.2) Creating the stat_pars file ............................... 6
5.3) Creating the dyn_pars file ............................... 11
5.4) Creating the FLY out32.tab table. ....................... 13
5.5) Creating the FLY quick-look table ql.tab
file ........ 14
5.6) Creating the Ewald grid tables ew_grid and ew_tab files .. 14
6) The executables FLY and the FLY_sort utility................. 14
6.1) Preparing the Module file ./FLY_3.1/src/fly_h.F ........ 14
6.2 mkfl_FLY and mkfl_FLY_sort makefiles and executables. ..... 15
6.3) Compiler options. ........................................ 16
6.4) Compilation .............................................. 16
7) Use the assistant to create the FLY job script.............. 16
8) Running a simulation with FLY................................ 16
9) Testcase..................................................... 17
9.1)Start the assistant ....................................... 17
9.2) Setting the ../Testcase directory ........................ 18
9.3) Creation of fly_h.F90 module ............................. 19
9.4) Creation of FLY_mpi in the ../Testcase directory ......... 19
9.5) Start mpd daemon ......................................... 21
9.6) Start the FLY_mpi executable ............................. 21
10) Tcl/Tk Graphic interface.................................... 22
10.1) The main graphical FLY interface ........................ 22
10.2) Creating the stat_pars file ............................. 24
10.3) Creating the dyn_pars file .............................. 24
10.4) Creating the out_32.tab file ............................ 25
10.5) Creating the ql.tab file ................................ 26
10.6) Creating the Ewald grid tables ew_grid and ew_tab files . 27
10.7) Preparing the Module file ./FLY_3.1/src/fly_h.F ........ 27
10.8) Creating mkfl_FLY and mkfl_FLY_sort makefiles ........... 28
10.9) Creating executable FLY and FLY_sort .................... 28
10.10) Interactive Run ........................................ 28
10.11) Batch Run .............................................. 28
11) Troubleshouting............................................. 29
30