The HDF5
Virtual File Layer (VFL)
and
Virtual File Drivers (VFDs)
Dana Robinson
The HDF Group
Efficient Use of HDF5 With High Data Rate X-Ray Detectors
Paul Scherrer Institut
May
May
30-31,
30-31
2012
HDF5 Workshop at PSI
HDF5 Library
HDF5 API
Virtual File Layer
VFD
VFD
disk
May 30-31, 2012
HDF5 Workshop at PSI
User-Supplied
VFD
HDF5 API
H5FDwrite()
Virtual File Layer
Virtual File Driver
function pointer
H5FD_sec2_write()
disk
May 30-31, 2012
internal generic
I/O call
HDF5 Workshop at PSI
VFD-specific
I/O call
HDF5 VFDs
sec21 (default)
log
MPI
Windows2
split
MPI-POSIX
STDIO3
multi
core (in-memory)
family
"logical" VFDs which
perform no I/O
themselves
1) Uses POSIX I/O (sec2 = "POSIX section 2")
2) Currently a wrapper for SEC2. There is no driver which uses Win32 API calls.
3) "How to write a VFD" demo driver. Not intended for production use.
May 30-31, 2012
HDF5 Workshop at PSI
The core VFD allows you to create/open HDF5 files in memory.
HDF5 API
H5FDwrite()
Virtual File Layer
Core VFD
H5FD_core_write()
File is (optionally)
written to disk on
close
memory
May 30-31, 2012
disk
HDF5 Workshop at PSI
The family VFD allows you to split a logical HDF5 file among many
smaller physical files.
HDF5 API
H5FDwrite()
Virtual File Layer
Family VFD
file A
H5FD_family_write()
file B
file C
disk
May 30-31, 2012
HDF5 Workshop at PSI
…
file Z
The multi (and split) VFD allows you to direct various categories of
HDF5 data to different files and disks.
Multi VFD
H5FD_multi_write()
raw data
superblock
global and local
heap data, B-trees,
object headers
file A
file B
file C
disk 1
May 30-31, 2012
disk 2
HDF5 Workshop at PSI
Selecting a "Terminal" VFD
A "terminal" VFD does not require a second VFD
1) Create the property list.
hid_t fapl_id = H5Pcreate(H5P_FILE_ACCESS);
2) Set the VFD using the appropriate API calls.
herr_t err = H5Pset_fapl_sec2(fapl_id);
3) Create/open your file.
hid_t fid = H5Fcreate("foo.h5", 0, H5P_DEFAULT, fapl_id);
May 30-31, 2012
HDF5 Workshop at PSI
Selecting a "Logical" VFD
A "logical" VFD requires a second, underlying VFD
Step 1
Create a file access property list
(FAPL) and set the VFD to use
the terminal VFD.
Step 2
Create a second FAPL and set the
VFD to use the logical VFD
(passing in the first VFD).
Logical VFD
Terminal VFD
Disk
Step 3
Open your file using the second
(logical) FAPL.
May 30-31, 2012
HDF5 Workshop at PSI
1) Create the first property list.
hid_t fapl_id = H5Pcreate(H5P_FILE_ACCESS);
2) Set the terminal VFD using the appropriate API calls.
herr_t err = H5Pset_fapl_sec2(terminal);
3) Create the second file create/access property list.
hid_t fapl_id = H5Pcreate(H5P_FILE_ACCESS);
4) Set the logical VFD using the appropriate API calls.
herr_t err = H5Pset_fapl_family(logical, SIZE, terminal);
5) Create/open your file.
hid_t fid = H5Fcreate("foo.h5", 0, H5P_DEFAULT, logical);
May 30-31, 2012
HDF5 Workshop at PSI
Stackable VFDs
Virtual File Layer
Non-Terminal VFD
Non-Terminal VFD
Terminal VFD
Disk
May 30-31, 2012
HDF5 Workshop at PSI