NeuroImage 19 (2003) 1033–1048
www.elsevier.com/locate/ynimg
The LONI Pipeline Processing Environment
David E. Rex, Jeffrey Q. Ma, and Arthur W. Toga*
Laboratory of Neuro Imaging, Department of Neurology, David Geffen School of Medicine at UCLA, Los Angeles, CA 90095-1769, USA
Received 3 December 2002; revised 25 February 2003; accepted 18 March 2003
Abstract
The analysis of raw data in neuroimaging has become a computationally entrenched process with many intricate steps run on increasingly
larger datasets. Many software packages exist that provide either complete analyses or specific steps in an analysis. These packages often
possess diverse input and output requirements, utilize different file formats, run in particular environments, and have limited abilities with
certain types of data. The combination of these packages to achieve more sensitive and accurate results has become a common tactic in brain
mapping studies but requires much work to ensure valid interoperation between programs. The handling, organization, and storage of
intermediate data can prove difficult as well. The LONI Pipeline Processing Environment is a simple, efficient, and distributed computing
solution to these problems enabling software inclusion from different laboratories in different environments. It is used here to derive a
T1-weighted MRI atlas of the human brain from 452 normal young adult subjects with fully automated processing. The LONI Pipeline
Processing Environment’s parallel processing efficiency using an integrated client/server dataflow model was 80.9% when running the atlas
generation pipeline from a PC client (Acer TravelMate 340T) on 48 dedicated server processors (Silicon Graphics Inc. Origin 3000). The
environment was 97.5% efficient when the same analysis was run on eight dedicated processors.
© 2003 Elsevier Science (USA). All rights reserved.
Keywords: Dataflow model; Visual programming; Data processing environment; Automated analysis
Introduction
Neuroimaging analysis often requires multiple processing steps as raw data is transformed into interpretable results. Studies focused on gross anatomic structure (Wright
et al., 1995; Rapoport et al., 1999; Rosas et al., 2002; Sowell
et al., 2002), functional anatomy (Woods, 1996; Corbetta et
al., 1998; Worsley et al., 1998; Dumoulin et al., 2000;
Zeineh et al., 2001; Pouratian et al., 2002), various physiological measurements (Woods et al., 1994; Maier et al.,
1995; McClure et al., 1998; Mega et al., 1999; Opitz et al.,
1999), cyto- (Geyer et al., 1996; Mega et al., 1997; White et
al., 1997; Amunts et al., 1999) and chemoarchitecture
(Zilles et al., 1991; Koutcherov et al., 2000), gene localization (Reiss et al., 1995; Lohmann et al., 1999; Thompson et
al., 2001; Cannon et al., 2002), and their interactions (Maz* Corresponding author. Department of Neurology, Room 4-238, 710
Westwood Plaza, Box 951769, Los Angeles, CA 90095-1769. Fax: ⫹1310-206-5518.
E-mail address: [email protected] (A.W. Toga).
ziotta et al., 2001) employ both common, as well as unique,
processing procedures as part of the analysis. Many of the
procedures are increasingly computationally intensive and
often automated.
Several software packages encapsulating these processing steps have evolved (Table 1). Some are monolithic
implementations, such as SPM T4 (Friston et al., 1990,
1991, 1995a, 1995b, Worsley and Friston 1995), BrainVoyager, SureFit and Caret (Carman et al., 1995; Drury et al.,
1996; Van Essen and Drury, 1997; Van Essen et al., 1998,
2001), and MEDx, which provide the user with all-in-one
approaches to analyzing data. Others are more modular
environments, such as AIR (Woods et al., 1998a, 1998b),
FSL (Jenkinson and Smith, 2001; Zhang et al., 2001; Smith,
2002), AFNI (Cox, 1996; Cox and Hyde, 1997), BrainSuite
(Shattuck and Leahy, 2001, 2002; Shattuck et al., 2001), the
Montreal Neurological Institute’s Brain Imaging Software
Toolbox (Collins et al., 1994; MacDonald et al., 1994,
2000; Sled et al., 1998), and FreeSurfer (Dale et al., 1999;
Fischl et al., 1999a, 1999b, 2001, 2002, Fischl and Dale,
1053-8119/03/$ – see front matter © 2003 Elsevier Science (USA). All rights reserved.
doi:10.1016/S1053-8119(03)00185-X
1034
D.E. Rex et al. / NeuroImage 19 (2003) 1033–1048
Table 1
Software packages for neuroimaging data analysis
Software
Available from
Website
SPM
BrainVoyager
SureFit and Caret
MEDx
AIR
FSL
AFN
BrainSuite
Wellcome Department of Cognitive Neurology
Brain Innovation B.V.
Van Essen Lab
Sensor Systems
Roger Woods, M.D.
FMRIB Image Analysis Group
Robert Cox, Ph.D.
Neuroimaging Research Group and the Laboratory
of Neuro Imaging
Brain Imaging Centre at the Montreal Neurological
Institute
CorTechs and the MGH NMR Center
Khoral Inc.
Scientific Computing and Imaging Institute at the
University of Utah
Advanced Visual Systems
University of Pittsburgh
Laboratory of Neuro Imaging
http://www.fil.ion.ucl.ac.uk/spm/
http://www.brainvoyager.com/
http://stp.wustl.edu/
http://medx.sensor.com/products/medx/
http://bishopw.loni.ucla.edu/AIR5/
http://www.fmrib.ox.ac.uk/fsl/
http://afni.nimh.nih.gov/afni/
http://neuroimage.usc.edu/brainsuite/
Brain Imaging Software Toolbox
FreeSurfer
Khoros w/Cantata
SCIRun
AVS/5
FISWidgets
LONI Pipeline Processing Environment
2000), providing the user with access to individual programs from each package.
There are advantages and disadvantages to monolithic
and modular approaches. In monolithic environments, there
is a unified complete analysis solution in one package. The
individual processing steps are already linked; there should
be no file format incompatibilities; and the environment
works as an integrated system. These monolithic environments, however, are less amenable to alteration. When new
procedures are needed, programming expertise and familiarity with code and interfaces are required to allow new
algorithms to be incorporated. Alternatively, researchers
may go outside of the environment requiring them to ensure
compatibility and force a modular approach on the analysis.
They may also choose to wait for the authors of the monolithic implementation to agree that the methodological enhancements are worthwhile and include them in the next
release. For these reasons, monolithic environments do not
always possess the most current and best algorithms for the
specific study at hand. Finally, these environments do not
lend themselves to distributed computing paradigms important in population-based, high-resolution, multifactorial
brain mapping studies.
Modular environments (Fig. 1) on the other hand, provide flexibility and complete control in the design of analyses. Many of these environments, like FSL, FreeSurfer,
and AFNI, provide enough modules, and an interface, for a
complete analysis akin to the monolithic environments.
Others, such as AIR and BrainSuite, focus their efforts for
particular needs of the brain mapping community. Programs
and subroutines from these packages can be called as individual steps in a compute sequence. Programs can be reconfigured and mixed and matched from various packages.
Users can also build their own programs to supplement or
replace steps in the sequence resulting in new methodolo-
http://www.bic.mni.mcgill.ca/software/
http://surfer.nmr.mgh.harvard.edu/
http://www.khoral.com/
http://www.sci.utah.edu/
http://www.avs.com/
http://neurocog.lrdc.pitt.edu/fiswidgets/
http://www.loni.ucla.edu/NCRR/Software/Pipeline.html
gies and analyses. To date, a disadvantage of modular environments is the difficulty of ensuring interoperability between the various steps. It is also difficult to become an
expert on all the packages available from a multitude of
laboratories. Often several different programs can each adequately accomplish a task in an analysis. Depending on the
dataset and analytic goals, one package may be better suited
than another. Choosing the most appropriate programs and
assembling the best analysis can be a complex and arduous
task. Usually this requires more expertise than a monolithic
environment, but offers considerably more flexibility.
Organization and logistics
Brain mapping studies process an increasingly large
amount of raw data. The data can originate from different
scanners and institutions and may be generated from different acquisition parameters and be stored in various formats.
During the processing of this data, the creation of many
intermediate datasets can be difficult to handle, organize,
and store. Ensuring all analysis steps have been performed
properly and consistently, while checking for failures in the
processing and confounding differences in the results, can
be troublesome for even the most diligent investigator.
Flaws can go unnoticed until a late stage in processing.
Some errors may never be detected.
Resources
Considerable computational power may be needed for
increasingly complex population-based analyses. Resource
limitations in memory, the number of available processors,
disk sizes and speeds, and network speeds can all reduce the
feasibility of an algorithm and its practical utility. New
techniques that use multivariate statistics (McKeown, 2000;
D.E. Rex et al. / NeuroImage 19 (2003) 1033–1048
Calhoun et al., 2001; Suzuki et al., 2002), multiple modalities (Dale and Halgren, 2001; Mohamed et al., 2001), and
vector or tensor field calculations (Chung et al., 2001; Arfanakis et al., 2002; Haueisen et al., 2002; Ruiz-Alzola et al.,
2002; Westin et al., 2002) are impractical, or impossible, on
large datasets without parallel and/or distributed computing
solutions.
Interoperability
Individual steps in an analytic sequence may not always
work with one another. The output of one step may be
incompatible with the input of its successor, especially if
these programs were developed at different institutions.
This is not limited to simple file conversions. Certain criteria may need to be fulfilled by the data itself. This can
include specifications such as allowed value ranges and data
types, spatial dimensions and dimension order, and limits on
voxel sizes. Data criteria might also include quality or
consistency limits such as signal-to-noise ratio, contrast-tonoise ratio, or the preservation of intensity gradients of a
certain kernel size. Ensuring the data is properly formatted
for the next processing step can take as much time and effort
as the analysis. This can become a large obstacle to many
collaborative efforts.
Reproducibility
A more general problem in neuroimaging is the difficulty
in reproducing analyses as described in a publication. Often
details are left out of the methodological description. These
details, though crucial for reproduction, aren’t needed for
the conveyance of the novel analyses presented in the published work.
Solutions
Scripts are often programmed to solve the difficulties of
a modular approach to brain mapping analysis. Scripting
languages, such as perl and tcsh, offer many powerful constructs and abilities, such as conditionals, loops, and job
control. However, many of these scripts are often written for
a single purpose or one time usage. In many situations they
may be highly modified over time to adapt to new analyses,
keeping old solutions tenuously intact that should have been
discarded. The ability to distribute these scripts is dependent
on the similarities between environments with no guarantees they will work without modifications. Issues such as
resource requirements, temporary files, heterogeneous hardware needs with remote executions, and parallel processing
are solved in ad hoc and often laboratory-specific fashions.
The programming effort required to fashion a script that
solves these issues in a standardized manner would be
considerable. These issues are addressed with the development of the LONI Pipeline Processing Environment.
The goal of the LONI Pipeline Processing Environment
1035
is to provide a cross-platform, distributed environment for
the design, distribution, and execution of brain mapping
analysis protocols. It is engineered to be intelligent, efficient, simple, and flexible. The environment was developed
with the aim of greater access to various techniques, algorithms, programs, and computing power. The improvements
in interoperability can facilitate collaboration between developers of specific tools and cooperation among those with
unique datasets. The LONI Pipeline Processing Environment makes studies with a very large number of subjects
more obtainable. It enables a high degree of sensitivity and
accuracy by supporting the mixing and matching of programs from various packages developed in different laboratories.
Methods
The LONI Pipeline Processing Environment
We developed an analysis strategy that provides the
ability to link together individual executable programs, or
modules, into a processing tree, or pipeline. A pipeline
contains the connectivity information between processing
modules that results in an analysis methodology. It allows
files, intermediate results, and other information to be accurately passed between individually connected modules. A
pipelet is the encapsulation of a pipeline that only completes
a subsection of the processing. Allowing pipelets to be
formed and treated as modules in larger pipelines provides
a conceptual abstraction layer. A pipeline execution environment controls the details of the computation requiring
only the connections, or pipes, between the modules that
establish the relationships of the pipeline to be specified.
Modules, pipelets, and pipelines can be saved to disk at any
stage of development and recalled at a later time for modification, use, or distribution.
Module and pipeline descriptions
The description of the modules and pipelines is implemented in the Extensible Markup Language, or XML (information available from http://www.w3c.org/XML/). The
LONI Pipeline Processing Environment does not require an
application programming interface, or API, to be used. Any
well-defined command-line driven processing program or
routine can be represented as a module, given default execution arguments and files, saved, and recalled for later use
and encapsulation in larger pipelines. The description of a
module is accomplished with the aid of a graphical user
interface that takes the user through the addition of input
arguments, output arguments, input streams, and/or output
streams. Arguments may be described as files of specific
types, flags, or values and, if need be, are added in the order
in which they should be provided to the underlying program. They may be optional or required and can be dupli-
Fig. 1. Modular environments allow the mixture of programs and algorithms to achieve tailored results. The flexibility afforded in these designs may provide
increased sensitivity with more accurate results, however, the expertise required to utilize various packages is considerable and the transfer of data from one
package to another is often problematic. Programs shown here include Octree Spatial Normalization (Kochunov et al., 1999), or OSN, AFNI (Cox, 1996;
Cox and Hyde, 1997), the Brain Surface Extractor (Shattuck et al., 2001), or BSE, Alignlinear (Woods et al., 1998a) and Tracer from AIR, and Mincresample,
Mincreshape, and Display (MacDonald et al., 1994) from MNI’s Brain Imaging Software Toolbox. All of these can work together with proper conversion
between the six file formats present.
Fig. 2. The LONI Pipeline Processing Environment is a visual programming interface for the simplified design, execution, and dissemination of neuroimaging
analyses. Modules can be added, deleted, and substituted for other modules within a pipeline analysis. The environment handles bookkeeping and information
transfer between modules and within a pipeline. Its Java-based design and client/server abilities provide a platform independent solution to the interoperation
of various modular packages for image processing in brain mapping.
D.E. Rex et al. / NeuroImage 19 (2003) 1033–1048
1037
Fig. 4. The global variables table separates important inputs and outputs from the inner workings of a pipeline. Variables may be defined for text, values,
files, or lists of files and accessed for any input, output, or new variable definition. Variables are accessed with a dollar sign and braces and may be modified
with a “:m” suffix, where m is the proper modifier, or the addition, “⫹,” or subtraction, “⫺,” of text. The modifiers “r” and “b” shown here represent the
input’s root, everything except a file’s extension, and base, the root without the path prefix, respectively.
Fig. 3. Modules and pipelines are presented in multiple organized and cross-referenced tree structures. Separate structures exist for the current user’s personal
modules and pipelines, the system the client is running on, and each Pipeline Server the client is connected to. Each structure can be subdivided into groups
of folders for the logical presentation of programs, packages, and analyses.
cated when multiple instances of the same argument type
are allowed in a program. Each additional argument, other
than a flag, provides an additional input or output for thedescribed module that may be used in a pipeline’s connection scheme.
Interface
The graphical user interface, or GUI, and execution engine for the LONI Pipeline Processing Environment is programmed as a lightweight environment in Java 1.3 (available from Sun Microsystems, http://java.sun.com/). It is
designed for maximum portability and platform independence. It will run on any system with a Java Virtual Machine of version 1.3 or newer. The design of the interface is
that of a visual programming environment (Fig. 2). The GUI
allows the user to link together the modules, change the flow
of a predesigned pipeline, or replace specific analysis modules for others. There is also a seamless client/server inter-
face that allows the execution of external modules, pipelets,
and pipelines defined on outside servers.
The LONI Pipeline Processing Environment presents
modules and pipeline analyses to the user in organized tree
structures (Fig. 3). There is a user specific section, a system
specific section, and a separate section for any connected
Pipeline server. Each of these sections can be subdivided
into folders, allowing the environment to be arranged into
different categories of modules and analyses. Modification
and utilization of modules or pipelines require placement in
the workspace. The module outputs are connected to other
modules’ inputs with pipes to form a pipelet or a complete
pipeline. Specific inputs and outputs may be defined as a
pipeline is created. Lists of inputs can be specified for batch
processing, comparisons, or contrasts across subject groups.
Outputs may be left as temporary files to be deleted when
they are no longer needed or saved to specified files for later
use.
Tables for the definition of global variables are available
1038
D.E. Rex et al. / NeuroImage 19 (2003) 1033–1048
Fig. 5. This server-based pipeline performs a Talairach registration (Collins et al., 1994), tissue classification (Shattuck et al., 2001), and cortical surface
generation (MacDonald et al., 1994) from T1-weighted MRI whole-head volumes. This is a fully automated pipeline requiring no input other than the files
to process. The display program from the MNI (http://www.bic.mni.mcgill.ca/software/Display/Display.html) was added as a client-side module to visualize
the results. Gray matter is shown in red, white matter in green, and CSF in blue.
in the LONI Pipeline Processing Environment. The architect of a pipeline uses these tables to make certain inputs
and outputs of a pipeline easily accessible (Fig. 4). The user
need not delve into the architecture of the underlying pipeline in order to change these inputs and save outputs. Instead, the appropriate global variables are changed to run
the pipeline on new data. Variable modifiers, such as accessing the directory a file is in, a filename without the
directory information, the base name of a file, or the extension on a file, may be invoked when using a global variable.
Variables also may be defined using previously defined
variables, modified versions of those variables, and explicit
changes to the variable text. This allows for outputs to be
defined based on input variables and sets of inputs to be
defined based on similar naming schemes. Inputs and out-
puts can be saved in the same directory, a similar directory
structure, or with related filenames.
Execution model
The LONI Pipeline Processing Environment uses a dataflow model (Karp et al., 1967) to provide its parallel processing architecture. The dataflow model is based on a
directed graph (Davis and Lowder, 1981) with processing
functions, or modules, as nodes and data pipes as edges
dictating the flow of data between the functions. The beginning of a data pipe is output data from one module. The
pipe directs the data to the inputs for other modules. In this
scheme, each module is self-aware and executes upon receiving all of its input data. When a module completes
D.E. Rex et al. / NeuroImage 19 (2003) 1033–1048
Fig. 6. The “Aligned BET” pipeline is an example where a good program
can be further enhanced by other programs. In native space, the Brain
Extraction Tool (Smith, 2002), or BET, often fails to fully remove the neck
musculature and other ventral structures. BET performs more reliably
when an interpolation induced smoothing and an automated crop is provided by resampling the unedited volume to ICBM305 space (Collins et al.,
1994). The resulting brain mask is transformed back to native space.
execution, the other modules that receive data from it are
signaled that needed inputs have arrived. The signaled modules will initialize execution if they are not waiting for
additional data. All modules that are not data dependent on
one another can execute in parallel without any explicit
specification. When no modules are left in a running state,
the pipeline has completed its execution.
All intermediate and output data in a dataflow are stored
in temporary files unless specified otherwise. Filenames
may be explicitly provided or dynamically generated using
the variable manipulation language described previously. A
unique filename is generated for each temporary file with a
filename extension based on the file type encoded in a
module’s description. Temporary files are discarded when a
workspace is cleared or a new pipeline is selected for
editing or execution.
A simple batch-processing scheme is provided in the
LONI Pipeline Processing Environment. In the default case,
a file list is provided as an input to a module and the
execution of each item in the list is in a separate, but
identical, module. This results in a listing of outputs in
one-to-one correspondence to the file list input. Full parallel
execution of file lists in the pipeline is allowed through the
dataflow execution model. No special cases need be derived. Modules may also have a file list input that is
grouped. All the items in the list are treated as identically
1039
typed input arguments to a single module. This is often the
case when group statistics are calculated. Data are first
preprocessed individually then combined for simultaneous
input for processing across the group. Argument lists and
multiple input lists to modules are also allowed.
To ensure that the dataflow model does not overwhelm
available computing power there is a first-in–first-out, or
FIFO, launch queue. Modules that are ready to execute will
wait on the queue for a processor to become available.
When the LONI Pipeline Processing Environment is set up
on a new computer the number of accessible processors is
set. This value is usually less than or equal to the number of
processors available on the platform. A module will move
from the launch queue to a running state only when the
number of currently running processes is less then the maximum allowed.
Grid computing and batch scheduling systems may also
be integrated with the LONI Pipeline Processing Environment to allow processing to occur over a cluster. In this case
the maximum number of simultaneous processes would be
set to the limits of the computational grid, and the environment would depend on the grid management software to
perform the process scheduling and resource allocation.
Modern grid management software can distribute processing based on available processors, memory, operating environments, access to specialized hardware, and various
other factors, providing a powerful and heterogeneous extension to the abilities of a pipeline environment.
Client/server
The client/server strategy built into the LONI Pipeline
Processing Environment is both application and cycle serving. It is platform independent using Java’s remote method
invocation, or RMI, libraries. When a client logs into a
Pipeline Server it is presented with all the modules, pipelets,
and full pipelines the server can run. The client user can
immediately select a pipeline from the server, load it with
data, and execute it on the server. Alternatively selected
modules or pipelets from the client and the server can be
connected together. These complex pipelines can be saved
in the user’s module listing and recalled for later use. Server
based pipelines and pipelets may only be accessed when the
user is connected to the appropriate servers.
When server modules or pipelets are executed their inputs are transferred to the server and the underlying modules are placed in the server’s launch queue for execution.
When completed, outputs are transferred back to the client
only if they are explicitly saved or needed for a client
module to run. Multiple server connections are allowed in a
pipeline session. Modules and pipelets may be mixed from
many different servers in a single workspace. The client will
act as a master to all the connected servers and dictate all the
file transfers and module launches that are to occur. If files
are provided to the pipeline through URLs, the client or
1040
D.E. Rex et al. / NeuroImage 19 (2003) 1033–1048
Fig. 7. This modified “AIR Make Atlas” pipeline utilizes chirp-z and 3D windowed-sinc interpolation for the linear and nonlinear registration steps,
respectively. It also includes a tissue classification step to generate probability fields for gray matter, white matter, and CSF locations in the atlas space. The
global variables table shows the important input and output files for this pipeline. The native volume and binary mask lists contain 452 subject volumes in
correspondence. The outputs include a full affine aligned atlas with gray, white, and CSF probability fields and a fifth order polynomial aligned atlas to the
linear defined space with gray, white, and CSF probability fields.
appropriate server will download them for use by the appropriate module.
Security
The Pipeline Server has been made secure in several
ways. Account access is encrypted and passwords are confirmed through a one-way hash. The client is not allowed to
launch any program on the server that is not defined in the
server’s process table. Static files on the server are accessible only to the server modules and may not be transferred
to the client for its use. Only temporary files written by the
pipeline are accessible to the client. Furthermore, only
server generated temporary files can be written to the server;
the clients cannot explicitly write any files. Finally, it is
recommended that the Pipeline server be run under a restricted account that only has access to the components it
needs. It is up to the server administrator to ensure that
modules do not contain programs that may be used to
compromise the server or any computer to which it may be
connected.
Results
Modules from AIR 5.2.3, BrainSuite, FSL 1.3, FreeSurfer, the Montreal Neurological Institute’s Brain Imaging
Software Toolbox, and the Laboratory of Neuro Imaging
have been described and are available through a Pipeline
server located at the Laboratory of Neuro Imaging. Included
in these definitions are modules for the skull stripping of
MRI data (Shattuck and Leahy, 2002; Smith, 2002), RF
inhomogeneity corrections (Sled et al., 1998; Shattuck et al.,
2001; Zhang et al., 2001), tissue classification (Shattuck et
al., 2001; Zhang et al., 2001), linear (Woods et al., 1992;
Collins et al., 1994; Woods et al., 1998a; Jenkinson and
Smith, 2001) and nonlinear (Woods et al., 1998a) registration of volumetric data, cortical surface extractions (MacDonald et al., 1994; Dale et al., 1999), skull and scalp
finding software (Shattuck and Leahy, 2002), and many
accessory tools for volumetric image processing.
Many individual processing pipelines have been built
and are available on the Pipeline server at the Laboratory of
Neuro Imaging. They are used for the nonlinear alignment
D.E. Rex et al. / NeuroImage 19 (2003) 1033–1048
1041
Fig. 8. The ICBM452 T1-weighted MRI atlas is an intensity average of 452 normal young adult brains existing in a linearly derived, through 12-parameter
affine transformations, least distant space from all 452 subjects. The final registration to this space is a fifth order polynomial warp allowing substructures
to be more clearly defined in the intensity average. Tissue classification for all 452 subjects is performed in ICBM452 space and probability fields for each
tissue type are derived in atlas space. White matter probabilities are shown in green, gray matter in red, and CSF in blue.
1042
D.E. Rex et al. / NeuroImage 19 (2003) 1033–1048
of a high-resolution template of gross anatomic delineations
to human imaging data, improved skull stripping utilizing a
rough registration to a cropped template, average positional
and intensity atlas construction from raw datasets, registration of datasets to an atlas space, and the Talairach space
(Collins et al., 1994) registration of raw data for tissue
classification and surface model generation (Fig. 5). These
pipelines were built from modules defined on the server and
require no other input than the datasets to be processed. The
use of these pipelines on a 64 processor Silicon Graphics
Origin 3000 has thus far generated over 300,000 module
executions with over 4 terabytes of data generated and/or
processed.
Data were analyzed from the International Consortium
for Brain Mapping (Mazziotta et al., 1995, 2001), or ICBM,
using the LONI Pipeline Processing Environment. Four
hundred fifty-two normal adult subjects from three imaging
centers were selected to create an average intensity atlas in
the average space for the group (Woods et al., 1999). The
raw data were skull-stripped using the Brain Extraction
Tool (Smith, 2002), or BET, in ICBM305 space (Collins et
al., 1994) and the masks of the subdural space were projected back onto the raw data. The pipeline for this procedure, “Aligned BET” (Fig. 6), is located on a LONI Pipeline
server, but was controlled by an Intel Pentium III 450 MHz
based laptop (Acer TravelMate 340T) with 192 Mb of
RAM. The LONI Pipeline server is running on a 64 processor Silicon Graphics Origin 3000 with 32 Gb of RAM,
though it was only granted 48 dedicated processors for this
test. The skull-stripping pipeline took 1 h 37 min to finish
with all 452 subjects, which comprised 1.1 Gb of compressed data. There were 3616 jobs started from the LONI
Pipeline server. It generated 4068 intermediate files using
27.0 Gb of disk, uncompressed, and produced an output of
452 masks defining the subdural space for each input MRI.
The masks compressed to 30 Mb of disk space.
The pipeline to build the atlas was derived from the “AIR
Make Atlas” pipeline on the same LONI Pipeline server.
This pipeline derives affine transformations (Woods et al.,
1998b) from all subjects to a single subject. These transforms are averaged to define a “least distant space” for
every subject to be transformed into. The data is transformed into the common space and an intensity average is
made. All the subjects are realigned with affine transformations to the average template and again averaged to form a
linearly defined atlas in both intensity and spatial domains.
A final step, to gain definition in the atlas, is a fifth order
polynomial nonlinear alignment (Woods et al., 1998b) to
the atlas space for each subject. Each subject is transformed
into the atlas space and averaged to form the final atlas.
The AIR Make Atlas pipeline was modified (Fig. 7) to
use chirp-z interpolation (Tong and Cox, 1999) to apply the
linear transformations and a (13 mm)3 windowed-sinc interpolation (Hajnal et al., 1995) for the nonlinear transformations. Additions were also made to apply the previously
generated masks of subdural space to the raw data as a first
step in the processing and to use the Partial Volume Classifier (Shattuck et al., 2001), or PVC, to generate gray
matter, white matter, and CSF masks of each subject in the
atlas space. These masks were averaged to produce gray
matter, white matter, and CSF probability fields in the atlas
space. The pipeline was run on 48 dedicated processors
from the SGI server and controlled by the Intel based laptop.
It took 19 h 2 min to generate the atlas and tissue based
probability fields from all 452 subjects (Fig. 8). There were
7697 jobs launched from the LONI Pipeline server generating 15,391 intermediate files taking up 46.3 Gb, uncompressed, of temporary disk space. The saved results spanned
nine volumes of compressed data occupying 25.4 Mb of
disk. They consist of a preliminary common space volume,
an affine generated atlas volume with gray, white, and CSF
probability field volumes, and a fifth order polynomial nonlinearly generated atlas volume in the affine average atlas
space with corresponding gray, white, and CSF probability
field volumes.
Additional atlases were created for the males, females,
left-handers, and right-handers in the dataset. The subjects
in the handedness atlases were selected to enforce an equal
number of males and females in each atlas. The modifications to the processing pipeline required to generate these
separate atlases consisted of changing the input file lists and
the location of the output files as appropriate. The processing statistics for each atlas are shown in Table 2.
Further analyses comparing nonlinear intersubject registrations of ICBM data have been performed using modules
from the Research Imaging Center at the University of
Texas Health Science Center at San Antonio, or UTHSCSA,
modules from AIR, and modules from the MNI. Converters
for the descriptor file format were provided by UTHSCSA
(information available from http://ric.uthscsa.edu/projects/
Fig. 9. The “Align Warp” procedure for nonlinear registration in AIR (Woods et al., 1998a) has been replaced by Octree Spatial Normalization (Kochunov
et al., 1999), or OSN, in this intersubject registration pipeline. For OSN to operate, tissue classifications are required to delineate gray matter, white matter,
and CSF. The data must be resampled to a 2563 voxel array as well. The proper format conversions are applied and the resulting nonlinear deformation fields
from the tissue-classified data are applied to the original MRI data.
Fig. 10. The processing efficiency of the LONI Pipeline Processing Environment is demonstrated by executing the atlas generation pipeline on different
numbers of subjects with different numbers of dedicated processors available. Optimal efficiency, when every additional processor provides a linear speed
increase, is approached as the number of subjects increases and dwarfs the number of processors available. The bottlenecks and single module executions
become a smaller fraction of the processing time as the number of subjects increases. The subject level parallelization dominates. On 48 processors the atlas
derived from 64 subjects was 59.3% efficient in its processor usage; the atlas derived from 336 subjects was 80.9% efficient. If, however, the number of
available processors dominates the number of parallel executions the time of completion is as long as the longest timed path through the analysis. The 16
subject atlas could not achieve more than a 10.4-fold speedup with the addition of processors due to its longest path.
D.E. Rex et al. / NeuroImage 19 (2003) 1033–1048
1043
Table 2
Atlas generation processing statistics
Atlas
Number of
subjects
Processing
time
Jobs
launched
Intermediate
file number
Intermediate
file space
All subjects
Males
Females
Right-handed
Left-handed
452
262
190
336
34
19 h 2 m
11 h 54 m
8 h 45 m
13 h 47 m
2 h 33 m
7697
4467
3243
5725
591
15391
8931
6483
11448
1179
46.3 Gb
27.1 Gb
19.5 Gb
34.4 Gb
3.6 Gb
Note. Shown are the processing numbers for generating each of the five atlases. The methodology for each atlas was identical and only the input files and
saved outputs were changed. Each atlas produced nine volumes of compressed output data occupying 25.4 Mb of disk.
chsn/des.html) and integrated with the LONI Pipeline Processing Environment. This allowed the Octree Spatial Normalization (Kochunov et al., 1999), or OSN, software from
UTHSCSA to be integrated in client environments and used
in place of server-based AIR or MNI modules for nonlinear
intersubject MRI registration. OSN requires input data to be
classified for gray matter, white matter, or CSF. PVC provided this prior to the OSN module execution and data were
resampled to match OSN’s input requirements (Fig. 9).
descriptions and are available to the user for modification.
An entire analysis that runs on T1-weighted MRI data may
only require a tissue classification step to have its input flag
changed from T1 to PD for a proton density dataset to be
analyzed. It is possible that this option is not available for
the current module being used. Another tissue classifier that
works on PD-weighted MRI volumes can replace it. Finally,
if there is no module available on the server to accomplish
a task, it can be installed on the client side. The transfer of
data between the servers and the client will be automatically
managed for the user.
Discussion
The total processing time required to assemble the
ICBM452 atlas and noted subatlases on the 48 dedicated
SGI processors was 2 days 9.5 h. Using sequential executions, or the LONI Pipeline Processing Environment with
only a single processor available, would take 39 times as
long,1 or 93 days, to generate the atlases. If this process
were executed at a command line by a single user, assuming
an 8-h work day, it will take up to 3X longer, or 279 days.
The pipelines found on the LONI Pipeline server are
common analysis methods. Interaction with the inner workings of the pipeline is not usually necessary. Providing
inputs, and sometimes outputs, in the global variables table
is all that is required for new datasets to be analyzed.
Outputs that are not explicitly defined by the user are often
derived from the input data and are visible in the global
variable table. Any undefined outputs are discarded. The
specific analysis pipelines used for a publication can be
placed on the server for others to access and execute on new
datasets.
If a pipeline is to be used with unforeseen data, or there
are new procedures to make an analysis more sensitive,
faster, or produce different results, it can be changed easily.
Flags, options, kernel sizes, etc. are all placed in module
1
The efficiency of the LONI Pipeline Processing Environment is less
than 100% due to adjustable delays in the environment’s execution scheme
and because of bottlenecks in dataflow. Delays on clients are to ensure Java
has time for “garbage collection.” Delays are present on servers to prevent
processes from accessing resources simultaneously. Delays are usually tens
to hundreds of milliseconds before each individual command execution
and are tunable for specific installations.
Interoperability and accessibility
The LONI Pipeline Processing Environment allows
many of the interoperability problems that exist between
modular packages used in the brain mapping community to
be solved in one environment and saved for later use. We
have shown the successful execution of pipelines that integrate work from the AIR, FSL, and BrainSuite packages as
well as individual programs from the MNI, LONI, and
UTHSCSA laboratories. These programs used different file
formats, Medical Image NetCDF (available from the MNI,
http://www.bic.mni.mcgill.ca/software/minc/minc.html), or
MINC, the descriptor file format, and two flavors of the
Analyze image format (available from the Mayo Clinic, http://
www.mayo.edu/bir/PDF/ANALYZE75.pdf; SPM modifications from the Wellcome Department of Cognitive Neurology, http://www.fil.ion.ucl.ac.uk/spm/distrib.html#AzeFmt),
and required a variety of preprocessing steps to be applied
before execution. The details of these pipelines can be
generalized for the robust interoperation of packages and
separated into pipelets for later use. These pipelets can be
further improved upon, and possibly replaced by more accurate steps.
The availability and accessibility of many algorithms to
perform similar tasks are part of the power of the LONI
Pipeline Processing Environment. Many seemingly redundant modules are included in the environment to provide
choices and to allow the tailoring of an analysis to specific
data and for specialized end products. When one package
fails on data presented in a manner not accounted for by its
authors, another package may fare better. For instance, we
1044
D.E. Rex et al. / NeuroImage 19 (2003) 1033–1048
have processed datasets where the Brain Extraction Tool
(Smith, 2002) performs more reliably and ones where the
Brain Surface Extractor (Shattuck and Leahy, 2002) performs more reliably. These differences exist due to the
sensitivity of an algorithm to noise, smoothness of gradients
in the data, different tissue volumes and tissue intensity
profile changes from disease, aging, or scanner protocol.
The output of a preprocessing module also may be more
suited to, or even aware of, the input requirements of a later
analysis step. A pipeline currently available on the LONI
Pipeline server uses both BFC (Shattuck et al., 2001) and
N3 (Sled et al., 1998) in parallel processing streams as
corrections for nonuniformity on the input MRI data. This is
to ensure the proper application of a tissue classifier (Shattuck et al., 2001) that BFC is packaged with as well as a
cortical surface extractor (MacDonald et al., 1994) that N3
is packaged with. The design of an algorithm often incorporates an expectation of the input data.
All of the modules and pipelines discussed in this paper
are available on the LONI Pipeline server for use by the
brain mapping community. This ability to distribute automated analysis programs and protocols via Pipeline servers
and the transfer of XML pipeline description files ensures
the accurate conveyance of new ideas from an article to the
workbench. The critical factors in the analysis are reduced
to the quality and structure of the input data. Collaborations
between groups can be facilitated by direct access to described methodology and the assurance of an identical analytic structure. Consolidating tools and analyses into a
common framework for the neuroimaging community to
utilize and enhance will also be beneficial. Improvements
can be disseminated along with the original protocols. The
LONI Pipeline Processing Environment is available from
http://www.loni.ucla.edu/NCRR/Software/Pipeline.html.
Accounts for access to the LONI Pipeline Server can be
requested at http://www.loni.ucla.edu/NCRR/NCRR_CollabApp.html. Access to the LONI Pipeline Server is currently
unlimited for the brain mapping and neuroimaging communities. Validated users may access as much computational
processing power as we can provide.
Computational abilities
The LONI Pipeline Processing Environment enhances
computational abilities in two ways, with distributed processing and through the dataflow model for parallel execution. The distributed processing provided by Pipeline servers allows the analysis of large amounts of data, the study of
complex data, such as tensor fields, and the use of datamining techniques to be available for every laboratory. The
Java-based servers are simple to set up requiring only the
number of available processors, a modules file, and a password file to be specified. The modules file is a Java Archive
file, or JAR, that contains the XML descriptions of modules,
pipelets, or full pipelines already defined on a system and
organized in a directory structure along with any desired
icons. Accounts to the server are created by an included
Java program to encode user names and encrypted passwords in a file. Pipeline servers require Java 1.3 or newer
and RMI to run. HTTP tunneling for firewall compatibility is
available so long as the RMI registry is connected to port 80.
The structure of the predominant number of analyses in
brain mapping is such that most functions are run on each
subject separately and only a few comparison steps are
required to run across all the subjects at once. In the atlas
generation pipeline there were 17 modules running independently on each of the 452 subjects and 10 that ran once with
all 452 subjects as input. In the skull-stripping pipeline there
were eight modules running independently on each of the
452 subjects and none that ran only once. Out of 11,310
module executions, 10 did not benefit from subject level
parallelizations in the analysis. Of those 10, three were in
the middle of the dataflow causing bottlenecks and only two
of the three did not benefit from the module parallelization
in our dataflow model.
Module and subject level parallelization of most analyses
provides close to optimal gains in required processing time
when the number of subjects is much greater than the
number of available processors (Fig. 10). If there are no
bottlenecks in a dataflow, gains are near optimal when the
number of subjects meets or exceeds the number of processors. The gains are only near optimal because particular
subjects may require more processing time and by chance
execute at the end of an analysis or close to last before a
bottleneck. These subjects will extend a sequence by however much longer they take to execute than the average
subject. Chance executions and bottlenecks that exist in an
analysis become a smaller and smaller factor as the number
of subjects is increased.
Another possibility is that the number of available processors exceeds the needs of a pipeline. In this situation the
time of execution is equivalent to the longest time path
through the analysis pipeline. Parallelization of the original
code for individual modules would enhance the efficiency
and speed of these analyses. Analyses that suffer bottlenecks from many group comparison steps would also benefit from parallelization of the code for group actions. Programs that are currently provided with parallel code may be
incorporated into a LONI Pipeline Processing Environment
using a grid computing solution that will allocate multiple
units of a resource, such as processors, to an individual
program when available. Future versions of the LONI Pipeline Processing Environment will allow a module to be
specified to use multiple processors. The required number of
processors will be provided to the executing module as a
block, and released back into the processor pool as a block
at the end of execution.
Similar outside efforts
Several other efforts complement the LONI Pipeline
Processing Environment. Khoros (Konstantinides and Ra-
D.E. Rex et al. / NeuroImage 19 (2003) 1033–1048
1045
Table 3
A comparison of data processing environments
Package
LONI Pipeline
SCIrun
Khoros w/ Cantata
AVS/5
FISWidgets
Main purpose
Protocol development
and distribution;
dissemination of
computing resources
Visual programming
environment and
module development
Visualization and
computation
environment
GUI builder and
sequential execution
environment
Platforms
API
Integration of
previous code
Java
Not needed
GUI-aided description
of any executable, no
recompiling
GUI-aided description
of any executable
Visualization and
steering, module
development,
computational efficient
processing
Linux, SGI IRIX
Yes
Recompilation with API
UNIX, Windows
Yes
Recompilation with API
UNIX, Linux
Yes
Recompilation with API
Compilation with API
Compilation with API
Compilation with API
Provides module
descriptors to run
commonly used neuro
imaging software
Yes
Many modules provided
for various
mathematical and visual
tasks
Yes
Many modules provided
for various
mathematical and visual
tasks
Yes
Many modules provided
for various mat
hematical and visual
tasks
Yes
Java
Yes, for wrappers
Java wrappers
encapsulate previously
written programs
Java code is written
from their API to
wrap programs
Commonly used neuro
imaging software that
has been wrapped
Yes
Yes
Yes
No
No
No
No
No
Yes, as long as they
share a common
filesystem
No
No
No
No
No
No
Free
Free
Commercial
Commercial
Free
Integration of new
code
Modules
Distributed
computing
Application serving
Protocol serving
Integration of
multiple
application
environments
Availability
No
Note. A comparison of the LONI Pipeline Processing Environment with other environments for scientific processing and visualization shows the
similarities and key differences between them. The LONI Pipeline Processing Environment stresses ease of integration, use, and distribution of methodology
and processing power.
sure, 1994; Barrera et al., 1998), with their Cantata visual
programming language, SCIRun (Parker et al., 1997),
AVS/5, and FISWidgets (Fissell et al., 2003), or Functional
Imaging Software Widgets, are particularly worthwhile examples of these endeavors (Table 1). They all possess key
differences, however, that drove the development of the
LONI Pipeline Processing Environment (Table 3).
Khoros (Konstantinides and Rasure, 1994; Barrera et al.,
1998) with Cantata is a commercial endeavor that provides
a similar visual programming environment to the LONI
Pipeline Processing Environment and also uses a dataflow
paradigm. Khoros, however, requires an API to link code
and algorithms into the environment. This requires a programmer to use the provided libraries to integrate new
modules into the processing scheme. The distributed processing scheme provided with Khoros does not allow independent external servers. Instead, servers must be tightly
linked with a common file system preventing independent
and secure external resources from being offered.
SCIRun (Parker et al., 1997) is a freely available environment for scientific modeling, processing, and visualization. It also uses a dataflow visual programming interface.
SCIRun focuses on steering an analysis or model for interactive visualization and therefore requires an integrated
programming effort with the environment. An API is provided that furnishes the developer with libraries, handles,
and data types to tightly link new code with their existing
model. This gives SCIRun complete control over the execution strategy, but limits external development. Most of the
benefits of the API are lost when studies with a large
number of subjects are being conducted. The benefit of
interactive steering becomes unlikely and subject level parallelization can do as well without the API. AVS/5 is a
commercially available visualization and processing environment similar in scope to SCIRun.
FISWidgets (Fissell et al., 2003) is a freely available API
and environment that allows Java-based GUIs to be programmed and attached to any command line program. It
comes with predesigned interfaces to many neuroimaging
analysis packages including AIR and AFNI. It allows the
packages to be integrated in a “GlobalDesktop” for sequential execution on a single workstation. The passing of
filenames and variables must be explicitly specified and
controlled by the user.
Similarities between the LONI Pipeline Processing Environment and these other environments include the visual
interface they all provide to enhance and ease the design of
analyses beyond that of script programming. These interfaces not only present protocols and analyses in an intuitive
manner, but also can take care of many housekeeping efforts, such as temporary files, overwrite protection, environ-
1046
D.E. Rex et al. / NeuroImage 19 (2003) 1033–1048
Table 4
Terms, acronyms, and abbreviations
Acronym
Meaning
Neuroimaging software
AFNI
AIR
BET
BFC
BSE
Caret
Analysis of Functional NeuroImages
Automated Image Registration
Brain Extraction Tool
Bias Field Corrector
Brain Surface Extractor
Computerized Anatomical Reconstruction and Editing
Toolkit
FISWidgets
Functional Imaging Software Widgets
FSL
FMRIB Software Library
N3
Non-parametric Non-uniform intensity Normalization
OSN
Octree Spatial Normalization
PVC
Partial Volume Classifier
SPM
Statistical Parametric Mapping
SureFit
Surface Reconstruction by Filtering and Intensity
Transformations
Computer terminology
API
FIFO
GUI
HTTP
JAR
MINC
NetCDF
RAM
RMI
URL
XML
Application Programming Interface
First In, First Out
Graphical User Interface
Hypertext Transfer Protocol
Java Archive file
Medical Image NetCDF
Network Common Data Format
Random Access Memory
Remote Method Invocation
Uniform Resource Locator
Extensible Markup Language
with provided binaries, scripts, or source code requiring a
minimal amount of effort. Also involved in this design
choice is the use of files for temporary intermediates between processes. Taking into account the speeds of modern
disks, the large amounts of cache available in storage systems, and the possibility of RAM disks for increased speed,
this design choice was preferred to the use of streams and a
required API. This approach provides the added benefit of
being able to quickly and easily read and transfer large files
using random access and bulk loaders. Also of use is the
semipermanence of a temporary file. It can be accessed,
viewed, and saved even after a pipeline has completed
processing without the need for a large, often unobtainable,
amount of memory on the executing computer. The standard input stream and standard output stream are still available in the description of a module and are utilized without
a file intermediate when passed directly between modules.
Another key difference the LONI Pipeline Processing
Environment possesses from the other environments is its
application and cycle server. Pipeline Servers don’t just
provide additional processing power in an internal, controlled environment. Instead they allow completely independent environments to be linked, declare their modules,
pipelets, and full pipeline analyses to the authenticated
client, and be utilized to their maximum potential. This
provides for a simple and effective means for the distribution and acquisition of new algorithms and protocols as well
as the computing power to run them.
Groups, laboratories, and companies
AVS
FMRIB
ICBM
LONI
MGH
MNI
SCI
SGI
UTHSCSA
Advanced Visual Systems
Functional Magnetic Resonance Imaging of the Brain
International Consortium for Brain Mapping
Laboratory of Neuro Imaging
Massachusetts General Hospital
Montreal Neurological Institute
Scientific Computing and Imaging Institute
Silicon Graphics Inc.
University of Texas Health Science Center at San
Antonio
Note. These lists are not exhaustive.
ment variables, and sanity checks, that must be carefully
controlled, or abandoned with possible consequences, in a
command-line scripting environment. All the environments
also provide the ability to save and distribute protocols,
either through a completely encapsulated environment, like
Khoros, AVS/5, and SCIRun, or through an application
server like the LONI Pipeline Processing Environment.
The LONI Pipeline Processing Environment differs from
these other environments in its lack of an application programming interface. The XML descriptor based inclusion,
instead of an API, allows any command-line argument
driven process to be easily encapsulated into the environment. Recompilation, at best, or reprogramming of the processes is not necessary with this approach, making it usable
Future
The goal of the LONI Pipeline Processing Environment
is to make the development and execution of any analysis as
simple as possible. This involves handling the details of an
analysis, providing the constructs necessary to include new
algorithms in the environment, and building general and
specific methodology.
In future releases, the LONI Pipeline Processing Environment will include automatic format conversions and
fully integrated visualization and tailored display of results.
Currently, modules providing access to individual external
converters or visualization programs are utilized. This will
be upgraded so that general conversions from one volumetric, surface, contour, or transformation format to another
similar format will be selected and applied automatically to
ensure compatibility between modules. Similarly, visualization schemes will be tailored in a data-driven manner. List
processing enhancements, conditionals, and iteration constructs are planned for inclusion as well. The integration of
database query, retrieval, and submission systems in the
LONI Pipeline Processing Environment are also planned.
Finally, in future releases, the LONI Pipeline Processing
Environment will provide intelligent agents to assist in the
design of a new analysis.
D.E. Rex et al. / NeuroImage 19 (2003) 1033–1048
Conclusions
The advanced techniques available in neuroimaging provide a landscape where studies of unprecedented size and
complexity are being formulated. The LONI Pipeline Processing Environment makes those studies accessible and
achievable in a reasonable amount of time. The ability to
find smaller effects using larger datasets, the assurance that
all data were analyzed identically and as desired, the time
saved during the processing and construction of an analysis,
and the lowered costs of shared computing resources are
current benefits of the LONI Pipeline Processing Environment. Future enhancements to this environment will provide
integrated access to visualization, databases, and intelligent
agents. We look forward to many additional modules and
pipelines being added in the near future.
Acknowledgments
This work was generously supported by grants from the
National Institute of Mental Health (1 P20 MH65166 and 5
P01 MH52176) and the National Center for Research Resources (2 P41 RR13642 and 2 M01 RR00865), with a
supplement by the Biomedical Informatics Research Network (2 P41 RR13642 [http://www.nbirn.net/]), funded by
the National Center for Research Resources. D.E.R. is supported, in part, by an ARCS Foundation scholarship and a
National Institutes of General Medical Sciences Medical
Scientist Training Program grant (GM08042). The authors
wish to thank Joseph D. Kulisics, Allan MacKenzie-Graham, Roger P. Woods, and the other members of the Laboratory of Neuro Imaging for their help and support.
References
Amunts, K., Schleicher, A., et al., 1999. Broca’s region revisited: cytoarchitecture and intersubject variability. J. Comp. Neurol. 412 (2), 319 –
341.
Arfanakis, K., Cordes, D., et al., 2002. Independent component analysis
applied to diffusion tensor MRI. Magn. Reson. Med. 47 (2), 354 –363.
Barrera, J., Banon, G.J.F., et al., 1998. MMach: a mathematical morphology toolbox for the KHOROS system. J. Electron. Imaging 7 (1),
174 –210.
Calhoun, V.D., Adali, T., et al., 2001. A method for making group inferences from functional MRI data using independent component analysis.
Hum. Brain Mapp. 14 (3), 140 –151.
Cannon, T.D., Thompson, P.M., et al., 2002. Cortex mapping reveals
regionally specific patterns of genetic and disease-specific gray-matter
deficits in twins discordant for schizophrenia. Proc. Natl. Acad. Sci.
USA 99 (5), 3228 –3233.
Carman, G.J., Drury, H.A., et al., 1995. Computational methods for reconstructing and unfolding the cerebral cortex. Cereb. Cortex 5 (6), 506 –
517.
Chung, M.K., Worsley, K.J., et al., 2001. A unified statistical approach to
deformation-based morphometry. NeuroImage 14 (3), 595– 606.
Collins, D.L., Neelin, P., et al., 1994. Automatic 3D intersubject registration of MR volumetric data in standardized Talairach space. J. Comput.
Assist. Tomogr. 18 (2), 192–205.
1047
Corbetta, M., Akbudak, E., et al., 1998. A common network of functional
areas for attention and eye movements. Neuron 21 (4), 761–773.
Cox, R.W., 1996. AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Comput. Biomed. Res. 29 (3),
162–173.
Cox, R.W., Hyde, J.S., 1997. Software tools for analysis and visualization
of fMRI data. NMR Biomed. 10 (4 –5), 171–178 [pii].
Dale, A.M., Fischl, B., et al., 1999. Cortical surface-based analysis. I.
Segmentation and surface reconstruction. NeuroImage 9 (2), 179 –194.
Dale, A.M., Halgren, E., 2001. Spatiotemporal mapping of brain activity
by integration of multiple imaging modalities. Curr. Opin. Neurobiol.
11 (2), 202–208.
Davis, A.L., Lowder, S.A. 1981. A simple management application program in a graphical data-driven programming language. Proc. IEEE
COMPCON 81, New York. 162–165.
Drury, H.A., Van Essen, D.C., et al., 1996. Computerized mappings of the
cerebral cortex: a multiresolution flattening method and a surface-based
coordinate system. J. Cogn. Neurosci. 8 (1), 1–28.
Dumoulin, S.O., Bittar, R.G., et al., 2000. A new anatomical landmark for
reliable identification of human area V5/MT: a quantitative analysis of
sulcal patterning. Cereb. Cortex 10 (5), 454 – 463.
Fischl, B., Dale, A.M., 2000. Measuring the thickness of the human
cerebral cortex from magnetic resonance images. Proc. Natl. Acad. Sci.
USA 97 (20), 11050 –11055.
Fischl, B., Sereno, M.I., et al., 1999a. Cortical surface-based analysis. II:
inflation, flattening, and a surface-based coordinate system. NeuroImage 9 (2), 195–207.
Fischl, B., Sereno, M.I., et al., 1999b. High-resolution intersubject averaging and a coordinate system for the cortical surface. Hum. Brain
Mapp. 8 (4), 272–284.
Fischl, B., Liu, A., et al., 2001. Automated manifold surgery: constructing
geometrically accurate and topologically correct models of the human
cerebral cortex. IEEE Trans. Med. Imaging 20 (1), 70 – 80.
Fischl, B., Salat, D.H., et al., 2002. Whole brain segmentation: automated
labeling of neuroanatomical structures in the human brain. Neuron 33
(3), 341–355.
Fissell, K., Tseytlin, E., et al., 2003. Fiswidgets: a graphical computing
environment for neuroimaging analysis. Neuroinformatics 1 (1), 111–
126.
Friston, K.J., Frith, C.D., et al., 1990. The relationship between global and
local changes in PET scans. J. Cereb. Blood Flow Metab. 10 (4),
458 – 466.
Friston, K.J., Frith, C.D., et al., 1991. Comparing functional (PET) images:
the assessment of significant change. J. Cereb. Blood Flow Metab. 11
(4), 690 – 699.
Friston, K.J., Holmes, A.P., et al., 1995a. Analysis of fMRI time-series
revisited. NeuroImage 2 (1), 45–53.
Friston, K.J., Holmes, A.P., et al., 1995b. Statistical parametric maps in
functional imaging: a general linear approach. Hum. Brain Mapp. 2,
189 –210.
Geyer, S., Ledberg, A., et al., 1996. Two different areas within the primary
motor cortex of man. Nature 382 (6594), 805– 807.
Hajnal, J.V., Saeed, N., et al., 1995. A registration and interpolation
procedure for subvoxel matching of serially acquired MR images.
J. Comput. Assist. Tomogr. 19 (2), 289 –296.
Haueisen, J., Tuch, D.S., et al., 2002. The influence of brain tissue anisotropy on human EEG and MEG. NeuroImage 15 (1), 159 –166.
Jenkinson, M., Smith, S., 2001. A global optimisation method for robust
affine registration of brain images. Med. Image Anal. 5 (2), 143–156.
Karp, R.M., Miller, R.E. 1967. Parallel program schemata: a mathematical
model for parallel computation. IEEE FOCS, 55– 61.
Kochunov, P.V., Lancaster, J.L., et al., 1999. Accurate high-speed spatial
normalization using an octree method. NeuroImage 10 (6), 724 –737.
Konstantinides, K., Rasure, J.R., 1994. The Khoros software development
environment for image and signal processing. IEEE Trans. Image Proc.
3 (3), 243–252.
1048
D.E. Rex et al. / NeuroImage 19 (2003) 1033–1048
Koutcherov, Y., Mai, J.K., et al., 2000. Organization of the human paraventricular hypothalamic nucleus. J. Comp. Neurol. 423 (2), 299 –318.
Lohmann, G., von Cramon, D.Y., et al., 1999. Sulcal variability of twins.
Cereb. Cortex 9 (7), 754 –763.
MacDonald, D., Avis, D., et al., 1994. Multiple surface identification and
matching in magnetic resonance images. Proc. Visualization Biomed.
Comput. 2359, 160 –169.
MacDonald, D., Kabani, N., et al., 2000. Automated 3-D extraction of
inner and outer surfaces of cerebral cortex from MRI. NeuroImage 12
(3), 340 –356.
Maier, M., Ron, M.A., et al., 1995. Proton magnetic resonance spectroscopy: an in vivo method of estimating hippocampal neuronal depletion
in schizophrenia. Psychol. Med. 25 (6), 1201–1209.
Mazziotta, J.C., Toga, A.W., et al., 1995. A probabilistic atlas of the human
brain: theory and rationale for its development. The International Consortium for Brain Mapping (ICBM). NeuroImage 2 (2), 89 –101.
Mazziotta, J., Toga, A., et al., 2001. A probabilistic atlas and reference
system for the human brain: International Consortium for Brain Mapping (ICBM). Philos. Trans. R. Soc. Lond. B Biol. Sci. 356 (1412),
1293–1322.
McClure, R.J., Keshavan, M.S., et al., 1998. Chemical and physiologic
brain imaging in schizophrenia. Psychiatr. Clin. N. Am. 21 (1), 93–122.
McKeown, M.J., 2000. Detection of consistently task-related activations in
fMRI data with hybrid independent component analysis. NeuroImage
11 (1), 24 –35.
Mega, M.S., Chen, S.S., et al., 1997. Mapping histology to metabolism:
coregistration of stained whole-brain sections to premortem PET in
Alzheimer’s disease. NeuroImage 5 (2), 147–153.
Mega, M.S., Chu, T., et al., 1999. Mapping biochemistry to metabolism:
FDG-PET and amyloid burden in Alzheimer’s disease. NeuroReport 10
(14), 2911–2917.
Mohamed, F.B., Vinitski, S., et al., 2001. Increased differentiation of
intracranial white matter lesions by multispectral 3D-tissue segmentation: preliminary results. Magn. Reson. Imaging 19 (2), 207–218.
Opitz, B., Mecklinger, A., et al., 1999. The functional neuroanatomy of
novelty processing: integrating ERP and fMRI results. Cereb. Cortex 9
(4), 379 –391.
Parker, S.G., Johnson, C.R., et al., 1997. Computational steering software
systems and strategies. IEEE Comput. Sci. Eng. 4 (4), 50 –59.
Pouratian, N., Sicotte, N., et al., 2002. Spatial/temporal correlation of
BOLD and optical intrinsic signals in humans. Magn. Reson. Med. 47
(4), 766 –776.
Rapoport, J.L., Giedd, J.N., et al., 1999. Progressive cortical change during
adolescence in childhood-onset schizophrenia. A longitudinal magnetic
resonance imaging study. Arch. Gen. Psychiatry 56 (7), 649 – 654.
Reiss, A.L., Abrams, M.T., et al., 1995. Neurodevelopmental effects of the
FMR-1 full mutation in humans. Nat. Med. 1 (2), 159 –167.
Rosas, H.D., Liu, A.K., et al., 2002. Regional and progressive thinning of
the cortical ribbon in Huntington’s disease. Neurology 58 (5), 695–701.
Ruiz-Alzola, J., Westin, C.F., et al., 2002. Nonrigid registration of 3D
tensor medical data. Med. Image Anal. 6 (2), 143–161.
Shattuck, D.W., Leahy, R.M., 2001. Automated graph-based analysis and
correction of cortical volume topology. IEEE Trans. Med. Imaging 20
(11), 1167–1177.
Shattuck, D.W., Leahy, R.M., 2002. BrainSuite: an automated cortical
surface identification tool. Med. Image Anal. 6 (2), 129 –142.
Shattuck, D.W., Sandor-Leahy, S.R., et al., 2001. Magnetic resonance
image tissue classification using a partial volume model. NeuroImage
13 (5), 856 – 876.
Sled, J.G., Zijdenbos, A.P., et al., 1998. A nonparametric method for
automatic correction of intensity nonuniformity in MRI data. IEEE
Trans. Med. Imaging 17 (1), 87–97.
Smith, S.M., 2002. Fast robust automated brain extraction. Hum. Brain
Mapp. 17 (3), 143–155.
Sowell, E.R., Thompson, P.M., et al., 2002. Mapping sulcal pattern asymmetry and local cortical surface gray matter distribution in vivo: maturation in perisylvian cortices. Cereb. Cortex 12 (1), 17–26.
Suzuki, K., Kiryu, T., et al., 2002. Fast and precise independent component
analysis for high field fMRI time series tailored using prior information
on spatiotemporal structure. Hum. Brain Mapp. 15 (1), 54 – 66.
Thompson, P.M., Cannon, T.D., et al., 2001. Genetic influences on brain
structure. Nat. Neurosci. 4 (12), 1253–1258.
Tong, R., Cox, R.W., 1999. Rotation of NMR images using the 2D chirp-z
transform. Magn. Reson. Med. 41 (2), 253–256.
Van Essen, D.C., Drury, H.A., 1997. Structural and functional analyses of
human cerebral cortex using a surface-based atlas. J. Neurosci. 17 (18),
7079 –7102.
Van Essen, D.C., Drury, H.A., et al., 1998. Functional and structural
mapping of human cerebral cortex: solutions are in the surfaces. Proc.
Natl. Acad. Sci. USA 95 (3), 788 –795.
Van Essen, D.C., Drury, H.A., et al., 2001. An integrated software suite for
surface-based analyses of cerebral cortex. J. Am. Med. Inform. Assoc.
8 (5), 443– 459.
Westin, C.F., Maier, S.E., et al., 2002. Processing and visualization for
diffusion tensor MRI. Med. Image Anal. 6 (2), 93–108.
White, L.E., Andrews, T.J., et al., 1997. Structure of the human sensorimotor system. I: Morphology and cytoarchitecture of the central sulcus.
Cereb. Cortex 7 (1), 18 –30.
Woods, R.P., 1996. Correlation of brain structure and function. Brain
Mapping: The Methods. Academic Press, San Diego, CA, pp. 313–
341.
Woods, R.P., Cherry, S.R., et al., 1992. Rapid automated algorithm for
aligning and reslicing PET images. J. Comput. Assist. Tomogr. 16 (4),
620 – 633.
Woods, R.P., Iacoboni, M., et al., 1994. Brief report: bilateral spreading
cerebral hypoperfusion during spontaneous migraine headache.
N. Engl. J. Med. 331 (25), 1689 –1692.
Woods, R.P., Grafton, S.T., et al., 1998a. Automated image registration: I.
General methods and intrasubject, intramodality validation. J. Comput.
Assist. Tomogr. 22 (1), 139 –152.
Woods, R.P., Grafton, S.T., et al., 1998b. Automated image registration: II.
Intersubject validation of linear and nonlinear models. J. Comput.
Assist. Tomogr. 22 (1), 153–165.
Woods, R.P., Dapretto, M., et al., 1999. Creation and use of a Talairachcompatible atlas for accurate, automated, nonlinear intersubject registration, and analysis of functional imaging data. Hum. Brain Mapp. 8
(2–3), 73–79.
Worsley, K.J., Friston, K.J., 1995. Analysis of fMRI time-series revisitedagain. NeuroImage 2 (3), 173–181.
Worsley, K.J., Cao, J., et al., 1998. Applications of random field theory to
functional connectivity. Hum. Brain Mapp. 6 (5– 6), 364 –367.
Wright, I.C., McGuire, P.K., et al., 1995. A voxel-based method for the
statistical analysis of gray and white matter density applied to schizophrenia. NeuroImage 2 (4), 244 –252.
Zeineh, M.M., Engel, S.A., et al., 2001. Unfolding the human hippocampus
with high resolution structural and functional MRI. Anat. Rec. 265 (2),
111–120.
Zhang, Y., Brady, M., et al., 2001. Segmentation of brain MR images
through a hidden Markov random field model and the expectationmaximization algorithm. IEEE Trans. Med. Imaging 20 (1), 45–57.
Zilles, K., Qu, M.S., 1991. Neurotransmitter receptors and cortical architecture. J. Hirnforsch. 32 (3), 343–356.