High-Performance Computing,
Computational Science, and
NeuroInformatics Research
Allen D. Malony
Department of Computer and Information Science
NeuroInformatics Center (NIC)
Computational Science Institute
University of Oregon
Outline

High-performance computing research




Computational science at UO



Projects
Computational Science Institute
Neuroinformatics research


Interactions and funding
Project areas
TAU parallel performance system
NeuroInformatics Center (NIC)
ICONIC Grid
April 29, 2004
PNNL UO Visit
High-Performance Computing Research

Strong associations with DOE national laboratories






DOE funding



Los Alamos National Lab
Lawrence Livermore National Lab
Sandia National Lab (Livermore)
Argonne National Lab
National Energy Research Supercomputing Center
Office of Science, Advance Scientific Computing
ASCI/NNSA
NSF funding


April 29, 2004
Academic Research Infrastructure
Major Research Instrumentation
PNNL UO Visit
Project Areas









Parallel performance evaluation and tools
Parallel language systems
Tools for parallel system and software interaction
Source code analysis
Parallel component software
Computational services
Grid computing
Parallel modeling and simulation
Scientific problem solving environments
April 29, 2004
PNNL UO Visit
TAU Parallel Performance System
Allen D. Malony
Sameer S. Shende
Department of Computer and Information Science
Computational Science Institute
University of Oregon
Parallel Performance Research

Tools for performance problem solving

Empirical-based performance optimization process
Performance
Tuning
hypotheses
Performance
Diagnosis
properties
Performance
Experimentation
characterization
Performance
Observation
April 29, 2004
Performance
Technology
• Instrumentation
• Measurement
• Analysis
• Visualization
PNNL UO Visit
Complexity Challenges for Performance Tools

Computing system environment complexity





Observation integration and optimization
Access, accuracy, and granularity constraints
Diverse/specialized observation capabilities/technology
Restricted modes limit performance problem solving
Sophisticated software development environments





April 29, 2004
Programming paradigms and performance models
Performance data mapping to software abstractions
Uniformity of performance abstraction across platforms
Rich observation capabilities and flexible configuration
Common performance problem solving methods
PNNL UO Visit
General Problems
How do we create robust and ubiquitous performance
technology for the analysis and tuning of parallel and
distributed software and systems in the presence of
(evolving) complexity challenges?

How do we apply performance technology effectively
for the variety and diversity of performance problems
that arise in the context of complex parallel and
distributed computer systems?
April 29, 2004
PNNL UO Visit
TAU Performance System



Tuning and Analysis Utilities
Performance system framework for scalable parallel and
distributed high-performance computing
Targets a general complex system computation model




Integrated toolkit for performance instrumentation,
measurement, analysis, and visualization



nodes / contexts / threads
Multi-level: system / software / parallelism
Measurement and analysis abstraction
Portable performance profiling and tracing facility
Open software approach with technology integration
University of Oregon , Forschungszentrum Jülich, LANL
April 29, 2004
PNNL UO Visit
TAU Performance System Architecture
Paraver
April 29, 2004
EPILOG
PNNL
UO Visit
TAU Performance System Status

Computing platforms


Programming languages


C, C++, Fortran 77/90/95, HPF, Java, OpenMP, Python
Communication libraries


IBM SP / Power4, SGI Origin 2K/3K, ASCI Red, Cray
T3E / SV-1 / X-1, HP (Compaq) SC (Tru64), HP
Superdome (HP-UX), Sun, Hitachi SR8000, NEX SX5/6, Linux clusters (IA-32/64, Alpha, PPC, PA-RISC,
Power, Opteron), Apple (G4/5, OS X), Windows
MPI, PVM, Nexus, shmem, LAMPI, MPIJava
Thread libraries

April 29, 2004
pthreads, SGI sproc, Java,Windows, OpenMP
PNNL UO Visit
TAU Performance System Status (continued)

Compilers


Application libraries (selected)


POOMA, MC++, ECMF, Uintah, VTF, UPS, GrACE
Performance technology integrated with TAU


Blitz++, A++/P++, PETSc, SAMRAI, Overture, PAWS
Application frameworks (selected)


Intel KAI (KCC, KAP/Pro), PGI, GNU, Fujitsu, Sun,
Microsoft, SGI, Cray, IBM (xlc, xlf), Compaq, Hitachi,
NEC, Intel
PAPI, PCL, DyninstAPI, mpiP, MUSE/Magnet
TAU full distribution (Version 2.x, web download)


April 29, 2004
TAU performance system toolkit and user’s guide
Automatic software installation and examples
PNNL UO Visit
Computational Science



Integration of computer science
in traditional science
Math
disciplines
Third model of
Geoscience
Computer
scientific
Science
research
Application of
Paleontology
high-performance
computation, algorithms
and networking


April 29, 2004
Biology
Neuroscience
Psychology
Parallel computing
Grid computing
PNNL UO Visit
Computational Science Projects at UO

Geological science


Bioinformatics






Zebrafish Information Network (ZFIN)
Evolution of gene families
Oregon Bioinformatics Tool
Neuroinformatics
Electronic notebooks
Domain-specific problem solving environments


Model coupling for hydrology
Dinosaur skeleton and motion modeling
Computational Science Institute
April 29, 2004
PNNL UO Visit
Computational Science  Cognitive Neuroscience

Computational methods applied to scientific research



Understand functional activity of the human cortex



Multiple cognitive, clinical, and medical domains
Multiple experimental paradigms and methods
Need for coupled/integrated modeling and analysis



High-performance simulation of complex phenomena
Large-scale data analysis and visualization
Multi-modal (electromagnetic, MR, optical)
Physical brain models and theoretical cognitive models
Need for robust tools: computational & informatic
April 29, 2004
PNNL UO Visit
Brain Dynamics Analysis Problem

Identify functional components





Different cognitive neuroscience research contexts
Clinical and medical applications
Interpret with respect to physical and cognitive models
Requirements: spatial (structure), temporal (activity)
Imaging techniques for analyzing brain dynamics

Blood flow neuroimaging (PET, fMRI)
spatial resolution  functional brain mapping
 temporal limitations to tracking of dynamic activities
 good

Electromagnetic measures (EEG/ERP, MEG)
 msec
temporal resolution to distinguish components
 spatial resolution sub-optimal (source localization)
April 29, 2004
PNNL UO Visit
Integrated Electromagnetic Brain Analysis
good spatial
poor temporal
Cortical Activity
Knowledge Base
Head Analysis
Structural /
Functional
MRI/PET
Experiment
subject
Dense
Array EEG /
MEG
Source Analysis
temporal
dynamics
Cortical
Activity Model
Constraint
Analysis
neural
constraints
spatial pattern
recognition
Individual
Brain Analysis
Component
Response Model
temporal pattern
recognition
Signal Analysis
Response Analysis
poor spatial
good temporal
April 29, 2004
Component Response
Knowledge Base
neuroimaging
integration
PNNL UO Visit
Experimental Methodology and Tool Integration
16x256
bits per
millisec
(30MB/m)
CT / MRI
segmented
tissues
EEG
NetStation
processed
EEG
Interpolator 3D
April 29, 2004
BESA
BrainVoyager
mesh generation
source localization
constrained to
cortical surface
EMSE
PNNL UO Visit
NeuroInformatics Center (NIC)

Application of computational science methods to
cognitive and clinical neuroscience problems





Advance techniques for integrated neuroimaging





Understand functional activity of the brain
Help to diagnosis brain-related disorders
Utilize high-performance computing and simulation
Support large-scale data analysis and visualization
Coupled modeling (EEG/ERP and MR analysis)
Advanced statistical factor analysis
FDM/FEM brain models (EEG, CT, MRI)
Source localization
Problem-solving environment for brain analysis
April 29, 2004
PNNL UO Visit
NIC Organization

Director, Allen D. Malony


Associate Director, Don M. Tucker


Ph.D., Computer Science; B.S., Physics
Computer Scientist, Sameer S. Shende


Ph.D., Computer Science; B.S., Physics
Computational Physicist, Sergei Turovets


Professor, Psychology; CEO, EGI
Computational Scientist, Kevin Glass


Associate Professor, Computer and Information Science
Ph.D., Computer Science; parallel computing specialist
Mathematician, Bob Frank

April 29, 2004
M.S., Mathematics
PNNL UO Visit
Funding Support

BBMI federal appropriation




NSF Major Research Instrumentation


DoD Telemedicine Advanced Technology Research
Command (TATRC)
Initial budget of approximately $750K
Oct. 1, 2002 through March 31, 2004
ICONIC Grid, awarded
New proposal opportunities


April 29, 2004
NIH Human Brain Project Neuroinformatics
NSF ITR
PNNL UO Visit
NIC Approaches

Optimize spatial resolution




Optimize temporal resolution




MRI structural information
Measurement of skull conductivity
Convergence / co-recording with MEG and fMRI
Use EEG/MEG time course for fMRI signal extraction
Decomposition of component analysis (ICA, PCA)
Single-trial analysis
Computational brain models


April 29, 2004
Boundary and finite element brain models
Brain information databases and atlases
PNNL UO Visit
EEG/ERP Methodology


Electroencephalogram (EEG)
Event-Related Potential (ERP)



Stimulus-locked measures of brain dynamics
Generated from subject- and trial-based analysis
Raw EEG datasets processed and analyzed



Segmentation to time series waveforms
Blink removal and other cleaning
ERP analysis
 Averaging
for increasing signal to noise
 Characterization with respect to trial conditions
 Results visualization

Source localization
April 29, 2004
PNNL UO Visit
EGI Geodesics Sensor Net


Electrical Geodesics Inc.
Dense-array sensor technology


256-channel geodesics sensor net



64/128/256 channels
AgCl plastic electrodes
Carbon fiber leads
Future optical sensors

April 29, 2004
EGI + LANL
PNNL UO Visit
EEG/ERP Experiment Management System


Support EEG-based cognitive neuroscience research
Based on experiment model



Management of experiment data




Experiment type
Subjects measured for trial types
Raw and processed datasets and derived statistics
Per experiment/subject/trial database
Secure protection and storage with selective access
Analysis tools and workflows


April 29, 2004
Generation of results (across experimental variables)
Analysis processes with multi-tool workflows
PNNL UO Visit
EEG/ERP Experiment Analysis Environment
raw
analysis workflow
processed datasets
/ derived results
virtual
services
storage
resources
April 29, 2004
compute resources
PNNL UO Visit
Source Localization

Mapping of scalp potentials to cortical generators


Single time sample and time series
Requirements

Accurate head model and physics
 High-resolution
3D structural geometry
 Precise tissue identification and segmentation
 Correct tissue conductivity assessment

Computational head model formulation
 Finite
element model (FEM)
 Finite difference model (FDM)
 Forward problem calculation

April 29, 2004
Dipole search strategy
PNNL UO Visit
Advanced Image Segmentation





Native MR gives high
gray-to-white matter
contrast
Edge detection finds region
boundaries
Segments formed by edge
merger
Color depicts tissue type
Investigate more advance
level set methods and
hybrid methods
April 29, 2004
PNNL UO Visit
Building Finite Element Brain Models


MRI segmentation of brain tissues
Conductivity model


Measure head tissue conductivity
Electrical impedance tomography
currents are injected
between electrode pair
 resulting potential measured
at remaining electrodes
scalp
CSF
 small


Finite element forward solution
Source inverse modeling


April 29, 2004
skull
cortex
Explicit and implicit methods
Bayesian methodology
PNNL UO Visit
Conductivity Modeling
Continuous
Solutions
Finite-Difference
Finite-Element
Boundary-Element
Finite-Volume
Spectral
Discrete Nodal Values
Tridiagonal
ADI
SOR
Gauss-Seidel
Gaussian elimination
 (x,y,z,t)
J (x,y,z,t)
B (x,y,z,t)
April 29, 2004
Governing
Equations
ICS/BCS
Discretization
System of
Algebraic
Equations
Equation (Matrix) Solver
Approximate
Solution
PNNL UO Visit
Source Localization Analysis Environment
raw
virtual
services
storage
resources
April 29, 2004
compute resources
PNNL UO Visit
NIC Computational Cluster (“Neuronic Cluster”)

Dell computational cluster

16 dual-processor nodes
 2.8
MHz Pentium Xeon
 4 Gbyte memory
 36 Gbyte disk
 Dual Gigabit ethernet adaptors
 2U form factor




Master node (same specs)
2 Gigabit ethernet switches
Brain modeling
Component analysis
April 29, 2004
PNNL UO Visit
NIC Relationships
Utah
UCSD
USC
OHSU
/ OGI
Internet2
Academic
CIS
April 29, 2004
Physics
Argonne
Sandia
NCSA
Labs / Centers
Intel
UO Departments
Psychology
BDL BEL
LANL
NIC
IBM
EGI
Industry
UO Centers/Institutes
CSI
BBMI
CNI
CDSI
NSI
PNNL UO Visit
NSF MRI Proposal



Major Research Instrumentation (MRI)
“Acquisition of the Oregon ICONIC Grid for
Integrated COgnitive Neuroscience Informatics and
Computation”
PIs



Senior personnel



Computer Science: Malony, Conery
Psychology: Tucker, Posner, Nunnally
Computer Science: Douglas, Cuny
Psychology: Neville, Awh, White
Approximately $1.2M over three years
April 29, 2004
PNNL UO Visit
ICONIC Grid
graphics workstations
interactive, immersive viz
other campus clusters
Internet 2
Gbit Campus Backbone
NIC
4x8
CIS
16
CIS
16
SMP
Server
Shared
Memory
Graphics
SMP
IBM p655
IBM p690
SGI MARS
SAN Storage System
April 29, 2004
CNI
2x8
NIC
2x16
Distributed Distributed
Memory
Memory
IBM JS20
Dell Pentium Xeon
5 Terabytes
PNNL UO Visit
Cognitive Neuroscience and ICONIC Grid

Common questions to be explored





Identifying brain networks
Critical periods during normal development
Network involvement in psychopathologies
Training interventions in network development
Research areas





April 29, 2004
Development of attentional networks
Brain plasticity in normal development and deprived
Attention and emotion regulation
Spatial working memory and selective attention
Attention and psychopathology
PNNL UO Visit
Computer Science and ICONIC Grid

Scheduling and resource management



PSEs for computational science



Provide scientists an entrée to the computational and
data management power of the infrastructure without
requiring specialized knowledge of parallel execution
Marine seismic tomograph, molecular evolution
Interactive / immersive three-dimensional visualization



Assign hardware resources to computation tasks
Scheduling of workloads for
Explore multi-sensory visualization
Merge 3D graphics with force-feedback haptics
Parallel performance evaluation
April 29, 2004
PNNL UO Visit