Symposium on Statistical Shape Models & Applications
Delémont, Switzerland
June 11‐13, 2014 Computational Anatomy Modeling of Abdominal Organs and Musculoskeletal Structures
Yoshinobu Sato
Graduate School of Information Science
Nara Institute of Science and Technology (NAIST)
Japan
Imaging-based
Computational
Biomedicine Lab
NAIST
Kyoto
Nara Institute of Science and Technology
Information Science
Material Science
Biological Science
Osaka University
NAIST
Tokyo
Osaka
Nara
Statistical Shape Models (SSMs) & Applications in this talk
Abdominal Organs PLSR prediction‐based conditional SSMs & probabilistic atlas
Implants & Host Bones
SSM & statistical distance maps Hierarchical SSM
Musculoskeletal Structures
Conditional SSM
Non‐conditional Conditional
Muscles
Outline
• Our computational anatomy project: Overview
• Anatomy modeling
– Abdominal anatomy
– Musculoskeletal anatomy
– Whole‐body anatomy
• Therapeutic modeling
– Surgeon’s expertise modeling • Artificial joint surgery (Total Hip Arthroplasty: THA)
Outline
• Our computational anatomy project: Overview
• Anatomy modeling
– Abdominal anatomy
– Musculoskeletal anatomy
– Whole‐body anatomy
• Therapeutic modeling
– Surgeon’s expertise modeling • Artificial joint surgery (Total Hip Arthroplasty: THA)
MEXT Grant‐in‐aid for Scientific Research, Japan
Computational Anatomy for Computer‐Aided Diagnosis and Therapy
Sep 2009 ‐ Mar 2014
Fund: $10 million
Principal Investigator: Prof. Hidefumi Kobatake
(TUAT: Tokyo University of Agriculture & Technology)
Eight core groups
Basic theories and technologies
Application systems
Clinical evaluations
http://www.comp‐anatomy.org/
Locations of eight core groups
Google search by “computational anatomy”.
The aim was to develop computational anatomy models of the human body (especially in torso), which represent inter‐subject variability of anatomy across a population, and their applications.
MEXT Grant‐in‐aid for Scientific Research, Japan
Computational Anatomy for Computer‐Aided Diagnosis and Therapy
Sep 2009 ‐ Mar 2014
Fund: $10 million
Principal Investigator: Prof. Hidefumi Kobatake
(TUAT: Tokyo University of Agriculture & Technology)
Eight core groups
Basic theories and technologies (Tokyo, Osaka, Gifu)
Application systems
Clinical evaluations
http://www.comp‐anatomy.org/
Google search by “computational anatomy”.
Osaka Univ.
(My former affiliation)
One of our goals: Complete understanding of whole‐body CT images
Conventional Representation of Human Anatomy
• Book Atlas
– Detailed illustrations of typical anatomy
• 3D Digital Atlas
– Detailed segmented 3D data of a specific subject
VOXEL‐MAN (Univ. Hamburg)
Visible Human data (NIH)
Frank H. Netter, Atlas of Human Anatomy
Semi‐automated segmentation http://www.voxel‐man.de/
They are constructed by Manual Drawing or Semi‐automated Segmentation.
They only show One Typical Example or One Particular Example.
3D Digital Atlas
VOXEL‐MAN (Univ. Hamburg)
One Particular Anatomy
Visible Human Data
Semi‐automated segmentation Reconstructed from Special data with Labor‐intensive efforts
Goal
Patient‐Specific Anatomy
Patient 3D Data (equivalent to Visible Human & VOXEL MAN)
Fully‐automated segmentation From Clinical data as Routine work
VOXEL‐MAN (Univ. Hamburg)
One Particular Anatomy
Visible Human Data
Semi‐automated segmentation Reconstructed from Special data with Labor‐intensive efforts
Goal
Patient 3D Data Patient‐Specific Anatomy
(equivalent to Visible Human & VOXEL MAN)
Fully‐automated segmentation From Clinical data as Routine work
Approach
Patient‐Specific Anatomy
Patient 3D Data (equivalent to Visible Human & VOXEL MAN)
Fully‐automated segmentation Reconstructed from Clinical data as Routine work
Shape & Location Priors in Bayesian Estimation
Computational Anatomy Models Representing Inter‐Patient Variability of Multiple Organs Atlas (training) datasets
….
….
Outline
• Our computational anatomy project: Overview
• Anatomy modeling
– Abdominal anatomy
– Musculoskeletal anatomy
– Whole‐body anatomy
• Therapeutic modeling
– Surgeon’s expertise modeling • Artificial joint surgery (Total Hip Arthroplasty: THA)
Target Abdominal Organs
Segmented Organs
Liver (brown) Spleen (blue violet)
Kidneys (pink)
Pancreas (yellow)
Gallbladder (green) Aorta and artery branches (red) • Inferior vena cava (IVC) and vein branches (cyan)
• Upper GI tract (cream yellow)
•
•
•
•
•
•
Toshi Okada, PhD
(Currently, University of Tsukuba)
Masatoshi Hori, MD
Organ segmentation via computational anatomy
Conventional framework
[Okada MICCAI 2007]
[Okada Acad Radiol 2008]
Abdominal CT
Automated
Segmentation
Target 3D data Patient anatomy
Computational Anatomy (CA) Model
Shape and location priors*
Intensity priors
Automated Construction
Training data
Manually‐traced organ shape data
Labeled DICOM data
*Inter‐Patient Anatomical Variability of Organ Shape and Location
Inter‐Patient Anatomical Variability of Organ Shape: Conventional Representation
Probabilistic Atlas (PA)
Voxel‐wise probability map of organ existence
in the normalized abdominal space
Segmentation ≒ Voxel‐wise MAP (Maximum a Posterior) estimation (Initialization is unnecessary after spatial normalization.)
[Park et al. TMI 2003]
[Okada et al. MICCAI 2007] Inter‐Patient Anatomical Variability of Organ Shape: Conventional Representation
Statistical Shape Model (SSM) (PCA of 3D shape)
Statistical constraints (inter‐patient variability) on shape and location in the normalized abdominal space
Segmentation ≒ Statistically constrained deformable model fitting ≒ Global MAP estimation (Initialization is needed.)
[Lamecker et al. 2004]
[Okada et al. MICCAI 2007] Roles of SSM from the mathematical viewpoint:
• Effective (Dimensionality) Reduction of Solution Space • Prior Probability Distributions in Bayesian Estimation
(fewer parameters for representing target shapes)
e2
Reduced mL‐d solution space for possible liver shapes (mL<<n)
eN
Prior
Likelihood
P( M | D)  P( M ) P( D | M )
P(M)
v2
v1
Posterior
Reduced mF‐d solution space for possible femur shapes (mF<<n)
e3
e1
n‐dimensional solution space representing all shapes
v1
Conventional Method [Okada MICCAI 2007] + 
A Single Organ Segmentation Method
Liver
Probabilistic Atlas (PA)
CT image
Spatial standardization
Prior
Statistical Shape Model (SSM)
[Okada MICCAI 2007] Intensity Model
Initial segmentation by PA
SSM refinement
Likelihood
Graph‐cut refinement
Segmentation result
+ 
Conventional Method [Okada MICCAI 2007] + 
A Single Organ Segmentation Method
Right kidney
Probabilistic Atlas (PA)
CT image
Spatial normalization
[Okada MICCAI 2007] Intensity Model
Prior
Statistical Shape Model (ML‐SSM)
Initial segmentation by PA
SSM refinement
Likelihood
Graph‐cut refinement
Segmentation result
+ 
Extension to multi‐organ modeling and segmentation
Organ segmentation via computational anatomy
[Okada MICCAI 2007]
Conventional framework
[Okada Acad Radiol 2008]
Abdominal CT
Automated
Segmentation
Target 3D data Patient anatomy
Computational Anatomy (CA) Model
Shape and location priors*
Intensity priors
Limitations
Automated Construction
Training data
Manually‐traced organ shape data
Labeled DICOM data
Inter‐relations among organs are not utilized.
Organ correlation graph (OCG)
Conditional shape & location prior (SSM & PA) network
P(Liver)
P(Spleen|Liver)
P(R‐Kidney|Liver)
P(L‐Kidney|Liver,Spleen)
P(Gallbladder|Liver)
P(Pancreas|Liver,Spleen)
[Okada, MICCAI 2013] PLSR (Partial Least Squares Regression)
Prediction‐based Conditional Priors
[Okada, MICCAI 2013] • Given predictor organs P, PLSR predicts the target organ shape. The prediction error E(P) is given by
E(P) = S ‐ S’(P) (S is true shape and S’(P) predicted shape.)
Training Phase
Training data
Predictor
Execution Phase
Predictor organs P
Target
…
PLSR predictor S’(P)
Predicted target shape S’
PLSR (Partial Least Squares Regression)
Prediction‐based Conditional Priors
[Okada, MICCAI 2013] • Given predictor organs P, PLSR predicts the target organ shape. The prediction error E(P) is given by
E(P) = S ‐ S’(P) (S is true shape and S’(P) predicted shape.)
• Among all possible combinations of predictor organs, predictor organs P minimizing prediction error E(P) are selected for each target organ, which define arcs of OCG (organ correlation graph).
Training Phase
Training data
Predictor
Execution Phase
Predictor organs P
Target
…
PLSR predictor S’(P)
Predicted target shape S’
Organ correlation graph
Conditional shape & location prior (SSM & PA) network
Anchor organ
P(Liver)
Predictor P(R‐Kidney|Liver)
organ
P(Gallbladder|Liver)
Predictor P(Spleen|Liver) organ
Predictor organ
P(L‐Kidney|Liver,Spleen)
P(Pancreas|Liver,Spleen)
[Okada, MICCAI 2013] Prediction‐based Statistical Atlas
Probabilistic Atlas (PA)
• Prediction error E is modeled as probabilistic atlas (PA) to generate less ambiguous PA. E = S ‐ S’ (S: True shape, S’: Predicted shape, E: Prediction error)
Conventional
Prediction‐based (Conditional)
P(Pancreas)
P(Pancreas|Liver,Spleen)
P(R‐Kidney) P(R‐Kidney|Liver)
P(Gallbladder)
P(Gallbladder|Liver )
Organ correlation graph (OCG)
Conditional shape & location prior (SSM & PA) network
Anchor organ
P(Liver)
Predictor
P(Spleen|Liver)
Predictor
Predictor
Predictor
P(R‐Kidney|Liver)
Predictor
P(Pancreas|Liver)
P(Gallbladder|Liver)
Probabilistic Atlas using Known Liver Shape
[Okada, MICCAI 2013] P(L‐Kidney|Liver)
Prediction‐based Statistical Atlas
Probabilistic Atlas (PA)
• Prediction error E is modeled as probabilistic atlas (PA) to generate less ambiguous PA. E = S ‐ S’ (S: True shape, S’: Predicted shape, E: Prediction error)
Conventional
Prediction‐based (Conditional)
Predictor: Liver
Predictor: Liver, Spleen, Kidneys
Prediction‐based Statistical Atlas
Statistical Shape Model (SSM)
• The prediction error E is also modeled using PCA in prediction‐
based SSM to obtain more constrained variability.
E = S ‐ S’ (S: True shape, S’: Predicted shape, E: Prediction error)
Conventional
P(Pancreas)
Prediction‐based (Conditional)
P(Pancreas|Liver,Spleen)
P(R‐Kidney) P(R‐Kidney|Liver)
P(Gallbladder)
P(Gallbladder|Liver )
Prediction‐based Segmentation Method
Segmentation results of predictor organs
CT image
Spatial standardization
Intensity Model
Prediction‐based PA
Initial segmentation by PA
ML‐SSM refinement
Prediction‐based SSM
Graph‐cut refinement
Segmentation result
Organ segmentation via computational anatomy
Multi‐organ interrelation modeling
Abdominal CT
Automated
Segmentation
Target 3D data [Okada Abd‐Img
WS 2011]
[Okada EMBC 2012]
Patient anatomy
Generic Computational Anatomy (CA) Models
Multi‐organ modeling inherent in anatomy
Automated
Customization
Customized Computational Anatomy (CA) Model
Target‐data specific model
Shape and location priors
Intensity priors
Automated Construction
Training data
Manually‐traced organ shape data
Labeled DICOM data
Intensity prior modeling (IM)
• In abdominal CT segmentation, we have to deal with a variety of contrast enhancement (CE) patterns.
• A new intensity prior model (IM) has to be constructed to deal with a new CE pattern.
Non (blood) contrast but oral contrast
Contrast‐enhanced
Venous phase Contrast‐enhanced
Early arterial phase
Contrast‐enhanced
Late arterial phase
Intensity prior modeling (IM)
• Supervised intensity modeling (IM) : Conventional
– Intensity prior modeling from “labeled” DICOM dataset
• A set of CT images and manual traces on them for each CE
• Unsupervised intensity modeling (IM): Proposed
– Intensity prior modeling from “unlabeled” DICOM dataset
• A set of CT images but no traces for each CE pattern – Target data specific (no training dataset for IM)
Non (blood) contrast but oral contrast
Contrast‐enhanced
Venous phase Contrast‐enhanced
Early arterial phase
Contrast‐enhanced
Late arterial phase
Organ segmentation via computational anatomy
Towards easily customizable and extendable systems
Target 3D data Abdominal CT
Automated
Segmentation
[Okada Abd‐Img
WS 2011]
[Okada EMBC 2012]
Patient anatomy
Generic Computational Anatomy (CA) Models
Multi‐organ modeling inherent in anatomy
Automated
Customization
Customized Computational Anatomy (CA) Model
Target‐data specific model
Shape and location priors
Intensity priors
Automated Construction
Training data
Manually‐traced organ shape data
Labeled DICOM data
Organ segmentation via computational anatomy
Towards easily customizable and extendable systems
Target 3D data [Okada MICCAI 2013]
Abdominal CT
Automated
Segmentation
Generic Computational Anatomy (CA) Models
Multi‐organ modeling inherent in anatomy
Automated
Customization
Customized Computational Anatomy (CA) Model
Imaging‐condition/Target‐data specific model
Shape and location priors
Automated
Customization
Automated Construction
Training data
Manually‐traced organ shape data
Patient anatomy
Unlabeled DICOM of specific imaging method/protocol
no
Intensity priors
Joint segmentation and intensity modeling
Organ segmentation via computational anatomy
Towards easily customizable and extendable systems
Target 3D data [Okada MICCAI 2013]
Abdominal CT
Generic Computational Anatomy (CA) Models
Multi‐organ modeling inherent in anatomy
Automated
Segmentation
Automated
Customization
Patient anatomy
Customized Computational Anatomy (CA) Model
Imaging‐condition/Target‐data specific model
Shape and location priors
Intensity priors
Automated Construction
Training data
Manually‐traced organ shape data
Joint segmentation and intensity modeling
Cope with Unknown Imaging Condition
Results
Experiments
• Upper abdominal CT data at two different hospitals were used.
– Non‐contrast (but artifact due to oral contrast) at NIH: 12 cases
– Venous phase at NIH: 25 cases
– Early and late arterial phases at Osaka Univ. Hospital
• Old protocol: Slice thickness 2.5 mm: 10 cases for each phase
• New protocol: Slice thickness 0.625 mm: 39 cases for each phase
– Totally, CT data of 134 cases (86 patients) with 4 different CE patterns were used.
• 2‐fold cross validation was performed. CT data with the same CE pattern as test data were not involved in any parameter tuning. • The segmentation methods were fully automated.
Non (blood) contrast but oral contrast
Contrast‐enhanced
Venous phase Contrast‐enhanced
Early arterial phase
Contrast‐enhanced
Late arterial phase
Case 1 (Osaka, Late arterial phase)
Prediction‐based CA
(Unsupervised IM)
Ground truth
Jaccard Index
Liver
Prediction
Prediction
(Unsupervised IM)
Basic
Conventional
(Unsupervised IM)
(IC-IM)
Conventional
Basic
(Supervised IM)
(Supervised IC-IM)
•
0.916
Conventional CA
(Unsupervised IM)
Conventional CA
(Supervised IM)
Spleen
R-Kidney
L-Kidney
Pancreas
Gallbladder
Aorta
IVC
0.941
0.980
0.963
0.747
0.543
0.935
0.681
0.936
0.985
0.964
0.430
0.591
0.833
0.467
0.940
0.984
0.963
0.578
0.933
0.817
0.438
Pancreas, aorta, and IVC were better segmented in the proposed prediction‐
based method than our conventional method.
Esophagus
GI‐tract
[Hirayama, 2013]
Ground truth
Duodenum
Conventional
Stomach
Prediction‐based
Prediction‐based
Conventional
Summary of abdominal multi‐organ segmentation
• Multi‐organ modeling and segmentation methods were proposed which effectively utilize the organ interrelations.
• Unsupervised intensity prior modeling combined with prediction‐based CA models can make the method adaptive to different CE patterns.
• Once key organs are segmented, other structures including GI‐
tract, vessel branches, and tumors are effectively segmented and anatomically identified.
Outline
• Our computational anatomy project: Overview
• Anatomy modeling
– Abdominal anatomy
– Musculoskeletal anatomy
– Whole‐body anatomy
• Therapeutic modeling
– Surgeon’s expertise modeling • Artificial joint surgery (Total Hip Arthroplasty: THA)
Musculoskeletal anatomy
Pelvis & Femur
Futoshi
Yokota, MS
Masaki Nobuhiko Takao, MD Sugano, MD
17 Muscles
Muscle tissues
Diseased hip joint
Unaffected hip
Primary
osteoarthritis
Secondary
osteoarthritis
( Crowe 1)
Secondary
osteoarthritis
( Crowe 2)
[Yokota, MICCAI 2013]
Collapsed
hip
100 CT data of Total Hip Arthroplasty (THA) patients:
All patients had healthy hip on one side and diseased the other
Approach of bone segmentation
[Yokota, MICCAI 2013]
1. Globally consistent initial segmentation using hierarchical hip SSM
2. Accurate segmentation of joint part using conditional SSMs
3. Final refinement by graph cut
More Robust
More Accurate
Specificity
>
Generality
<
Conditional femoral head SSM
[de Bruijne MICCAI 2006] Hierarchical hip SSM
[Okada, MICCAI 2007]
Conditional SSM
[Yokota, MICCAI 2013]
[de Bruijne MICCAI 2006] Given part
Pelvis and distal femur
Conditional
femoral head SSM
Standard
femoral head SSM
Segmentation by Hierarchical SSM fitting
• Initial rough segmentation of bone regions using simple thresholding where joints part is not separeted.
[Yokota et al. MICCAI 2009]
Segmentation by Hierarchical SSM fitting
• Coarse fine fitting of hierarchical SSM is performed.
[Yokota et al. MICCAI 2009]
Segmentation by Hierarchical SSM fitting
• Coarse fine fitting of hierarchical SSM is performed.
– Initial fitting of combined pelvis and femur SSM
– Subsequent fitting of pelvis & femur SSMs with consistency constraint
– Fitting and edge updating are repeated.
[Yokota et al. MICCAI 2009]
Results
Red: pelvis Green: femur
Primary
osteoarthritis
Secondary
osteoarthritis
( Crowe 1)
Secondary
osteoarthritis
( Crowe 2)
Collapsed hip
CT image
Ground truth Independent SSMs Conditional SSM
Musculoskeletal anatomy
Pelvis & Femur
Muscle tissues 17 Muscles
Different patients
Hierarchical multi‐atlas label fusion
[Yokota, CAOS 2012]
Automatically segmented patient label images
Skin, pelvis & femur
Initial
bone & skin segmentation
Target CT image
Second stage: 5 selected muscle segmentation
First stage: Muscle tissue segmentation
….
38 datasets
Label images for spatial normalization (cancelation of variability)
Muscle tissue ….
….
5 selected muscles
Final stage: 17 muscle segmentation
Best Technical Paper Award
Automatically segmented patient label image
Final
segmentation
….
38 datasets
Atlas datasets
2 datasets
Intensity images
Label images for label fusion
Musculoskeletal segmentation
Results
[Yokota, CAOS 2012]
Front views
Back views
Three‐stage
Two‐stage
Single‐stage
(1.9 mm error)
(3.0 mm error)
(4.1 mm error)
Musculoskeletal segmentation
Results
Original CT images
Ground truth
[Yokota, CAOS 2012]
Three‐stage
Two‐stage
Single‐stage
(1.9 mm error)
(3.0 mm error)
(4.1 mm error)
Outline
• Our computational anatomy project: Overview
• Anatomy modeling
– Abdominal anatomy
– Musculoskeletal anatomy
– Whole‐body anatomy
• Therapeutic modeling
– Surgeon’s expertise modeling
• Artificial joint surgery (Total Hip Arthroplasty: THA)
MEXT Grant‐in‐aid for Scientific Research, Japan
Computational Anatomy for Computer‐Aided Diagnosis and Therapy
Sep 2009 ‐ Mar 2014
Fund: $10 million
Principal Investigator: Prof. Hidefumi Kobatake
(TUAT: Tokyo University of Agriculture & Technology)
Tokyo
Eight core groups
Gifu
Basic theories and technologies
Yamaguchi Osaka
TUAT
Nagoya
Application systems
Kyushu
Tokushima
Clinical evaluations
Locations of eight core groups
http://www.comp‐anatomy.org/
Google search by “computational anatomy”.
One of our main goals: Complete understanding of whole‐body CT images
Example of collaboration
Abdominal module (Tokyo & Osaka)
Prof. Masutani
(Univ. of Tokyo
Currently, Hiroshima City Univ.)
Abdominal Bounding‐box Localization
Landmark Localization
Abdominal Multi‐organ Segmentation
Random forest regression
Training data
,
Tokyo
,…,
,
,…,
Tokyo & Osaka
Osaka
Musculoskeletal modules
(Gifu, Osaka
Tokushima) Prof. Niki
(Univ. Tokushima)
Lung module (Tokushima)
Prof. Fujita
(Gifu Univ.)
Vessel modules (Nagoya, Osaka)
Prof. Mori
(Nagoya Univ.)
Non‐contrast CT
Fully‐automated Segmentation
Outline
• Our computational anatomy project: Overview
• Anatomy modeling
– Abdominal anatomy
– Musculoskeletal anatomy
– Whole‐body anatomy
• Therapeutic modeling
– Surgeon’s expertise modeling
• Artificial joint surgery (Total Hip Arthroplasty: THA)
Cup planning of mildly and severely diseased pelvises: Our problem
Mildly diseased case
•
Severely diseased case
The position and size of the acetabular cup should be basically determined so as to recover the original anatomy of the acetabulum. Cup planning of mildly and severely diseased pelvises: Our problem
Mildly diseased case
•
•
Severely diseased case
The position and size of the acetabular cup should be basically determined so as to recover the original anatomy of the acetabulum. Although it is not so difficult to predict the original anatomy for mildly diseased case, it is somewhat difficult for severely diseased acetabulum due to its severe deformation and shift.
Bone‐Implant Statistical Model (1)
Prior probability of likely spatial relations between patient bone and implant
Surgical Plan Database
Otomaru et al.
CAOS 2009
Pelvis‐Cup Statistical Model P(Xpelvis, Xcup)
Statistical Shape Model (SSM)
Statistical Analysis
Cup Plan
Patient Pelvis Shape Data: D
Automated Planning
Maximize P(Xpelvis, Xcup)P(D|Xpelvis)
Maximum a Posterior (MAP) Estimation
Bone‐Implant Statistical Model (2)
Prior probability of likely spatial relations between patient bone and implant
Surgical Plan Database
Otomaru et al.
Med Image Anal
2012
Femoral Cavity ‐ Stem Statistical Model P(Xfemur, Xstem)
Statistical Distance Map (SDM)
penetration 0 gap
Stem Plan
Patient Femoral Cavity Shape Data: D
Automated Planning
Maximize P(Xfemur, Xstem)P(D|Xfemur)
Maximum a Posterior (MAP) Estimation
Summary of this talk
• Statistical shape models (SSMs) and other statistical atlas representation incorporating interrelations among multiple organs (structures) are presented.
• Their applications were demonstrated to
– Abdominal organs
– Musculoskeletal structures
– Bone implant surgical planning
• These problems are formulated as MAP estimation based on Bayes theorem, where SSMs are regarded as prior probability distributions.
Thank you! Sunrise at Yakushi Temple, Nara, Japan