Atlas based liver segmentation using nonrigid
registration with a B-spline transformation
model
Pieter Slagmolen1,2 , An Elen1 , Dieter Seghers1 , Dirk Loeckx1 ,
Frederik Maes1 , and Karin Haustermans2
1
Medical Image Computing (ESAT/PSI), Faculties of Medicine and Engineering,
UH Gasthuisberg, Herestraat 49, B-3000 Leuven, Belgium.
2
Department of Radiation Oncology, UH Gasthuisberg, Leuven, Belgium
[email protected]
Abstract. Liver segmentation is an important step for the therapeutic
decision making in liver surgery. However, manual segmentation is timeconsuming and tedious and so the need for accurate and robust automatic
segmentation methods for clinical data arises.
In this work an atlas in combination with nonrigid registration is used
to segment the liver in actual clinical CT images. First, the atlas is
built on twenty training images using nonrigid registration with a novel
surface distance penalty. Next, this atlas is nonrigidly registered to ten
test images. Currently, the user interaction is limited to the initialization
of a rigid registration and to the definition of a region of interest for the
nonrigid registration. Future work will focus on replacing the remaining
user interaction with fully automatic procedures.
Results are promising with an average overlap error of 10.4% and an
average RMS distance of 5.0mm for the ten test images. Errors occur
mainly at sites where the atlas is ill-defined such as the border between
the heart and the liver.
1
Introduction
Computer aided planning of liver surgery can greatly improve the choice of a
suitable treatment strategy [1]. Liver segmentation is a crucial step for this computer aided planning. In a more general context, segmentation still is a bottleneck
for the breakthrough of many computer assisted procedures because it remains
tedious, time-consuming and subjective. New algorithms are constantly being
developed and published. However, the step from research to clinical practice
has proven to be very difficult because algorithms need to perform robustly and
accurately on actual clinical data [2].
In this work a clinical dataset of ten liver patients is automatically segmented.
We use a method based on an atlas and nonrigid registration similar to the one
used in [3]. The atlas is built from a training database of 20 images with manual
segmentations.
First, an overview will be given of how the atlas was built. Next, the registration of the atlas to the individual images is described. Finally, segmentation
results are given for the training images and for the test images.
2
2.1
Materials and Methods
Images and Segmentations
A database of thirty CT images is available. Twenty randomly selected images
are used for training while ten other test images are used for validation.
All CT images are enhanced with contrast agent and scanned in the central
venous phase on a variety of scanners (different manufacturers, 4, 16 and 64
detector rows). As it is CT, all datasets have been acquired in transversal direction. The pixel spacing varies between 0.55 and 0.8mm, the inter-slice distance
varies from 1 to 3mm. There is no overlap between neighbouring slices.
All segmentations were created manually by radiological experts, working
slice-by-slice in transversal view. The first tool they employed was an intensitybased region grower. In case of leakage, these leaks were removed by drawing
manual cut-lines. The segmentation is defined as the entire liver tissue including
all internal structures like vessel systems, tumours etc. In general, a vessel counts
as internal if it is completely surrounded by liver tissue (in the transversal view).
The large vessels that enter the liver (V.Cava and portal vein) are segmented
in the part which is enclosed by liver tissue, i.e. as the convex hull of the liver
shape in that area. The segmentations are available as binary maps.
2.2
Atlas Building
Affine Registration and Resampling First, one image with an average liver was
chosen from the training set (image 1 in training set). All other images and
their corresponding segmentations were affinely registered and resampled to this
image. Affine registration was performed using the MIRIT software and is based
on the maximization of mutual information [4]. However, due to the diversity of
the images, initialization of the affine registration is needed and therefore images
were manually shifted to provide an initial, rough alignment. Our future work
will contain the automatization of this initialization. The resulting resampled
images Ii and segmentations, Si all have a size of 512 x 512 x 183 voxels with a
voxelsize of 0.74 x 0.74 x 1.5 mm.
Nonrigid registration Nonrigid registration is performed using a B-spline transformation model [5, 6]. A grid of mesh control points is positioned over the
reference image and the displacements of these control points act as parameters
for the deformation field. A gradual refinement of the grid allows more local deformations to be modelled. Again, mutual information is used as the similarity
measure.
For the atlas building two penalty terms are optimized along with the mutual information. First, the smoothness penalty will disfavour unlikely transformations by promoting a smooth transformation field. Next, a surface distance
penalty will minimize the distance between the segmentations on reference and
floating image. This penalty is described in detail in the next paragraph.
The cost function Ec to optimize is a linear combination of the mutual information Emi and the two penalty factors Esm and Esd , each with their own
weighting factor.
Ec = ωmi Emi + ωsm Esm + ωsd Esd
Optimization is carried out using a multiresolution approach. Starting from
downscaled images and a coarse mesh, the image and/or mesh resolution are
increased at each stage. Within each stage, the optimal set of parameters is
sought using a limited memory quasi Newton optimizer.
Surface Distance Penalty We introduce a new registration penalty, which penalizes the remaining distance between surfaces of corresponding structures in
reference and floating image. First, the known surfaces in the reference and floating images are approximated with a triangular mesh using the marching cubes
algorithm [7]. The thus obtained mesh points in the reference image are considered as a dense sampling of the reference image surface. At each optimization
step of the registration, the inverse transformation is applied to the coordinates
of these samples.
At the same time, a polyharmonic Radial Basis Function (RBF) is fitted
through the mesh points of the floating image surface [8]. The used fast RBF
method constructs a signed distance function, which evaluates to zero in the mesh
points, to one in a set of points outside the surface at unit distance along the
normal of each mesh triangle and to minus one in a corresponding set of points
inside the surface. This signed distance function of the floating image surface
is evaluated in the transformed samples of the reference image surface. The
mean squared value of these results is considered as a measure for the remaining
distance between the two surfaces and is minimized during optimization.
Mean Morphology For each of the training images Ii a mean deformation field
Ti is determined. Each image from the training set (floating image) is registered
to all other training images (reference image).
Ii→j = Tij (Ii )
For all 20 training images, 19 registrations were calculated resulting in a
total of 380 registrations. Consequently, for each image Ii , 19 deformation fields
Tij are available that define its transformation to any other training image.
Averaging these 19 deformation fields for each image gives a mean deformation
map Ti for all training images.
Ti =
1 X
Tij
n−1
j6=i
All images and segmentations are then deformed with their corresponding
mean deformation map to produce 20 deformed images I i and 20 deformed
segmentations S i .
I i = Ti (Ii )
S i = Ti (Si )
Mean Intensities The 20 deformed images each are biased towards their original
image. To overcome this bias, all images and segmentations are averaged thus
producing a single atlas image I Atlas with the corresponding atlas segmentation
S Atlas .
n
I Atlas =
1X
Ii
n i=1
n
S Atlas =
1X
Si
n i=1
To overcome possible cut-off problems at the resampling with some images,
empty slices are added to the cranial side of the atlas to produce an atlas image
that contains 512 x 512 x 210 voxels with a 0.74 x 0.74 x 1.5 voxelsize. The
resulting atlas is shown in figure 1.
Fig. 1. Atlas image I Atlas and corresponding Atlas segmentation S Atlas
2.3
Atlas Registrations
The atlas built in the previous section is used to segment the training and
test images by nonrigidly registering the atlas image to all these images. In
the following, the image we want to segment will be referred to as the target
image.
Affine Registration and Resampling First, the target image is affinely registered
and resampled towards the atlas image after manual initialization to overcome
large differences in imaging position. The resampling ensures that the parameters
for the nonrigid registration such as the multiresolution settings are reproducible
and can be kept constant for all possible target images.
Region of interest To decrease registration time and to focus the registration on
the region of the liver, a region of interest (ROI) is defined for the target image.
This region of interest is chosen around the liver and is currently manually
defined. The segmentation also works without the ROI but this makes the next
step significantly slower. An automatic detection of the ROI is possible since it
doesn’t need to be very precise. However, this has not yet been implemented.
Atlas Segmentation The nonrigid registration used to register the atlas image
is the same as the one used to construct the atlas but without the surface distance penalty which obviously can’t be used for segmentation. The resulting cost
function is:
Ec = ωmi Emi + ωsm Esm
Registration results were best when the atlas was deformed, and thus the
target image remained fixed. The reason behind this is that the atlas image is
much smoother than the target image. If we would perform the registration the
other way round, the nonrigid transformation field would be tempted to wipe
out smaller image details in the target image, trying to make it as smooth as
the atlas image.
After nonrigid registration, the atlas segmentation is deformed with the found
deformation field and it is thresholded at 50% of the maximum value. To finish
the segmentation a morphological opening operation is performed and possible
unconnected segments are removed. Finally, the segmentation is resampled back
to the original test image.
3
3.1
Results
Atlas Building
The results of the nonrigid registrations used for the atlas building are shown
in Table 1. The correspondence between the segmentations is very high in most
cases. Some registrations failed due to folding induced by the surface distance
penalty. However, this occurs in very few registrations and thus their influence
on the mean deformation field is minimal.
The evaluation metrics used here are the volume overlap error, the volume
difference, the average surface distance, the RMS surface distance and the maximum surface distance [9]. The same registrations but without surface registration
penalty yielded much poorer results with an average overlap error of 14.5% and
volume difference of 8.95%. Thus, the resulting atlas will be more accurate when
using the surface distance penalty.
Table 1. Results of the comparison metrics for the atlas building [9]. These parameters
have been calculated on a total of 380 registrations.
Overlap Error Volume Diff. Avg Dist. RMS Dist. Max. Dist.
[%]
[%]
[mm]
[mm]
[mm]
Average
7.30
1.40
1.52
3.70
35.28
Standard Dev
4.07
5.11
1.55
4.07
21.64
Median
6.00
0.43
1.02
2.22
28.36
3.2
Training Set
The results shown in Table 2 are calculated by segmenting the trainig images
with the proposed method and comparing these automatic segmentations with
the manual segmentations made by an experienced radiologist.
3.3
Testing Set
The results shown in Table 3 are calculated by segmenting the test images with
the proposed method in comparison with the manual segmentations. The results
on the training images are slightly better than the results on the test images.
This is probably because the atlas contains information about each training
image and not about the test images. Figure 2 gives some visual examples of our
segmentations in an easy, intermediate and difficult case. In the difficult case (test
image 3), our method was unable to include the tumour in the segmentation.
This is reflected by the score for this segmentation which is lower than the
average (see Table 3).
Table 2. Results of the comparison metrics [9] for the training database
Training Overlap Error Volume Diff. Avg Dist. RMS Dist. Max. Dist.
Image
[%]
[%]
[mm]
[mm]
[mm]
1
9.71
-1.53
1.84
3.69
35.50
2
7.15
-3.72
1.18
2.12
18.00
3
5.68
-1.04
1.00
2.00
24.23
4
8.41
6.24
1.82
5.18
70.08
5
7.56
5.44
1.16
3.22
34.81
6
10.44
8.05
2.26
5.01
44.79
7
6.08
2.02
0.94
1.74
22.71
8
8.66
-3.15
1.68
3.16
33.26
9
6.71
3.59
1.08
2.45
23.02
10
15.72
17.60
2.84
5.08
28.44
11
11.01
4.66
2.68
6.50
53.12
12
5.91
4.47
1.12
2.71
36.10
13
12.13
10.30
2.94
7.26
51.36
14
15.36
-6.10
3.37
6.90
53.85
15
7.63
6.97
1.58
3.37
35.12
16
5.43
1.97
1.13
2.31
28.12
17
6.11
4.37
0.98
2.00
21.05
18
6.19
3.68
1.35
3.47
36.77
19
8.28
0.51
2.48
7.84
66.02
20
4.93
-0.69
0.78
1.44
14.18
Average
8.46
3.18
1.71
3.87
36.53
Table 3. Results of the comparison metrics [9] and corresponding scores for all ten
test cases.
Dataset Overlap Error Volume Diff. Avg. Dist. RMS Dist. Max. Dist. Total
[%] Score [%] Score [mm] Score [mm] Score [mm] Score Score
1 9.2
64
5.2
72
2.1 48
6.0 17 49.8 34
47
2 13.1
49 10.4
45
2.4 40
5.9 18 48.3 36
38
3 14.5
43 -5.5
71
2.8 30
6.4 11 49.4 35
38
4 5.9
77
2.1
89
0.9 77
1.8 76 12.7 83
80
5 6.8
73 -0.5
98
1.2 70
2.4 67 21.3 72
76
6 8.9
65 -4.0
79
2.1 47
6.5 10 59.6 22
45
7 15.1
41 14.4
23
3.4 15
9.6
0 70.5
7
17
8 8.9
65
4.7
75
1.8 55
4.6 36 37.4 51
56
9 12.0
53
8.2
56
1.6 60
3.5 51 24.5 68
58
10 9.9
61
1.9
90
1.6 60
3.3 54 31.0 59
65
Average 10.4
59
3.7
70
2.0 50
5.0 34 40.5 47
52
3.4
Registration Time
Table 4 gives the average time needed to perform a single segmentation (mean
of the segmentation time for the ten test images). The largest portion of time
is spent on calculating the nonrigid registration. About two minutes of user
Fig. 2. From left to right, a sagittal, coronal and transversal slice from a relatively easy
case (1, top), an average case (4, middle), and a relatively difficult case (3, bottom). The
outline of the reference standard segmentation is in red, the outline of the segmentation
of the method described in this paper is in blue. Slices are displayed with a window of
400 and a level of 70.
interaction are required for each image. Removing the region of interest from
the registration would decrease this manual interaction but consequently would
increase registration time significantly. The manual initialization of the affine
registration could be made obsolete by making the affine registration more robust
to account for large variations in imaging position.
4
Discussion and Future Work
A segmentation framework was presented based on nonrigid registration with an
atlas image. Results show that our segmentation is able to segment all images
and that it doesn’t fail on any of the proposed images. For the 20 training images
an average volume overlap error of 8.5% and an RMS surface distance of 3.9mm
are obtained. For the 10 test images an average volume overlap error of 10.4%
and an RMS surface distance of 5.0mm are obtained.
Table 4. Time needed to perform a full segmentation of a single image.
Step
Average time
Manual Initialization
1 min
Affine Registration
30 sec
Resampling
30 sec
Definition Region of Interest
1 min
Nonrigid Registration
59 min
Binary Operations
1.5 min
Resampling to Original
30 sec
Total Time
64 min
A few shortcomings of this method still need to be solved. First, some registrations in the atlas building process still fail due to folding induced by the
surface distance penalty. This could be solved by decreasing the weigth of the
surface distance penalty in the first multiresolution stages. Even though the influence of a single failed registration is limited in the atlas due to the averaging of
the deformation field, a slight increase in accuracy for the atlas can be expected.
Next, manual initialization of the affine registration and manual definition
of a region of interest for the segmentation make this a semi-automatic method
rather than a fully automatic one. Both initializations can probably be done
automatically in the future but are not yet implemented.
Fig. 3. Similar intensities in the target image can cause the registration to fail locally.
This is mainly the case at the border between the heart and the liver (left). Also, in
one case (test image 7) a part of the bowel was mistaken for a part of the liver. The
outline of the manual segmentation is in red, the outline of our segmentation is in blue.
Slices are displayed with a window of 400 and a level of 70.
Finally, when looking at the images with the corresponding segmentations in
Figure 2 and Figure 3, an additional shortcoming of this method becomes clear.
Small differences in intensities can cause it to fail in certain regions. In Figure 2,
a tumor is not included in the segmentation where it should have been. In Figure
3, either a part of the heart or a part of the bowel are incorrectly included in the
segmentation. This occurs mainly where the liver and a surrounding organ with
similar intensity lie closely together as is the case in the examples shown. Also,
the atlas is ill-defined at the border between the heart and the liver (see Figure
1) and this obviously limits the information available for registration in this
region. To overcome these problems statistical information about the shape and
intensities of the liver could be implemented as an additional penalty. This would
discourage unlikely shape changes or intensity profiles. Our future work will try
to extend our atlas approach with this statistical information as we expect this
could diminish the problems we encounter because they usually involve unlikely
shape (liver-heart boundary) or intensity (tumour) information.
References
1. Meinzer, H.P., Thorn, M., Cardenas, C.E.: Computerized planning of liver surgery–
an overview. Computers & Graphics 26(4) (2002) 569–576
2. Heimann, T., Wolf, I., Meinzer, H.P.: Active shape models for a fully automated
3d segmentation of the liver - an evaluation on clinical data. In Larsen, R., Nielsen,
M., Sporring, J., eds.: MICCAI (2). Volume 4191 of Lecture Notes in Computer
Science., Springer (2006) 41–48
3. Seghers, D., D’Agostino, E., Maes, F., Vandermeulen, D., Suetens, P.: Construction
of a brain template from MR images using state-of-the-art registration and segmentation techniques. In Barillot, C., Haynor, D.R., Hellier, P., eds.: Lecture notes in
computer science. Volume 3216 of Lecture Notes in Computer Science., Springer
(2004) 696–703
4. Maes, F., Collignon, A., Vandermeulen, D., Marchal, G., Suetens, P.: Multimodality
image registration by maximization of mutual information. IEEE Transactions om
Medical Imaging 16(2) (1997) 187–198
5. Loeckx, D., Maes, F., Vandermeulen, D., Suetens, P.: Nonrigid Image Registration Using Free-Form Deformations with a Local Rigidity Constraint. In Barillot,
C., Haynor, D.R., Hellier, P., eds.: MICCAI (1). Volume 3216 of Lecture Notes in
Computer Science., Springer (2004) 639–646
6. Loeckx, D.: Automated nonrigid intra-patient image registration using B-splines.
PhD thesis, K.U.Leuven, http://hdl.handle.net/1979/298 (2006)
7. Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3D surface construction algorithm. In: SIGGRAPH ’87: Proceedings of the 14th annual conference
on Computer graphics and interactive techniques. Volume 21., New York, NY, USA,
ACM Press (1987) 163–169
8. Carr, J.C., Beatson, R.K., Cherrie, J.B., Mitchell, T.J., Fright, W.R., McCallum,
B.C., Evans, T.R.: Reconstruction and Representation of 3D Objects with Radial
Basis Functions. In: SIGGRAPH ’01: Proceedings of the 28th annual conference on
Computer graphics and interactive techniques, New York, NY, USA, ACM Press
(2001) 67–76
9. Gerig, G., Jomier, M., Chakos, M.: Valmet: A New Validation Tool for Assessing
and Improving 3D Object Segmentation. In Niessen, W.J., Viergever, M.A., eds.:
MICCAI. Volume 2208 of Lecture Notes in Computer Science., Springer (2001)
516–523