Automated Identification of
Filaments in Cryo-electron
Microscopy Images
Yuanxin Zhu, Bridget Carragher,
David Kriegman, Ronald A. Milligan,
and Clinton S. Potter
(Submitted to Journal of
Structural Biology)
Date Issued: July 16, 2001
The Beckman Institute
Imaging Technology Group
Technical Report 01-012
Copyright © 2001 Board of Trustees of the University of Illinois
The Beckman Institute for Advanced Science and Technology
Imaging Technology Group
405 N Mathews
Urbana, IL 61801
[email protected]
http://www.itg.uiuc.edu
Automated Identification of Filaments in Cryo-electron Microscopy Images
Yuanxin Zhu, Bridget Carragher, #David J. Kriegman, *Ronald A. Milligan, Clinton S. Potter
Beckman Institute, and #Department of Computer Science, University of Illinois at UrbanaChampaign, Urbana, Illinois 61801
*The Scripps Research Institute, La Jolla, California
Corresponding Author:
Yuanxin Zhu
Beckman Institute for Advanced Science and Technology
405 N. Mathews Avenue
Urbana, Illinois 61801
Tel: 217-265-0875; Fax: 217-244-6219; e-mail: [email protected]
1
Abstract
Since the foundation for the three-dimensional image reconstruction of helical objects from
electron micrographs was laid more than 30 years ago, there have been sustained developments
in specimen preparation, data acquisition, image analysis, and interpretation of results.
However, the boxing of filaments in large numbers of images—one of the critical steps toward
the reconstruction at high resolution—is still constrained by manual processing even though
interactive interfaces have been built to aid the tedious and sometimes inaccurate boxing process.
This article describes an accurate approach for automated detection of filamentous structures in
low-contrast images acquired in defocus pairs using cryo-electron microscopy. The performance
of the approach has been evaluated across various magnifications and at a series of defocus
values using tobacco mosaic virus (TMV) preserved in vitreous ice as a test specimen. By
integrating the proposed approach into our automated data acquisition and reconstruction system,
we are now able to generate a three-dimensional map of TMV to approximately 10Å resolution
within 24 hours of inserting the specimen grid into the microscope.
Keywords: cryo-electron microscopy, automation, three-dimensional reconstruction of helical
objects, perceptual organization, phase correlation
2
1. Introduction
An important role of cryo-electron microscopy (cryo-EM) is to provide unique information about
biological structure from the molecular to the cellular level and thus to yield insight into
biological function. As pointed out by Nogales and Grigorieff (2001), the most promising
technique for the structural characterization of large molecular machines is EM and image
reconstruction. One of the principal bottlenecks to cryo-EM is the very large number of images
that must be acquired and processed for the structural analysis. This requirement results from the
necessity for using extremely low doses of electrons (10e/Å2) to image the specimen in order to
avoid beam induced radiation damage. As a result, the acquired images have a very low signalto-noise ratio and in order to reconstruct a three-dimensional electron density map from the twodimensional projections, a very large number of images must be averaged together. The exact
number of images required to complete a three-dimensional reconstruction of a macromolecule
depends on the size of the macromolecule and the desired resolution. However, it is generally
agreed that in order to interpret a structure to atomic resolution, data from possibly hundreds of
thousands of copies of the macromolecule must contribute to the average (Henderson, 1995).
This in turn requires the collection of thousands to tens of thousands of electron micrographs.
One of the current constraints in the field is the manual data acquisition and analysis methods
that are slow and labor-intensive.
Using helical specimens as a driver, we have been developing an integrated system for cryo-EM
that would allow a three-dimensional electron density map to be automatically generated within
hours of a specimen being inserted in the microscope and with no manual intervention required
3
by an operator. This promises to improve both the quality and quantity of data collection and
processing. Towards this goal, we have already automated the acquisition of images from the
electron microscope (Potter et al, 99; Carragher et al., 2000). The system performs all the
operations normally performed by an experienced microscopist including identifying suitable
areas of vitreous ice at low magnification, determining the presence and location of specimen on
the grid, adjusting imaging parameters under low dose conditions and acquiring images at high
magnification to either film or a digital camera. The system can acquire up to 1000 images a day
and the overall performance of the automated system is equivalent to that of an experienced
microscopist.
More than 30 years ago, DeRosier and Moore (1970) laid the foundation for the reconstruction of
three-dimensional images of biological structures with helical symmetry from electron
micrographs. Since then, various research groups have developed a variety of tools to aid the
image processing of helical objects (Unwin and Klug, 1974; Stewart et al., 1981; Wagenknecht
et al., 1981; Egelman, 1986; Milligan and Flicker, 1987; Bluemke et al., 1988; Carragher et al.,
1988; Bremer et al., 1994; Whittaker et al., 1995; Owen et al., 1996). Several interactive
interfaces (for example, the IMGBOX, Owen et al., 1996) have been built for extracting (boxing
out regions of) filaments from large numbers of micrographs where multiple filaments in various
orientation and length may be present. However these are essentially manual methods and
limited both in the time required for the process and their accuracy.
Integration of the automated data acquisition with the three-dimensional reconstruction process
requires an automation of the boxing process, i.e. an automated approach for the identification of
4
the helical filaments in two-dimensional projection images. This automation is particularly
promising since the boxing step is often the bottleneck for high-resolution reconstruction
(Nogales et al., 2001). To our best knowledge, no technique for this purpose has been reported
so far. On the other hand, research on automated selection of single particles is very active. A
number of methods for automatically identifying single particle images have been proposed (van
Heel 1982; Frank and Wagenknecht, 1984; Harauz and Fong-Lochovsky, 1989; Lata et al, 1995;
Thuman-Commike and Chiu, 1995; Martin et al, 1997; Ludtke et al, 1999). Most of them are
based on the technique of template matching, namely first generating a rotationally averaged
reference particle image (template) and then locating particles by seeking peaks in the space
formed by cross-correlating the template with the entire image.
In order to improve
performance, Ludtke et al (1999) explored using multiple reference templates. Besides its heavy
computational requirement, cross-correlation is sensitive to background noise. As Martin et al
(1997) mentioned, it is commonly agreed that it is very difficult to identify peaks in the crosscorrelation maps computed from low-contrast images acquired using cryo-EM.
This paper presents an accurate approach for automated identification of filamentous structures
in low-dose, low-contrast images acquired in defocus pairs using cryo-EM. Images are normally
acquired in defocus pairs from any chosen specimen area using the Leginon system (Carragher et
al, 2000; Potter et al, 1999). The first image is acquired at very near to focus (NTF) conditions
(e.g., -0.3 ~ -0.15µm, where “-” and “+” indicate under-focus and over-focus respectively) and
the second one at a farther from focus (FFF) conditions (e.g., -3 ~ -2µm), see Fig. 2, for example.
The time interval between the two exposures is less than 20s due to the time required to read out
the digital image from the camera. There are at least two major advantages of using a defocus
5
pair of images. First, by combining the two images in the defocus pair, relatively high contrast at
both low and high spatial frequencies can be attained. Second, the moderately strong lowresolution signals in the FFF images make it possible for us to develop algorithms to identify
filamentous structures automatically. The idea of using a defocus pair of images has been
explored by several other researchers. For example, Fuller (Fuller 1987) and Schrag et al. (1989)
used the “out-of-focus” image for determination of the virus orientation and the “in-focus” image
for the reconstruction. Zhou and Chiu (Zhou and Chiu, 1993; Zhou, 1995) used a focal pair of
images to obtain relatively high contrast at both low and high spatial frequencies. They also
exploited the center and distance information determined for the “far-from-focus” micrograph to
aid particle selection in the “close-to-focus” micrograph.
Our approach begins with a three-level perceptual organization algorithm to detect filaments in
the FFF images, see Fig. 1 for a schematic overview. Because the NTF images are acquired very
close to focus (e.g. -0.15µm), the contrast in such images is extremely low. On the other hand,
the NTF image in a defocus pair covers almost the same specimen area as the FFF image. The
relative distance between filaments within the NTF image should be the same as that in the FFF
image. We therefore circumvent directly identifying filaments in the NTF image and instead
detect filaments in the FFF image which is then aligned using phase correlation techniques with
the NTF image. The filaments in the NTF image are then extracted using the coordinates that are
identified in the FFF image and shifted according to the result of the alignment.
The proposed approach was tested and evaluated by applying it to high magnification
micrographs of tobacco mosaic virus (TMV) preserved in vitreous ice. Performance is assessed
6
in terms of the percentages of correctly identified filaments, false alarms (incorrectly identified
areas), and false dismissals (unboxed filaments). The calculation is based on a comparison to the
filaments that would have been selected by an experienced operator. False dismissals do not
constitute a serious type of error as long as there are large numbers of filament images available
and the percentage of falsely dismissing is low. This is consistent with algorithms for identifying
single particles (Martin et al., 1997). False alarms do pose a severe error since they are not
projections of real helical filaments and thus introduce additional error into the reconstruction
process. Therefore, a preferred filament detection approach should produce a high percentage of
correctly identified filaments while maintaining a low percentage of false dismissals and a very
low percentage of false alarms.
2 Materials and Methods
2.1 Image acquisition
Our automated acquisition system (Potter et al., 1999, Carragher et al., 2000) was used to record
high magnification images of helical filaments of TMV preserved in vitreous ice. TMV is a
well-characterized helical virus (Jeng et al., 1989). It is often used as a transmission electron
microscopy (TEM) standard for calibration of magnification and determining resolution and
provides a good test specimen for evaluating a new technique. Defocus pairs of images were
collected using a Philips CM200 TEM equipped with a CCD camera. At the beginning of our
work, images were acquired to a 1K × 1K Gatan MSC CCD camera at the magnification of
66,000x. At this magnification, the pixel size is 2.8Å and the accumulated dose for high
magnification image area was about 10e/Å2. An example defocus pair of images acquired in this
7
way is shown in Fig. 2a and 2b. The images show projections of TMVs whose structures form
helical filaments 18 nm in diameter and 300 nm in length. Later, we have experimented with
images acquired to a newly installed 2K × 2K Tietz CCD camera. With the 2K × 2K camera,
defocus pairs of images were acquired at the magnification of either 66,000x or 88,000x. In the
case of 88,000x, the pixel size is 1.5Å and the accumulated dose for high magnification image
area was about 12e/ Å2.
2.2 Introduction to perceptual organization
In humans, perceptual organization is the ability to immediately detect relationships such as
collinearity, parallelism, connectivity, and repetitive patterns among image elements (Lowe,
1985). Researchers in human visual perception, especially the Gestalt psychologists (Koffka,
1935; Kohler, 1947), have long recognized the importance of finding organization in sensory
data. The Gestalt law of organization states the common rules by which our visual system
attempts to group information (Koffka, 1935; Kohler, 1947]). Some of these rules include
proximity, similarity, common fate, continuation and closure. In computer vision, as Mohan and
Nevatia (Mohan and Netvatia, 1992) proposed, perceptual organization takes primitive image
elements and generates representations of feature groupings that encode the structural
interrelationships between the component elements. Many perceptual organization algorithms
have been proposed (Lowe, 1987; Mohan and Nevatia, 1992; Sarkar and Boyer, 1993, Shi and
Malik, 2000), and surveys of these works can be found (Palmer, 1983; Lowe, 1985; Sarkar and
Boyer, 1994; Williams and Thornber, 1996).
2.3 Detecting filaments in FFF images
8
We will model the filament detection process in FFF images as a three-level perceptual
organization based on the classificatory structure proposed by Sarkar and Boyer (1994) who
arrange the features to be organized into four categories: signal, primitive, structural, and
assembly. At each level, feature elements that are not grouped into higher-level features are
discarded. Briefly, at the signal level, we use a Canny edge detector (Canny, 1986) to detect
weak filament boundaries.
A Hough transform followed by detecting end points of line
segments and merging collinear line segments is developed at the primitive level to organize
discontinuous edges into line segments with a complete description. At the structural level, line
segments are grouped into filamentous structures, areas enclosed by pairs of line segments, by
seeking parallelism and utilizing high-level knowledge. In addition, statistical method is used to
separate two filaments if they are joined together end to end.
Edge detection
For noisy images, it is necessary that at the signal level, edge detection be used to enhance lowlevel feature elements by organizing interesting boundary points into edge chains and
suppressing the possible effects of noise on higher-level perceptual organization. Edge detection
is typically a three-step process including noise smoothing, edge enhancement and edge
localization (Trucco and Verri, 1998). We adopt probably the most widely used edge detector in
today’s computer vision community, the Canny edge detector (Canny, 1986), to detect these
weak boundaries. Canny edge detection uses linear filtering with a Gaussian kernel to smooth
noise and then computes the edge strength and direction for each pixel in the smoothed image.
Candidate edge pixels are identified as the pixels that survive a thinning process called nonmaximum suppression, followed by hysteresis thresholding on the thinned edge magnitude
9
image using a low threshold and a high threshold. All candidate edge pixels below the lower
threshold are labeled as a non-edge and all pixels above the low threshold, which can be
connected to any pixel above the high threshold through a chain of edge pixels, are labeled as
edge pixels.
Selecting the input parameters of the Canny algorithm is a critical step because the resulting edge
quality varies greatly with the choice of parameters. At the signal level, the input parameters
were selected to optimize the quality of edges for the purpose of higher-level perceptual
organization. As Heath et al. (1997) described, three parameters need to be specified for the
Canny edge detector. The first is the standard deviation of the Gaussian filter specified in pixels.
The second and third parameters are respectively the low and high thresholds required by the
hysteresis thresholding. The best single overall parameter set for the Canny edge detector,
identified by the performance measure method and evaluated on a wide variety of real images
(Heath et al., 1997), does not produce a successful result as applied to images in our case. To
select the input parameters suitable for the highly noisy, low-contrast cryo-EM images in our
case, we first select a range of parameters that samples the space broadly enough but not too
coarsely for each parameter, and then chose the best suitable input parameter by visual
inspection of the resulting edge images. While this selection does not guarantee that optimal
input parameter set was identified, it does generate good results in a timely way. Fig. 3a shows
the edges detected in the image shown in Fig. 2b with a suitable parameter set.
Line detection and end points computing
10
At the primitive level, the Hough transform (HT) (Hough, 1962), followed by searching end
points of the line segments and merging collinear line segments, is used to organize noisy and
discontinuous edges into complete line segments. To simplify the computation, the ρ − θ rather
than slope-intercept parameters are used to parameterize lines (Duda and Hart, 1972). The use of
the HT to detect lines in an image involves the computation of HT for each pixel in the image,
accumulating votes in a two dimensional accumulator array and searching the array for peaks
holding information of potential lines present in the image. The accumulator array becomes
complicated when the image contains multiple lines, with multiple peaks, some of which may
not correspond to lines but are artifacts of noise in the image. Therefore, we iteratively detect
the dominant line in the image, remove its contribution from the accumulator array, and then
repeat to find the dominant of the remaining lines.
The peaks in the accumulator array provide only the shortest distance of the line to origin ( ρ )
and the angle that the normal makes with the x-axis ( θ ) of the image plane. They do not provide
any information regarding the length, position, or end points of the line segments, which are vital
to the detection of acceptable filaments. Several algorithms have been proposed in the literature
for finding the length and end points of a line from the Hough accumulator array (Yamato et al.,
1990; Atiquzzaman and Akhtar, 1994; Karmat-Sadekar and Ganesan, 1998). The drawback of
these algorithms is that they are computationally intensive. Our integrated system requires that
the computation be performed on the order of seconds to maximize throughput.
We extend an algorithm, due to McLaughlin and Alder (1997), for the computation of the end
points of a line segment directly from the image, based on accurate line parameters detected
11
using the HT. Accurate line parameters can be detected using the HT since the edge detection
process has removed most noise. At each iteration of detecting the dominant line, we find the
global maximum in the accumulator array and from this compute the equation (parameters) of
the corresponding line. A simple post-processing step is used to find the beginning and end
points of the line, which involves moving a small window (e.g. 15× 15 pixels) along the line and
counting the number of edge pixels in the window. This count will rise above a threshold value
at the beginning of the line and decrease below the threshold at the line end. Collinear line
segments are merged by allowing gaps in the line segment while moving the small window along
the line.
Having found two end points of the line segment, we next tag all pixels lying on the line segment
and remove the contribution of these pixels to the Hough accumulator array.
The Hough
accumulator array is now becoming equivalent to that of an image that had not contained the
detected line segment. The above processing is repeated with the new global maximum of the
accumulator array. This continues until the maximum is below a threshold value which, found
from experimentation, is recalculated at each iteration. Edges that are not grouped into line
segments are discarded after primitive-level perceptual organization. Fig. 3b shows an example
of the line segments with acceptable length obtained by primitive-level perceptual organization
where we can see that the end points of those line segments were accurately identified and
collinear line segments were successfully merged.
Detection of filamentous structures
12
At the structural level, line segments are actively grouped into filaments by seeking parallelism
and employing knowledge obtained from training data. A filament is the area enclosed by two
line segments with a particular structural relationship.
Besides parallelism, the two line
segments should also meet some heuristics: (i) The distance between the two line segments
should be between the specified maximal and minimal values, both of which are determined by
statistical analysis on training data. (ii) The shorter one of the two line segments should be no
less than one third of the length of the longer line segment.
(iii) There should be some
overlapped area between the two line segments. The overlapped area can be measured from the
correspondence between the end points of the two line segments. At least one third of either one
should overlap with the other line segment. The area enclosed by two line segments that meet
these requirements is considered as a possible filament. Finding every possible filament requires
that the algorithm search for all possible matches in a neighborhood of each line segment. As a
result, there may be situations where a line segment has multiple matches (filaments). In this
case, the match with the largest overlapped area wins the competition. Therefore, each line
segment can be a boundary of only one filament. Like the ungrouped edges in the primitivelevel, unmatched line segments are discarded.
In selecting filaments from the images that will be passed to the three-dimensional reconstruction
algorithms, more stringent constraints must be used to limit the number of filaments in order for
the reconstruction process to be accurate and efficient. First, only the area enclosed by the
overlapped part of the two line segments are clipped as the detected filament. Secondly, as
described earlier, filaments may be joined end to end along their length (see Fig. 3d, for
example) and must be individually segmented prior to the three-dimensional image
13
reconstruction. Thus our algorithm must separate a filament aggregation into multiple segments
if it consists of two or more filaments that are joined together end to end (the longest filament in
the image shown in Fig. 3e, for example). Finally, filaments that are shorter than the minimal
length required for further analysis are discarded.
Separation of end-to-end joins
To determine whether a detected filament actually consists of two or more shorter filaments that
are end-to-end joined together, we have adopted a four-step approach. (i) Rotate the image
around its center so that the inclined filament becomes horizontal, and then cut the filament out
of the rotated image. (ii) Enhance possible end-to-end joins by linearly filtering the filament
image with a Gaussian kernel and then computing the gradient magnitude for each pixel in the
smoothed image. (iii) Sum along the columns of the image to generate a one-dimensional
projection in which the global peak corresponds to the position of a possible end-to-end join.
(iv) Determine the location of the end-to-end joins by thresholding based on the statistics
(maxium, mean, and standard deviation) of the one-dimensional projection. Using this method
the end-to-end join was identified in Fig. 3d and the longest filament as shown in Fig. 3e is
extracted for further analysis.
After examining a set of filament images, we found that the metric defined as (m − µ ) W can be
used to accurately discriminate whether a filament contains end-to-end joins, where m and µ
are respectively the maximum and mean of the one-dimensional projection, and W is the width
of the filament. First, a threshold value is learned from the set of images by visual inspection.
Then a filament is classified as one containing one or more end-to-end joins if the value of the
14
metric is larger than the threshold value and the filament will be separated along the position
indicated by the global peak in the one-dimensional projection. A recursive approach is used
when there are multiple end-to-end joins within one filament. In other words, a filament is split
into two sub-filaments along its dominant split point indicated by the global peak in the onedimensional projection. The sub-filaments are examined using the same approach and split if
they contain end-to-end joins. This process is repeated until all filaments do not contain any
end-to-end joins.
2.4 Detecting filaments in NTF images
As mentioned earlier, one of the advantages of image acquisition in defocus pairs is to use
information about filaments in the FFF image of the defocus pair to aid filament finding in the
NTF image of the pair. We first align the NTF images to their corresponding FFF images then
extract filaments in the NTF images using the coordinates identified in the corresponding FFF
images. Fig. 2 shows an example of a pair of NTF and FFF images with a total of 18 filaments
successfully targeted.
Cross-correlation has been the most widely exploited tool for aligning pairs of images acquired
under different conditions (for review, see Frank, 1980). The cross-correlation function (CCF) is
usually calculated by first forming the cross power spectrum (the product of multiplying the
complex conjugate of the Fourier transform of the first image by the transform of the second)
and then inverse-transforming the cross power spectrum back to real space. Ideally, the CCF
exhibits a well-posed local peak, the position of which indicates the relative displacement of the
15
two images.
However, as Al-Ali and Frank (1980) observed, the cross-correlation peak
deteriorates rapidly with increasing defocus difference between a pair of images. Our initial
experiment confirmed that the peak shape of the CCF between a pair of NTF and FFF images is
generally unrecognizable (see Fig. 4b, for example).
In order to improve the peak shape of the CCF under noise or strong background variations,
several variants involving linear or non-linear modification in Fourier space has been previously
proposed (for review, see Saxton, 1994).
Particularly, Saxton (1994) described two
modifications for accurate alignment of sets of images. The first modification, called a phase-
compensated CCF, was supposed to address the general case, when the transfer functions with
which the images have been recorded are arbitrarily complex, but requires at least an
approximate knowledge of the transfer functions. The second modification, a phase-doubled
CCF, was proposed particularly for axial imaging with contrast dominated either by phase or
amplitude (as is true in our case). It does not require any knowledge of the transfer functions
(Saxton, 1994). The phase-doubled CCF is calculated as the inverse Fourier transform of the
phase difference between two images (the cross power spectrum of the two images divided by its
modulus). We note that the phase-doubled CCF is essentially the same as the phase correlation
alignment method developed by Kuglin and Hines (1976). As indicated by the latter, the phase
correlation is a highly accurate alignment technique that exhibits an extremely narrow correlation
peak (see Fig. 4a, for example) and is generally insensitive to narrow bandwidth noise and
conventional image degradations.
16
We therefore adopted the phase correlation technique to align pairs of defocus images. In
summary, the algorithm for aligning a defocus pair of images using the phase correlation
technique consists of four steps: (i) A 2D fast Fourier transform is computed for the NTF and
FFF images which have the same dimensions, say N × N , resulting in two complex
N × N elements arrays, F1 and F2 . (ii) The phase difference matrix is derived by forming the
*
cross power spectrum, F1 F2 , and dividing by its modulus; (iii) The phase correlation function
(PCF) is then obtained as a real N × N element array by taking the inverse FFT of the phase
difference matrix. (iv) The relative displacement of the two images is finally determined by
searching the position of the highest peak in the PCF. In general, the position of the PCF surface
peak is a continuous function of image displacement. Since the PCF surface peak is very sharp,
it should be straightforward to measure subpixel (non-integer) displacements through the use of
interpolation with a few data points around the peak. In our case, it is accurate enough to align a
defocus pair of images by finding an integer displacement between the two images (see Fig. 2c,
for example).
4. Experiment Result and Analysis
Identification of filaments in FFF images
We have tested and evaluated the performance of the proposed approach by applying it to high
magnification images of helical filaments of tobacco mosaic virus (TMV) acquired under various
imaging conditions.
As mentioned in the Introduction, the performance of the proposed
approach is measured in terms of the percentages of correctly identified filaments, false
dismissals and false alarms, as compared to the filaments that would have been selected by an
17
experienced user. A visual screening of the original images outlined with the detected objects
indicates a variety of problems with the targeted image areas which do not represent filaments
suitable for further processing. Among these problems are poor quality filaments (overlapped by
other filaments and/or severely distorted by noise) and false alarms (regions that do not
correspond to any filaments). To simplify the account of the performance, we have assigned all
poor quality filaments to the category of false alarms.
Table 1 summarizes the overall performance of the proposed approach as applied to the detection
of TMV filaments during 3 independent acquisition sessions. Training data was obtained from
141 high magnification images collected in a separate session. The training data is used to derive
high-level knowledge, such as range of filament width in pixels, threshold for merging two
collinear but disconnected line segments, threshold for determining parallelism of two line
segments, and so on, which is required by our grouping algorithms. Table 1 indicates that on
average 88% of TMV filaments can be automatically identified with a false dismissal rate of
12%. While we would like to reduce the false dismissals, the yield is currently sufficient to
produce a 3D map during a single data acquisition session (see Figure 5). On average, over 85%
of filaments that contain end-to-end joins can be identified and accurately separated along their
joins.
Table 1 also tells us that there are quite a number of false alarms (21% on average). Most of
those false alarms are poor quality filaments which are not suitable for further processing). For
instance, among the 165 false alarms in experiment session 1 are 124 poor quality filaments and
41 image regions that do not represent TMV filaments. Since only boundary information was
18
exploited to group pairs of parallel line segments into filaments, we are not surprised that a
certain number of poor quality filaments were identified during the process of three-level
perceptual organization. We could decrease the false alarm rates by exploring quality metrics of
identified image regions and pruning those poor quality filaments. However, this is unnecessary
since our automated reconstruction process has the ability to reject those poor quality filaments
using criteria related to the identification of helical layer lines in the Fourier transform of the
extracted filament.
Alignments of defocus pairs of images
As mentioned earlier, it is much easier to identify the global peak on the phase correlation
surface than on the cross correlation surface when aligning a defocus pair of images. Using the
phase correlation technique, the alignment between defocus pairs of images, where there are
filaments selected in the FFF images, can be achieved with almost 100% accuracy. We have
also done a statistical analysis on the relative displacements of defocus pairs of images acquired
in experiment Session 2 (see Figure 4c). The statistics indicate that the displacements along the
X and Y -axis of the image plane range from –49 to 45 pixels (2.2Å/pixel) and from –30 to 42
pixels, respectively, and their standard deviations are respectively 14.51 and 8.75 pixels. These
variations demonstrate that the drift between each defocus pair of images must be measured
explicitly.
Helical reconstruction of a three-dimensional map of TMV
Filaments extracted automatically in experiment Session 2 using the techniques described above
were used to reconstruct a three-dimensional electron density map of TMV using standard
19
helical reconstruction techniques (Whittaker et al, 1995; Carragher et al, 1996). A total of 725
filaments were automatically extracted from the near to focus images and submitted to automatic
helical processing routines. The processing time required to extract the filaments from a defocus
pair of images was approximately 2~3 minutes.
During the helical processing phase, the
extracted filaments are assessed at various steps in order to determine whether they are suitable
for further processing and inclusion into the final three-dimensional map. One of these steps
involves finding and measuring helical layer lines in the Fourier transform of the filament. A
second step extracts the helical layer lines and examines them to determine the most probable tilt
of the filament out of the projection plane. Filaments which result in measurements out of
expected tolerance ranges for these values are rejected and not processed any further. At the end
of these procedures a total of 147 filaments were passed along to the next stage of the
reconstruction process which is to align the filaments to each other and average them together to
produce a final three-dimensional map. Filaments which do not align well to the average map
after three rounds of alignment, are rejected from the final map. A total of 72 filaments
contributed to the final map which is shown in Figure 5. Examination of the final set of
filaments used in the calculation of the map indicates that all the false positives were rejected
during the analysis.
Preliminary analysis of the map shows that it has the characteristic
appearance expected for TMV and indicates a resolution of approximately 10 Å.
5. Summary and Concluding Remarks
This paper presents an accurate approach for automatic identification of filamentous structures in
low-dose, low-contrast images acquired in defocus pairs using cryo-electron microscopy. A
20
defocus pair consists of a near-to-focus (NTF) image acquired first followed by a further-fromfocus (FFF) image.
In the first stage of the approach, a three-level perceptual
organization algorithm is developed to successfully detect filaments in the FFF image. At each
level, feature elements that are not grouped into higher-level features are discarded. At the
signal (low) level, edges (mostly filament boundaries) distorted by strong noise are detected
using the Canny edge detector. At the primitive (middle) level, collinear discontinuous edges are
organized into line segments with a complete description using the Hough transform followed by
detecting end points of line segments. At the structural (high) level, line segments are grouped
into filamentous structures, the area enclosed by a pair of line segments, by seeking parallelism
and exploiting high-level knowledge obtained from training data. In the case of two filaments
joining together end to end, a statistical method with an accuracy of over 85% is developed to
locate the splitting boundary and thus separate them along the boundary. In the second stage, the
NTF image is aligned to the FFF image using the phase correlation technique. Filaments in the
NTF image are then extracted using the coordinates identified in the FFF image and shifted
according to the alignment result.
The proposed approach has been tested and evaluated by applying it to high magnification cryoEM images of helical specimens. Experimental result indicates that on average over 88% of a
large number of filaments can be accurately detected by the proposed approach with a low
percentage of false dismissals, which is desirable for the purpose of structural reconstruction.
We showed in a test case reconstructing a three-dimensional map of TMV that the number of
filaments extracted automatically from an acquired data set was sufficient to result in a map at a
resolution consistent with the imaging conditions used to acquire the data. Thus we have
21
achieved the immediate goal of fully automatic filament detection from images acquired in
defocus pairs using cryo-electron microscopy.
The development of automated detection of filamentous structures will not only facilitate the
traditional three-dimensional reconstruction of helical objects using cryo-electron microscopy,
but also expedite the development of a present integrated system for cryo-electron microscopy.
After integrating the filament-finding package into our automated image acquisition and threedimensional reconstruction system, we are currently able to produce a three-dimensional map of
TMV to a resolution of approximately 10Å during a single data acquisition session (within 24
hours). With the integrated system, we not only reduce the time required for building a threedimensional map, but also can provide feedback about the quality of a specimen while it is inside
a microscope. This feedback in turn can be used to guide decisions as to the data collection on
the specimen grid.
During the visual screening of original images outlined with detected filaments, we observed that
our current algorithm only extracts the straight portions of curved filaments.
Although this
currently does not pose a serious problem, we plan to enhance the proposed approach to detect
curved filaments and then straighten the curved filaments using standard methods (Egelman,
1986, for example) in the future. It is straightforward to extend our approach to identify curved
filaments if we consider curved filaments as piecewise straight ones.
7. Acknowledgements
22
The authors gratefully acknowledge Jim Pulokas and the rest of the Leginon team at the
Beckman Institute for collecting the data and Ruben Diaz-Avalos for supplying the TMV. This
material is based upon work supported by the National Science Fundation (Award No. DBI9904547 and DBI-9730056) and the National Institutes of Health (Award No. GM61939-01).
23
References
Al-Ali, L., and Frank, J. (1980) Resolution estimation in electron microscopy, Optik 56, 31-40.
Atiquzzaman, M., and Akhtar, M.W. (1994) Complete line segment description using the Hough
transform, Image and Vision computing 12, 267-273.
Bluemke, D. A., Carragher, B., and Josephs, R. (1988) The reconstruction of helical particles with
variable pitch, Ultramicroscopy 26, 255-270.
Bremer, A., Henn, C., Goldie, K. N., Engel, A., Smith, P. R., and Aebi, U. (1994) Towards atomic
interpretation of F-actin filament three-dimensional reconstruction, J. Mol. Biol. 242, 683700.
Canny, J. (1986) A computation approach for edge detection, IEEE Trnas. Patt. Analy. and Machine
Intell. 8, 679-698.
Carragher, B., Bluemke, D. A., Gabriel, B., Potel, M. J., and Josephs, R. (1988) Structural analysis of
polymers of sickle cell hemoglobin. I. Sickle hemoglobin fibers, J. Mol. Biol. 199, 315-331.
Carragher, B., Whittaker, M., and Milligan, R. A. (1996) Helical processing using Phoelix, J. Struct.
Biol. 116, 107-112.
Carragher, B., Kisseberth, N., Kriegman, D., Milligan, R.A., Potter, C.S., Pulokas, J., and Reilein, A.
(2000) Leginon: An automated system for acquisition of images from vitreous ice specimens,
J. Struct. Biol.132, 33-45.
DeRosier, D. J., and Moore, P. B. (1970) Reconstruction of three-dimensional images from electron
micrographs of structures with helical symmetry, J. Mol. Biol. 52, 355-369.
Duda, R. O., and Hart, P. (1972) Use of the Hough transformation to detect lines and curves in
pictures, Communications of the ACM, 15, 11-15.
24
Egelman, E. H. (1986) An algorithm for straightening images of curved filamentous structures.
Ultramicroscopy 19, 367-374.
Fuller, S. D. (1987) Cell 48, 923.
Frank, J. (1980) The role of correlation techniques in computer image processing, Computer
processing of Electron Microscope Images (ed. By P.W. Hawkes), 187-222, Berlin: springer.
Frank, J., and Wagenknecht, T. (1984) Automatic selection of molecular images from electron
micrographs, Ultramicroscopy 12, 169-176.
Henderson, R. (1995) The potential and limitations of neutrons, electrons, and X-rays for atomic
resolution microscopy of unstained biological macromolecules, Quart. Rev. Biophys. 28, 171193
Harauz, G., and Fong-Lochovsky, A. (1989) Automatic selection of macromolecules from electron
micrographs by component labeling and symbolic processing, Ultramicroscopy 31, 333-344.
Heath, M. D., Sarkar, S., Sanocki T., and Bowyer, K. W. (1997) A robust visual method for assessing
the relative performance of edge-detection algorithms, IEEE Trnas. Patt. Analy. and Machine
Intell.19, 1338-1359.
Hough, P.V.S. (1962) Method and means for recognizing complex patterns, U.S.Patent 3,069,654.
Jeng, T.-W., Crowther, R. A., Stubbs G., and Chiu, W. (1989) Visualization of Alpha-helices in
tobacco mosaic virus by cryo-electron microscopy, J. Mol. Biol. 205, 251-257.
Karmat-Sadekar V., and Ganesan, S. (1998) Complete description of multiple line segments using the
Hough transform, Image and Vision Computing 16, 597-613.
Kuglin C. D. and Hines D. C. (1975) The phase correlation image alignment method, Proc. of Int’l
Conf. Cybernetics and Society, 163-165.
Koffka, K. (1935) Principles of Gestalt psychology, Harcourt, Brace and World, 1935.
25
Kohler, W. (1947) Gestalt psychology, Liveright, New York, 1947.
Lata, K. R., Penczek, P., and Frank, J. (1995) Automatic particle picking from electron micrographs,
Ultramicroscopy 58, 381-391.
Lowe, D. G. (1985) Perceptual Organization and Visual Recognition, Boston: Kluwer Academic
Publishers.
Lowe, D. G., (1987) Three–dimensional object recognition from single two – dimensional images,
Artificial Intelligence, 31, 355-395.
Ludtke, S. J., Baldwin, P. R., and Chiu, W. (1999) EMAN: Semiautomated software for highresolution signle-particle reconstructions, J. Struct. Biol. 128, 82-97.
McLaughlin, R. A., and Alder, M. D. (1997) Technical report – the Hough transform versus the
Upwrite, Tech. Rep. TR97-02, Univ. of Western Australia, CIIPS, Dept. of Electrical and
Electronic Eng.
Martin, I. M. B., Marinescu, D. C., Lynch, R. E., and Baker T. S. (1997) Identification of spherical
virus particles in digitized images of entire electron micrographs, J. Struct. Biol. 120, 146157.
Milligan, R. A., and Flicker, P. F. (1987) Structural relationships of actin, myosin, and tropomyosin
revealed by cryo-electron microscopy, J. Cell Biol. 105, 29-39.
Mohan, R. and Nevatia, R. (1992) Perceptual organization for scene segmentation and description,
IEEE Trans. Patt. Analy. And Machine Intell.14, 616-634.
Nogales, E., and Grigorieff, N. (2001) Molecular machines: Putting the pieces together, J. Cell Biol.
152, F1-F10.
Owen, C. H., Morgan, D. G., and DeRosier, D. J. (1996) Image analysis of helical objects: The
Brandeis helical package, J. Struct. Biol. 116, 167-174.
26
Palmer, S. E. (1983) The psychology of perceptual organization: A transformational approach, In J.
Beck, B. Hope, and A. Rosenfeld, editors, Human and Machine Vision, 269-339, New York:
Academic Press.
Potter, C. S., Chu, H., Frey, B., Green, C., Kisseberth, N., Madden, T.J., Miller, K.L., Nahrstedt, K.,
Pulokas, J., Reilein, A., Tcheng, D., Weber D., and Carragher B. (1999) Leginon: A system
for fully automated acquisition of 1000 micrographs a day, Ultramicroscopy, 77, 153-161.
Sarkar, S., and Boyer, K. L. (1993) Integration, inference, and management of spatial information
using bayesian networks: perceptual organization, IEEE Trans. Patt. Analy. And Machine
Intell.15, 256-273.
Sarkar S., and Boyer, K. L. (1994) Computing Perceptual Organization in Computer Vision,
Singapore: World Scientific.
Saxton, W. O. (1994) Accurate alignment of sets of images, J. Microscopy 174, pt 2, 61-68.
Schrag, J. D., Prasad, B. V. V., Rixon, F. J., and Chiu, W. (1989) Cell 56, 651.
Shi, J., and Malik, J. (2000) Normalized cuts and image segmentation, IEEE Trans. Patt. Analy. And
Machine Intell.22, 888-905.
Stewart, M., Kessler, M. W., and Levine, R. J. C. (1981) Structure of Limulus telson muscle thick
filaments, J. Mol. Biol. 153, 781-790.
Thuman-Commike, P. A., and Chiu, W. (1995) Automatic detection of spherical particles from spotscan electron microscopy images, J. Micros. Soc. Am. 1, 191-201.
Trucco, E., and Verri, A. (1998) Introductory techniques for 3-D computer vision, New Jersey:
Prentice Hall.
Unwin, P. N. T., and Klug, A. (1974) Electron microscopy of the stacked disk aggregate of tobacco
mosaic virus protein. I. Three-dimensional image reconstruction, J. Mol. Biol. 87, 641-656.
27
van Heel, M. (1982) Detection of objects in quantum-noise-limited images, Ultramicroscopy 8, 331342.
Wagenknecht, T., DeRosier, D. J., Shapiro, L., and Weissborn, A. (1981) Three-dimensional
reconstruction of the flagellar hook from Caulobacter crescentus, J. Mol. Biol. 151, 439-465.
Williams, L. R., and Thornber, K. K. (1998) A comparison of measures for detecting natural shapes
in cluttered backgrounds, in Proceedings, 5th European Conf. Computer Vision, Vol. 2, 43248.
Whittaker, M., Carragher, B. O., and Milligan, R. A. (1995) PHOELEX: A package for semiautomated helical reconstruction, Ultramicroscopy 58, 245-259.
Yamato, J., Ishii I., and Makino, H. (1990) Highly accurate segment detection using Hough
transformation, Sys. Comp. Japan, 21, 68-77.
Zhou, Z. H., and Chiu, W. (1993) Prospects of using an IVEM with FEG to record images towards
atomic resolution, Ultramicroscopy 49, 407-416.
Zhou, Z. H. (1995) High resolution 3-Dimensional electron cryomicroscopy and reconstructions of
herpes simplex virus capsids. Ph.D. thesis, Baylor College of Medicine.
28
Table 1. Results of automated detection of TMV filaments in 3 independent experiment sessions.
Session No.
Image
1
Magnification
3
Average
66,000x 66,000x 88,000x
acquisition Average dose (e/Å2)
conditions
2
10.69
11.83
11.87
Defocus value for NTF images (µm)
-0.3
-0.3
-0.15
Defocus value for FFF images (µm)
-2
-2
-3
Image size
1K × 1K
2K × 2K 2K × 2K
Pixel size
2.8Å
2.2Å
1.5Å
Number of defocus pairs of images
206
411
178
Total number of filaments in the FFF images
542
686
755
Correctly detected filaments (%)
90
89
85
88
False dismissals (%)
10
11
15
12
False alarms (%)
25
16
21
21
Images were acquired in defocus pairs: a near to focus (NTF) image is recorded first followed by
a farther from focus (FFF) image from any chosen specimen area. The identified image regions
during filament detection consist of the correctly detected filaments and false alarms as defined
in the text. Total number of filaments is determined by visual screening of the images as
described in the text, which is equal to the sum of correctly detected filaments and false
dismissals. The percentage of correctly detected filaments and false dismissals is calculated
from the total number of filaments. The percentage of false alarms is calculated from the total
number of identified image regions.
29
FFF
Extraction of filaments in
the NTF image.
Extraction of filaments in the FFF
image by a three-level perceptual
organization algorithm.
Acquisitions of high magnification
images in defocus pairs (first a NTF
image and then a FFF image)
NTF
Edge detection
Discontinuous
edges grouped into
line segments
Line segments
organized into
filaments
Shift
information
Alignment of the NTF
image to the FFF
image.
Extract filaments in the
NTF image using
coordinates identified
in the FFF image.
Figure 1: Schematic overview of the automated filament finding approach.
30
35.98nm
35.98nm
(a)
(b)
(c)
(d)
Figure 2: Illustration of filament finding in a defocus pair of images of specimens of tobacco
mosaic virus acquired (6,6000x) to a 1K × 1K CCD camera using cryo-electron microscopy. The
image shown in (a) was acquired first at a very near-to-focus (NTF) condition (-0.3µm) and the
one shown in (b) was recorded second at much farther from focus (FFF) (-3µm). Following
filament finding in the FFF image, alignment of the NTF image to the FFF image results in 18
filaments found in the pair of images shown in (c) and (d) respectively where targeted filaments
outlined by pairs of parallel line segments.
31
(a)
(b)
(d)
(e)
(c)
Figure 3: Illustration of filament finding in the FFF image of a defocus pair using a three-level
perceptual organization algorithm and the separation of end-to-end aggregation. (a) Result of
edge detection (signal-level perceptual organization). (b) Line segments with acceptable length
obtained by the primitive-level perceptual organization. (c) 9 filaments were detected after
structural-level organization. (d) An extracted region from the image represents two filaments
aggregated end to end as indicated by the arrow. (e) The longest filament in (d) after separation
along the end-to-end join.
32
(a)
(b)
(c)
Figure 4: Illustration of the alignment of NTF images to FFF images. (a) and (b) show the phase
correlation and cross-correlation surfaces obtained from the pairs of images shown in Figure 2a
and 2b. The peak (indication of relative displacement of the two images) on the phasecorrelation surface is much more well-posed than that on the cross-correlation surface. (c) A
plot of the relative displacements calculated using phase correlation of defocus pairs of images
acquired in experiment Session 3, where each dot represents a defocus pair of images and its
coordinates indicate the relative displacements. The maximum displacement along the
horizontal direction (X-axis of the image plane) ranges from –49 to +45 pixels (2.2Å/pixel), and
that along the vertical direction (Y-axis of the image plane) ranges from –30 to +42 pixels.
33
Figure 5: A three-dimensional map of TMV at a resolution of approximately 10Å reconstructed
from filaments extracted automatically using the techniques described in the text. The starting
point was a dataset of 411 defocus pairs (2kx2K digital images) collected at a nominal
magnification of 66,000x (2.2Å/pixel) and a dose of 11e/ Å2 for a near to focus image. A total of
725 filaments were automatically extracted from the near to focus images and submitted to
automatic helical processing routines. A wedge of data has been cut out of the map to expose the
inner surfaces.
34