'$ &' !* BIOINFORMATICS
Homology modeling, model and software
evaluation: three related resources
'$&' ')#!,/ $. "#& $# '(/ #)*' '&* &
)+ )#& &+) ') &+# &!#&)#&! & #'+"&'$'!.
'%(,+#&! #$)! )%&.
-& , &
Abstract
Motivation: Homology modeling is rapidly becoming the
method of choice for obtaining three-dimensional coordinates for proteins because genome projects produce sequences at a much higher rate than NMR and X-ray
laboratories can solve the three-dimensional structures. The
quality of protein models will not be immediately clear to
novices and support with the evaluation seems to be needed.
Expert users are sometimes interested in evaluating the
quality of modeling programs rather than the quality of the
models themselves.
Results: Three servers have been made available to the
scientific community: a homology modeling server, a model
quality evaluation server and a server that evaluates models
built of proteins for which the structure is already known,
thereby implicitly evaluating the quality of the modeling
program.
Availability: The modeling-related servers and several
structure analysis servers are freely available at
http://swift.embl-heidelberg.de/servers/
Contact: [email protected]
Introduction
The routine sequencing of entire genomes is providing an
overwhelming flood of sequence information. Every day
∼2000 nucleotide sequences are deposited in publicly accessible databases (Stoesser et al., 1997). Compared with
this, the numbers of protein structures that are solved and
deposited stand in marked contrast at four per day on average. Thus, the structure gap—the difference between the
number of known sequences and the number of experimentally determined three-dimensional structures—is widening
rapidly. Consequently, homology modeling is becoming the
technique of choice for routine structure ‘determinations’.
The CASP bi-annual modeling competition (Dunbrack et al.,
1997; Mosimann et al., 1995) has made it clear that model
building by homology can provide remarkably good results
if the sequence identity is high between the protein to be modeled and the template protein (>75%), but at low sequence
Oxford University Press
identities (<40%) modeling errors become the rule, rather
than the exception.
In the fields of experimental structure determination, it is
now becoming a common practice to validate the coordinates with validation programs such as WHAT_CHECK
(Hooft et al., 1996a; WHAT_CHECK help:
http://swift.embl-heidelberg.de/whatcheck/),
SURVOL
(Pontius et al., 1996) and PROCHECK (Laskowski et al.,
1993). The program WHAT_CHECK, for example, provides >70 different classes of checks, and this program has
been used to search for violations (the PDBREPORT database: http://swift.embl-heidelberg.de/pdbreport/) in all files
in the PDB (Bernstein et al., 1997). The second server presented here executes those checks of the WHAT_CHECK
program that are relevant for the validation of models built
by homology.
The CASP bi-annual modeling competition (Dunbrack et
al., 1997; Mosimann et al., 1995) has created the realization
that homology modeling is a viable science, the results of
which are not black box magic, but verifiable observations.
CASP has spurred many researchers into improving the
quality of model-building-by-homology software. The
judges of this competition so far have used several quality
criteria, but the all-atom RMS deviation between the model
and the corresponding real structure still is their major determinant of model quality. It is, however, becoming generally accepted that this RMS deviation is a bad measure.
Abagyan et al. (1997) suggested a local contact-based score,
called the CAD score, as a better determinant of model quality. Although the CAD score captures the intuitive feeling
about model quality better than the RMS deviation, it also
describes only one aspect of the model quality. The third
server presented here provides a long list of different types
of comparisons when a model and the corresponding real
structure are submitted.
Modeling
Since the WHAT IF program (Vriend, 1990) performed well
for the high-sequence-identity test cases in both CASP competitions, we decided to make this modeling program avail-
523
R.Rodriguez et al.
Fig. 1. WWW layout of the modeling server.
able as a WWW-based server (see Figure 1) that can be used
for high-sequence-identity modeling tasks. The server is not
as automatic as, for example, the SwissModel server
(Peitsch, 1996); it does not search for the optimal template
and it does not yet model insertions. The search for an optimal template will perhaps be implemented in the near future,
but ab initio modeling of loops has, in the two CASP competitions, been proven to be too difficult for today’s techniques (Dunbrack et al., 1997; Mosimann et al., 1995) (especially when limited CPU time is available, like in a WWWbased server set-up) and will only be implemented in the
server after a major technical or scientific breakthrough.
However, we are presently working on the incorporation of
a module that will allow for the insertion of very short loops
(one or two residues), and hope to make this module available soon.
The algorithms implemented in the WHAT IF modeling
option have been described before (DeFilippis et al., 1994;
Chinea et al., 1995; Rodriguez and Vriend, 1997) and will
only be summarized very briefly here. The strength of the
WHAT IF modeling module comes from the fact that residues that are conserved between the model and the template
are modified as little as possible, and from the use of posi-
524
tion-specific rotamers for residues that have to be exchanged
going from the template to the model. The study by De Filippis et al. (1994) indicated that one has the best chance to
model conserved residues correctly by not modifying them
at all. The study by Chinea et al. (1995) showed that the use
of position-specific rotamers (Jones and Thirup, 1986) is a
very good solution for the problem of selecting the best rotamer from among the many different possibilities that exist
at each position in the model. A rotamer distribution for a
certain residue type at a certain position—called a positionspecific rotamer distribution—is determined by extracting
from a database of non-redundant protein structures (Hobohm et al., 1992) all suitable fragments of five or seven
residues (seven in helix and strand; five in the case of irregular local backbone). Suitable fragments are those that have a
local backbone conformation similar to the one around the
evaluated position, and have the same residue type at the central position. In the server implementation, the RMS deviation of the backbone alpha carbons is maximally allowed to
be 0.5 Å.
A list of models built by homology, for which the real
structure is known but not used in the modeling process, is
available (Rodriguez and Vriend, 1997) to give the user an
Homology modeling, model and software evaluation
impression of the quality of this model building by homology server. The user can, of course, also get an impression
of the quality of this modeling server by building a couple of
models of known proteins and submitting these models to the
WWW-based model precision determination server.
The casual user is probably better served with the SwissModel server, but the user who wants to improve the model
quality by iteratively changing the alignment or even the
template structure will probably want to use the server described here. The process of merging/combining our server
and the SwissModel server is in progress.
Model validation
Most types of violations that WHAT_CHECK can detect
have already been described elsewhere (WHAT_CHECK
help: http://swift.embl-heidelberg.de/whatcheck/; Vriend
and Sander, 1993; Hooft et al., 1996a,b, 1997). The
WHAT_CHECK options that are aimed at solving typically
crystallographic problems, such as crystal cell dimension
deviations, B-factor distributions, or inconsistencies in the
administrative crystallographic records (Hooft et al., 1994),
are not useful for the validation of models built by homology
and are not implemented in the model validation server.
Since models are often, for a large part, kept the same as
the template structure, we also have to check elementary geometric aspects of a model (bond lengths, bond angles, planarities, chiral centers). The techniques used to determine misthreading in X-ray structures can equally well be used to determine alignment errors that underlie errors in models. The
Ramachandran plot is probably the most powerful determinant of the quality of protein coordinates (Laskowski et al.,
1993; Hooft et al., 1997). When the ‘quality’ of the Ramachandran plot of the model is significantly worse than that of
the template, then it seems likely that there are significant
differences between the backbone of the template and the
structure to be modeled. WHAT_CHECK spends a large
fraction of its CPU time determining if any Asn, His or Gln
side chains need to be rotated by 180_ about their χ2, χ2 or
χ3 angle, respectively. The modeling process will sometimes
change the side chain torsion angles that are required for
optimal hydrogen bonding. However, as it is not yet possible
to model reliably more than a few water molecules per protein, the hydrogen bond checking facilities in the model validation server cannot be as precise as in the structure validation server. A detailed description of all the differences
between model validation and structure validation was given
by Hooft et al. (WHAT_CHECK help: http://swift.embl-heidelberg.de/whatcheck/).
Modeling program validation
We have added to the WHAT IF (Vriend, 1990) software a
module that compares a model with an experimentally determined structure and provides a wide spectrum of comparison
values. As a service to the modeling community, and to allow
interested scientists to check for themselves the quality of
their favorite modeling program, we have made this model
validation server, called WHAT_MODQ, available via the
WWW. We will use the terms ‘template’, ‘model’ and ‘real
structure’, respectively, for the protein used to model from,
the model that was built by homology, and the experimentally determined structure the modeling of which was attempted.
In contrast to experimental structures that should always
be as good as possible all over the structure, the biological
questions that initiated the modeling determine what is important in a model and what therefore needs to be modeled
with the greatest precision. For example, if the model is supposed to suggest a new lead for drug design, it is important
that the active site is modeled with great precision, but further
away from the active site this precision becomes of less interest to the drug designer.
A problem that can be encountered is domain displacements. If one domain rotates with respect to another, a very
poor RMS score will result, but the model can at every position be of sufficient local precision to answer all biological
questions correctly. It is, therefore, important to augment the
global RMS values with RMS scores that are based on local
comparisons only.
Crystallographic symmetry contacts are a problem that has
not yet been taken into account upon judging the CASP
entries. Nevertheless, it is not a rare event if in a small protein
more than a quarter of all side chain conformations are influenced by crystal contacts. Crystal contacts in the template
can be determined, and a good modeling program will try to
compensate for them. Crystal contacts in the model, however, cannot be predicted and thus any residue that makes a
symmetry contact in the real structure should not be taken
into account if the quality of the model is judged.
To cope with the aforementioned problems, the
WHAT_MODQ server provides a large series of comparisons between model and real structure. These tests are designed to dissect as clearly as possible the nature of the
modeling errors and to help the modeler find the algorithmic
origin of those problems. WHAT_MODQ provides RMS
deviations, but also the linear (or robust) errors because these
numbers are less influenced by one or two outliers. These
numbers are provided for all helical, strand, loop, buried and
accessible residues. In several categories, symmetry-contacting and not-symmetry-contacting residues are separated.
Several local comparison parameters, such as one similar to
525
R.Rodriguez et al.
Fig. 2. Model comparison server output. The small table on the top is repeated for all residue classes listed below it. Series of dots indicate that
some output has been removed in this figure for brevity.
Abagyan’s CAD score (Abagyan et al., 1997) and a moving
average short fragment RMS deviation, are provided. The 10
poorest modeled residues are listed, and plots of the superposition of model and real structure are provided for all secondary structure elements.
Of course the WHAT_MODQ server deals correctly with
a large series of problems such as missing or misnamed
atoms or, for example, a tyrosine that roughly has a 180_
rotation about the Cβ–Cγ bond in either the model or the real
structure. Figure 2 shows some examples of the output.
The quality of a model built by homology is partly a subjective measure because the relative importance of different
errors is partly determined by the biological questions that
526
the model is supposed to answer. The WHAT_MODQ server
tries to accommodate the user by providing the widest possible range of comparisons between the model and the real
structure. We suggest that both the template and the real
structure are submitted to the BIOTECH structure validation
server (http://biotech.embl-heidelberg.de:8400/) or that the
quality of these two real structures are looked up in the
PDBREPORT database (Hooft et al., 1996a;
http://swift.embl-heidelberg.de/pdbreport/). It would be unfair to compare a model with a residue in the real structure
that sits ‘the wrong way around’ or is otherwise not correct.
The three servers described above are part of a larger series
of structure and model analysis servers (see Figure 3).
Homology modeling, model and software evaluation
Fig. 3. The WHAT IF-based modeling related servers.
Acknowledgements
We thank G.Padron, B.Bywater, C.Sander and A.Tramontano for stimulating discussions. We thank B.Altenberg and
K.Krmoian for technical assistance, and J.Weare for technical assistance and for critically reading the manuscript. This
work was partly funded by EC grants CT960166 and
CT960189, and by the BMBF RELIWE grant.
References
Abagyan,R.A. and Totrov,M.M. (1997) Contact area difference
(CAD): a robust measure to evaluate accuracy of protein models. J.
Mol. Biol., 268, 678–685.
Bernstein,F.C., Koetzle,T.F., Williams,G.J.B., Meyer,E.F.,Jr,
Brice,M.D., Rodgers,J.R., Kennard,O., Shimanouchi,T. and Tasumi,M. (1977) The Protein Data Bank: a computer-based archival file
for macromolecular structures. J. Mol. Biol., 112, 535–542.
Chinea,G., Padron,G., Hooft,R.W.W., Sander,C. and Vriend,G. (1995)
The use of position specific rotamers in model building by
homology. Proteins, 23, 415–421.
DeFilippis,V., Sander,C. and Vriend,G. (1994) Predicting local structural changes that result from point mutations. Protein Eng., 7,
1203–1208.
Dunbrack,R.L., Gerloff,D.L., Bower,M., Chen,X., Lichtarge,O. and
Cohen,F.E. (1997) The second meeting of the critical assessment of
techniques for protein structure prediction (CASP2). Fold. Des., 2,
R27–R41.
Hobohm,U., Scharf,M., Schneider,R. and Sander,C. (1992) Selection
of representative protein data sets. Protein Sci., 1, 409–417.
Hooft,R.W.W., Sander,C. and Vriend,G. (1994) Reconstruction of
symmetry-related molecules from protein data bank (PDB) files. J.
Appl. Crystallogr., 27, 1006–1009.
Hooft,R.W.W., Sander,C., Vriend,G. and Abola,E. (1996a) Errors in
protein structures. Nature, 381, 272.
Hooft,R.W.W., Sander,C. and Vriend,G. (1996b) Verification of
protein structures: Side-chain planarity. J. Appl. Crystallogr., 29,
714–716.
Hooft,R.W.W., Sander,C. and Vriend,G. (1997) Objectively judging
the quality of a protein structure from a Ramachandran plot.
Comput. Applic. Biosci.
Jones,T.A. and Thirup,S. (1986) Using known substructures in protein
model building and crystallography. EMBO J., 5, 819–823.
527
R.Rodriguez et al.
Laskowski,R.A., MasArthur,M.W., Moss,D.S. and Thornton,J.M.
(1993) PROCHECK: a program to check the stereochemical quality
of protein structures. J. Appl. Crystallogr., 26, 283–291.
Mosimann,S., Meleshko,R. and James,M.N.G. (1995) A critical
assessment of comparative molecular modelling of tertiary structures of proteins. Proteins, 23, 301–317.
Peitsch,M.C. (1996) ProMod and Swiss-Model: Internet based tools
for automated comparative protein modelling. Biochem. Soc. Trans.,
24, 274–279.
Pontius,J., Richelle,J. and Wodak,S. (1996) Deviations from standard
atomic volumes as a quality measure for protein crystal structures. J.
Mol. Biol., 264,121–136.
528
Rodriguez,R. and Vriend,G. (1997) Professional gambling. In Vergoten,G. and Theophanides,T. (eds), Biomolecular Structure and
Dynamics. Kluwer Academic, Dordrecht.
Stoesser,G., Sterk,P., Tuli,M.A., Stoehr,P.J. and Cameron,G.N. (1997)
The EMBL nucleotide sequence database. Nucleic Acids Res., 25,
7–13.
Vriend,G. (1990) WHAT IF: A molecular modelling and drug design
program. J. Mol. Graph., 8, 52–56.
Vriend,G. and Sander,C. (1993) Quality control of protein models:
Directional atomic contact analysis. J. Appl. Crystallogr., 26, 47–60.