Volume 02 / Issue 01 / January 2014
boa.ac.uk
Page 57
© 2014 British Orthopaedic Association
Evidence vs. Anecdote
in Foot and Ankle Surgery
Judith Baumhauer, M.D., M.P.H. Professor and Associate Chair of Orthopaedic Surgery
University of Rochester Medical School, Rochester, NY
Evidence: The available body of facts
suggesting a belief is true or valid.
Anecdote: A short, amusing or interesting
story that need be neither true nor valid.
Diagnosis and treatment decisions are often
based on sketchy information passed down
through the years; eventually becoming
dogma; gradually assuming the mantle of
fact masquerading as knowledge. We must
revisit some of these established treatment
trends to determine if there is an evidence
base to support their use.
The ancient Chinese developed the concept of evidence based medicine,
however it was in the early 1990’s that Gordon Guyatt and colleagues
brought the current concept of evidence based medicine to the fore1. This
led to an evidence based hierarchy that has been peer reviewed published
and well accepted2 (Table 1).
Table 1: Levels of Evidence in Medicine2
Level 1
Randomised Clinical Trials (RCT)
Level 2
Lower quality RCT
Level 3
Case controlled Studies
Level 4
Case Series
Level 5
Expert opinion
The quality of the information
obtained influences the level
of evidence ranking. Quality
indicators include the type of study,
quality of the study design, per
cent follow up, statistical methods
and enrollment criteria utilised for
the clinical trial3 (Table 2).
Table 2: Quality indicators for
Clinical Research in Medicine *
Type of Study
Quality of the Study Design
% of Patient Follow up
Statistical methods
Enrollment criteria
Judith Baumhauer
This article is based on the
Naughton Dunn Lecture given by
Judith Baumhauer at the BOA
Annual Congress in Birmingham
2013.
* Wright RW, Brand RA, Dunn
W, Spindler KP: How to write a
systematic review. Clin Orthop
Relat Res. 2007 Feb;455:23-9.
The rigour of these quality
measures impacts the level of
evidence assigned to the research.
Foot and ankle research
has primarily been Level 4
retrospective cases series. A
recent publication by Hunt and
Hurwit (2013)4 reviewed our foot
and ankle literature to assess
the level of evidence over a 9-10
year time period. They found
70% Level 4 studies, 12% Level
3, 9% Level 2 and only 9% Level
1 studies. This was actually an
improvement over earlier papers
looking at the level of evidence
in foot and ankle literature5!
Although current publications
record the level of evidence,
older literature did not. Here we
will look back on some well held
principles of foot and ankle care
from older literature to determine
whether or not these “peer
reviewed” papers were in fact
based on evidence or anecdote.
>>
Volume 02/ Issue 01 / January 2014
boa.ac.uk
Page 58
JTO Peer-reviewed Articles
Journal of Trauma and Orthopaedics: Volume 02, Issue 01, pages 57-62
Title: Evidence vs. Anecdote in Foot and Ankle Surgery
Author/s: Judith Baumhauer
The index citation tool on the Web
of Science was used to provide
a listing of the most highly cited
peer reviewed papers in the world
literature cross referenced for
foot and ankle over the last 30
years. Identifying a highly cited
paper and assessing whether
or not the science behind that
paper used quality research
principles would give us some
insight as to whether or not we
are perpetuating dogma or if we
are truly using high level evidence.
“The rupture of posterior tibial
tendon causing flat foot - surgicaltreatment” authored by Roger
Mann and Francesca Thompson
in 19856 was cited 213 times. In
the conclusions that are drawn
from the evidence of the article:
“Isolated FDL transfer to the
navicular arrests the flat foot
deformity, relieves pain and
restores inversion power of the
hindfoot at two years.” Is this
evidence or anecdote?
Using quality research measures,
this paper is defined as a Level 4
paper, a case series with a small
subject number (17). There is no
control arm therefore all patients
underwent FDL tendon transfer to
the navicular. The authors did not
utilise any validated methodology
or outcomes measures. There
was no standardised radiographic
methodology available at that
time and no functional outcomes
were obtained. The authors
did use standardised surgical
techniques and post-operative
treatment protocols for this patient
group. There were no statistical
analyses performed on any of the
parameters measured in this paper.
The duration of follow up was a
minimum of two years. Based on
this information and assessment
of the quality measures utilised
to determine the conclusion
statements, it would seem that this
case series described a surgical
treatment however; it did not
prove that this surgical intervention
relieves pain and restores inversion
power of the hindfoot. It is
therefore a well-meaning anecdote.
There are numerous case series
available in the foot and ankle
literature (Hunt et al). These
case series do not have a level
of evidence to support universal
adoption of the suggested
treatment as in the use of an
isolated FDL transfer to navicular
however; they do provide
information that can lead to
further study. Pinney published
a symposium on evidence based
medicine: “What is it and how
should it be used?”7. A section
looked at case series (level 4)
and case reports (level 5)8. These
published Level 4 and 5 studies
allow clinicians and researchers to
be exposed to new ideas which
may be the stepping stones for
more advanced study. Friedman,
JAMA 19999 commented about
Level 4 and 5 studies. “These
lower level studies provide detailed
observations and descriptions of
new diseases or conditions. They
also provide pilot study data for
future power analyses and aid in
study design. These Level 4 and 5
studies also may have a quick turn
around on time sensitive issues
opposed to Level 1 and 2 studies
which often can take years to get
the available data and conclusions
into the literature. These case
series and case reports may be
the only options for orphan or very
rare conditions and they also allow
us to detect drug side effects in an
expedient manner.”
Hoffman et al10 published an article
on the negative implications of
case series and case reports. They
looked into the percentage of case
series and case reports that have
been used as stepping stones for
Level 1 or Level 2 studies. They
highlighted that most new ideas,
from these lower level evidence
papers, are not substantiated with
more rigorous research. They
argue that the case series and case
reports contain misleading data
and conclusions with small subject
numbers and wide variation. The
data is often qualitative and not
quantitative and it lacks validated
outcome measures as we saw with
the FDL to navicular paper. They
summarise their conclusions with
a comment “does more harm than
good by focusing on the bizarre.”
Despite the negative comments
made by Hoffman and his
colleagues, there are historic
examples of Level 4 and 5,
evidence that have made great
contributions to the advancement
of patient care. In the 1980’s, Drs.
Conant and Volverg recognised a
unique tumor, Kaposi’s sarcoma
in eight homosexual males11. It is
their recognition of this link that
led to the recognition of AIDS
and HIV virus transmission. In
1952, Dr. Jonas Salk produced
a polio vaccine consisting of a
dead injectable virus, and in 1957
Albert Sabin, M.D. produced
an attenuated vaccine tested in
19 children that could be taken
orally12. Salk was credited with the
eradication of polio in the United
States while Sabin’s oral vaccine
has been utilised throughout the
world. These Level 4 case series
changed care around the world.
Mr. Naughton Dunn, M.D.
recognised the importance of case
series and the re-evaluation of his
patients to improve patient care.
In his presidential address in 1928
he stated, “If we refer to a modern
textbook on orthopaedic surgery
we find that all the principles on
which we rely for the treatment of
infantile paralysis are recorded, so
that I have nothing new to offer.
My only reason for selecting this
subject for discussion is that so
many alternative treatments are
advised that a frank review of
these and the results of our own
practice and experience may be
helpful” 13. Mr. Dunn recognised
that anecdotal information had
been perpetuated by repetition.
He recognised the need for a frank
assessment of these treatments.
Mr. Dunn published a case series
entitled “Calcaneal cavus and its
treatment” in 191914. He looked at
a wedge resection of the midfoot
to correct a cavus foot along with
a soft tissue release of the Achilles.
He provided pre- and postoperative radiographs and clinical
photographs of his patients. This
type of surgery and treatment is
still utilised today.
In summary, the rupture of the
posterior tibial tendon causing
flat foot – surgical – treatment by
Drs. Mann and Thompson was
a meaningful case series but not
critical evidence. It led to a number
of higher level evidence studies
with control groups comparing
different treatment options for
posterior tibial tendon dysfunction.
It did advance the science.
Volume 02 / Issue 01 / January 2014
boa.ac.uk
Page 59
© 2014 British Orthopaedic Association
Hallux Metatarsophalangeal-Interphalangeal Scale (100 points total )
Pain (40 points)
None
40
Mild, occasional
30
Moderate, daily
20
Severe, almost always present
The fifteenth most commonly
cited Web of Science article in
foot and ankle was out of JAMA
entitled “Preventing foot ulcers
in patients with diabetes”15. The
conclusions are “substantial
evidence supports screening all
patients with diabetes to identify
those at risk for foot ulceration.
These patients might benefit from
certain prophylactic interventions
including patient education,
prescription foot wear, intensive
podiatric care, and evaluation for
surgical interventions.” If we look
at the quality research indicators,
this paper was in fact a Level 1
systematic review. The authors
looked at 165 articles, 22 of which
were randomised controlled trials.
The primary outcome measures
utilised in the study included
ulceration or amputation with
or without intervention. They
reviewed the factors resulting in
diabetic foot ulceration including
peripheral neuropathy, vascular
insufficiency, increased plantar
pressures, poor glucose control
and smoking and examined
interventions aimed at decreasing
ulceration or amputation. These
interventions included patient
or physician education, custom
foot wear, orthoses, preventative
surgery, and routine foot exams.
In reading this article and the
referenced supportive publications
from the systematic review, it was
clear that patient or physician
education, custom foot wear,
orthotics and preventative surgery
had no impact on decreasing foot
ulceration or amputation. The
isolated factor that decreased
ulceration and amputation was
routine foot examination. It
appears that seeing the patient in
a routine period of time such as
every six months does decrease
the risk of foot ulceration.
The conclusions stating that the
patients might benefit from certain
prophylactic interventions should
only include routine foot exams.
The other aspects of patient or
physician education, custom
foot wear, orthoses, preventative
surgery are only options and not
substantiated by this publication.
It is often the case that a significant
amount of resources and time
is spent on patient education,
physician education, custom foot
wear, orthotics and preventative
surgery for diabetic patients
suffering from ulcerations. The
support for these treatments might
actually be regarded as anecdote
rather than evidence. A better
summary that is supported by the
evidence would have included only
routine foot exams.
Mr. Naughton Dunn recognised the
importance of clinical outcomes
in assessing our patients and
stated in a paper published in
1922 “Orthopaedic surgery is so
closely associated with function
that perhaps in no other branch
of surgery is the patient in a better
position to judge of the practical
success or failure resulting from any
operative procedure”17. Naughton
Dunn was certainly ahead of his
time in recognising the importance
of patient derived outcomes.
The number one citation from
the Web of Science was a paper
entitled “Clinical rating systems
for the ankle-hindfoot, midfoot,
hallux, and lesser toes” 16. This
particular paper has been cited
over 1,200 times. The summary
statement from this article is “Four
rating systems were developed
by the American Orthopaedic
Foot and Ankle Society to provide
a standard method of reporting
clinical status of the ankle and foot.
The systems incorporate both
subjective and objective >>
0
Function (45 points)
Activity Limitations
No limitations
10
No limitation to daily activities, such as employment
responsibilities, limitation of recreational activities
7
Limited daily and recreational activities
4
Severe limitation of daily and recreational activities
0
Footwear requirements
Fashionable, conventional shoes, no insert required
10
Comfort footwear, shoe insert
5
Modified shoe or brace
0
MTP joint motion (dorsiflexion plus plantarflexion)
Normal or mild restriction (75o or more)
10
Moderate restriction (30o-74o)
5
Severe restriction (less than 30o)
0
IP joint motion (plantarflexion)
No restriction
5
Severe restriction (less than 10o)
0
MTP-IP stability (all directions)
Stable
5
Definitely unstable or able to dislocate
0
Callus related to hallux MTP-IP
No callus or asymptomatic callus
5
Callus, symptomatic
0
Alignment (15 points)
Good, hallux well-aligned
15
Fair, some degree of hallux malalignment observed, no symptoms 8
Poor, obvious symptomatic malalignment
0
Volume 02/ Issue 01 / January 2014
boa.ac.uk
Page 60
JTO Peer-reviewed Articles
Journal of Trauma and Orthopaedics: Volume 02, Issue 01, pages 57-62
Title: Evidence vs. Anecdote in Foot and Ankle Surgery
Author/s: Judith Baumhauer
factors into numerical scales
to describe function, alignment
and pain.” An example of the
hallux metatarsophalangealinterphalangeal scale is provided
in Table 3. From: Kitaoka HB,
Alexander IJ, Adelaar RS, Nunley
JA, Myerson MS, Sanders M.
Clinical rating systems for the
ankle-hindfoot, midfoot, hallux,
and lesser toes. Foot Ankle Int.
1994 Jul;15(7):349-53.
These clinical rating scales are
anatomic outcomes instruments
that are clinician derived and
administered. There are four
anatomic scales and each
includes the items of pain,
function and alignment in a point
allocation system that totals 100
points. It takes about five minutes
to complete.
Despite high numbers of citations
for this paper, many subsequent
publications have raised questions
over the validity and reliability of
the clinical rating scales18-20. This
paper, the number one cited foot
and ankle paper identified by the
Web of Science, is anecdotal and
proven to be inappropriate for
future use.
With the sun setting on the AOFAS
clinical scoring systems, comes
an opportunity to re-evaluate
what type of outcomes measures
might be appropriate for foot and
ankle. The PROMIS is a Patient
Reported Outcomes Measurement
System that has been developed
in collaboration with Northwestern
University in Chicago, IL and
National Institute of Health21. It
allows healthcare providers to
assess patient reported outcomes
through the utilisation of a
technique called item response
theory (IRT) and computer
adaptive testing (CAT). Through
the PROMIS system, patients are
asked questions in a variety of
different domains including lower
extremity physical functioning.
The American Orthopaedic Foot
and Ankle Society is currently
organising pilot projects to look
at the feasibility of the PROMIS
physical functioning CAT tool2224
. First steps included gathering
data from 10 academic centres
to optimise the data collection
and the utilisation process. With
the next step PROMIS will be
rolled out into a sample of private
practices that often do not have
the infrastructure for outcomes
assessment. Additionally, the
Society has been examining the
bank of questions that are currently
utilised in the lower extremity
physical functioning domain and
comparing it to other legacy scales
such as the clinical rating scales.
In summary, evidence is really
based on quality research. There
are quality measures that can be
used to evaluate publications.
We need to be critical in the
assessment of research that
influences the treatment of patients
to determine whether or not the
foundation of any research is
evidence or anecdote. All levels
of evidence have value; however,
taking research directly to our
patients needs a critical eye to
avoid dogma. n
References
1. Cook DJ, Jaeschke R, Guyatt
GH (1992). “Critical appraisal
of therapeutic interventions in
the intensive care unit: human
monoclonal antibody treatment
in sepsis. Journal Club of the
Hamilton Regional Critical Care
Group”. J Intensive Care Med 7
(6): 275–82.
2. Sackett DL, Rosenberg
WM, Gray JA, Haynes RB,
Richardson WS: Evidence
based medicine: what it
is and what it isn’t. BMJ.
1996;312(7023):71.
3. How to write a systematic
review. Wright RW, Brand
RA, Dunn W, Spindler KP.
Clin Orthop Relat Res. 2007
Feb;455:23-9.
4. Hunt KJ, Hurwit D: Use of
patient-reported outcome
measures in foot and
ankle research. J Bone
Joint Surg Am. 2013 Aug
21;95(16):e118(1-9)..
5. Barske HL, Baumhauer J:
Quality of research and level
of evidence in foot and ankle
publications. Foot Ankle Int.
2012 Jan;33(1):1-6.
6. Mann RA, Thompson FM:
Rupture of the posterior
tibial tendon causing flat
foot. Surgical treatment.
Bone Joint Surg Am. 1985
Apr;67(4):556-61.
7. Pinney S. Symposium:
evidence-based medicine:
what is it and how should it be
used? Foreward. Foot Ankle
Int. 2010 Nov;31(11):1033.
8. Baumhauer J: The use and
misuse of Level IV and Level V
evidence. Foot Ankle Int. 2010
Nov;31(11):1037-9.
9. Friedman MA, Woodcock
J, Lumpkin MM, Shuren JE,
Hass AE, Thompson LJ. The
safety of newly approved
medicines: do recent market
removals mean there is a
problem? JAMA. 1999 May
12;281(18):1728-34.
10. Hoffman JR: Rethinking case
reports. West J Med. 1999
May;170(5):253-4.
11. Rubenstein SA, Jenkin WM,
Conant MA, Volberding
PA: Disseminated Kaposi’s
sarcoma in male homosexuals.
J Am Podiatry Assoc. 1983
Aug;73(8):413-7.
12. Katz SL. From culture to
vaccine--Salk and Sabin.
N Engl J Med. 2004 Oct
7;351(15):1485-7.
13. Dunn, Naughton: Presidential
Address: The surgery
of muscle and tendon in
relation to infantile paralysis.
Proceedings of the Royal
Society of Medicine. Section
of Orthopaedics. October
2, 1928. John Bale, Sons &
Danielsson, pub.
14. Dunn, N: Calcaneo Cavus and
It’s Treatment. J Orthopaedic
Surg. 1:12:1919
15. Singh N, Armstrong DG, Lipsky
BA. Preventing foot ulcers in
patients with diabetes. JAMA
293(2):217-28, 2005.
Volume 02/ Issue 01 / January 2014
boa.ac.uk
Page 62
JTO Peer-reviewed Articles
Journal of Trauma and Orthopaedics: Volume 02, Issue 01, pages 57-62
Title: Evidence vs. Anecdote in Foot and Ankle Surgery
Author/s: Judith Baumhauer
16. Kitaoka HB, Alexander IJ,
Adelaar RS, Nunley JA,
Myerson MS, Sanders M.
Clinical rating systems for the
ankle-hindfoot, midfoot, hallux,
and lesser toes. Foot Ankle
Int. 1994 Jul;15(7):349-53.
17. Dunn N.Stabilizing Operation
in the Treatment of Paralytic
Deformities of the Foot. Proc
R Soc Med. 1922;15(Surg
Sect):15-22.
18. Guyton GP: Theoretical
limitations of the AOFAS
Scoring System: An Analysis
using Monte Carlo Modeling.
FAI 22:779; 2001.
19. Baumhauer JF, Nawoczenski
DA, DiGiovanni BF, Wilding
GE: Reliability and Validity
of the American Orthopaedic
Foot and Ankle Society Clinical
Rating Scale: A Pilot Study for
the Hallux and Lesser Toes.
FAI 27:1014, 2006.
20. Pinsker E and Daniels TR:
AOFAS Position Statement
regarding the future of the
AOFAS Clinical Rating
Systems. FAI 32:841, 2011.
21. Cella D, Yount S, Rothrock N,
Gershon R, Cook K, Reeve
B, Ader D, Fries JF, Bruce B,
Rose M; PROMIS Cooperative
Group. The Patient-Reported
Outcomes Measurement
Information System (PROMIS):
progress of an NIH Roadmap
cooperative group during its
first two years. Med Care. 2007
May;45(5 Suppl 1):S3-S11.
22. Hung M, Baumhauer JF, Latt
LD, Saltzman CL, SooHoo NF,
Hunt KJ; National Orthopaedic
Foot & Ankle Outcomes
Research Network. Validation
of PROMIS ® Physical Function
computerized adaptive tests
for orthopaedic foot and
ankle outcome research.
Clin Orthop Relat Res. 2013
Nov;471(11):3466-74.
23. Hung M, Nickisch F, Beals
TC, Greene T, Clegg DO,
Saltzman CL. New paradigm
for patient-reported outcomes
assessment in foot & ankle
research: computerized
adaptive testing. Foot Ankle
Int. 2012 Aug;33(8):621-6.
24.Hung M, Franklin JD, Hon SD,
Cheng C, Conrad J, Saltzman
CL. Time for a Paradigm
Shift With Computerized
Adaptive Testing of General
Physical Function Outcomes
Measurements. Foot Ankle
Int. 2013 Oct 7. [Epub ahead
of print]
Correspondence:
Judith Baumhauer, M.D., M.P.H.
Professor and Associate
Chair of Orthopaedic Surgery
University of Rochester
Medical School
601 Elmwood Avenue
Rochester NY 14642
Email: judy_baumhauer@URMC.
Rochester.edu
Don’t forget that videos from
Congress are now available
online, including all keynote
lectures and numerous
sessions on data and
research:
http://www.boneandjoint.org.
uk/boacongress2013/menu
This includes the popular and
thought-provoking Howard
Steel Lecture by Mark
Stevenson: ‘The Future ... and
what to do about it’.