Research and applications
A survey of SNOMED CT direct users, 2010:
impressions and preferences regarding content
and quality
Gai Elhanan,1,2 Yehoshua Perl,1 James Geller1
< Additional materials are
published online only. To view
these files please visit the
journal online (www.jamia.org/
content/18/Suppl_1.toc).
1
Department of Computer
Science, New Jersey Institute
of Technology (NJIT), Newark,
New Jersey, USA
2
Halfpenny Technologies, Inc,
Blue Bell, Pennsylvania, USA
Correspondence to
Dr Gai Elhanan, Computer
Science Department, New
Jersey Institute of Technology,
University Heights, Newark, NJ
07102-1982, USA;
[email protected]
Received 27 April 2011
Accepted 11 July 2011
Published Online First
11 August 2011
ABSTRACT
Objective Little information exists concerning SNOMED
CT (systematized nomenclature of medicinedclinical
terms) users. This report describes current impressions
and preferences of direct SNOMED CT users regarding
coverage, quality, and concept details, and the change
request mechanism.
Design A 43-question anonymous survey distributed
electronically to relevant online communities.
Measurements Data on user demographic
characteristics, modes and purposes of use, means and
frequencies of access, satisfaction with SNOMED CT
content coverage and quality and with the change
request mechanism were recorded.
Results The survey was conducted in January 2010 and
elicited 215 responses. Details regarding users’ profiles,
modes of use and access were reported elsewhere. The
coverage of SNOMED CT was perceived to be at least
85% complete by 42% of responders, and 60% were at
least satisfied with its quality. Various deficiencies were
encountered at least ‘somewhat often’ by 28e61% of
responders. Incorrect data were more bothersome than
missing data. Users indicated that significant resources
should be allocated to more consistent and complete
conceptual representations and to further enhance
content coverage. Enhanced synonym coverage and the
introduction of textual definitions were important to
users (54% and 63%, respectively).
Limitations A survey format with limited control over
recruitment and selection bias. Lack of information
regarding the SNOMED CT version used by responders.
Conclusion Despite overall satisfaction, direct users
indicated a strong desire to improve consistency, quality,
and completeness of conceptual representations and
concept details, as well as a continued desire to expand
coverage. The survey provides much needed data for
informed decisions regarding the use and development
goals of SNOMED CT. Focused periodical surveys are
warranted.
INTRODUCTION
In the current alphabet soup of US and worldwide
healthcare-related terminologies, SNOMED CT1
(systematized nomenclature of medicinedclinical
terms) is the premier clinical terminology.
SNOMED CT (SCT) is a comprehensive clinical
terminology that provides standard clinical content
and expressivity for clinical documentation and
reporting.2 Software applications can use its
concepts, hierarchies, and relationships as a common
reference point for data analysis.
SCT is poised to play a major role in US healthcare because of the health information technology
i36
(HIT) component (HITECH) of the American
Recovery and Reinvestment Act of 2009.3 HITECH
is designed to jumpstart the transition of US
medical institutions and providers to electronic
health information systems.4 The final rule5 for the
initial set of standards for HIT includes SCT as one
of two options to ‘enable a user to electronically
record, modify, and retrieve a patient’s problem list
for longitudinal care’. To accelerate the adoption
and ‘meaningful use’ of certified electronic health
records (EHRs) by providers and institutions,
a system of incentives and penalties is proposed.6
Use of SCT or the International Classification of
Diseases, 9th edition, Clinical Modification to
encode up-to-date problem lists of current and
active diagnoses for at least 80% of all patients is
proposed as one of the indications of ‘meaningful’
EHR use. Moreover, SCT is slated to become the
exclusive encoding system for problem lists by 2015.
Despite SCT’s momentum and many endorsements, actual use within clinical applications is
sporadic and not well understood, with notable
exceptions such as Kaiser Permanente’s (KP)
Convergent Medical Terminology (CMT7) and the
Veterans Health Administration’s Enterprise Reference Terminology (ERT8). Many vendors await
clearer business case and clients’ demand.9 Little is
known about the use of SCT embedded within HIT
applications, or about users who interact with SCT
in a direct manner.10 HIT end users are far removed
from SCT by enhanced ‘interface’ versions and
third party data subsets and are limited in their
ability to provide input. Hence, we focus our study
on the direct users. These are the designers, developers, managers, and implementers of HIT applications that utilize SCT, as well as researchers and
educators in the field of healthcare informatics. In
January 2010, we conducted a survey of direct users
of SCT. The survey’s data on direct user’s profiles
and use of SCT were presented at the AMIA 2010
symposium.11 Here we present the survey’s results
on users’ impressions and preferences related to
SCT’s content and qualities.
BACKGROUND
SNOMED CT is a description logic based clinical
terminology. Originating with SNOP (systematized
nomenclature of pathology),12 SNOMED underwent several transitions, the most important of
which was the convergence of SNOMED reference
terminology (SNOMED RT) and the UK clinical
terms version 3 (formerly known as the Read
Codes)13e15 in 1999. SCT contains more than
310 000 active concepts with an extensive network
J Am Med Inform Assoc 2011;18:i36ei44. doi:10.1136/amiajnl-2011-000341
Research and applications
of relationships among them. Since 2007, the International
Health Terminology Standards Development Organization
(IHTSDO16) has owned and is tasked with the continued
development, promotion, and implementation of SCT and its
derivative products. IHTSDO has representation from member
nations, vendors, standards organizations, researchers, and other
user groups.
Each SCT concept has a fully specified name, its unique
descriptive term, a preferred term, and, typically, a set of synonyms. Concepts are organized into 19 top-level, singly rooted
hierarchies to capture broad, clinically related groupings, such as
‘clinical finding’, ‘substance’, or ‘body structure’. Concepts
within one hierarchy are linked by IS-A (subsumption) relationships such that each hierarchy forms a directed acyclic
graph. Concepts also have ‘attribute relationships’ (attributes)
directed to other concepts. Attributes can serve in either definitional or qualification capacities. Each kind of attribute is
defined to span from a source hierarchy to a target hierarchy.2
Each concept is further classified by its status of logical definition: fully defined versus primitive. A primitive concept is
underspecified in the sense of not having enough attributes to
distinguish it from its parents.
The US National Committee for Vital and Health Statistics
views SCT as a comprehensive, high-quality terminology.17 In
content coverage studies, SCT’s precursors consistently
performed better than other sources.18e21 However, these
studies were conducted more than a decade ago. In 2003, SCT
was found to cover 88.4% of diagnosis/problem list terms used
by clinicians within a computerized physician order entry
system.22 In 2004, the Veterans Health Administration (VA)
evaluated SCT and concluded that it has promise as a coding
system for clinical problems.23 Brown et al24 found the sensitivities of SCT as a reference terminology and as an interface
terminology were measured at 83% and 53%, respectively, for
a general medical evaluation template. Rosenbloom et al25 found
SCT suitable to provide standardized representations of information created by two interface terminologies. The study
suggested, however, that enriching SCT semantics would
improve representation of the external terms.
Future ‘meaningful use’ of EHRs will be partly defined by
encoding problem lists by SCT codes.5 6 To facilitate use, the
NLM prepared the UMLS CORE (clinical observations recording
and encoding).26 Among the source terminologies in the UMLS,
SCT covers the highest percentage (81%) of the UMLS CORE
concepts. In the e-prescribing domain, the Food and Drug
Administration has adopted the VA and KP problem list subset
of SNOMED27 as the terminology to represent indications in
electronic labels. In an evaluation of mapping indication phrases,
SCT as a whole covered 90.3%, while the clinical finding
hierarchy covered 79.5%.28
Despite incremental improvements in coverage in the specific
domain of diagnosis/problem list, SCT may not yet be suitable
for implementation in EHRs. For example, the concept ‘acute
myocardial infarction’, despite being fully defined in SCT, is not
linked hierarchically to the concept ‘ischemic heart disease’ (a
primitive) (July 2010 release),29 nor is there any physiological
attribute linking it to the associated myocardial ischemic
process. No reverse associations exist either. In fact, when large
organizations use SCT in their HIT applications, significant
enhancements are required. KP’s CMT7 and the VA’s ERT8 are
such examples. However, the end results of such enterprise
efforts are not necessarily compatible with current standards
efforts, and cannot be copied by smaller institutions. We note
that KP has recently donated the CMT to the IHTSDO30 and
J Am Med Inform Assoc 2011;18:i36ei44. doi:10.1136/amiajnl-2011-000341
that access to the content to support HIT goals will be provided
by the IHTSDO and the NLM.31
Issues with SCT’s underlying structure and conceptual representations may affect potential secondary use such as decision
support. More than 50% of SCT’s concepts are primitives (July
2010 release) lacking the attributes necessary for conceptual
definitions. Issues with part of attribute representations, discrepancies in defined semantics, and definitional inconsistencies
between ancestors and descendants have been highlighted.32e34
Critical reviews of the present architecture of SCT have been
presented, addressing both logical and ontological issues.35 36
In 250 scientific publications over 40 years of use,10 few
studies have dealt with the use of SNOMED in clinical practice,
and 11 or fewer have dealt with SNOMED’s content coverage
or data quality. No surveys such as those of the UMLS userbase37 38 have been conducted in the last decade. Thus, despite
its vast potential, little is known about direct user satisfaction
with SCT and their perceptions and preferences regarding
associated shortcomings.
METHODS
We designed a 43-item questionnaire with five sections. The first
three sections surveyed demographics and employment experience information, purpose of use of SCT, and means and
frequency of access to SCT releases. The results of these sections
were published in an AMIA symposium paper.11 A fourth
part consisted of questions investigating the respondent’s satisfaction with SCT content coverage and quality. The last part
was designed to evaluate direct users’ involvement and satisfaction with the change request mechanism of SCT. This paper
covers the data obtained in the last two sections of the survey.
The questionnaire contained 31 mandatory questions and 12
optional questions, of which six were free-text questions. The
fourth and fifth sections contained 27 questions: six free-text, 16
mandatory, and five non-free-text optional questions. Four
questions allowed multiple choices. The survey was designed
using freely available Google Docs tools and was presented as
a Google Docs form on the web. Incomplete responses (ie, to
only some of the mandatory questions) could not be submitted.
The questionnaire can be viewed online.39
The call to participate in the survey was sent by email to
members of various user and discussion groups. Among the user
groups were the IHTSDO user community, the UMLS user
mailing list maintained by the NLM, 10 AMIA discussion
groups, as well as other ontology-related and nursing groups.
Additionally, a mailing list was compiled on the basis of
a PubMed search of authors of SNOMED-related articles
published since 2006. Each message included a short introduction to our research center (SABOC40) and stressed the importance of participation. The respondents were encouraged to
forward the questionnaire to their colleagues. All responses were
anonymous, and no employer-specific data were collected.
The survey was conducted between January 7 and January 25,
2010. The request for participation was sent in three phases to
the same distribution list. The initial call did not include
a deadline. A second call 12 days later included a deadline within
7 days. The third call was a last-chance participation reminder
within 24 h of deadline expiration.
All responses were collected and initially summarized by
Google Docs tools. Data were exported into an Excel spreadsheet
for processing of information contained in ‘Other ’ fields and for
further analysis. Sector analysis is used when appropriate. The
c2 test was used for statistical analysis where appropriate
(non-overlapping).
i37
Research and applications
between one professional category and other categories. Each
left column depicts the percentage of respondents in each category that indicated that they were members of that profession
alone.
To clarify sector analysis and to avoid ‘small numbers’ issues,
we grouped certain categories. The following categories were
created: researchers and educators (Res/Edu), engineers, developers, and programmers (Eng/Dev), physicians, nurses, and
medical students (Clinical), and administrators and managers
(Admin). For each of the above groups, corresponding subgroups
of respondents who were members of only a single group (pure)
were createddthat is, pRes/Edu, pEng/Dev, pClinical, and
pAdmin. The resulting groups and subgroups are presented in
figure 1.
RESULTS
A total of 215 responses were received. Between 87 and 190
responses were recorded for the non-free-text, non-mandatory
questions. For the free-text questions, between 49 and 113
responses were recorded. A spreadsheet containing the raw data
is publicly available.41 To maintain anonymity, free-text
responses were removed.
The results for the first three sections of the survey, covering
user demographics and uses of SCT, were published elsewhere.11
Of the respondents, 65% were from North America, 36% were
from academic or research institutions, 28% from the industry
sector, and 23% worked for governments.11 Although the survey
was conducted in January 2010, the results do not necessarily
reflect a specific release of SCT, especially since only 14% of users
indicated that they load new releases immediately.11
Coverage perceptions
In our AMIA symposium paper,11 we reported that, on a scale of
1e5 where 1 is ‘not at all’ and 5 is ‘very satisfied’, 58% of
respondents indicated that they are at least satisfied (4 or 5)
with SCT’s coverage, and 10% were very satisfied. However,
users were also asked two additional questions: ‘What is your
Profession’s overlap and segments
11
As we previously reported on the distribution of professions
among the respondents, the professional categories were not
disjoint. We observed substantial overlaps between professional
categories. Each right column in figure 1 displays the overlap
Pure: 23 (28%)
All: 83
Pure: 23 (31%)
All: 74
Pure: 17 (44%)
All: 39
Res/Edu
Eng/Dev
Clinical
Admin
94
44%
All
Respond.:
No.
%
%Pure
90
80
Pure
69
32%
70
60
54
25%
50
Lib
Stu-Oth
Admin
Manag
Stu-Med
38%
40
25
12%
Nurse
Phys
21
10%
18
8%
32%
9
4%
n
Res
br
ar
ia
t-O
th
er
ra
to
r
St
ud
en
er
Ad
m
in
is
t
M
an
ag
t-M
ed
.
St
ud
en
Nu
rs
e
ic
ia
n
Ph
ys
r
gi
ne
e
En
er
ev
el
op
at
or
Pr
og
./D
Ed
uc
rc
he
r
0
ea
2
1%
56%
2
1%
Eng
Prog/Dev
Edu
Li
33%
10
28%
28%
20
30%
23%
25
12%
43%
29
13%
30
Re
s
All
100
Pure: 44 (42%)
All: 104
Figure 1 Composition of professions. Each pair’s left column on the bar chart shows the ratio between members of only one specific profession
(pure) and the overall numbers of respondents that indicated participation in that profession as well as other professions (mixed) (see top legend). Each
pair’s right column depicts the internal distribution of responders within a profession. Please note that the proportion of pure members of any
profession on the left column is greater than the one depicted on the right because of non-excluded overlaps between the other professional groups
included in the right column. To avoid ‘small numbers’ issues, related professions were combined into four broad categories, listed on top. Above each
broad category are the numbers of its pure (and percentage) and all members. These combined groups form the basis for sectorial analysis. Admin,
administrator; Edu, educator; Eng, engineer; Lib, librarian; Manag, manager; Phys, physician; Prog./Developer or Prog/Dev, programmer/developer;
Res, researcher; Student-Med. or Stu-Med, student-medicine; Stu-Oth, student-other.
i38
J Am Med Inform Assoc 2011;18:i36ei44. doi:10.1136/amiajnl-2011-000341
Research and applications
overall perception of SNOMED-CT’s coverage/completeness of
your domains of interest?’ (Q19), and ‘How important is it to
expand SNOMED-CT’s coverage?’ (Q21, scale of 1e5, ‘not
important at all’ to ‘extremely important’). Although the questions did not explicitly specify whether they addressed pre- or
post-coordinated content, the responses to the following question
(Q23, 113 responses) clearly indicate that responders addressed it
in precoordinated context. Figure 2 summarizes the results for
Q19 through Q21 by professional groups. Coverage was considered to be more than 85% complete by 42% of respondents, and
12% considered it above 95%. Furthermore, 69% considered it
important (4 or 5) to expand content coverage, and 36% considered it extremely important. Within professional groups, Admins
and especially pAdmins tended to be less extreme regarding
coverage completeness, and were more likely to indicate importance of coverage expansion (p¼0.01 and p¼0.02, respectively).
Only 10% of those who indicated that they perceive SCT’s
domain coverage as 85% or more also indicated that coverage
level was not sufficient for their needs. On the other hand, 37%
of respondents indicated that a level of coverage completeness of
less than 85% may still be sufficient for their needs. Of users
that perceived coverage completeness as 85% or greater, 63%
indicated that it was at least important to expand coverage, as
were 67% of users who indicated that coverage is satisfactory for
their needs. Twelve out of 25 respondents (48%) that perceived
SCT’s coverage as better than 95%, and 55% of those who
indicated that coverage is very satisfactory for their needs still
indicated a desire to expand coverage.
Figure 3 depicts the respondents’ preferences (Q22) as to
which content areas should be expanded (a threshold of 20%
was used). ‘Clinical finding’ and ‘procedure’ were the two most
dominant candidate roots for content expansion (47% and 31%
of users, respectively). As figure 3 shows, the hierarchies targeted
for expansion differed by groups. Only ‘clinical finding’ was
targeted by all groups.
A free-text question (Q23) could not clearly identify
a predominant missing content area. In Q24, respondents to
Q23 were neutral regarding the urgency of introducing their
preferred missing content area.
Use of SCT’s root hierarchies
Each of SCT’s root hierarchies was used by at least 24% of the
respondents. The three most popular hierarchies were ‘clinical
finding’ (92%), ‘body structure’ (72%), and ‘procedure’ (71%)
(Q8). These were also the three most important hierarchies for
responders (83%, 42%, and 49%, respectively) (Q9). The three
least utilized hierarchies were ‘record artifact’, ‘physical force’,
and ‘environment or geographical location’ (24%, 25%, and 28%,
respectively). No reportable differences were observed per
professional groups or employment sectors for Q8 and Q9.
Concept information
In Q25 (scale of 1e5, ‘not satisfied at all’ to ‘very satisfied’), users
were asked about their satisfaction with the current level of
SCT’s concept details (ie, synonyms, hierarchy, and attributes).
Overall, 45% of respondents indicated they were at least satisfied
Figure 2 Coverage perception: Completeness (left), satisfaction (center), and desire (right) for coverage expansion. The categories for completeness
are by percentage ranges whereas the other two questions are on a scale of 1e5.
J Am Med Inform Assoc 2011;18:i36ei44. doi:10.1136/amiajnl-2011-000341
i39
Research and applications
Figure 3 Root hierarchies as candidates for content expansion by employment and professional groups. Shown are only root hierarchies that
received 20% or more of the votes within each group. The last column on the right of each group denotes a ‘None of the above’ selection (ie, no root
hierarchy requires expansion) and therefore was not subject to the 20% threshold.
with the current level of concept details (8% ‘very satisfied’), and
only 13% indicated they were not satisfied (1 or 2). Q26, Q27,
and Q29 (all questions on a scale of 1e5, ‘not important at all’ to
‘extremely important’) covered the perceived importance of
adding synonyms, attributes, or textual concept definitions
(respectively). Of the respondents, 54e60% indicated that it was
at least important to expand/add each of these parameters
(20e27% ‘very important’), despite the level of satisfaction
indicated above.
In sector analysis (figure 4), we observed that Admins tended
to report fewer negative impressions regarding the status of
concept details than pRes/Edus (p¼0.01). Administrators
assigned less importance to Q26, Q27, and Q29, whereas pRes/
Edus and pClinicals assigned the most importance to expansion
of concept attributes, and pEng/Devs and pClinicals to textual
definitions. More respondents from academia and research organizations assigned importance to expansion of SCT attributes
than those working in the industry (p<0.01 for both).
Frequency of SCT deficiencies and user attitudes toward them
More than 64% of responders indicated that they rarely
encounter missing or incorrect parents, incorrect attributes,
attribute values, or children (Q31). Missing concepts, synonyms,
attributes, or children were indicated as at least often encountered
by 23%, 17%, 9%, and 16% of responders, respectively (figure 5).
No reportable differences were observed for different groups.
i40
However, when encountering such deficiencies, respondents
from all groups indicated that they find incorrect content much
more bothersome than missing content (Q30). As shown in
figure 5, 54e60% of respondents indicated that they are ‘very
much’ bothered by incorrect parents, children, attributes, and
attribute values. Between 28% and 35% indicated that such
findings are only ‘somewhat’ bothersome. The attitude toward
missing content was more forgiving, and 49e67% indicated that
they are ‘somewhat’ bothered by such findings.
Quality and allocation of resources
Overall, 60% of respondents were at least satisfied with SCT’s
quality. Only 7% were very satisfied (Q34, scale of 1e5, ‘not
satisfied at all’ to ‘very satisfied’). Of all respondents, 6% indicated some degree of dissatisfaction (3e11%). In Q35 (scale of
1e5, ‘not important at all’ to ‘extremely important’), 63% of
respondents indicated that it is very important to conduct
quality assurance and auditing of SCT’s content, and 88%
indicated that it is at least somewhat important.
On independent scales (each up to 100%), respondents were
hypothetically willing to allocate at least 30% of IHTSDO’s
budget on the following activities: QA/auditing (67% of
respondents), coverage expansion (63%), and expansion of
concept details (56%) (Q36, figure 6). Of pAdmins, 88% were
willing to invest at least 30% of IHTSDO’s budget in QA/
auditing. Fewer pRes/Edus (51%) were willing to invest at least
J Am Med Inform Assoc 2011;18:i36ei44. doi:10.1136/amiajnl-2011-000341
Research and applications
Figure 4 Respondents’ attitudes toward SNOMED CT’s concept details: overall satisfaction and importance of expanding/adding synonyms,
attributes and textual definitions (right-to-left).
30% of IHTSDO’s budget in coverage expansion, and only 42%
of pEng/Devs were willing to do so for expanding concept
details.
SCT auditing and change request process
In Q38, 83 respondents (39% of all respondents) indicated that
they were actively involved in auditing of SCT (51% of
government employees, 59% of pAdmins, 30% of pClinicals),
and 79 respondents (37%) indicated that they submit SCT
change requests, but only 48 respondents (22%) indicated doing
both. Change request submissions were distributed quite evenly
between IHTSDO, National Centers, and CAP-STS (30%, 39%,
and 33%, respectively).
Q40 through Q42 (all three questions on a scale of 1e5, ‘not
satisfied at all’ to ‘extremely satisfied’) concentrated on overall
users’ satisfaction with the change request process, the rate of
acceptance, and the rate of implementation of change requests.
Most responses indicated a neutral position (55%, 48%, and
51%, respectively), with more respondents indicating some
degree of satisfaction (30%, 33%, and 26%, respectively).
Free-text questions
Six free-text questions were included in the survey. One hundred
and fifty participants (70%) responded to at least one free-text
question (2.5 free-text questions on average), and 25% responded
to at least four free-text questions. The responses were diverse
and beyond the scope of this paper. However, we observed
J Am Med Inform Assoc 2011;18:i36ei44. doi:10.1136/amiajnl-2011-000341
a common thread across the responses. Many free-text respondents indicated that they would rather have SCT concentrate on
improving quality than on expanding content coverage. Specifically, the issues of numerous primitive concepts, lack of desired
attributes, and overall patchy modeling and coverage were
common in this thread.
Those who use the change request process also indicated that,
although the process has improved recently, it seems to be
structured toward new content rather than for resolving existing
issues.
DISCUSSION
This is a timely survey for American HIT. After establishing
itself as a premier international clinical terminology, aspects of
SCT are soon to become mandated components of EHRs.5 In
discussions regarding the role of SCT in HIT applications, many
presumptions are being made. Direct users serve as an interface
between end users and terminology designers. Thus, they
present a balanced view of practical and conceptual issues
regarding SCT. However, little up-to-date information exists
about their attitudes toward SCT.10
This study suffers from the inherent deficiencies of its survey
format. Moreover, in this survey we had little control over
recruitment and selection bias, affecting our ability to generalize
our observations. Additionally, we did not control for the SCT
release addressed by the respondents. As reported previously,11
many direct users take their time to load new releases or use
i41
Research and applications
100%
90%
80%
Not al all
Rarely
70%
Somewhat often
Somewhat
Often
60%
All the time
Very much
50%
Q31 Q30
40%
30%
20%
10%
0%
Missing
concepts
Missing
synonyms
Missing
parents
Incorrect
parents
Missing
attributes
Incorrect
attributes
Incorrect
attribute
values
Missing
children
Incorrect
children
Figure 5 Frequency of encounters with (Q31, left column), and attitude toward (Q30, right column), various SNOMED CT deficiencies.
multiple versions (27%). However, we were pleasantly surprised
by the number of complete responses (215), and the overall
commitment of the respondents, as reflected by the high
number of detailed free-text responses. Despite the encouraging
number of responses, our ability to perform meaningful sectorial
analysis was limited by ‘small numbers’ issues and the multitude of perspectives to examine. We believe that it is likely that
the views presented here and in our previously published
report11 correctly reflect those of the community of direct SCT
users and provide much needed current information for
informed decision making.
The goals of SCT’s direct user community are not necessarily
aligned with its design. The tension between ‘reference’ and
‘interface’ terminologies was omnipresent in the responses. It is
also reflected in recent discussions within the IHTSDO member
community.42 43 One such discussion43 closely resembles many
of the opinions reflected in our respondents’ answers. Continued
focus on coverage expansion and precoordination versus
enhanced efforts to contain expansion for maintainability’s
sake, and improved consistency and concept representation are
dominant themes. However, our survey did not address pre- and
post-coordination capabilities in SCT, and the issue was rarely
discussed in the free-text responses. Our discussion of the study
is conducted in the context of this important, ongoing debate
within SCT’s direct user community.
The results indicate that most direct users are satisfied with
SCT’s coverage and quality, despite its limitations. These findings are a tribute to the individuals and organizations that
tirelessly push SCT forward. As responses to the free-text
i42
questions and other discussions42 43 indicate, some direct users
are frustrated by deficiencies in SCT, especially incorrect
conceptual representations. Respondents indicated that significant resources should be allocated to improve overall quality as
reflected by issues such as insufficient/incorrect concept details
and coverage. We note that even those who are satisfied, or very
satisfied with coverage, or many of those who call for better
quality, still indicate that they would like coverage further
increased. Our survey was not designed to detect a cut-off point
at which the desire to expand coverage is curtailed. In fact, such
a point may not exist.
More than 50% of SCT’s concepts (July 2010 release) are
primitives, existing logical definitions are suboptimal,32e36 only
36% of concepts have synonyms (averaging 0.51 per concept),
and by design, SCT does not include textual definitions.
Such findings may be perceived as shortcomings that may
limit usability for end users. To compensate and meet expectations, organizations may resort to developing or acquiring
external extensions and dedicated subsets, which may diverge
from current standardization efforts. Even in the context of
the HITECH initiative,3e6 the use of SCT’s concepts for
encoding problem lists is mostly in the context of dedicated
lists.26 27 In discussions,42 43 there seems to be a general agreement that SCT should not be used ‘as-is’ in patient-facing
applications.
Thus, it is important not to ignore the issues about which
users clearly expressed their preferences. It is at least important
(categories four and five on a 1e5 scale) to 55% of users to
expand synonym coverage; 54% desired to have more complete
J Am Med Inform Assoc 2011;18:i36ei44. doi:10.1136/amiajnl-2011-000341
Research and applications
Figure 6 Hypothetical allocation of resources (as percentages of IHTSDO’s budget) toward QA/auditing, coverage expansion, and expansion of
concept details.
attribute coverage and 60% wished to have textual concept
definitions. The latter is most likely a byproduct of SCT’s
success. As the user base expands, users indicate that they
require more than the ‘conceptual definitions only’ approach.
Similarly, LOINC44 recently added textual definitions/
descriptions45 to assist users.
The balance between the various qualities of a clinical
terminology changes as the user community and use patterns
evolve. As terminologies mature, users increasingly become
concerned with the correctness, consistency, and accuracy of the
terminology,46 47 similarly to software products.48 SCT direct
users’ desire to invest in quality assurance and improve quality
parameters reflects SCT’s mature status. The findings of this
study are reminiscent of those of a study on the UMLS.37
Governing bodies such as IHTSDO should pay close and
continuous attention to users’ feedback, even if it does not
conform to the original design, and adapt accordingly. The data
provided by this survey may be used for that purpose. However,
mechanisms for periodic focused assessments and feedback from
users are lacking, and should be established and formalized as
part of the life-cycle of such a terminology. The existing change
request mechanism should not be viewed as an alternative for
user feedback.
SCT evolved over many years into an extensive body of
knowledge. It is not clear that the preferences expressed in this
study are prioritized on the IHTSDO’s roadmap. Moreover,
responding to the expressed desires of direct users will require
J Am Med Inform Assoc 2011;18:i36ei44. doi:10.1136/amiajnl-2011-000341
large-scale efforts that are likely to drain existing resources and
interfere with ongoing and planned projects. Alternative mechanisms should be considered to allow rapid implementation of
such requests. Embracing the extended user community and
providing a more inclusive, controlled editing environment may
be conducive to attaining such goals. However, such operations
will require the development of effective quality assurance
processes.
The findings of this study and the preceding AMIA symposium paper11 provide the most current perspectives on the use
and perceptions of SCT by direct users. Since SCT is becoming
ubiquitous in today’s HIT, we propose that focused surveys be
conducted on a scheduled basis, to better track trends related
to its manner of use, perceived deficiencies, and help define
development and improvement goals.
CONCLUSION
Most direct users of SCTare at least somewhat satisfied with the
level of coverage, level of concept details and overall quality. For
most parameters, positive responses outweigh negative impressions. High levels of satisfaction or dissatisfaction are relatively
low. Users indicated continued desire to improve consistency,
quality, and completeness of conceptual representations and
concept details (synonyms, attributes, and textual definitions),
coupled with a continued desire to expand coverage. The survey
provides much needed, timely information for informed decisions regarding the use and goals of SCT. Similar surveys should
i43
Research and applications
be conducted periodically, so that changes in users’ expressed
attitudes can be tracked and help direct development goals.
22.
Funding This work was partially supported by the NLM under grant
R-01-LM008912-01A1.
23.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
REFERENCES
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
IHTSDO. SNOMED CT. http://www.ihtsdo.org/snomed-ct/ (accessed Aug 2010).
IHTSDO. SNOMED Clinical Terms User Guide. 2009. http://www.ihtsdo.org/
fileadmin/user_upload/Docs_01/SNOMED_CT/About_SNOMED_CT/Use_of_
SNOMED_CT/SNOMED_CT_User_Guide_20090731.pdf (accessed Aug 2010).
American Recovery and Reinvestment Act of 2009. Public Law 111-5. 111th
Congress. http://www.gpo.gov/fdsys/pkg/PLAW-111publ5/pdf/PLAW-111publ5.pdf
(accessed Jun 2011).
Blumenthal D. Launching HITECH. N Engl J Med 2010;362:382e5.
HHS. 45 CFR Part 170. RIN 0991- AB58. Health Information Technology: Initial Set of
Standards, Implementation Specifications, and Certification Criteria for Electronic
Health Record Technology; Final Rule. http://www.edocket.access.gpo.gov/2010/
pdf/2010-17210.pdf.
CMS. CMS-0033-P. RIN 0938-AP78. Medicare and Medicaid Programs; Electronic
Health Record Incentive Program. http://www.healthit.hhs.gov/portal/server.pt/
gateway/PTARGS_0_10741_876941_0_0_18/Meaningful%20Use_7.16.09.ppt
(accessed Aug 2010).
Dolin RH, Mattison JE, Cohn S, et al. Kaiser Permanente’s convergent medical
terminology. Stud Health Technol Inform 2004;107:346e50.
Lincoln MJ, Brown SH, Nguyen V, et al. U.S. Department of Veterans Affairs
Enterprise Reference Terminology strategic overview. Stud Health Technol Inform
2004;107:391e5.
Giannangelo K, Fenton SH. SNOMED CT survey: an assessment of implementation
in EMR/EHR applications. Perspect Health Inf Manag 2008;5:7.
Cornet R, de Keizer N. Forty years of SNOMED: a literature review. BMC Med Inform
Decis Mak 2008;8(Suppl 1):S2.
Elhanan G, Perl Y, Geller J. A survey of direct users and uses of SNOMED CT: 2010
status. AMIA Annu Symp Proc 2010;13:207e11.
College of American Pathologists. Systematized Nomenclature of Pathology.
Chicago, IL: College of American Pathologists, 1971.
Wang AY, Barrett JW, Bentley T, et al. Mapping between SNOMED RT and clinical
terms version 3: a key component of the SNOMED CT development process.
Proc AMIA Symp 2001:741e5.
Wang AY, Sable JH, Spackman KA. The SNOMED clinical terms development
process: refinement and analysis of content. Proc AMIA Symp 2002:845e9.
Spackman KA. Rates of change in a large clinical terminology: three years
experience with SNOMED Clinical Terms. AMIA Annu Symp Proc 2005:714e18.
IHTSDO. http://www.ihtsdo.org/about-ihtsdo/.
Cohn SP. Seventh Annual Report to Congress on the Implementation Of the
Administrative Simplification Provisions of the Health Insurance Portability and
Accountability Act of 1996. 2005. http://www.ncvhs.hhs.gov/050908rpt.htm
(accessed Aug 2010).
Humphreys BL, McCray AT, Cheh ML. Evaluating the coverage of controlled health
data terminologies: report on the results of the NLM/AHCPR large scale vocabulary
test. J Am Med Inform Assoc 1997;4:484e500.
Campbell JR, Carpenter P, Sneiderman C, et al. Phase II evaluation of clinical coding
schemes: completeness, taxonomy, mapping, definitions, and clarity. CPRI Work
Group on Codes and Structures. J Am Med Inform Assoc 1997;4:238e51.
Campbell JR, Payne TH. A comparison of four schemes for codification of problem
lists. Proc Annu Symp Comput Appl Med Care 1994:201e5.
Chute CG, Cohn SP, Campbell KE, et al. The content coverage of clinical
classifications. For The Computer-Based Patient Record Institute’s Work Group on
Codes & Structures. J Am Med Inform Assoc 1996;3:224e33.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
Wasserman H, Wang J. An applied evaluation of SNOMED CT as a clinical
vocabulary for the computerized diagnosis and problem list. AMIA Annu Symp Proc
2003:699e703.
Penz JF, Brown SH, Carter JS, et al. Evaluation of SNOMED coverage of Veterans
Health Administration terms. Stud Health Technol Inform 2004;107:540e4.
Brown SH, Rosenbloom ST, Bauer BA, et al. Direct comparison of MEDCIN and
SNOMED CT for representation of a general medical evaluation template. AMIA Annu
Symp Proc 2007:75e9.
Rosenbloom ST, Brown SH, Froehling D, et al. Using SNOMED CT to represent two
interface terminologies. J Am Med Inform Assoc 2009;16:81e8.
NLM. The CORE Problem List Subset of SNOMED CT. http://www.nlm.nih.gov/
research/umls/Snomed/core_subset.html (accessed Aug 2010).
NLM. UMLS Enhanced VA/KP Problem List Subset of SNOMED. http://www.umlsks.
nlm.nih.gov/uPortal/frame.jsp?umlsks-frame¼http://www.nlm.nih.gov/research/
umls/licensedcontent/vakpproblemlist.html&page_header¼VA/KP%20Problem%
20List (accessed Aug 2010).
Mantena S, Schadow G. Evaluation of the VA/KP problem list subset of SNOMED as
a clinical terminology for electronic prescription clinical decision support. AMIA Annu
Symp Proc 2007:498e502.
Rector AL, Brandt S, Schneider T. Getting the foot out of the pelvis: modeling
problems affecting use of SNOMED CT hierarchies in practical applications. J Am
Med Inform Assoc 2011;18:432e40.
IHTSDO. News. http://www.ihtsdo.org/index.php?id¼184&tx_ttnews[tt_news]¼
151&cHash¼ff2c089926d35950cbd6fc92a4f39821 (accessed Jun 2011).
NLM, UMLS. http://www.nlm.nih.gov/research/umls/cmt/cmt_faq.html (accessed
Dec 2010).
Schulz S, Hanser S, Hahn U, et al. The semantics of procedures and diseases in
SNOMED CT. Methods Inf Med 2006;45:354e8.
Schulz S, Hahn U, Rogers J. Semantic clarification of the representation of
procedures and diseases in SNOMED CT. Stud Health Technol Inform
2005;116:773e8.
Agrawal A, Elhanan G, Halper M. Dissimilarities in the logical modeling of
apparently similar concepts in SNOMED CT. AMIA Annu Symp Proc
2010;2010:212e16.
Schulz S, Suntisrivaraporn B, Baader F, et al. SNOMED reaching its adolescence:
ontologists’ and logicians’ health check. Int J Med Inform 2009;78(Suppl 1):S86e94.
Rector AL, Brandt S. Why do it the hard way? The case for an expressive
description logic for SNOMED. J Am Med Inform Assoc 2008;15:744e51.
Chen Y, Perl Y, Geller J, et al. Analysis of a study of the users, uses, and future
agenda of the UMLS. J Am Med Inform Assoc 2007;14:221e31.
Fung KW, Hole WT, Srinivasan S. Who is using the UMLS and howdinsights from
the UMLS user annual reports. AMIA Annu Symp Proc 2006:274e8.
SNOMED-CT Usage Questionnaire. http://www.spreadsheets.google.com/viewform?
hl¼en&formkey¼dDJYUHF5VWZZSjBSNGJkVk5iaWF2bXc6MA (accessed Jun
2011).
SABOC. http://cs.njit.edu/woohvr/SABOC/index.html (accessed Jul 2011).
SNOMED Questionnaire Data. https://www.spreadsheets.google.com/ccc?key¼
0Aol6i3iU6Nn2dDlmbnpMYXNucFFfR2MwWU9WWEtsd0E&hl¼en&authkey¼
CM7ft_QK (accessed Jun 2011).
IHTSDO. https://www.thecap.basecamphq.com/projects/388747/posts/31226264/
comments#63756401 (accessed Aug 2010).
IHTSDO. https://www.thecap.basecamphq.com/projects/4407802/posts/
34698173/comments#74405402 (accessed Aug 2010).
McDonald CJ, Huff SM, Suico JG, et al. LOINC, a universal standard for identifying
laboratory observations: a 5-year update. Clin Chem 2003;49:624e33.
Friedlin J, McDonald CJ. An evaluation of medical knowledge contained in
Wikipedia and its use in the LOINC database. J Am Med Inform Assoc
2010;17:283e7.
Min H, Perl Y, Chen Y, et al. Auditing as part of the terminology design life cycle.
J Am Med Inform Assoc 2006;13:676e90.
Geller J, Perl Y, Halper M, et al. Special issue on auditing of terminologies. J Biomed
Inform 2009;42:407e11.
Software Assurance. https://buildsecurityin.us-cert.gov/swa/ (accessed Jul 2011).
PAGE fraction trail=9
i44
J Am Med Inform Assoc 2011;18:i36ei44. doi:10.1136/amiajnl-2011-000341