05/04/2013
Coding with
Confidence
Hilary Vass
(Global Clinical Dictionary Manager
AstraZeneca UK Limited)
Tomás Moraleda
(Medical Officer – MSSO)
25th Annual
EuroMeeting
4-6 March 2013
RAI, Amsterdam
Netherlands
Disclaimer
The views and opinions expressed in the following PowerPoint slides are
those of the individual presenter and should not be attributed to Drug
Information Association, Inc. (“DIA”), its directors, officers, employees,
volunteers members,
volunteers,
members chapters,
chapters councils,
councils Special Interest Area
Communities or affiliates, or any organization with which the presenter is
employed or affiliated.
These PowerPoint slides are the intellectual property of the individual
presenter and are protected under the copyright laws of the United
States of America and other countries. Used by permission. All rights
reserved Drug Information Association
reserved.
Association, DIA and DIA logo are registered
trademarks or trademarks of Drug Information Association Inc. All other
trademarks are the property of their respective owners.
2
1
05/04/2013
About MedDRA
MedDRA was developed under the auspices of the
International Conference on Harmonisation of Technical
R
Requirements
i
t ffor Registration
R i t ti off Ph
Pharmaceuticals
ti l ffor
Human Use (ICH). The activities of the MedDRA
Maintenance and Support Services Organization (MSSO)
are overseen by an ICH MedDRA Management Board,
which is composed of the six ICH parties (EU, EFPIA,
MHLW, JPMA, FDA, PhRMA), the Medicines and
Healthcare products Regulatory Agency (MHRA) of the UK
UK,
the Health Canada, and the WHO (as Observer).
3
Learning Objectives
• Describe how to code clinical safety
data accurately and consistently with
MedDRA
• Apply the principles described in the
ICH-endorsed “MedDRA Term
Selection: Points to Consider”
Consider
document
4
2
05/04/2013
Workshop Overview
• MedDRA refresher
• “MedDRA
“M dDRA T
Term S
Selection:
l ti
P
Points
i t tto
Consider” document
• Browsing and coding tips and tricks
• Practical exercises
• Best
B t practices
ti
5
MedDRA Refresher
6
3
05/04/2013
MedDRA Definition
MedDRA is a clinically-validated
international medical terminology used by
regulatory authorities and the regulated
biopharmaceutical industry. The
terminology is used through the entire
regulatory process, from pre-marketing to
post marketing and for data entry
post-marketing,
entry,
retrieval, evaluation, and presentation.
7
Scope of MedDRA
OUT
Not a drug
dictionaryy
Patient
demographic
terms
Clinical trial study
design terms
IN
Frequency qualifiers
Diseases
Diagnoses
Signs
Symptoms
Therapeutic indications
Investigation names &
Numerical values for
qualitative results
results
Medical & surgical procedures
Medical, social, family history
Medication errors
Product quality, device issues
Severity descriptors
Terms from other
terminologies
Not an equipment, device,
diagnostic product dictionary
8
4
05/04/2013
MedDRA Structure
System Organ Class (SOC) (26)
High Level Group Term (HLGT) (334)
High Level Term (HLT) (1,717)
Preferred Term (PT) (20
(20,057)
057)
Lowest Level Term (LLT) (71,326)
MedDRA Version 16.0
9
9
System Organ Classes
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Blood and lymphatic system disorders
Cardiac disorders
Congenital, familial and genetic disorders
Ear and labyrinth disorders
Endocrine disorders
Eye disorders
Gastrointestinal disorders
General disorders and administration site
conditions
Hepatobiliary disorders
Immune system disorders
Infections and infestations
Injury, poisoning and procedural
complications
Investigations
Metabolism and nutrition disorders
•
•
•
•
•
•
•
•
•
•
•
•
Musculoskeletal and connective tissue
disorders
Neoplasms benign, malignant and unspecified
(i l cysts
(incl
t and
d polyps)
l )
Nervous system disorders
Pregnancy, puerperium and perinatal
conditions
Psychiatric disorders
Renal and urinary disorders
Reproductive system and breast disorders
Respiratory, thoracic and mediastinal disorders
Ski and
Skin
d subcutaneous
b
tissue
i
disorders
di d
Social circumstances
Surgical and medical procedures
Vascular disorders
10
5
05/04/2013
MedDRA hierachy
SOC = Cardiac disorders
HLGT = Cardiac arrhythmias
HLT = Rate and rhythm disorders NEC
PT = Arrhythmia
LLT
Arrhythmia
NOS
LLT
Arrhythmia
LLT
Dysrhythmias
LLT (Non-current)
Other specified cardiac
dysrhythmias
11
Non-Current Terms
• Non-current terms are flagged at the LLT
level within MedDRA
• Not recommended for continued use
• Retained within the terminology to
preserve historical data for retrieval and
analysis
• Terms that are vague, ambiguous, outdated truncated,
dated,
truncated or misspelled
• Terms derived from other terminologies
that do not fit MedDRA rules
12
6
05/04/2013
MedDRA Codes
• Each MedDRA term assigned an 8-digit
numeric code
• The code is non-expressive
• Codes can fulfill a data field in various
electronic submission types (e.g., E2B)
• Initially assigned alphabetically by term
starting with 10000001
– New terms are assigned sequentially
• Critical for “upversioning”
13
A Multi-Axial Terminology
• Multi-axial = the representation of a
medical concept in multiple SOCs
– Allows grouping by different classifications
– Allows retrieval and presentation via different
data sets
• Purpose of Primary SOC
– Determines which SOC will represent a PT
during cumulative data outputs
– Is used to support consistent data
presentation for reporting to regulators
14
7
05/04/2013
A Multi-Axial Terminology (cont)
SOC = Respiratory, thoracic and
mediastinal disorders
HLGT = Respiratory tract
infections
HLT = Viral upper respiratory
tract infections
SOC = Infections and
infestations
HLGT = Viral infectious
disorders
HLT = Influenza viral
infections
PT = Influenza
15
A Multi-Axial Terminology (cont)
• PTs in the following SOCs only appear in
that particular SOC and not in others; i.e.,
ie
they are not multi-axial:
– Investigations
– Surgical and medical procedures
– Social circumstances
16
8
05/04/2013
Rules for Primary SOC Allocation
• PTs for diseases, signs and symptoms are assigned to
prime manifestation site SOC
• Congenital
C
it l and
dh
hereditary
dit
anomalies
li tterms h
have SOC
Congenital, familial and genetic disorders as Primary
SOC
• Neoplasms terms have SOC Neoplasms benign,
malignant and unspecified (incl cysts and polyps) as
Primary SOC
– Exception: Cysts
y
and polyps
y have prime manifestation site SOC
as Primary SOC
• Infections and infestations terms have SOC Infections
and infestations as Primary SOC
17
Primary SOC Priority
• If a PT links to more than one of the
exceptions, the following priority will be
used to determine primary SOC:
1st: Congenital, familial and genetic
disorders
nd
2 : Neoplasms benign, malignant and
unspecified
nspecified (incl ccysts
sts and pol
polyps)
ps)
rd
3 : Infections and infestations
18
9
05/04/2013
Conditions vs. Investigations
PT
HLT
HLGT
Pregnancy test
positive
Reproductive Endocrine
Investigations
hormone
investigations
analyses
(incl sex
hormones)
Pregnancy
Normal
pregnancy,
labour and
delivery
Pregnancy,
labour,
delivery and
postpartum
conditions
SOC
Pregnancy,
puerperium
and perinatal
conditions
Be careful to distinguish between a condition and an
investigation or a result of an investigation
19
“MedDRA
Term Selection:
Points to Consider”
Document
10
05/04/2013
MedDRA PTC Documents
• There are two PTC
documents
– Term Selection
– Data Retrieval and
Presentation
• Using MedDRA is a
big step forward
• Using MedDRA the
same way is a leap
toward harmonization
21
www.diahom
Why Do We Need Coding
Conventions?
• Differences in medical aptitude of coders
• Consistency concerns (many more
“choices” to manually code terms in
MedDRA compared to older terminologies)
• Even with an autoencoder, will still need
manual coding and review
22
11
05/04/2013
MedDRA Term Selection:
Points to Consider
• An ICH-endorsed guide for MedDRA users
• Developed to promote medically accurate
and consistent use of MedDRA in
exchange of data (ultimately, for
“medically meaningful” retrieval and
analysis)
• Current version available on MedDRA
MSSO Web site
(http://www.meddramsso.com/subscriber_library_ptc.asp)
23
MedDRA Term Selection PTC (cont)
• In cases with more than one option for
g terms,, the “preferred
p
option”
p
is
selecting
identified but this does not restrict
MedDRA users from selecting one of the
alternative options. Organizations should
be consistent in their choice of option.
g ((Appendix)
pp
)
• Section 4.1 – Versioning
– 4.1.1 Versioning methodologies
– 4.1.2 Timing of version implementation
24
12
05/04/2013
General Term Selection Principles
•
•
•
•
•
•
•
•
•
•
Quality of Source Data
Quality Assurance
Do Not Alter MedDRA
Always Select a Lowest Level Term
Select Only Current Lowest Level Terms
When to Request a Term
Use of Medical Judgment in Term Selection
g More than One Term
Selecting
Check the Hierarchy
Select Terms for All Reported Information, Do Not Add
Information
25
Do Not Alter MedDRA
• MedDRA is a standardized terminology
pre-defined term hierarchyy
with a p
• Users must not make ad hoc structural
alterations, including changing the primary
SOC allocation
• If terms are incorrectly placed, or the
concept is missing
missing, submit a change
request to the MSSO
26
26
13
05/04/2013
Synonym Lists
• Can be derived from existing term lists or
directly from verbatims
• For recurring, but unusual, verbatims – onetime assignment to a MedDRA term
• Enforces consistency by limiting choices once
MedDRA term is assigned
• Increases likelihood of autoencoding “hit”
• Natural outgrowth of a legacy data conversion
• Maintenance required
27
Synonym List Examples
Verbatim
LLT
Throbbing above
temple
Aching all over head
Pulsing pain in head
Headache
Muscular pain in legs
Myalgia of lower
extremities
Comment
LLT Myalgia of
lower extremities is
a better choice than
LLT Muscular pain
since it captures
both the event and
body site
28
14
05/04/2013
Always Select a Lowest Level Term
• Lowest Level Term that most accurately
reflects the reported verbatim
information should be selected
• Degree of specificity may be
challenging
– Example: “Abscess on face”  select
“Facial
Facial abscess,
abscess ” not simply “Abscess”
Abscess
29
Term Selection Points
• Diagnoses and Provisional Diagnoses with or without
Signs and Symptoms
• Death and Other Patient Outcomes
• Suicide and Self-Harm
• Conflicting/Ambiguous/Vague Information
• Combination Terms
• Age vs. Event Specificity
• Body Site vs. Event Specificity
• Location Specific vs. Microorganism Specific Information
• Modification of Pre-existing Conditions
• Exposures During Pregnancy and Breast Feeding
• Congenital Terms
• Neoplasms
• Medical and Surgical Procedures
• Investigations
30
15
05/04/2013
Term Selection Points (cont)
• Medication/Administration Errors, Accidental Exposures and
Occupational Exposures
• Misuse, Abuse and Addiction
g
via Product
• Transmission of Infectious Agent
• Overdose, Toxicity and Poisoning
• Device-related Terms
• Drug Interactions
• No Adverse Effect and “Normal” Terms
• Unexpected Therapeutic Effect
• Modification of Effect
• Social Circumstances
• Medical and Social History
• Indication for Product Use
• Off Label Use
• Product Quality Issues
31
Diagnoses and Provisional Diagnoses
SINGLE DIAGNOSIS
DEFINITIVE DIAGNOSIS
Single diagnosis without signs
and symptoms
PROVISIONAL DIAGNOSIS
Single provisional diagnosis
without signs and symptoms
•Diagnosis (only possible option) •Provisional diagnosis (only
possible option)
Example: “Myocardial
infarction
f
”  select
“Myocardial infarction”
Example: “Possible myocardial
infarction
f
”  select
“Myocardial infarction” (select
term as if definitive diagnosis)
Similar principles apply for multiple diagnoses
32
16
05/04/2013
Diagnoses and Provisional Diagnoses
(cont)
SINGLE DIAGNOSIS
DEFINITIVE DIAGNOSIS
PROVISIONAL DIAGNOSIS
Single
g diagnosis
g
with signs/
g
symptoms
Single
g provisional
p
diagnosis
g
with
signs/symptoms
•Preferred: Diagnosis only
•Preferred: Provisional
diagnosis and signs/symptoms
Example: “Anaphylactic
Example: “Possible myocardial
reaction with rash, dyspnea,
hypotension and laryngospasm”
hypotension,
 select “Anaphylactic
reaction”
infarction with chest pain,
dyspnea, diaphoresis”  select
dyspnea
“Myocardial infarction” “Chest
pain”, “Dyspnea”, and
“Diaphoresis”
33
Diagnoses and Provisional Diagnoses
(cont)
SINGLE DIAGNOSIS
DEFINITIVE DIAGNOSIS
PROVISIONAL DIAGNOSIS
Single diagnosis with signs/
symptoms
Single provisional diagnosis with
signs/symptoms
•Alternate: Diagnosis and
signs/symptoms
•Alternate: Signs/symptoms
only (as provisional diagnosis
may change
Example: “Anaphylactic reaction Example: “Possible myocardial
with rash,
rash dyspnea,
dyspnea hypotension,
hypotension infarction with chest pain,
pain
and laryngospasm”  select
dyspnea, diaphoresis”  select
“Anaphylactic reaction”, “Rash”, “Chest pain”, “Dyspnea”, and
“Dyspnea”, Hypotension”, and “Diaphoresis”
“Laryngospasm”
34
17
05/04/2013
Diagnoses and Provisional Diagnoses
(cont)
• Always include signs/symptoms not associated
with diagnosis
g
Reported
LLT Selected
Myocardial infarction
Myocardial infarction, chest
Jaundice (note that jjaundice is
pain dyspnea
pain,
dyspnea, diaphoresis,
diaphoresis
not typically associated with
ECG changes and jaundice
myocardial infarction)
35
Conflicting/Ambiguous/
Vague Information
• First, attempt to obtain more specific information
Reported
LLT Selected
Hyperkalemia with a
serum potassium of 1.6
mEq/L
Serum potassium
abnormal
GU pain
Pain
Congestion
Unevaluable event
Comment
LLT Serum potassium
abnormal covers both of
the reported concepts
(note: serum potassium of
1.6 mEq/L is a low result,
not high)
“GU” could be either
“genito-urinary” or “gastric
ulcer”.
l
” Since
i
“ i ” is
“pain”
i
definite, select LLT Pain
“Congestion” reported
alone is vague; this can
refer to multiple organs
and physiologic processes
36
18
05/04/2013
Combination Terms
• One condition is more specific than the other
Reported
LLT Selected
Arrhythmia due to atrial
fibrillation
Atrial fibrillation
• A MedDRA combination term is available
Reported
p
LLT Selected
Retinopathy due to diabetes
Diabetic retinopathy
37
Combination Terms (cont)
• If splitting provides more clinical information,
select more than one term
• In
I allll cases off combination
bi ti tterms, apply
l medical
di l
judgment
Reported
LLT Selected
Diarrhea and vomiting
Diarrhea
Vomiting
Wrist fracture due to fall
Wrist fracture
Fall
38
19
05/04/2013
Location Specific vs. Microorganism
specific infection
• No MedDRA term including both microorganism
and anatomic location
Reported
LLT Selected
Preferred
Option
Chlamydial infection
Respiratory infection
Respiratory
chlamydial
infection
Comment
Represents both
microorganism
specific infection
and anatomic
location
Represents
location-specific
infection
Represents
microorganism
specific infection

Respiratory infection
Chlamydial infection
39
Body Site vs. Event Specificity
• MedDRA term includes body site and event
information
Reported
LLT Selected
Skin rash on face
Rash on face
• No MedDRA term that includes body site and
event. Event information has priority.
Reported
Skin rash on chest
LLT Selected
Comment
Skin rash
In this instance,
there is no available
term for a skin rash
on the chest
40
20
05/04/2013
Body Site vs. Event Specificity (cont)
• No MedDRA term that includes body site and
event. Exercise judgment; body site may take
priority.
Reported
Cyanosis
y
at injection
j
site
LLT Selected
Comment
Injection
j
site
reaction
Cyanosis implies a
generalized disorder.
In this example,
selecting
g LLT Cyanosis
y
would result in loss of
important medical
information and
miscommunication.
41
Procedure and diagnosis are reported
• If a procedure is reported with a diagnosis, the
preferred option is to select terms for both the
procedure and diagnosis. Alternatively, select a
term only for the diagnosis.
Reported
LLT Selected
Preferred Option
Liver transplantation due to liver
injury
Liver transplantation Liver injury

Comment
Selecting term for
the procedure may
indicate severity of
the condition
Liver injury
42
21
05/04/2013
EU PV Directive:
Impacts on MedDRA
and MTS:PTC
Document
Directive Covered Topics
• Impacts on MedDRA and MTS:PTC
document
• ISO ICSR [E2B (R3)]
• Patient Reporting
• Safety Signal Detection Responsibilities
• New CT 3 (Clinical Trials Safety
Reporting)
www.diahom
44
Drug
Inform
22
05/04/2013
EU PV Directive
http://ec.euroa.eu/health/files/eudralex/vol-1/dir_2010_84_en.pdf
Drug Information Association
45
www.diahome.org
EU Pharmacovigilance Directive
Chapter 17
p
ap
pharmacovigilance
g
Member States should operate
system to collect information that is useful for the
monitoring of medicinal products, including
information on suspected adverse reactions arising
from use of a medicinal product within the terms of
the marketing authorisation as well as from use
outside the terms of the marketing authorisation,
authorisation
including overdose, misuse, abuse and medication
errors, and suspected adverse reactions associated
with occupational exposure.
www.diahom
46
Drug
Inform
23
05/04/2013
Amended Definition of
Adverse Reaction
• Noxious and unintended effects resulting from authorized
use of a medicinal product at normal doses
• There is regulatory interest in “special
special situations
situations”
•
•
•
•
•
•
Off label use
Overdose
Misuse
Abuse
Medication errors
Occupational exposure
• MedDRA covers most such concepts and can expand as
needed through Change Requests
• MTS:PTC provides definitions and coding examples.
47
Medication Errors
48
24
05/04/2013
Medication Errors (cont.)
• Appendix B in MedDRA Introductory Guide
• To assist in understanding and appropriate
use of medication error and product quality
issue terms in regulatory reporting
• Medication error descriptions developed by
FDA and MSSO
• Note that Occupational Exposures has
been added to section name
49
Medication Errors (cont.)
• 3.15.1 Medication/Administration Errors
– 3.15.1.1 Medication errors reported with clinical
consequences
– 3.15.1.2 Medication errors and potential medication
errors reported without clinical consequences
– 3.15.1.3 Medication errors in the context of labeled
i t
interactions
ti
– 3.15.1.4 Do not infer a medication error
50
25
05/04/2013
Medication Errors and Product
Quality Issues
• There can be overlap between product quality
issues and medication errors
• An example of both a product quality issue and
a medication error is provided
51
Accidental Exposures
52
26
05/04/2013
Occupational Exposures
53
Occupational Exposures (cont.)
54
27
05/04/2013
Misuse, Abuse and Addiction
55
Misuse – Terms in MedDRA
• Currently 7 terms in MedDRA
• Note that misuse is considered intentional
• LLT Unintentional device misuse is therefore not a
misuse concept
56
28
05/04/2013
Misuse (cont.)
57
Misuse (cont.)
58
29
05/04/2013
Abuse
59
Abuse (cont.)
60
30
05/04/2013
Addiction
61
Addiction (cont.)
62
31
05/04/2013
Expansion of Medication Error
Terms
• Prior to MedDRA Version 8.0, only one term
existed - PT Medication error
• Medication error section expanded in v8.0
– Added HLGT Medication errors in SOC Injury,
poisoning and procedural complications
•
•
•
•
•
HLT Maladministrations
HLT Medication errors due to accidental exposures
HLT Medication monitoring errors
HLT Overdoses
HLT Medication errors NEC
63
Off Label Use
3.27 – Off Label Use
The concept of “off label use” relates to situations
where the product is intentionally used for a medical
purpose not in accordance with the authorized
product information.
64
32
05/04/2013
Off Label Use (cont.)
65
Off Label Use (cont)
66
33
05/04/2013
Off Label Use (cont)
67
FDA- Defined
Coding Errors
34
05/04/2013
FDA-Defined Coding Errors
• Missed Concepts
– All medical concepts described after the product is
taken
k should
h ld be
b coded
d d
– Example: “The patient took drug X and developed
alopecia, increased LFTs and pancreatitis”.
Manufacturer only codes alopecia and increased
LFTs (missed concept of pancreatitis)
– Example: “The patient took drug X and developed
interstitial nephritis which later deteriorated into renal
failure”. Manufacturer only codes interstitial nephritis
(missed renal failure concept)
Acknowledgement: Dr. Toni Piazza-Hepp, Office of Surveillance
and Epidemiology, CDER
69
FDA-Defined Coding Errors (cont)
• “Soft Coding”
– Selecting
g a term which is both less specific
p
and less
severe than another MedDRA term is “soft coding”
– Example: “Liver failure” coded as hepatotoxicity or
increased LFTs
– Example: “Aplastic anemia” coded as unspecified
anemia
– Example: “Rash
Rash subsequently diagnosed as Stevens
Johnson syndrome” coded as rash
Acknowledgement: Dr. Toni Piazza-Hepp, Office of Surveillance
and Epidemiology, CDER
70
35
05/04/2013
Animal Pharma: Challenging
Verbatims
–
–
–
–
–
–
–
–
–
–
Deer ria
Big fat ugly cow
Hippo tension
Wanted to take an elephant dump
Mousy feeling in chest
Beasting R arm
Menstrual clams
Seeing
gp
people
p in room,, seeing
g chickens at window
Seeing stars and chicken farting
Patient recently began new job where he works
around chicken wings and barbecue sauce
71
MedDRA
Data Retrieval
and Presentation:
Points to Consider
36
05/04/2013
Data Retrieval and Presentation PTC
• An ICH-Endorsed Guide for MedDRA users on Data
Output
• Developed by an ICH Expert Working Group
• Provides data retrieval and presentation options for
industry or regulatory purposes
• Objective is to promote understanding of implications
that various options for data retrieval have on
accuracy and consistency of final output
• Current version available on MedDRA MSSO Web site
(http://www.meddramsso.com/subscriber_library_ptc.asp)
73
Data Retrieval PTC: Points Addressed
• General Principles
–
–
–
–
–
Qualityy of Source Data
Documentation of Data Retrieval and Presentation Practices
Do Not Alter MedDRA
Organization-Specific Data Characteristics
Characteristics of MedDRA that Impact Data Retrieval and
Analysis
– MedDRA Versioning
• General Queries and Retrieval
• Standardised MedDRA Queries
• Customized Searches
74
37
05/04/2013
Definition of SMQ
• Result of cooperative effort between CIOMS and ICH
(MSSO)
• Groupings of terms from one or more MedDRA
System Organ Classes (SOCs) related to defined
medical condition or area of interest
• Included terms may relate to signs, symptoms,
diagnoses, syndromes, physical findings, laboratory
and other physiologic test data
data, etc
etc., related to
medical condition or area of interest
• Intended to aid in case identification
75 75
SMQs in Production - Examples
As of Version 16.0, a
total of 90 in production
•
•
•
•
•
•
•
•
Agranulocytosis
Anaphylactic reaction
Cerebrovascular disorders
Convulsions
Depression and
suicide/self injury
suicide/self-injury
Hepatic disorders
Ischaemic heart disease
Lack of efficacy/effect
• Peripheral neuropathy
• Pregnancy and neonatal
topics
• Pseudomembranous colitis
• Rhabdomyolysis/myopathy
• Severe cutaneous adverse
reactions
• Systemic lupus
erythematosus
• Chronic kidney disease
• Malignant lymphomas
• Hypersensitivity
76
38
05/04/2013
Thank You
39
05/04/2013
Practical
Experience
Applying Coding
Principles and
Conventions
Hilary Vass
(Global Clinical Dictionary Manager
AstraZeneca UK Limited)
Tomás Moraleda
(Medical Officer – MSSO)
Overview
• Examples: assessing verbatims and
selecting MedDRA® terms
• Interactive Ex.s
• Pitfalls and solutions
• Tips for coding medication errors and
product quality issues and for handling
abbreviations
• Sharing best practices
2
1
05/04/2013
Assessing the reported term
• First step: Consider what is being reported
–
–
–
–
–
–
–
–
–
–
Is it a clinical condition - Diagnosis, sign or symptom?
Is it an indication?
Is it a test result?
Is it trauma?
Is it a procedure?
Is it a medication error?
The type of report will
influence the way you
Is it a product quality issue?
search for a suitable LLT.
Is it a social circumstance?
It may indicate in which
Is it a device issue?
SOC you expect to find
the closest match.
Is it a procedural complication?
– Is it a combination of these?
3
MedDRA Browsing Tips
•
•
•
•
•
•
•
A good browser is a key component
Use “top-down” and “bottom-up” approaches
First, try using actual words from reporter
Consider synonyms, e.g., “Liver” and “Hepatic”
Use word stems, e.g., “Pancrea”
Search different word orders, “and”, “or”, etc.
Use available resources for difficult verbatim terms (web
search medical dictionaries
search,
dictionaries, colleagues)
• Check the hierarchy
• Look at the “neighbors”
4
2
05/04/2013
Example 1: Complications and
Outcomes
• “Death due to liver failure secondary to
Hep B liver cirrhosis”
cirrhosis
5
Example 1: Assessing the Verbatim
• “Death due to liver failure secondary to
Hep B liver cirrhosis”
cirrhosis
– Hep B = original medical condition
– Liver cirrhosis = complication of Hep B
– Liver failure = complication of liver cirrhosis
and Hep B, proximal cause of death
– Death = outcome
6
3
05/04/2013
Example 1: Term Selection
• “Death due to liver failure secondary to Hep B
liver cirrhosis”
• Check for applicable combination terms – none
• Hep B: LLT Hepatitis B  SOC Infections and
infestations
– Do not use non-current LLT Hep B (PTC 2.5)
– Query if abbreviation is unacceptable
– Would
W ld nott need
d to
t code
d if patient
ti t known
k
to
t
have had Hep B at baseline (PTC 3.5.5)
7
Example 1: Term Selection, cont.
• Liver cirrhosis, Liver failure: LLT Liver
cirrhosis and LLT Liver failure both 
SOC Hepatobiliary disorders
– Neither term is adequately expressed by
the other or by LLT Hepatitis B
• Death: outcome
– Do not code (PTC 3
3.2.1)
2 1)
8
4
05/04/2013
Example 2: Complications and
Provisional Diagnoses
• “Sepsis leading to shock (from
spontaneous bacterial peritonitis or bowel
perforation)”
9
Example 2: Assessing the
Verbatim
• “Sepsis leading to shock (from
spontaneous bacterial peritonitis or bowel
perforation)”
10
5
05/04/2013
Example 2: Term Selection
• “Sepsis leading to shock (from spontaneous
bacterial peritonitis or bowel perforation)”
• Check for applicable combination terms:
– LLT/PT Septic shock found
– Better than coding to both LLT/PT Sepsis and
LLT/PT Shock (preserves relationship)
• SBP or bowel perforation:
– Potential causes of the septic shock
– Uncertain if both conditions were actually
present (likely a differential diagnosis)
11
Example 2: Term Selection, cont.
• Code both provisional diagnoses for best
p
capture:
– LLT Spontaneous bacterial peritonitis
PT Peritonitis bacterial
SOC Infections and infestations
– LLT Bowel perforation
PT Intestinal perforation
SOC Gastrointestinal disorders
12
6
05/04/2013
Examples 1 & 2: Actual Verbatim
• “Death due to liver failure secondary to
Hep B liver cirrhosis and sepsis from
spontaneous bacterial peritonitis or bowel
perforation”
13
Ex. 1: Complications. Which
ones would you choose?
• “Retinal disease from HIV with near total
blindness (R and L)”
L)
– LLT Retinal damage
– LLT Retinal disorder
– LLT HIV disease
– LLT Blindness
– LLT HIV retinopathy
– LLT Blindness, both eyes
14
7
05/04/2013
Ex. 1: Suggested Terms
• “Retinal disease from HIV with near total
)
blindness ((R and L)”
– LLT Retinal damage (goes to injury SOC)
– LLT Retinal disorder (combo term available)
– LLT HIV disease (combo term available)
– LLT Blindness (not the most specific LLT)
– LLT HIV retinopathy (good combo term)
– LLT Blindness, both eyes (most specific
term for this reported condition)
15
Ex.1: Why not also code the HIV?
• If patient known to have had HIV at
baseline ((can’t tell here))
• Combination term was available
• Check hierarchy for PT AIDS retinopathy:
– Primary SOC: Infections and infestations, HLT
Retroviral infections
– Secondary SOCs: Eye disorders,
disorders Immune
system disorders
• Coded event will display in infections SOC
16
8
05/04/2013
Ex. 2: Lab Tests. Which ones
would you choose?
• “Testing showed increased serum
creatinine and BUN,
BUN with increased
BUN/creatinine ratio.”
a) LLT Increased serum creatinine, LLT BUN
increased
b) LLT Increased serum creatinine, LLT BUN
increased LLT Blood urea nitrogen/creatinine
increased,
ratio increased
c) LLT Renal function tests NOS abnormal
17
Ex. 2: Suggested Terms
• “Testing showed increased serum creatinine
and BUN,, with increased BUN/creatinine ratio.”
a) LLT Increased serum creatinine, LLT BUN
increased (does not capture the abnormal ratio)
b) LLT Increased serum creatinine, LLT BUN
increased, LLT Blood urea nitrogen/creatinine ratio
increased
c)) LLT Renal
R
l ffunction
ti ttests
t NOS abnormal
b
l (do
(d nott
lump together per PTC 3.14.4; loss of specificity;
also not NOS, as abnormalities were specified)
18
9
05/04/2013
Pitfalls and Solutions, 1
• Do not lose information by lumping
• Do not add information by coding to a
diagnosis that is not explicitly stated in the
verbatim  Reread verbatim carefully,
resist any internal or external pressure to
code to unstated diagnoses
– If the physician who actually saw the patient
was not comfortable committing to a
diagnosis, why should you be?
19
Pitfalls and Solutions, 2
• Terms that sound like something they aren’t (eg,
LLT Blood urine))  Read through
g the other PTs
under that HLT
– LLT/PT Blood urine vs. LLT/PT Blood urine present
(lab test name vs. lab test result)
• Not finding the most specific term  Combine
bottom-up and top-down searches, use judicious
translations
– For “osteoporosis due to advanced age”, select
LLT/PT Senile osteoporosis, not LLT/PT Osteoporosis
20
10
05/04/2013
Pitfalls and Solutions, 3
• “Autocoder specials” (ie, inappropriate
terms selected byy autocoder))  Review
all weight-based autocoding carefully,
apply medical judgment
• Inadvertently sacrificing detail at the PT
level to capture detail in an LLT  Check
hierarchyy
– For “intermittent migraine headaches”, use
LLT Migraine headache  PT Migraine, not
LLT Intermittent headache  PT Headache
21
Ex. 3: Medication Errors and
Product Quality Issues
• “Eye clinic nurse reported accidentally
using a vial of an unpreserved injectable
medication on more than one patient and
complained that the warning on the label
stating that it was for single use only was
too small to read. One of the patients
de eloped an injection site infection
developed
infection.”
22
11
05/04/2013
Tips for Coding Medication Errors
and Product Quality Issues
• Spend some time reading through the available
terms and hierarchies, for familiarity
y
• Read the LLTs to better understand the meaning
of the PTs:
– For example, PT Wrong technique in drug usage
process includes LLTs like Wrong injection technique,
Tablet crushed incorrectly, and Inhalation not
administered correctly.
correctly
• Code all elements: medication errors, product
quality issues, and associated adverse events
23
Ex.3: Assessing the Verbatim
• “Eye clinic nurse reported accidentally
using a vial of an unpreserved injectable
medication on more than one patient and
complained that the warning on the label
stating that it was for single use only was
too small to read. One of the patients
de eloped an injection site infection
developed
infection.”
24
12
05/04/2013
Ex. 3: Medication Error
Which option would you choose?
• “Eye clinic nurse reported accidentally
using a vial of an unpreserved injectable
medication on more than one patient.”
a) LLT Medication error
b) LLT Circumstance or information capable of
leading to a medication error
c)) LLT Multiple
M lti l use off single-use
i l
product
d t
d) LLT Poor quality drug administered
25
Ex. 3: Suggested Terms
• “Eye clinic nurse reported accidentally
using a vial of an unpreserved injectable
medication on more than one patient.”
a) LLT Medication error (not specific)
b) LLT Circumstance or information capable of
leading to a medication error (error occurred)
c)) LLT Multiple
M lti l use off single-use
i l
product
d t
d) LLT Poor quality drug administered
(presumes poor quality)
26
13
05/04/2013
Ex. 3: Product Quality Issue
Which option would you choose?
• “Eye clinic nurse complained that the
warning on the label stating that it was for
single use only was too small to read.”
a) LLT Product label issue
b) LLT Product quality issue
c) LLT Product label missing text
27
Ex. 3: Suggested Terms
• “Eye clinic nurse complained that the
warning on the label
label…was
was too small to
read.”
a) LLT Product label issue (note: label refers to
the actual label on the product)
b) LLT Product quality issue (too general)
c)) LLT Product
P d t llabel
b l missing
i i ttextt (text
(t t was
there, just too small)
28
14
05/04/2013
Ex. 3: Adverse Effect
Which option would you choose?
• “One of the patients developed an
injection site infection
infection.”
a) LLT Infection
b) LLT Injection site infection
c) LLT Eye infection
29
Ex. 3: Suggested Terms
• “One of the patients developed an
injection site infection
infection.”
a) LLT Infection (too general)
b) LLT Injection site infection
c) LLT Eye infection (presumes eye)
30
15
05/04/2013
Ex. 4: Overdose & Self-Harm
• “Patient with reactive depression due to
the recent passing of his spouse
attempted suicide by intentionally taking a
handful of his diuretic tablets.”
31
Ex.4: Assessing the Verbatim
• “Patient with reactive depression due to
the recent passing of his spouse
attempted suicide by intentionally taking a
handful of his diuretic tablets.”
32
16
05/04/2013
Ex. 4: Depression/Loss
Which options would you choose?
• “Patient with reactive depression due to
the recent passing of his spouse”
spouse
1) LLT Depression or LLT Reactive depression?
2) LLT Death of spouse or no code at all?
SOC Social circumstances
33
Ex. 4: Suggested Terms
• “Patient with reactive depression due to the
recent p
passing
g of his spouse”
p
1) LLT Depression or LLT Reactive depression?
LLT Reactive depression (same PT, but more specific)
2) LLT Death of spouse or no code at all?
SOC Social circumstances
See PTC 3.23 (generally do not enter as AE; could
enter under medical/social history; consult local
conventions)
34
17
05/04/2013
Ex. 4: Overdose/Self-Harm
Which options would you choose?
• “Patient attempted suicide by intentionally
t ki a h
taking
handful
df l off his
hi diuretic
di ti ttablets.”
bl t ”
1) LLT Attempted suicide or
LLT Suicidal behavior?
2)) LLT Diuretic abuse or
LLT Drug overdose deliberate self-inflicted or
LLT Drug toxicity due to intentional overdose?
35
Ex. 4. Suggested Terms
• “Patient attempted suicide by intentionally
taking
g a handful of his diuretic tablets.”
1) LLT Attempted suicide (exact match)
LLT Suicidal behavior (different PT)
2) LLT Diuretic abuse (refers to something else)
LLT Drug
ug o
overdose
e dose de
deliberate
be ate se
self-inflicted
cted
LLT Drug toxicity due to intentional overdose
(no toxicity was reported)
36
18
05/04/2013
Ex. 5. “Exposures”
While preparing to administer of a blood
transfusion,, a nurse stabbed herself with the
needle and subsequently developed acute
hepatitis B, with dark urine, fever, loss of
appetite, yellow skin and mono-arthritis. Lab
tests showed HBsAg 12 weeks later. Further
investigation revealed that transfusion blood
was contaminated with HBV and that the
nurse had received one only dose of HBV
vaccine.
Ex. 5. Assessing verbatims
While preparing to administer of a blood
transfusion,, a nurse stabbed herself with the
needle and subsequently developed acute
hepatitis B, with dark urine, fever, loss of
appetite, yellow skin and mono-arthritis. Lab
tests showed positive HBsAg 12 weeks
later Further investigation revealed that
later.
transfusion blood was contaminated with
HBV and that the nurse had received one
only dose of HBV vaccine.
19
05/04/2013
Ex. 5. Adverse Event. Which ones
would you choose?
• Acute hepatitis B
• “dark
dark urine”
urine ..?
?
– Bilirubin urine
– Urine bilirubin
increased
– Choluria
• Skin? ..
– Yellow skin
– Jaundice
• Monoarthritis
• Investigations
– HBsAg
• Appetite lost
39
Ex. 5. AE. Suggested terms
• Acute hepatitis B
• “dark
dark urine”
urine ..?
?
– Bilirubin urine
– Urine bilirubin
increased
– Choluria
• (since it is an
observation, not a test)
• Skin? ..
– Yellow skin
– Jaundice
• Monoarthritis
• Investigations
– HBsAg
• Appetite lost
40
20
05/04/2013
Ex. 5 (AE). [ Dx + S/S = Dx only ]
Single definitive diagnosis with signs/symptoms:
Preferred diagnosis only
• Acute hepatitis B
• “dark urine” ..?
– Bilirubin urine
– Urine bilirubin
increased
– Choluria
Ch l i
• Skin? ..
– Yellow skin
– or Jaundice ?
• Monoarthritis
• Investigations
– HBsAg
• Appetite lost
note that, despite hepatitis B has pain in
joints, a localized monoarthritis is
not typically associated with it)
41
Ex. 5 (AE). Investigation consistent
with Dx
When investigation results are reported with a diagnosis,
select only a term for the diagnosis if investigation results
are consistent with the diagnosis.
• Acute hepatitis B
• “dark urine” ..?
– Bilirubin urine
– Urine bilirubin
increased
– Choluria
• Skin? ..
– Yellow skin
– Jaundice
• Monoarthritis
• Investigations
– HBsAg
• Appetite lost
42
21
05/04/2013
Ex. 5. Exposure. Which would you
choose?
– Transmission of an
infectious agent via product
• Transfusion-transmitted
i f ti
infectious
di
disease ?
– Hepatitis B immunisation
– Incomplete course of
vaccination
– Occupational exposure
while preparing product for
administration
– Needle stick/puncture
– Accidental injury
– Accidental exposure
while
e ad
administering
se gd
drug
ug
– Exposure to device
contaminated with body
fluid
• (under PT Exposure to
contaminated device)
– Exposure
p
to bodyy fluid
– Exposure via blood
• (under PT Injury associated
with device)
43
Ex. 5. Exposure. Table 1
SOC
HLGT
PT
LLT
Genrl-
Complications associated with Complications associated
device
with device NEC
HLT
Injury associated with
device
Needle stick/puncture
Infec-
Ancillary infectious topics
Transmission of an
Transmission of an infectious
infectious agent via product agent via product
Infectious transmissions
Transmission of an
Transfusion-transmitted
infectious agent via product infectious disease
Inj&P-
Exposures, chemical injuries
and poisoning
Inj&P-
Injuries NEC
Inj&P-
Medication errors
Surg-
Therapeutic procedures and
supportive care NEC
Exposures to agents or
circumstances NEC
Exposure to body fluid
Exposure to body fluid
Exposure to contaminated
device
Exposure to device
contaminated with body fluid
Occupational exposures
Occupational exposure to
product
Occupational exposure while
preparing product for
administration
Pathways and sources of
exposure
Exposure via blood
Exposure via blood
Non-site specific injuries
NEC
Accidental exposures to
product
Injury
Accidental injury
Accidental exposure to
product
Accidental exposure while
administering drug
Maladministrations
Incomplete course of
vaccination
Hepatitis B immunisation
Incomplete course of
vaccination
Hepatitis B immunisation
Immunisations
44
22
05/04/2013
Ex. 5. Exposure. Suggested terms
– Transmission of an
infectious agent via product
• Transfusion-transmitted
i f ti
infectious
di
disease ?
– Hepatitis B immunisation
– Incomplete course of
vaccination
– Occupational exposure
while preparing product for
administration
– Needle stick/puncture
– Accidental injury
– Accidental exposure
while
e ad
administering
se gd
drug
ug
– Exposure to device
contaminated with body
fluid
• (under PT Exposure to
contaminated device)
– Exposure
p
to bodyy fluid
– Exposure via blood
• (Under PT Injury associated
with device)
45
Ex. 5. Exposure. Table 2.
SOC
Genrl‐
HLGT
Complications associated with device
PT
Injury associated with device
LLT
Needle stick/puncture
Hepatic and hepatobiliary disorders
HLT
Complications associated with device NEC
Cholestasis and jaundice
Hepat‐
Infec‐
Jaundice
Jaundice
Ancillary infectious topics
Infectious transmissions
Viral infectious disorders
Exposures, chemical injuries and poisoning
Hepatitis viral infections
Exposures to agents or circumstances NEC
Transmission of an infectious agent via product
Transmission of an infectious agent via product
via product
Acute hepatitis B
Exposure to body fluid
Transmission of an infectious agent via product
Transfusion‐transmitted infectious disease
Acute hepatitis B
Exposure to body fluid
Infec‐
Inj&P‐
Exposure to contaminated device
Exposure to device contaminated with body fluid
Occupational exposure while preparing product for administration
Occupational exposures
Occupational exposure to product
Pathways and sources of exposure Exposure via blood
Exposure via blood
Inj&P‐
Inj&P‐
Injuries NEC
Medication errors
Non‐site specific injuries NEC
Accidental exposures to product
Injury
Accidental exposure to product
Maladministrations
Incomplete course of vaccination
Accidental injury
Accidental exposure while administering drug
Incomplete course of vaccination
Inv‐
Hepatobiliary investigations
Liver function analyses
Bilirubin urine
Urine bilirubin increased
Bilirubin urine
Urine bilirubin increased
Inv‐
Microbiology and serology investigations
Appetite and general nutritional disorders
Joint disorders
Urinary tract signs and symptoms
Virus identification and serology
Hepatitis B surface antigen
HBsAg
Appetite disorders
Decreased appetite
Appetite lost
Arthropathies NEC
Urinary abnormalities
Monarthritis
Choluria
Monoarthritis
Choluria
Yellow skin
Yellow skin
Metab‐
Musc‐
Renal‐
Skin‐
Surg‐
Epidermal and dermal conditions
Dermal and epidermal conditions NEC
Therapeutic procedures and supportive Immunisations
care NEC
Hepatitis B immunisation
Hepatitis B immunisation
46
23
05/04/2013
Ex. 6. “Do Not Add Information”
Neutropenic patient under chemotherapy
treatment presented with retinitis with cotton
like lesions in fundoscopy. One week later,
patient was hospitalized with high fever,
nuchal rigidity and obnubilation. CSF was
positive for Candida app, and
hematogenous candidiasis was suspected.
suspected
47
Ex. 6. Which would you choose.
•
•
•
•
•
•
•
Chemotherapy
Neoplasm malignant
Neutropenia
Retinitis
Candida retinitis
Cotton wool spots
Fundoscopy
abnormal
•
•
•
•
•
High temperature
Nuchal rigidity
Obnubilation
Candidiasis
Disseminated
candidiasis
• Candida test positive
• Candidal meningitis
48
24
05/04/2013
Ex. 6 Suggested terms
•
•
•
•
•
•
•
Chemotherapy
Neoplasm malignant
Neutropenia
Retinitis ¿?
Candida retinitis ¿?
Cotton wool spots
Fundoscopy
abnormal
•
•
•
•
•
High temperature
Nuchal rigidity
Obnubilation
Candidiasis ¿?
Disseminated
candidiasis ¿?
• Candida test positive
• Candidal meningitis
49
Ex. 7. About “versioning”
A 49-year-old female developed intestinal
graft-versus-host
graft
versus host disease (GVHD) which
prolonged hospitalization. The patient
received her first study platelet transfusion
on 08 July 2010. The patient’s medical
history includes NK leukemia and vaginal
hysterectomy.
hysterectomy
50
25
05/04/2013
A 49-year-old female developed intestinal
graft-versus-host
graft
versus host disease (GVHD) which
prolonged hospitalization. The patient
received her first study platelet transfusion
on 08 July 2010. The patient’s medical
history includes NK leukemia and vaginal
hysterectomy.
hysterectomy
51
New terms and “re-coding”
Reported event 10.0
10.1
GVHD (unspec.) GVHD (as a
whole). No
GVHD skin
diferentiation.
Undetermined
Skin
GVHD liver
Liver
GVHD intestine
Intestine
52
26
05/04/2013
New terms and “re-coding”
Reported event 10.0
10.1
GVHD (unspec.) GVHD (as a
whole). No
GVHD skin
diferentiation.
Undetermined
Skin
GVHD liver
Liver
GVHD intestine
Intestine
53
Version update
v13.1
v14.0 V14.1
Study A
data lock
Study B
Study C
v15.0 v15.1
data lock
2 MedDRA
versions per
year (March
and
September)
Integrated
analysis:
MedDRA v15.1
data lock
Pharmacovigilance
54
27
05/04/2013
Ex. 8. Procedure and diagnosis are
reported
• A 75-year-old male diagnosed with coronary
insufficiency developed anginal pain on 21 Feb
2010 resulting in hospitalization while enrolled in a
randomized, open-label study. The patient’s
medical history is significant for triple bypass graft
surgery 2006, postural hypotension and stage D
prostate carcinoma. On 26 Feb 2010, the patient
percutaneous transluminal coronaryy
underwent p
angioplasty with stent placement in the right
coronary and proximal left anterior descending
arteries.
55
Ex. 8. Assessing the Verbatim
• A 75-year-old male diagnosed with coronary
insufficiency developed anginal pain on 21Feb10
resulting in hospitalization while enrolled in a
randomized, open-label study. The patient’s
medical history is significant for triple bypass graft
surgery 2006, postural hypotension and stage D
prostate carcinoma. On 26Feb10, the patient
percutaneous transluminal coronary
y
underwent p
angioplasty with stent placement in the right
coronary and proximal left anterior descending
arteries.
56
28
05/04/2013
Ex. 8. Time profile and term types
• A 75-year-old male diagnosed with ..
– coronary insufficiency ..
• .. developed anginal pain on 21Feb10 resulting in
hospitalization while enrolled in a randomized, openlabel study.
– The patient’s medical history is significant for triple bypass
graft surgery 2006, postural hypotension and stage D
prostate carcinoma.
• On 26Feb10, the patient underwent percutaneous
transluminal coronary angioplasty with stent
placement in the right coronary and proximal left
anterior descending arteries.
57
Tips for Handling Abbreviations
• Train investigators to avoid reporting
abbreviations
• Consider
C
id context
t t iin th
the verbatim
b ti th
thatt might
i ht
disambiguate the abbreviation
• Consider using the ARGH Biomedical Acronym
Resolver to look up various meanings of
acronyms and their frequency of use in Medline
– Longer acronyms might be specific enough to code
• Consider specifying that a certain medical
acronym dictionary will be used (>1 meaning 
query)
58
29
05/04/2013
Funny Verbatims
•
•
•
•
•
•
•
•
“Trauma of right knee during skiing in FRANCE”
Endorses smoking marijuana once per day”
day
“Endorses
“Just her time to go”
“Tongue stud”
“Brain feels like a lava lamp”
“Fungus left feet”
“Feeling like Gumby”
“Loss of sensibility in pubic area”
59
Summary
In this workshop we:
• Reviewed key principles in the “MedDRA
MedDRA
Term Selection: Points to Consider”
document
• Learned practical approaches to coding
consistently, accurately, and with
confidence
fid
• Engaged in practical Ex.s and shared best
practices
60
30
05/04/2013
Learner
Assessment
Learner Assessment
Concerning the pre-defined primary SOC
allocations in MedDRA, which of the following
statements is true?
a. They should never be changed
b. Coders should choose whichever primary SOC
they prefer when coding
primary
y SOCs to
c. Data reviewers should alter the p
fit the product’s AE profile
d. They should be changed only through the
MSSO’s Change Request process
62
31
05/04/2013
Learner Assessment
According to the preferred option in the MTS:PTC,
what would be the appropriate LLT selection(s) for
the verbatim “Abdominal
Abdominal cramps
cramps, vomiting
vomiting,
diarrhea, (food poisoning), Athlete’s foot”?
a. Abdominal cramps, Vomiting, Diarrhea, Food
poisoning
b. Acute gastroenteritis
c. Food poisoning, Athlete’s foot
d. Athlete’s foot
63
Learner Assessment
According to the MTS:PTC, what would be the
appropriate LLT selection(s) for the verbatim
“Markedly elevated ALT and AST after starting
study drug”?
a. Hepatitis drug-induced
b Transaminases increased
b.
c. ALT increased, AST increased
d. Hypertransaminasemia
64
32
05/04/2013
Learner Assessment
According to the MTS:PTC, what would be the
appropriate LLT selection(s) for the verbatim
“Patient expired because of a drug administration
error (gave wrong antiarrhythmic)”?
a.
b
b.
c.
d.
Cardiac death, Drug toxicity
Wrong drug administered
Expired drug used
Death, Drug administration error
65
Thank you!
(¿? please)
33