Curriculum Inventory
for Curriculum Deans
and Administrators
Terri Cameron, MA
Director, Curriculum Programs
Walter Fitz-William, MPP
We will cover:
•
Why Upload to the Curriculum Inventory?
•
MedBiquitous Standard
•
Standardized Vocabulary
•
Physician Competency Reference Set (PCRS)
•
XML Export
•
Data Upload Processes
•
Data Verification Report
•
Documenting Multiple Iterations
•
Documenting Clerkships
•
CI Upload Process
•
Data Validation Issues
www.aamc.org/cir
What Is the Curriculum Inventory?
• Streamlined international curriculum data
repository
• Not a local curriculum management system
• No data entry screens – all data is uploaded
from local curriculum management systems
• Provides graphical interpretations of aggregate
and historical curriculum-related data
(includes LCME A/Q Part II data)
• Serves as the premier source for
benchmarking and educational
research in medical education
www.aamc.org/cir
Why participate?
•
The Curriculum Inventory is the culmination of a long
history of the AAMC collecting information about medical
education content, structure, and pedagogy to support:
• Medical Education Research (medical school faculty
and external researchers)
• Continuous Quality Improvement / Benchmarking
• Support of Curriculum Committees
• Reporting on new trends
• Responding to legislative inquiries or providing data
when important issues are being discussed
• Respond to media inquiries
•
The Curriculum Inventory is only effective when the data
represents most, if not all, medical schools.
www.aamc.org/cir
Why participate?
•
CI data can be used to assist in accreditation
preparation:
• Verification Report includes how CI data looks when
used to populate LCME Summary Tables related to
curriculum
• CI for CQI and Accreditation (CICA) Task Force is
creating a set of Best Practices for documenting
curricula in curriculum management systems and
assisting us in setting up reports that schools can
use to respond to other elements and questions
related to curricula. This information can be used for
continuous quality improvement AND accreditation
•
Only participating schools can request detailed
Curriculum Inventory Reports
www.aamc.org/cir
Curriculum Inventory Participation
2012-2013
2013-2014
2014-2015
90 Schools
120 Schools
(82 of 90 retained;
28 new schools)
135 Schools
(114 retained;
21 new schools)
2015-2016
141 Verified
(10 new
schools)
8 Schools that participated previously did not participate in 20152016 due to changes in curriculum management systems, changes in
administration, and/or accreditation preparation.
2012-2013 2013-2014 2014-2015 2015-2016
US Medical Schools
134
85
115
128
Canadian Medical
Schools
US Osteopathic
Schools (Pilot)
5
5
5
5
n/a
n/a
3
2
www.aamc.org/cir
How can the CI help my school?
•
•
•
•
•
Educational research
Trends in medical education
Benchmarking
Tracking competency based education
Graphical interpretations of aggregate and
historical curriculum-related data
• All schools (US and/or Canadian)
• By healthcare profession
• By country / geographical region
• By level of training (UME / GME)
www.aamc.org/
cir
www.aamc.org/cir
https://www.aamc.org/initiatives/cir/curriculumreports
60+ Reports –
Use search feature to
find charts.
Detailed CI Reports are
not included in this list
– they are only available
to Participating
Schools. A list has been
strategically hidden in
the middle of this page
(we are working on
updates for the site) –
send requests for the
reports to [email protected].
Curriculum Inventory Reports
• Two sources of data:
• LCME Annual Questionnaire Part II
• 100% of US Medical Schools
• Institutional and policy data, as well as
‘overview’ curriculum data
• Curriculum Inventory
• 92% of US Medical Schools
• 29% of Canadian Medical Schools
• Detailed curriculum data
• The two sources complement each other
www.aamc.org/
cir
www.aamc.org/cir
Curriculum Inventory Reports
Curriculum Inventory data will augment
LCME Annual Questionnaire data with
details such as:
• Number of courses where content is
covered (total, mean, median, range)
• Number of events where content is
covered (total, mean, median, range)
• Instructional and assessment
methods used in events where
content is covered
• Resources used in events where
content is covered
• Competencies linked to events and
courses where content is covered
Curriculum Inventory
Reports are accessed
over 1,250 times per
month
Accessed over 1,000
times in 2015
Jan 2015 Feb 2015 Mar 2015 Apr 2015 May 2015 Jun 2015
202
102
149
92
84
70
Jul 2015 Aug 2015 Sep 2015 Oct 2015 Nov 2015
83
79
99
57
38
YTD
Total
1055
Drop-down
includes
ONLY
participating
schools
Email needs to be
from your
institution
Options to include data from:
• All Schools
• US MD
• CA MD
www.aamc.org/cir
https://www.aamc.org/initiatives/cir/384428/ciincontexthome.html
www.aamc.org/cir
CI in Context: May 2017
•
•
•
Tackling the Opiate
Epidemic Through
Educational Innovation
and Partnership
Melissa Fischer, MD
Med, University of
Massachusetts Medical
School
https://www.aamc.org/in
itiatives/cir/384428/ciinc
ontexthome.html
www.aamc.org/cir
Terminology
Terminology
• Previous Academic Year
(LCME model – if current year is 2016-2017,
previous academic year is 2015-2016)
• Academic Level (formerly called ‘year of curriculum’)
• Sequence Block
(Course, Module, Unit, Block, Clerkship)
• Nested Sequence Block (Optional)
(a way to show a variety of organizational methods, including an ordered set of courses
within a phase of instruction, a series of electives, selectives, and parallel blocks)
• Integration Block (Optional)
(a way to describe how content related to a specific topic, discipline, or theme is
incorporated or integrated across Phases, Sequence Blocks, and other curricular
structures)
• Tracks (Optional)
(a way to document tracks that meet the LCME definition of tracks)
• Event (Instructional or Assessment Session)
(e.g., Lecture, Lab, Discussion, Assessment)
www.aamc.org/cir
Terminology, Continued
• Standardized Vocabulary
(Instructional Methods, Assessment Methods)
• PCRS
(Physician Competency Reference Set)
• Expectations
(Competency / Objective / Milestone / EPA)
• Verification Report
(Curriculum Inventory data report used to verify curriculum details)
• MedBiquitous (www.medbiq.org)
• XML
(Extensible Mark-up Language: similar to HTML (Hypertext Markup
Language) used to generate web sites, XML is used to exchange
data between systems)
www.aamc.org/cir
®
Mission: To develop and
promote technology
standards for the health
professions that advance
lifelong learning, continuous
improvement, and better
patient outcomes.
Not-for-profit, member-driven,
standards development organization
CI Standardized Vocabulary
 Use local terms for institutional data entry and
reporting; match to standardized vocabulary to
upload to CIR for aggregate reporting
 Instructional Methods
 Assessment Methods
 Resources
 Detailed descriptions available at
https://www.aamc.org/initiatives/cir/curriculumadministrators
https://www.aamc.org/download/464874/data/curriculuminventorystandardizedvocabulary.pdf
Two versions – one showing changes from 2012 Version; one
without mark-up.
www.aamc.org/cir
www.aamc.org/cir
Matching Local Terminology to
CI Standardized Terminology
Local Terminology:
Instructional Methods
CI Standardized Terminology
Clinical Skills
Clinical Experience - Outpatient
Procedure Training
Clinical Experience - Inpatient
Resource Session
Lecture
Standardized Patient
Simulation (with Resource: Standardized
Patient)
Wrap-Up Session
Discussion, Small Group
• The MedBiquitous Curriculum Inventory Standardized
Terminology Subcommittee reviews the terminology each year,
based on literature and terminology schools report as difficult to
match (challenges)
• Submit matching ‘challenges’ to [email protected]
or post comments in iCollaborative
www.aamc.org/cir
Physician Competency Reference Set
(PCRS)
Competencies vs Objectives
• Competencies, Learning
Objectives, Milestones, and
Entrustable Professional Activities
(EPAs) are all separate concepts
with explicit definitions and
purposes.
• For the Curriculum Inventory, they
are all “Expectations”.
• The category element can be used
to distinguish whether an
Expectation is a learning objective,
competency, etc., but there are no
requirements to use that option at
this time.
Expectations
www.aamc.org/cir
Physician Competency Reference Set
(PCRS)
Partial URI
Physician Competency Reference Set (AAMC 2013)
aamc-pcrs-comp-c0100
PATIENT CARE: Provide patient-centered care that is compassionate,
appropriate, and effective for the treatment of health problems and
the promotion of health
aamc-pcrs-comp-c0101
Perform all medical, diagnostic, and surgical procedures considered
essential for the area of practice
Gather essential and accurate information about patients and their
condition through history-taking, physical examination, and the use
of laboratory data, imaging, and other tests
aamc-pcrs-comp-c0102
aamc-pcrs-comp-c0103
aamc-pcrs-comp-c0104
aamc-pcrs-comp-c0105
Organize and prioritize responsibilities to provide care that is safe,
effective, and efficient
Interpret laboratory data, imaging studies, and other tests required
for the area of practice
Make informed decisions about diagnostic and therapeutic
interventions based on patient information and preferences, up-todate scientific evidence, and clinical judgment
Matching School Expectations to
PCRS
• Program-Level (School) Expectations must be
mapped to PCRS competencies
•
May NOT be mapped to PCRS domains
•
‘Other’ provided for instances where there is
not a direct match between expectations
• Event- and Sequence Block-Level
Expectations may NOT be mapped to PCRS
•
PCRS
Program
Sequence
Block
Event and Sequence Block Expectations may be
mapped to each other or to Program Expectations
or not at all
•
This mapping is pulled from your curriculum
management system
Event
www.aamc.org/cir
Matching Program Objectives to PCRS
(Crosswalk)
PCRS
Program Objectives
2. Knowledge for Practice: Demonstrate knowledge
of established and evolving biomedical, clinical,
epidemiological and social-behavioral sciences,
as well as the application of this knowledge to
patient care.
2.1 Demonstrate an investigatory and analytic approach
to clinical situations
2.2 Apply established and emerging bio-physical
scientific principles fundamental to health care for
patients and populations
2.3 Apply established and emerging principles of clinical
sciences to diagnostic and therapeutic decisionmaking, clinical problem-solving, and other aspects
of evidence-based health care
2.4 Apply principles of epidemiological sciences to the
identification of health problems, risk factors,
treatment strategies, resources, and disease
prevention/health promotion efforts for patients and
populations
2.5 Apply principles of social-behavioral sciences to
provision of patient care, including assessment of the
impact of psychosocial and cultural influences on
health, disease, care-seeking, care compliance, and
barriers to and attitudes toward care
2.6 Contribute to the creation, dissemination,
application, and translation of new health care
knowledge and practices
2.99 Other knowledge for practice
2. Demonstrate a broad working knowledge of the
fundamental science, principles, and processes
basic to the practice of medicine.
2.1 Understand the clinical relevance of scientific inquiry.
2.2 Apply this knowledge in a judicious and consistent
manner to prevent common health problems and
achieve effective and safe patient care.
2.3 Demonstrate the ability to evaluate emerging
knowledge and research as it applies to diagnosis,
treatment and the prevention of disease.
2.4 Utilize state of the art information technology
and tools to retrieve, manage and use biomedical
information in the care of individuals.
2.5 Utilize state of the art information technology
and tools to retrieve, manage and use biomedical
information in the care of populations.
2.6 Understand the indications, contraindications, and
potential complications of common clinical
procedures.
2.7 Perform the basic clinical procedures expected
of a new PGY-1.
Sequence Blocks, Events, and
Expectations Mapping
• Some schools map relationships between
Event and Sequence Block Expectations
• Some schools map relationships between
Program and Sequence Block
Expectations OR Event Expectations
• Some schools do not have any mapping
for Sequence Block Expectations OR
Event Expectations
• The MedBiquitous Competency
Framework supports any of these options
PCRS
Program
Sequence
Block
Event
www.aamc.org/cir
• Expectations mapping will be
pulled from your school’s
curriculum management
system, if you have mapped
your curriculum expectations
• The only mapping Curriculum
Administrators must document
outside of the curriculum
management system is
matching School (Program)
Expectations to PCRS
School CMS CI
Sequence Blocks, Events, and
Expectations Mapping
PCRS
Program
Sequence
Block
Event
www.aamc.org/cir
Sequence Blocks, Events, and
Expectations Mapping
• Mapping Program Expectations to
PCRS
PCRS and Event and/ or Sequence
Block Expectations to Program
Program
Expectations allows for reporting
Sequence
Block
of PCRS to the Event / Sequence
Event
Block Level:
• Expectations being mapped across curricula
• Instructional methods used to teach expectations
• Assessment methods used to assess achievement of
expectations
• Content areas in which expectations are taught
www.aamc.org/cir
CI Verification Report
•
Institution (Program Information)
•
All Academic Levels (Dates)
•
All Sequence Blocks
• Expectations / Expectation Mapping (if provided)
•
All Events
• Expectation Mapping (if available)
• Instructional Methods
• Primary Instructional Methods
• Assessments
• Events with Assessment Method(s) only
www.aamc.org/cir
www.aamc.org/cir
CI Verification Report
CI Verification Report
CI Verification Report
CI Verification Report
CI Verification Report
CI Verification Report
CI Verification Report
Curriculum Inventory
Process
Curriculum Inventory Process
• Portal
•
•
•
Manage Data (Home Screen)
• Download Verification Report
• After successful data upload
• Reject or Verify Data
Manage Users (Four Roles)
• Curriculum Dean (Pre-populated)
• Primary CI Admin (Pre-Populated)
• CI Admin
• CI Viewer
• Schools are encouraged to add Vendor with CI
Viewer role (work with vendor to create e-mail)
Manage Vendors
• Vendor must be selected by school in Portal or data
will not upload
www.aamc.org/cir
Curriculum Inventory Process
• Data
• Submit Data
• ‘Portal Open’ Notification: August 1
• System Check for (MB CI Standard, Terminology,
Competencies)
• Verify Data
• Verification Report available: soon after data is
successfully submitted (15 – 30 minutes)
• Deadline for verifying data: September 30
• ‘Portal Closed’ Notification: October 1
•
Create Snapshot of data
www.aamc.org/cir
Curriculum Inventory Process
• Reminders and Notifications
•
•
Primary CI Admin / Curriculum Dean
Dean
• Ramifications
•
•
No access to Curriculum Inventory Reports
No documentation for continuous quality improvement and
accreditation
• Portal
•
Only available actions:
• Download Verification Report
• Manage Users
• Manage Vendors
www.aamc.org/cir
Upload Process
• Vendor / Sender Dependent
• CMS exports XML file
• File is uploaded through CI Portal
• Web services uploads from CMS to AAMC
• XML file is manually created
• File is uploaded through CI Portal
www.aamc.org/cir
School Portal
Curriculum Inventory Process
Please submit and verify data prior to 09/30.
• An activity log displays all activity – data upload attempts and
status, Verification Report downloads, and data rejections and
verifications.
www.aamc.org/cir
Curriculum Inventory Process
09/30/2015.
August 30
August 30.
www.aamc.org/cir
Curriculum Inventory Process
09/30/2015.
August 30
August 30
Curriculum Inventory Process
09/30/2015
.
August 30
30
August
August
August 30
30
• The data and Verification Report will remain available until the
upload process for the next academic year begins.
www.aamc.org/cir
CI Submission Errors
• Two types of submission errors:
• Core: upload file is not processed; notification goes
to sender; no status update in CI Portal Activity Log
• Incorrect or missing AAMC Institution ID
• Sender not selected by school in CI Portal
• Upload includes data for more than one school
• Upload does not pass XML format check
• Business Rules: Error message is sent to CI Portal
• Program Level Competencies not matched to
Physician Competency Reference Set (PCRS)
• Instructional and / or Assessment Method that is
not on Standardized Vocabulary list
www.aamc.org/cir
AAMC CI Business Rules
www.aamc.org/cir
AAMC CI Business Rules
www.aamc.org/cir
What Data Should be
Included in CI Uploads?
Curriculum Inventory Data
• Events
•
•
Duration in hours and / or minutes
Instructional Methods
•
•
Assessment Methods
•
•
•
•
•
Can have multiple IMs but one must be designated Primary
Summative vs Formative
Assessment Events
Resources
Expectations (Learning Objectives / Outcome
Objectives / Competencies)
Keywords
•
(School-generated, LCME “Hot Topics,” curriculum content
terms for LCME documentation, USMLE Content Outline,
MESH, UMLS, etc.)
www.aamc.org/cir
Curriculum Inventory Data
• Sequence Blocks
•
•
•
ALL Sequence Blocks: Courses, Clerkships, Electives
Duration in Days AND / OR Begin / End Dates
• Clerkships in Days AND Begin / End Dates
• Days for single iteration (5 days = 1 week)
• Begin / End Dates for Iteration Period
Expectations (Learning Objectives / Outcome Objectives /
Competencies)
• Nested Sequence Blocks (Optional)
•
•
•
Organ Systems
Integrated Clerkships / Longitudinal Clerkships
Selectives
• Integration Blocks (Optional)
•
•
Themes
Vertical Integration / Horizontal Integration
www.aamc.org/cir
Issues/Challenges
• Creating a crosswalk to map ‘local’ vocabulary to
Standardized vocabulary
• Implementing the concept of Resources for Instructional
and Assessment Methods
• Linking course and session objectives to program
objectives / competencies
• Clerkship data – what needs to be documented; where
can the data be found?
• Electives data – what needs to be documented; where
can the data be found?
• Uploading CI data in a manner that will meet needs for
continuous quality improvement and accreditation
www.aamc.org/cir
CI Upload Process
• Biggest Hurdles:
• Report Start and End Dates
• Institution Information
• Competencies
• Program to PCRS (related)
• Hierarchy Levels (broader / narrower)
• Academic Levels
• Start and End Dates
• Sequence Blocks in multiple levels
• Unique Curricula
• Multiple options for students
www.aamc.org/cir
CI Upload Process
• Biggest Hurdles:
• Data accuracy
• Organizing data
• Documenting competencies in XML
• CI Verification Report
• Assessment Counts
• Summative / Formative
• Instructional Method Hour Counts
• Constantly-changing environment
• Communication!
www.aamc.org/cir
Documenting Multiple Iterations
•
Goal is to document what the majority of
students experience
• CMS and CI document faculty expectations
of how students will progress through the
curriculum – NOT individual student progress
• Challenges:
• Small Groups
• Multiple Clerkship Iterations
• Multiple Sites
• Multiple Tracks
• Multiple Campuses
www.aamc.org/cir
Documenting Multiple Iterations
•
Upload options:
• Choose one student / student group for
upload
• System automatically uploads first or last
instance of each iteration
• Create ‘Model’ instance of each iteration
• Upload all iterations
• Issue: Are these iterations distinguishable
as multiple iterations?
www.aamc.org/cir
Documenting Multiple Iterations
•
Aggregate report options:
• Quarantine outliers
• Calculate averages per institution for all
iterations of Sequence Blocks
• Change explanations or add footnotes for
numbers of schools
www.aamc.org/cir
Documenting Clerkships:
•
Goal is to document as many hours as possible
• Clerkship weeks x 50 hours
• Didactics
• Include X-Ray Conference, M&M,
Tumor Board, Grand Rounds
• Individual activities
• Scheduled clinical activities
• Clinics, surgeries, mentors, attending
rounds, resident rounds
• Unscheduled clinical activities
• Divide ‘undocumented hours’ using
ambulatory / inpatient percentage
www.aamc.org/cir
Documenting Clerkships:
•
Metadata
• Objectives and Keywords
• Course syllabus
• Didactic / Conference Slides
• Evaluation system data
• Faculty / Student documentation
• Competencies
• Keywords
www.aamc.org/cir
Best Practices for Curriculum
Documentation
• Centralized
• Review syllabi and educational materials
• Interview faculty
• Distribute worksheets to faculty
• Provide an online tool for faculty
• Hire medical students
• Search software
• Natural language processing
www.aamc.org/cir
Best Practices for Curriculum
Documentation
• Distributed
• Faculty document as educational materials are
uploaded to LMS
• Faculty provide course coordinators with
mapping terminology
• Course Directors collect and document all
mapping terminology for their courses
www.aamc.org/cir
Preparing for 2016-2017
• Staging (for testing uploads) available now!
•
•
Web Services (requires use of SOAPUI)
Staging Portal (requires staging-specific credentials
– send request for access to [email protected])
• Webinars/Virtual Focus Groups:
•
•
•
•
Three-part Developer Series
Curriculum Deans and Administrators
Curriculum Mapping
Documenting Challenging Content
www.aamc.org/cir
Documenting Resources:
•
Use of resources is optional, but encouraged
• Documents use of Standardized Patients,
Virtual Patients, etc.
•
Beginning with 2017-2018, if Resources are
uploaded, Standardized Terminology UIDs will
be required
www.aamc.org/cir
What’s next?
• Two major initiatives:

Data validation tool
 Keyword synonym check
www.aamc.org/
cir
www.aamc.org/cir
CI Data Analysis
• Issue 1: copying one course over to all other
courses
• Issue 2: keywords that just have letters or
punctuation
• Issue 3: synonyms
• Issue 4: no titles for competency objects
www.aamc.org/cir
CI for CQI and Accreditation Task Force
Charge:
• Review the types of data in the Curriculum
Inventory
• Analyze how well that CI data and the way it is
formatted responds to the requirements of the
DCI
• Develop best practices to assist schools as they
upload to the CI in preparation for completing a
DCI
www.aamc.org/cir
For more information
• www.aamc.org/cir
• Reports
• Webinars and training
• Information about Curriculum Management System
vendors
• Tools for Developers
• Curriculum Inventory School Portal
• [email protected]
www.aamc.org/
cir
www.aamc.org/cir
Questions?