The Importance of Accuracy Measurement
Standards for Computer-Integrated
Interventional Systems
T. Haidegger1, P. Kazanzides2, I. Rudas3, B. Benyó1 , Z. Benyó1
1
Budapest University of Technology and Economics, Dept. of Control Engineering and
Information Technology, Laboratory of Biomedical Engineering, Budapest, Hungary
2
Johns Hopkins University, Center for Computer-Integrated Systems and Technology,
Baltimore, MD, USA
3
Óbuda University, Budapest, Hungary
Outline
•
•
•
•
•
•
Euron GEM Sig Workshop
Why CIIS is important
What can go wrong
Current metrics
Performance of CIIS systems
Current efforts
Case study
Outline
¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers
Standardization efforts
¤ Case study ¤ Conclusion
¤
Medical technology
You name it!
Euron GEM Sig Workshop
• CIS: Computer-Integrated Surgery
• CIIM: Computer-Integrated Interventional Medicine
• CAS: Computer-Assisted Surgery
Computer-Aided Surgery
• MIS: Minimally Invasive Surgery
• Surgical CAD/CAM
• CASD Computer Aided Surgical Design
Credit: DLR
• CASM Computer Aided Surgical Manufacturing
• Surgical Total Quality Management
Outline
¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers
Standardization efforts
¤ Case study ¤ Conclusion
¤
Computer-Integrated Surgery
Operating room
Taylor et al. 2008
Image-guided surgery
Euron GEM Sig Workshop
$1.6 billion estimated market, several competitors
• Visualization and targeting in surgery
Credit: Renishaw
• Robotic navigation through IG
Outline
¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers
Standardization efforts
¤ Case study ¤ Conclusion
¤
Motivation
Euron GEM Sig Workshop
Facing the challenges in CIS
• Human-in-the-loop control
– Leave the mapping to the surgeon
• Registration (image) based
Credit: Intuitive Surgical Inc.
Outline
¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers
Standardization efforts
¤ Case study ¤ Conclusion
Credit: ISS Inc.
– Human oversight
¤
Sources of errors in CIS
Euron GEM Sig Workshop
• Imaging errors
• Volume model generation errors
• Treatment planning errors
• Registration errors
• Errors introduced by hardware fixturing
• Intra-operative data noise
• Inherent inaccuracies of surgical tools and
actions
Achievable accuracy
• Intrinsic (technical) accuracy (0.1–0.6 mm)
• Registration accuracy (0.2–3 mm)
• Application accuracy (0.6–10 mm)
Outline
Credit: Intuitive Surgical Inc.
¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers
Standardization efforts
¤ Case study ¤ Conclusion
¤
Metrics in use
Inherited from the industry
Inherent accuracy of system components
 Accuracy vs. repeatability
Problems with measurements
Accuracy of treatment delivery is important
 Difficult to measure routinely
 Some used pre- vs. post-operative CT
Alternative use of phantoms (artifacts)
 Try to replicate clinical conditions as much as possible
 Ultimate goal is:
task specific measurement of uncertainty
Simon et al. 1995
Metrics in use II
Euron GEM Sig Workshop
In medical imaging
Handling point-based registration
(Maurer et al. 1997)
• Fiducial Localization Error (FLE)
• Fiducial Registration Error (FRE)
• Target Registration Error (TRE)
• TRE and FRE are not correlated
• Trend to use wrong metrics
• Seriously constraining assumptions
e.g. isotropic Gaussian noise
Outline
¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers
Standardization efforts
¤ Case study ¤ Conclusion
¤
Metrics in use III
Euron GEM Sig Workshop
ISO 10360 standard
Acceptance and reverification tests for coordinate measuring
machines (CMMs) – 1994 (current version: 2001)
 Volumetric Length Measuring Error : a set of 5 gauges has to be
measured 3 times with one probing at each end, in 7 directions
 Volumetric Probing Error: sphere has to be measured with 25
equally distributed probing
Outline
¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers
Standardization efforts
¤ Case study ¤ Conclusion
¤
Accuracy numbers
Euron GEM Sig Workshop
With commercially available navigation systems
• manufacturers provide STDs
Credit: NDI Inc.
Outline
¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers
Standardization efforts
¤ Case study ¤ Conclusion
¤
Intrinsic
accuracy
Euron GEM Sig Workshop
Repeat.
Application
accuracy
0.05
2
Robot
Company
Puma 200
Memorial Medical Center
ROBODOC
Int. Surgical Systems Inc.
Curexo Tech. Corporation
0.5 – 1.0
1.0 – 2.0
NeuroMate
Inn. Medical Machines Int.
Int. Surgical Systems Inc.
Renishaw plc
0.75 / 0.6
0.36 ± 0.17
0.86 ± 0.32
1.95 ± 0.44
da Vinci
Intuitive Surgical Inc.
da Vinci S
Intuitive Surgical Inc.
CyberKnife
Accuray Inc.
B-Rob I
ARC Seibersdorf Research
B-Rob II
ARC Seibersdorf Research
SpineAssist
Mazor Surgical Technologies
0.15
1.35
1.02 ± 0.58
1.05 ± 0.24
0.42 ± 0.4
0.93±0.29
1.48 ± 0.62
0.66 ± 0.27
1.1 ± 0.8
0.87 ± 0.63
All values are in mm.
Accuracy numbers II
Standardization efforts
Euron GEM Sig Workshop
Call for standardized reporting on accuracy in neurosurgery
(Widmann et al. 2009)
(A) Study design and evaluation methods:
•
•
•
•
•
•
Study design (controlled, non-controlled)
Study type (phantom, anthropomorphic phantom, cadaver, patient)
Target design (anatomical, artificial, spherical, conical, pivot, size)
Error type (TRE, TPE-total, TPE-lateral, TPE-longitidinal, TPE-angular)
Space of TPE measurement (navigation space, postop. image space, physical
Statistics (mean, standard deviation, range, statistical tests)
(B) Specifications of the navigation technology:
•
•
•
•
Image modality (CT, MRI, scanner characteristics)
Scan parameters (slice thickness, voxel size, image sequence, coils etc.)
Tracking technology (optoelectronic, electromagnetic)
Registration procedure
• Targeting technique
Standardization efforts III
Euron GEM Sig Workshop
ASTM working group F04.05 (2004–)
• “Standard Practice for Measurement of Positional
Accuracy of Computer Assisted Surgical Systems (CAOS)“
• Focus of the group:
• computer-assisted navigation and surgery
• accuracy of surgical robots (mostly in manual control mode)
• development of surg. robots and phantom (artifact) devices
• testing of stimulation devices
• drug-delivery and physiologic monitoring devices
• Standards for procedures: e.g., cutting, drilling, milling, reaming
• Distinct applications: joint replacement, implant nailing, plating,
osteotomy, etc.
• Certain imaging modalities: fluoroscopy, CT, MRI, ultrasound
Outline
¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers
Standardization efforts
¤ Case study ¤ Conclusion
¤
Standardization efforts IV
Euron GEM Sig Workshop
ASTM working group F04.05
• Nebraska University phantom
• 47 identical fiducial points
0.75 mm deep cone-shape holes
• Stainless steel, CNC machined
Credit: University of Nebraska
• Coordinates derived from CMM measurements
Outline
¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers
Standardization efforts
¤ Case study ¤ Conclusion
¤
Standardization efforts II
Euron GEM Sig Workshop
NIST Phantom (2007)
• Computer-Assisted Orthopaedic Hip Surgery(CAOHS) Artifact
• Designed to mimic hip joint
– Quantify task specific measurement uncertainty
• Uses magnetic ball-and-socket joint
Courtesy of Nicholas Dagalakis, NIST
Outline
¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers
Standardization efforts
¤ Case study ¤ Conclusion
¤
Standardization efforts
ANSI/AAMI ES60601-1:2005
Euron GEM Sig Workshop
Medical electrical equipment
• New amendment for medical robots (due August 2013)
• defining medical software systems and associated technical
requirements
• streamlining the application of risk management
• clarifying the definition of essential performance
• identifying essential performance and mitigating the risk
• “We started this process rolling because the subcommittee of the
International Standards Organization, ISO TC 184, which deals with
robots and robotic devices, now has a work program to develop
standards for medical robotic and personal care robots.”
/Chuck Sidebottom/
Outline
¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers
Standardization efforts
¤ Case study ¤ Conclusion
¤
Case study
Skull base robot at JHU
• NeuroMate 5DOF robot (FDA approved)
• StealthStation surgical navigator
(FDA approved, Medtronic Navigation Inc.)
• 6DOF force sensor
(JR3 Inc.)
• Surgical bone drill
(Anspach eMax)
• Visualization PC
• Control PC
Euron GEM Sig Workshop
Euron GEM
Sig Workshop
System
operation – cooperative
control
System
operations
Accuracy measurements I
Euron GEM Sig Workshop
Using the Nebraska phantom
– NeuroMate robot
• Multiple complete measurements with the
Nebraska phantom
• 0.36 mm FRE based on 23 points
• 0.34 ± 0.17 mm (TRE) for the rest 24 points
• Used the phantom for Pivot Calibration as well
– StealthStation navigation system
• With hand-held probe (3 trials)
» 0.51 ± 0.42 mm TRE (FRE: 0.52 mm)
• With the Robot Rigid Body (5 trials)
» 0.49 ± 0.22 mm TRE (FRE: 0.49 mm)
Outline
¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers
Standardization efforts
¤ Case study ¤ Conclusion
¤
Accuracy measurements II
Euron GEM Sig Workshop
Determining application accuracy
• Foam block cutting
Overall accuracy: 0.79 ± 0.82 mm
Placement error: 0.6 ± 0.3 mm
Dimensional error: 0.6 ± 0.8 mm
Outline
¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers
Standardization efforts
¤ Case study ¤ Conclusion
¤
System verification
Euron GEM Sig Workshop
• Cadaver tests
– Application accuracy
average cross-sections: 1 mm
– Cadaver experiments show
maximum overcut: 2.5–3 mm
Xia et al. 2008
Outline
¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers
Standardization efforts
¤ Case study ¤ Conclusion
¤
Conclusion
Euron GEM Sig Workshop
The field of Computer-Integrated Interventional Medicine
– Rapidly growing
– Conquering new areas, inventing procedures
– Systems are getting more complex and partially automated
Standards are needed for system verification
– Tests can make systems comparable
Current metrics and efforts
–
–
–
–
Outline
ISO 10360 standard
ASTM working group F04.05
NIST CAOHS Artifact
ANSI/AAMI ES60601-1 extension
¤ Introduction ¤ Motivation ¤ Metrics in use ¤ Accuracy numbers
Standardization efforts
¤ Case study ¤ Conclusion
¤
Thank you for your attention!
[email protected]