Novel Techniques for Autonomous Robotic Suturing
Der-Lin Chow, Russell Jackson, Taoming Liu, Wyatt Newman, M. Cenk Çavuşoğlu
ABSTRACT
Robot Assisted Minimally Invasive Surgery (RAMIS) has been successfully used in many surgical procedures.
However, these applications are still limited to direct teleoperation by surgeons and are typically slower than
conventional open surgery due to the confined workspace, poor endoscopic vision, and insufficient sensing.
While fully autonomous robotic surgery will continue to be a challenge for the foreseeable future, low level
suturing tasks such as needle driving, suture pulling, and knot tying could be performed by robots automatically.
By adopting advanced sensing techniques for visual and tactile feedback, as well as new skills for needle
grasping and suture handling, our research team has completed real time suture tracking, optimized needle
driving, and fast knot tying. All of these skills are crucial to autonomous robotic surgery.
Electrical Engineering and Computer Science,
Case Western Reserve University
Improved Knot-Tying Method for Autonomous Robotic Surgery
Suturing consists of needle driving followed by suture pulling and followed by knot tying. After the needle is
driven through the tissue, two grippers then hand over hand pull the suture from the needle exit point until a
short length (suture tail) is left from the needle entrance point. One gripper then holds the suture and wraps
loops about the other gripper. The gripper with suture loops wrapped on it then grasps the suture tail. Two
grippers then pull the suture in opposite directions and tension the knot. These procedures are tedious and
time consuming for RAMIS due to the limited workspace. A fully automated knot tying would benefit surgeons
and patients.
One of the challenges for autonomous robot knot-tying is to manage interference of the deformable suture
with robot tools or with the suture itself in motions. Four looping methods - “Spiral”, “Rolling Arc”, “Double
Cross” and “Binary-Star” (below, from left to right) had been developed to properly handle suture twist and sag
properties by tensioning the suture during knot tying procedures and successfully tying knots with pre-planned
trajectories (where the location of suture tail is known to the system). A one-throw knot can be completed in
8.5s which is comparable to human speed.
• Suture Thread Detection and Tracking
In order to successfully complete a surgical suture knot, the robotic grippers must be able to accurately grasp
the suture thread end while simultaneously tracking the remaining length of thread.
• Thread Segmentation
It is possible to segment the thread from the image space by looking at the ‘vesselness ‘ of the suture thread in
the image space. The problem of segmenting small tubular structures in images has been explored previously
[6]. Since the thread is also a small tubular structure, it can be segmented using the same technique.
The ‘vesselness’ of an image feature is determined by looking at the
scale space representation of the image. The scale space image is
found by convolving the image with a Gaussian image kernel of known
width.
• Vision Guided Knot-Tying
If the second derivative Hessian matrix of the scale space image is
computed, then the eigenvalues (|λ1| < |λ2|) of the Hessian correspond to
the two principle directions of the image at each pixel. The direction
corresponding to λ1 is the direction of smaller change.. This is equivalent
to the Gaussian kernel traversing along the thread (the kernel traverses
along the ridge of the thread). The direction corresponding to λ2 is the
direction of the greatest change.
The ratio between the two eigenvalues corresponds to the strength of
the ‘vesselness’ of the image at each pixel. Ideally, one eigenvalue
would be 0 while the other would be large.
We then extend this capability with the use of stereo vision to locate
the suture tail. For the present research, the stereo-vision task is
simplified with the use of colored fiducials mounted to the gripper. By
utilizing the colored fiducials and the static coordinate frame transform
from the fiducial pose to the gripper tip, our system can correctly
identify the pose of the gripper tip (blue circles on camera images).
Experimental System
The different methods of suturing and knot tying are validated using the complete robot system shown above, including two robots –
ParaDex (left) and ABB IRB140 (right), two custom-designed surgical grippers (attached to robot arm) and a stereo camera system.
The direction of the corresponding eigenvectors corresponds to the
direction of the thread. This information can be used to estimate the
vessel direction as well as the ‘vesselness’. The segmented image (left)
is falsely colored to indicated direction.
• Stereo Camera System
The Cameras system is comprised of a pair of 1280x960 pixel cameras (left)
that are capable of detecting objects to an accuracy of 1 mm. The colored
fiducials (center and right) allow the cameras to infer the location of the
gripper tip using geometric transforms. Other objects, including sutures, tissue,
and needles are also tracked using the camera system.
This algorithm is highly parallelizable and a graphics card is used to
improve the segmentation performance.
•
Generally, automated scene analysis is difficult and error prone, whereas surgical operations demand high
reliability. To address this challenge, the operator (surgeon) interacts directly with the scene to assist with
selecting points of interest (POI) by clicking on the suture tail segment (red circles) on left and right camera
views. Once the spatial coordinates of POI are identified, we can correctly grasp the suture tail and complete
knot tying procedures.
Excluding waiting time for clicking and image processing, which could be sped up with code optimization, a
complete surgeon’s knot (two throws) requires only 70s with our automatic knot- tying algorithm, which is
comparable to current speed under teleoperation (71s to tie a single knot on a ring using Da Vinci).
Optimal Needle Grasp Selection
NURBS Curve Model
• Trajectory Optimization of Robotic Suturing
Once the thread is segmented, the thread curve is modeled using a three dimensional Non Uniform Rational
B-Spline (NURBS). The NURBS models the thread using a relatively small collection of control points. The
control points are connected using polynomials. This allows the thread to be efficiently modeled as a smooth
curve.
The needle grasping pose of the gripper during needle driving in suturing is automatically determined by a proposed selection algorithm.
Inappropriate needle grasp increases operating time requiring multiple regrasps to complete the desired task. Once an optimal needle
trajectory in a well-defined suturing scenario which obeys the best practices of needle path planning that are used by surgeons is given,
the proposed algorithm will select the optimal needle grasp from the perspective of kinematics.
The NURBS model is also capable of tracking the thread and updating
the control points in real time. This is accomplished by projecting the
thread model into the stereoscopic camera images . The control points
are updated by updating the curve points to maximize the local
segmented image value . Since the curve points are a linear combination
of the control points, the updated curve points can be used to directly
update the control points using linear algebra.
Firstly, the algorithm will evaluate the completion feasibility of a needle driving motion by checking violation of manipulator’s joint limitation
and collision possibilities between robot links or between robot and environment. If it successfully passes the feasibility algorithm, then
the algorithm will use manipulability, dexterity and torque metrics as three performance metrics to score each feasible needle grasp. The
optimal needle grasp will be selected by comparing the resulting performance scores.
Experimentally, the system is capable of operating in real time. The
thread can be tracked at approximately 15 Hz.
Acknowledgements
This work was supported in part by NSF under grant CISE IIS-0905344.
Most surgical tools have long cylinder shapes and can be coarsely modeled as a
collection of spheres. A workspace can be defined in terms of cubic voxels (1mm
in our system). Based on the criteria set forth, a MatLab simulation algorithm can
efficiently evaluate the metrics of motion – the swept volumes for gripper paths,
collision check during motion, the minimum cubic volume containing gripper
motion and path length for motion speed.
The metrics can be used to evaluate applicability and feasibility of a predefined
skill, such as knot tying, with respect to a specific work volume, portal locations,
and robotic kinematics.
• Needle Grasp Parameterization
The way that the surgical manipulator grasps a needle determines how
the desired reference needle trajectory is converted to the manipulator
motion that performs the needle drive. The needle grasp with the surgical
end-effector can be parameterized with 4 DoFs as shown (left).
The kinematics of the pose of the needle grasp assume that the needle
translates starting from the tip of the gripper and couples three different
rotations. Specifically,
specifies the insertion translation along the
negative X axis of G frame.
,
and
are three rotations that
determine the orientation of the needle. The zero configuration of needle
grasp parameterization is when
= = =0, the needle frame N
is aligned with the gripper frame G. Here,
is the insertion translation
distance for . ,
and
are the corresponding rotation angles for
,
and , respectively.
.
References
1. Der-Lin Chow, Wyatt Newman, “Improved Knot-Tying Methods for Autonomous Robot Surgery”', in 2013 IEEE
International Conference on Automation Science and Engineering (CASE), August, 2013, pp.461-465.
2. Jackson, R.C.; Çavuşoğlu, M.C. “Modeling of needle-tissue interaction forces during surgical suturing,” in Robotics and
Automation (ICRA), 2012 IEEE International Conference on, may 2012, pp. 4675 –4680.
3. Jackson, R.C.; Cavusoglu, M.C., "Needle path planning for autonomous robotic surgical suturing," Robotics and
Automation (ICRA), 2013 IEEE International Conference on , pp.1669,1675, 6-10 May 2013
4.T. Liu, M.S. Thesis “Design and Prototyping of a Three Degrees of Freedom Robotic Wrist Mechanism for a Robotic
Surgery System,” Dept. EECS, Case Western Reserve University, Cleveland, Ohio, January 2011.
5. P. Boonvisut and M.C. Çavuşoğlu, “Estimation of Soft Tissue Mechanical Parameters From Robotic Manipulation
Data,” IEEE/ASME Transactions on Mechatronics, 2012.
6. A.F. Frangi, et aI., "Multiscale vessel enhancement filtering", MICCAI Proceedings, SpringerVerlag, 130-137, 1998.