Biomechatronics
Graham Brooker
What is Biomechatronics?
• Mechatronic engineering can
be defined as the synergistic
combination of mechanical,
electronic, computer and
control systems
• Biomechatronics is the
application of mechatronic
engineering to human biology
Graham Brooker | IBBW 2011
2
Some Aspects of Biomechatronic (and
other) Research
•
•
•
•
•
Hearing and balance
Vision
Heart replacement
Respiration
Movement
Graham Brooker | IBBW 2011
3
Hearing and Balance
• Chris Todd. “An Inexpensive System for Assessment
of Galvanic Vestibular Sway Response”
• Elisabeth Magdas. “Visual Vestibular Stimulation
using Virtual Reality”
Graham Brooker | IBBW 2011
4
An Inexpensive System for Assessment
of Galvanic Vestibular Sway Response*
• Build a cheap and versatile method of measuring
patient sway in response to Galvanic Vestibular
Stimulation
–
–
–
–
–
Inertial Measurement Unit (IMU)
Electrical Stimulator & electrodes
Nintendo Wii Balance Board
Microcontroller
Bluetooth Enabled
* In conjunction with Dr. Miriam Welgampola, RPAH
Graham Brooker | IBBW 2011
5
Patient Stimulator
Low Cost Inertial Measurement Unit
• Sparkfun SEN10010 6DOF IMU
• Cost $60
Comparisonbetweenmeasuredandrollcorrected
accelerationsinthexandydirections
Graham Brooker | IBBW 2011
7
Results: Stimulator
• Output voltages as a function of load resistance for a
fixed 1mA bipolar current
Graham Brooker | IBBW 2011
8
Results: Wii
• Comparison between the Centre of Pressure
measured by a Kistler force plate and the Wii
Graham Brooker | IBBW 2011
9
Results: Wii and Gyro
• Comparison between the displacement of the Centre
of Mass generated by the Wii and the tilt angle
generated by integrating the gyro roll output and
correcting for drift
GyroDrift
Driftcompensatedgyro
Wii
Graham Brooker | IBBW 2011
10
Results: IMU
• Gyro drift cannot be reliably compensated for and this
results in poor sway displacement measurements
when compared to the PhaseSpace data
– Additional angle measure to compensate for drift & Kalman
filter to fuse data
– More expensive IMU with lower drift gyros Attitude heading
reference system (ARHS)
• Comparison between PhaseSpace and Motion Node
IMU displacements
Graham Brooker | IBBW 2011
11
Visual Vestibular Stimulation using
Virtual Reality
• Using VR to gain an understanding of how human
beings respond to and interact with unexpected
motion in computer designed environments
• Visual Stimulus
– Puzzle solving 3D game called Portal (Valve)
– Vancouver 2010 Winter Olympics – downhill ski (Sega)
• Feedback
– EMG signals from the medial gastrocnemius (calf) muscle
– Body tilt
– Heart rate
• Unpredicted feedback generated by flipping the head
tracker module by 90deg
Graham Brooker | IBBW 2011
12
System Configuration
Graham Brooker | IBBW 2011
13
The VR Setup
6o
Data Acquisition & Control Setup
ServoforHeadTracker
DataAcquisition
Tilt&ECGMeasurement
EMGMeasurement
6o
Player Characteristics
• Player Statistics (15 players)
–
–
–
–
10 were male most in the 22-25 year age bracket
3 had posture or back problems
All rated their sense of balance as “good”
Most were myopic and two needed to wear glasses under the
VR goggles
• Primary Gameplay Platform
–
–
–
–
–
–
13 were regular or fairly regular game players
4 on PCs
1 on Playstation
0 on XBox
2 on Nintendo
6 on handheld consoles (mobile phones etc)
Graham Brooker | IBBW 2011
16
Portal
• Puzzlesolving3Dgameprovedtobeconfusingto
noviceplayers
Vancouver 2010 Winter Olympics
• Skiing and Snowboarding were found to give best
solo play “in game” experience
Analysing Tilt, EMG and Heart Rate
• Three factors were measured at all times: Tilt, EMG
and Heart Rate
• Videos were reviewed to note if/what specifically
triggered at which moment in game-play
• Recordings were also reviewed in general per person
to get an understanding of their game-play style
Results
• Heart rate decreased during game-play (particularly
during puzzle solving aspects of Portal) and
increased on exit
• EMG amplitude and peak body tilt increased as
game-play went on
• Body motion mimicked the movements within the VR
environment (particularly in the downhill skiing game)
• Head tracker flip results were disappointing
Graham Brooker | IBBW 2011
20
Sensory Substitution
•
•
•
•
Alex Ballantyne. “Handheld Text Reader and
Image Pre-processing for Optical Character
Recognition”
Michelle Noguez. “Ultrasound Prosthesis for
the Blind”
Jay Park. “Ultrasonic Blind Aid”
Phill Marathakis. “Electrotactile Stimulated
Sensory Substitution for Use as a Sight Aid”
Graham Brooker | IBBW 2011
21
Handheld Text Reader and Image Preprocessing for OCR
• Robust mobile phone based device to translate
everyday text into speech
–
–
–
–
–
Android platform
Image capture
Compensating for poor illumination
Dewarping
Interfacing to Tesseract OCR
Graham Brooker | IBBW 2011
22
Lighting and Shading
• Thresholding is the key to Tersseract performance
• Uniformly dark or overexposed images could be
processed
• Graded variations in shading were more difficult to
handle
• 2D moving average to determine local intensity
statistics resulted in good letter reconstruction
Graham Brooker | IBBW 2011
23
Text Dewarping
• Shape from shading
• Spatial tracing of text line
using fuzzy c-means
algorithm
– Generates c clusters
– Connect cluster centres
together (not aligned vertically)
– Create mesh warped in 1-D
– De-warp image onto rectilinear
mesh
Graham Brooker | IBBW 2011
24
Light and Shading Results
Graham Brooker | IBBW 2011
25
De-warping Results
Tesseract performanceonwarpedtext
Tesseract performanceondewarpedtext
Graham Brooker | IBBW 2011
26
Ultrasound Imaging with Vibrotactile
Stimulation
Jay’sSensor
Michelle’sPrototype
Graham Brooker | IBBW 2011
27
Electrotactile Stimulation
• Array of 4x6 electrodes, with an
array of 3x5 common positive
electrodes in between the 4x6
array
• Programmable constant current
source provides individual
excitation 0.45 to 1.45mA to each
active pin
• Downsampled video image
provides PWM signal to modulate
each current source
Proofofconceptprototypeof
tonguearray
Graham Brooker | IBBW 2011
28
Prosthetic Vision
• Paul Wong. “Assessing the Feasibility of a
Smartphone-based image Processor for a Visual
Prosthesis”*
• Lauren Meredith. “Simulated prosthetic Vision: the
Role of Head and Eye Movement in the Visual
Prosthesis”**
* with Gregg Suaning (AVPG, UNSW)
** with Spencer Chen (AVPG, UNSW)
Graham Brooker | IBBW 2011
29
Smartphone-based image Processor for
a Visual Prosthesis
• Uses Android operating system on either HTC Desire
or Nexus One handsets
• Main issues are
– Rapid evolution of hardware and operating systems (running
just to stand still)
– Frame rate latency
Graham Brooker | IBBW 2011
30
Simulated prosthetic Vision: the Role of Head
and Eye Movement in the Visual Prosthesis
• Subjects required to perform two visual tasks within a VR
environment that provided simulated prosthetic vision:
– Room identification
– Find object in room
• Performance was evaluated with different combinations
of head and eye tracking to displace the visual mosaic
Graham Brooker | IBBW 2011
31
Modes
• Head tracking
• Eye tracking
• Independent head movement for scanning, eye
movements for stabilisation
• Natural combination of head and eye tracking
Graham Brooker | IBBW 2011
32
Results
• Analysis considered time taken to achieve goal and
success rate.
• Findings:
– The appropriate viewing mode may be different depending on
the particular task required
– For tasks requiring general scanning, viewing mode 3 was
inappropriate as it was hard to control head and eye
movements independently
– For tasks requiring fixation on a particular object, viewing
modes 2 and 4 were not successful as subjects found it
difficult to use eye movements to fixate
Heart Replacement
• Still waiting for a suitable
student
Graham Brooker | IBBW 2011
34
Respiration
• Jacqui Casali. “Doppler SIDS Monitor: A preventative
Intervention to Reduce the Incidence of SIDS”
• Renee Doherty. “Non-invasive Positive Pressure
Ventilation”*
• Peshala Kariawasam. “Mechanical Lung Simulation for
Sleep Disordered Breathing”*
• Chris Larkin. “Development of a Cost Effective Fleisch
Pneumotachograph”
* In conjunction with Resmed
Graham Brooker | IBBW 2011
35
Doppler SIDS Monitor: A preventative
Intervention to Reduce the Incidence of SIDS
• Use the phase information from a 24GHz Doppler
radar to observe the breathing patterns of a baby
Grace– AshtonDrakeLullaby
andGoodnightBreathingBaby
Dollwithconductivewaistcoat
Inthenursery
Labtests
Graham Brooker | IBBW 2011
36
Results
• Amplified voltage out of the Doppler radar operating
as a phase detector
Icomponent
IandQcomponents
Graham Brooker | IBBW 2011
37
Processing
•
•
•
•
Moving mean estimation
Dynamic upper and lower threshold determination
Peak excursion detection
Timer
Graham Brooker | IBBW 2011
38
Mechanical Lung Simulation for Sleep
Disordered Breathing
• Requirements
– Extremely low cost
– Programmable waveform
Graham Brooker | IBBW 2011
39
Results
Graham Brooker | IBBW 2011
40
Development of a Cost Effective Fleisch
Pneumotachograph
• Project Aim: Make a Fleisch Pneumotachograph and
calibration system from materials available in the 3rd
world
OperationalprinciplereliesonPoiseuille’s Law
Fleisch cellmadefrom190
syringeneedles
Graham Brooker | IBBW 2011
41
Calibration Process
0.2
3.9
0.1
3.8
3.75
Filterand
Integrate
3.7
3.65
3.6
3.55
3.5
0
1
2
3
4
Time (sec)
5
6
Flow (V) & Volume (V.s)
Differential Pressure (Volts)
3.85
0
-0.1
-0.2
-0.3
7
Measuredvoltageoutof
differentialpressure
transducer
DIYSpirometer
Graham Brooker | IBBW 2011
-0.4
-0.5
1.5
2
2.5
3
3.5
Time (sec)
4
4.5
5
5.5
Knownvolume=1.4lit
Thereforescalefactor
K=3110ml/spervolt
42
Calibration Results
• Calibration results for different sources of air flow
show a slight decrease in K with increased flow rate
Graham Brooker | IBBW 2011
43
Results
10
6
4
HeavyBreathing
2
0
-2
0
5
10
15
Time (sec)
20
25
30
10
8
6
Maximum
expiratoryflowdata
Flow Rate (lit/s)
Flow (Lit/s) & Volume (Lit)
8
4
2
0
-2
-1
0
Graham Brooker | IBBW 2011
1
2
3
Lung Volume (lit)
4
5
6
ShallowBreathing
44
Low Cost Phone Interface
• Standard fluid filled
manometer tube changes
inductance in tank circuits of
a pair of RF oscillators
• Mixer and amplifier
complete a beat frequency
oscillator (BFO)
• Stable signal in the audio
range with a frequency
proportional to the flow rate
Graham Brooker | IBBW 2011
45
Movement
• James Bencke. “A Myoelectric
Interface for the Control of a Robotic
Arm”
• Ryan Nguyen. “Motivational
Physiotherapy Device”
• Bulmaro Valdes. “Robotic Orthotic
Device for Finger Rehabilitation”
• Eloise Matheson. “Assistive Robotic
Power Glove”
• Kanchan Bist. “Remote Monitoring
(Goniometry of the Knee)”*
• Clare Young. “A Prototype for
Measuring the Function of the Human
Knee Using Accelerometers”*
• Sarah McDonald. “Modified Pegboard
Test”
*InconjunctionwithSORI
Graham Brooker | IBBW 2011
46
A Myoelectric Interface for the Control of
a Robotic Arm
Graham Brooker | IBBW 2011
47
Hardware
PrototypePCB
Highpassfilterandbuffer
Precisionrectifier
Graham Brooker | IBBW 2011
48
Analog Processing
Rawsignal
withcommon
modenoise
Signalafterhigh
CMRR
instrumentationamp
Afterrectification
LPfilter
Graham Brooker | IBBW 2011
49
Results
• Contraction sequence
– Relaxed
– Mid contraction
– Maximum contraction
Graham Brooker | IBBW 2011
50
Robotic Orthotic Device for Finger
Rehabilitation
• Project Aim:
• To develop an easily used robotic device to help with
the rehabilitation of post stroke and other patients
with neurological damage
– Open loop exercise of individual fingers, finger sequences or
complete hand
– Closed loop servo assistance to help promote neuroplastic
restructuring
– Programmable resistance force to increase muscle strength
Graham Brooker | IBBW 2011
51
Hardware
• RC servo controlled cam
mechanism
• Linear bearings
• Custom finger attachment
Graham Brooker | IBBW 2011
52
Software Interface
• Labview GUI
• Communicates with Arduino microcontroller
• PWM control of RC servos
Graham Brooker | IBBW 2011
53
Results
Graham Brooker | IBBW 2011
54
Assistive Robotic Power Glove
• Project aimed to develop a general purpose
assistive orthotic for the human hand:
– Reduce problems with hand fatigue and grip strength in
astronauts on EVA
– Assist patients with reduced hand strength to function
normally
– Operate as a rehabilitation tool to redevelop hand
functionality
Graham Brooker | IBBW 2011
55
Design Prototype I
Pneumatic
Muscle
Actuator
Linear Variable
Displacement
Transducer
Plastic bearing
for wrist
Position of
flexible force
sensors
Graham Brooker | IBBW 2011
Flexible
Fibreglass
Exoskeleton
56
Flexible Exoskeleton Tests
• Cylindrical composite structures tested to failure
StructureofExoskeletonElement
Max
Force
(N)
Max
Displacement
(mm)
3layersGRP smallradius
19.6
20
6layersGRPsmallradius
34.3
44
6layersGRPlargeradius
49.0
10
Carbon/Kevlar/Carbon smallradius
34.3
5
4layersGRPtriangularshell
14.7
8
4layersCarbon Fibretriangularshell 34.3
4
Graham Brooker | IBBW 2011
57
Implementation - Prototype I
Graham Brooker | IBBW 2011
58
Results
• Inadequate extension due to material properties
• Composite material fatigue
Flexion
Extension
Graham Brooker | IBBW 2011
59
Design Prototype II
Pneumatic
Muscle
Actuators
Plastic Sleeve
Bearings
Graham Brooker | IBBW 2011
Steel
Exoskeleton
Plate
Restorative
Torsion
Springs
60
Implementation Prototype II
Graham Brooker | IBBW 2011
61
Results: Prototype II
Graham Brooker | IBBW 2011
62
Knee Rehabilitation
• Develop easy to attach wireless monitoring devices to
measure knee recovery after orthopaedic surgery
• Two projects conducted in conjunction with Sydney
Orthopaedic Research Institute (SORI)
– Accelerometer and gyro based measurement of shock
absorption capability
– Goniometer based measurement of knee flexion and
extension
Graham Brooker | IBBW 2011
63
Goniometer Hardware
• Resistive flex sensors were mounted in two positions
in front and laterally on the knee
• Resistive bridge cct generates voltage proportional to
sensor flex
• ADC
• Bluetooth
• PC Data logger
Graham Brooker | IBBW 2011
64
Accelerometer and Gyro Hardware
3axisaccelerometerandgyro
Magneticattachmentof
accelerometer/gyromodules
Dataacquisitionand
BluetoothWireless
Graham Brooker | IBBW 2011
65
Comparative Trials of Accelerometer
Attachment Method
• Two types of trials
– 5 trials with sleeve
– 5 trials with sports
tape
Accelerometer
Results
Accelerationprofilesonthetibiaand
femurindicateshockabsorbingcapability
Graham Brooker | IBBW 2011
Integratedgyrooutputshows
righttibialanglewrt ground
67
Goniometer Results
• Comparison between goniometer
results and gait lab measurements
– Calibration problems
– Front sensor shows good correlation
– Lateral sensor catches and produces
poorer results
Pearson'scoefficient
Pearson'scorrelationcoefficient
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Frontsensorand
motioncapture
Lateralsensorand
motioncapture
0
2
4
6
8
Blocknumber
Graham Brooker | IBBW 2011
68
Modified Pegboard Test*
• Project aimed to develop a modified pegboard test along
with testing procedures to monitor the progression of
Parkinson’s disease
• It should be able to gauge the severity of core symptoms
– Tremor – shaking or trembling
– Rigidity – muscle stiffness
– Bradykinesia – slowness of movement
• Other requirements:
–
–
–
–
Suitable for at home testing
Simple and easy to use
Quick, to allow multiple tests every day
Robust and reliable hardware and software
* In conjunction with Philip Leong EIE
Graham Brooker | IBBW 2011
69
Configuration
• Battery powered wireless peg
• USB powered pegboard
• PC
Graham Brooker | IBBW 2011
70
The Peg
• Battery powered
– Fitted with power light
– Charged through pegboard
• 3-axis accelerometer
– Set to ±8g
– Data rate of 100Hz
• Bluetooth™
– Baud set to 115k
The Pegboard
• User interface
– LCD screen
– Three buttons
• Light emitting diodes
• Optical switches
• Home position
– Charging peg
• USB Serial
– Baud set to 9600
– Acts as power source
Computer
• Data collection
– Matlab
• User interface
• Collect and store
• Storage
– As an Excel file
• Analysis
– Matlab
• Access data
• Calculate and display
Product Testing
• Initial testing
– Random paths
– Ten times daily
• Clinical trial testing
– Pre-set paths
– Twice daily
• Method
– Nine sets of tests
– Left and right hands
– Before and after medication
Results
• Measured data analysed using two methods:
– Method 1:
• Compilation of movement time, reaction time, and
accelerometer data
– Method 2:
• Displacement plot of the subject’s motion path
Subject 3 Movement Data - Test Number 6 (Path #1)
200
Displacement
Home
S1
S2
S3
S4
S5
S6
100
Y Axis (mm)
Acceleration(m/s2)
Subject 1 Left Hand "OFF" - Last Dose 3h10min ago (9:25am 5/8/2011)
50
0
0
-100
-50
0
50
100
150
200
Readings
250
300
-200
-300
350
-200
-100
0
100
X Axis (mm)
200
300
Subject 3 Movement Data - Test Number6 (Path #1)
200
0
-50
Displacement
Home
S1
S2
S3
S4
S5
S6
100
0
50
100
150
200
Readings
250
300
350
X Axis
Y Axis
Z Axis
React(high)/Mov(low)
Y Axis (mm)
Acceleration(m/s2)
Subject 2 Left Hand - CONTROL (9:46am 5/8/2011)
50
0
-100
-200
-300
-200
-100
0
100
X Axis (mm)
200
300
All right – any questions
PoemandillustrationscourtesyLeunig &SMH
Graham Brooker | IBBW 2011
76