PST TP - 215
Real-Time Non-Rigid Image Registration
(Image Morphing) Engine
Propulsion Science and Technology, Inc. (PST)
Randall Miles, Donna Yosmanovich, Michael Thorwart, Margaret Taylor
GPU Technology Conference 2016
San Jose, CA
April 5, 2016
1
Willow
https://goo.gl/0OxGAk
The film, Willow, directed by George Lucas and staring Warwick
Davis, was the first example of extensive use of image morphing
in film. Dennis Muren, Creative Director at Industrial Light and
Magic, said the scene took weeks of computer work at the time,
but could be done on a cell phone today.
2
Example: Image Morphing
Michael Jackson’s music
video “Black and White” is
another early example of
image morphing. In the
video, faces of individuals of
various nationalities are
morphed from one to another
as they dance and smile on
the screen. The frame shown
here capture the transition
from a western male to an
asian female.
3
Outline
• Introduction
– Who is PST?
– Why are we interested in image registration?
• What is image registration?
– A basic definition
– Examples
• Solution: Image Morphing Toolkit
– The library
– Examples
• Conclusion
4
Who is PST?
(Propulsion Science and Technolgy)
Scientists with a passion for
• Science
• Technology
• and rockets!
Our historical focus: modeling exhaust
plumes and wakes
Physics based modeling and simulation is time consuming!
5
Limitations of Modeling
• Models are used to simulate rocket plume characteristics
– Sensor testing, HWIL
– Simulations
– Algorithm testing
• Pre-built databases of modeling results can alleviate the
runtime limitations
6
Limitations of Modeling
• Models are used to simulate rocket plume characteristics
– Sensor testing, HWIL
– Simulations
– Algorithm testing
• Pre-built databases of modeling results can alleviate the
runtime limitations
• Pre-built databases introduce other limitations:
•
Nearest neighbor image interpolations require large number of images
and produces discontinuities in the extracted imagery
• Inflexible to new conditions
• Image registration techniques can be used to address these
limitations
7
Outline
• Introduction
– Who is PST?
– Why are we interested in image registration?
• What is image registration?
– A basic definition
– Examples
• Solution: Image Morphing Toolkit
– The library
– Examples
• Conclusion
8
Definitions:
Image registration: Process of aligning
multiple images according to
corresponding features in each.
Image morphing: (an application image
registration) Images are progressively
aligned to common intermediate
conditions, creating an interpolation
between images.
9
Transformations
Translation
Shear
Rotation
Non-rigid transformation
(warp)
10
Image Morphing
•
•
•
•
Image morphing is the calculation of transformed images that smoothly
blend between two input images.
Corresponding features must be identified and tracked (using landmark
points, in green below).
This requires a transformation of the two input images using the
landmark points to an intermediate condition.
A simple average of the images would cause two overlapping ghosted
images.
11
Program Motivation - Jitter
• Tightly spaced database points
minimize the step-wise changes
between points
• Jitter (discontinuities) still occur;
especially when the image is changing
quickly, or the viewing conditions
change quickly
• Morphing allows smooth interpolation of
the images
12
Outline
• Introduction
– Who is PST?
– Why are we interested in image registration?
• What is image registration?
– A basic definition
– Examples
• Solution: Image Morphing Toolkit
– The library
– Examples
• Conclusion
13
Image Morphing Tool (IMT)
• C++/CUDA library that fully automates image
morphing procedures
– Solves thin plate spline warping transformation
– Incorporates hole filling methodology
– Capable of working with multiple core computers as well as
single CPU devices
– GPU-enabled for real time morphing (500+ Hz)
– Designed for general use
• Application specific tools (e.g. PMT, for plumes)
– Automated landmark identification tailored to missile plume
characteristics  provide feature landmark positions
– Interface to source images from customer specified
database
14
Thin-Plate-Spline Warps
• The TPS algorithm defines a transformation
f ( x , y )  ( X ( x , y ), Y ( x , y ))
giving new coordinates for points in the
original image.
• The transformation is determined by
minimizing the curvature of the surface
(integrated squared second derivative)
I f   R ( f  2 f  f )xy
2
2
xx
2
xy
2
yy
15
IMT Flow Chart
At high level, the process
flow is relatively simple:
• Given: set of input
conditions
• IMT routines to return
landmarks and images.
• IMT performs the
required transformations, and returns
the result.
16
Process Within the IMT
Step 1 – Select images that will be morphed ‘from’
Step 2 – Retrieve landmarks on images
Step 3 – Compute new landmarks
Step 4 – Warp images to new landmarks
Step 5 – Cross dissolve the warped images
Landmarks from
Image 1
Landmarks from
Image 2
Solve for new
landmarks
Warp 1
Warp 2
Morph
17
Database Interpolation
•
Linear interpolation is
currently used to
determine target
landmark positions
• 4 input images are
needed for interpolation
in 2 dimensions.
• In a database with 3
independent variables,
8 input images are
needed
• IMT determines the
required images and
retrieves them
automatically
18
Code Example
// Set up 3D morphing object
getImageFcn imageDatabaseFcn();
getlmFcn
landmarksfunc();
#ifdef CUDA
morph3D_cuparam param(landmarksfunc.nlm(),ni,nj,3);
typedef morph3D_cuda<getImageFcn,getlmFcn> morph3D;
#elif PARALLEL
TPSwarp_param param(landmarksfunc.nlm(),ni,nj);
typedef morph3D_parallel<getImageFcn,getlmFcn> morph3D;
#else // SERIAL
TPSwarp_param param(landmarksfunc.nlm(),ni,nj);
typedef morph3D_serial<getImageFcn,getlmFcn> morph3D;
#endif
float *imageMorph=new valueType[numPix];
morph3D mymorph3D(&nodes,&landmarksfunc,&imageDatabaseFcn,&param);
//Perform image interpolation (cond1, cond2, & cond3 must be defined)
While (!done)
{
getConditions(*cond1,*cond2,*cond3);
mymorph3D.morph(cond1,cond2,cond3,imageMorph);
// do something with the image
}
delete[] imageMorph;
19
GPU Acceleration
The IMT was developed to support
•
serial,
•
parallel (multi-core) and
•
CUDA
•
Windows and Linux.
GPU accelleration:
•
Windows: ~60x’s increase in speed
•
Linux: ~80x’s increase in speed
CUDA techniques:
• Pinned memory transfers from CPU to GPU
• cuBLAS functions
• Custom kernels
• CUDA Streams
• Concurrent memory copies
20
GPU Acceleration
First pass CUDA implementation:
initialize()
on image 1
transform()
on image 1
map() on
image 1
intialize()
on image 2
transform()
on image 2,
etc
Second pass CUDA implementation:
initialize() all
8 images
transform()
all 8 images
map() all 8
images, etc
Each piece of the IMT morph (initialize, transform, and map)
are naturally parallel processes. Our first implementation
focussed on the parallel nature of each process. In our second
pass, however, we found that using “batched” methods (e.g.
cublasSgetrfBatched(…)), significantly accelerated the
computation.
21
Performance
Operating System Processor GPU
Method
Windows 7
i7-4820K
Serial
Linux
i7-4820K
Serial
Windows 7
i7-4820K
Parallel
Windows 7
i7-4820K
Titan
CUDA
Linux
i7-4820K
Titan
CUDA
Time in Hz
9
9
37
530
720
These frame rates were calculated for 128x256 pixel images, using 21 landmarks,
interpolating in three database dimensions, operating on dual images. (Each “frame” here
represents two images together.)
22
Additional Applications
 Thermal images of uniforms can be
warped
 Coupled with registration
techniques, allows a quantitative
comparison
 Applications include: quantitative
comparisons for thermal
properties of textiles, degradation
of parts and materials, part failure
prediction.
23
Application Example
Non-rigid image registration can be used to make quantitative
comparisons between measurements of images that change with time.
The images here show two MRI scans of a knee. The knee on the
right is bent more than the knee of the left, making registration and
differencing difficult.
24
Application Example
Image comparison without registration
Simply overlaying the images, and displaying the differences
does not produce usable results. Rigid registration techniques, in
this case, can do little better.
25
Application Example
Image comparison with registration
Non-rigid registration, as done with the IMT, can align features
within the images, thus allowing for quantitative comparisons.
26
Future Applications
• Apply measured data to increase fidelity
of models through parametrically
warping model predictions (increase
realism)
• Apply image morphing principles to 3D
fields of data
• Test IMT with new applications
• Develop more tools to automate
landmark selection
27
IMT Solution
• Jitter
– IMT resolves jitter issues
– Allows smoothly varying interpolations over any series
of conditions
• Reduce Database Size and Cost
– Based on current error analysis, implementing the IMT
can reduce database size by an order of magnitude
• Quantitative comparisons for data analytics
• Image stabilization for deforming images
• Extreme high-speeds possible due to GPU
acceleration
• This capability could be deployed in embedded
systems, on boards like the Jetson
28
Acknowledgements
I’d like to acknowledge the Air Force SBIR
program for their investment, and thank
Nvidia for their generous support of our
program.
29