NEW COURSE:
SKETCH RECOGNITION
Analysis, implementation, and comparison of sketch recognition
algorithms, including feature-based, vision-based, geometry-based,
and timing-based recognition algorithms; examination of methods to
combine results from various algorithms to improve recognition using
AI techniques, such as graphical models.
Learn how to make your drawings come alive…

Sezgin
• Finds corners in a polygon or in a complex
shape.
• First: Polygon
Direction
Direction of each stroke segment = arctan2(dy,dx)
Add check to make sure graph continuous (e.g., add 2pi)
Curvature
Change in direction for each segment
Speed
Speed of each segment (already computed in Rubine)
Threshold Curvature
• Threshold = mean curvature
Threshold Speed
• Threshold = 90% of mean
Select Vertices for Each
• Max of all sections above threshold
• Fd = curvature points
Select Vertices for Each
• Max of all sections above threshold
• Fs = speed points
Initial Fit
• Intersection of Fd and Fs (and of course
the endpoints)
Local neighborhood: i-k, i+k
• Si is a stroke point
• k is number of stroke point on either side
to search
• Le = Euclidian distance from Si-k to Si+k
• K not defined
– Lets try k = 3 … AND
– Variable k where min(Le > 6 pixels)
Curvature Certainty Metric for Fd
• di = curvature at point i
• l = stroke segment curve length from Si-k to
Sj+k
• Lets call it CCM(vi)
Speed Certainty Metric for Fs
• Lets call it SCM(vi)
Vertex Possibilities:
•
•
•
•
Fd: Possibilities based on curvature
Fs: Possibilities based on speed
CCM(Fd) : Curvature Certainty Metric
SCM(Fd) : Speed Certainty Metric
• H0 = Initial Hybrid Fit = intersection of Fd
and Fs (and endpoints)
Selecting new points to add
• Hi’ = H0 + max(SCM(Fs – H0) )
• Hi’’ = H0 + max(CCM(Fd – H0))
• Pick highest scoring from each list
• Try to add it to the H0
Make Line Segments for Hi’ and
Hi’’
• For each vertex vi in Hi – create a list of
line segments between vi and vi+1
• If there are n points in the vertex selection,
there will be n-1 line segments
Calculate Error
for Both Hi’ and Hi’’
• S = total stroke length (not Euclidean
distance)
• ODSQ = orthogonal distance squared
• ODSQ(s,Hi) = distance of stroke point s to
appropriate line segment (previous slide)
Distance from point to line segment
• if isPointOnLine(point, line)
•
theDistance = 0;
• else
•
array1 = getLineAxByC(line);
•
array2 = getAxByCPerpendicularLine(line, point);
•
•
A = [array1(1), array1(2); array2(1), array2(2)];
•
b = [array1(3); array2(3)];
•
intersectsPoint = linsolve(A, b)';
•
if isPointOnLine(intersectsPoint, line)
•
theDistance = getLineLength([intersectsPoint; point]);
•
else
•
dist(1) = getLineLength([line(1,:); point]);
•
dist(2) = getLineLength([line(2,:); point]);
•
theDistance = min(dist);
•
end
• end
Select either Hi’ or Hi’’
• Pick the one with the lower error
• (in the paper this is also the one with the
`higher score’ – score in this sense is just
an internal ranking metric)
• H1 = Hi’ or Hi’’ (the one with the lower
error)
• We have added one vertex point.
Repeat
•
•
•
•
Create new Hi’ and Hi’’
Hi’ = H1 + max(SCM(Fs – H1) )
Hi’’ = H1 + max(CCM(Fd – H1))
(Note one of them will be the same as the
previous time.)
• Recompute error and continue
When to Stop
•
•
•
•
Stop when error is below the threshold.
… what is threshold?
Not in paper
Lets
–
–
–
–
–
Compute the error for H0 = e0
Compute the error for Hall (all the chosen v) = eall
We want something in the middle: close to eall
.1*(e0-eall) + eall
You will try other thresholds in your implementation
Implications
• Anything that you can describe
geometrically you can build sketch system
for
Curves
• Stroke between corners can be curve or
line
• Is line if l2/l1 is close to 1.
– Lets use .9 < l2/l1 *1.1 < 1
• Else is curve
Sezgin Bezier Curve Fitting
•
•
Want to replace with a Bezier curve
http://www.math.ubc.ca/people/faculty/cass/gfx/bezier.html
– Bezier Demo
•
2 endpoints and 2 control points
Bezier curve equation
• http://www.cl.cam.ac.uk/Teaching/2000/AGraphHCI/SMEG/node3.ht
ml
• http://www.moshplant.com/direct-or/bezier/math.html
• P0 (x0,y0) = p1, P1 = c1, P2 = c2, P3 = p2
• x(t) = axt3 + bxt2 + cxt + x0
• y(t) = ayt3 + byt2 + cyt + y0
• cx = 3 (x1 - x0)
bx = 3 (x2 - x1) - cx
ax = x3 - x0 - cx - bx
• cy = 3 (y1 - y0)
by = 3 (y2 - y1) - cy
ay = y3 - y0 - cy - by
Sezgin Bezier Curve Fitting
• Control points
• Endpoints: u = p1, v = p2
• T1= tangent vector – initial direction of stroke at
point p1
• T2 = tangent vector of p2 – initial direction of
stroke at point p2
• K = stroke length / 3
– 3 is common approximation
• c1=k*t1 + v
• c2 = k*t2 + u
Want to Test our Approximation
• Perhaps this is really a very complex curve
which can’t be fit with a simple Bezier
curve
– E.g., the treble clef of a musical staff
• Discretize the curve. (It doesn’t say into
how many parts – I leave that up to you.)
– Linear approximation for each part
If error too high
• Break our curve down the middle into two
curves, and try again.
Matlab Curve Fitting
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
function [estimates, model] = fitcurvedemo(xdata, ydata)
% Call fminsearch with a random starting point.
start_point = rand(1, 2);
model = @expfun;
estimates = fminsearch(model, start_point);
% expfun accepts curve parameters as inputs, and outputs sse,
% the sum of squares error for A * exp(-lambda * xdata) - ydata,
% and the FittedCurve. FMINSEARCH only needs sse, but we want to
% plot the FittedCurve at the end.
function [sse, FittedCurve] = expfun(params)
A = params(1);
lambda = params(2);
FittedCurve = A .* exp(-lambda * xdata);
ErrorVector = FittedCurve - ydata;
sse = sum(ErrorVector .^ 2);
end
end
Circles and Ellipses
•
•
•
•
Least squares fit
Make a bounding box of stroke and form
Oval is formed from that bounding box.
Circle is easy:
– Find out how far the circle is from the center =
d, return (d-r)^2
– Ellipse more complicated
Project Suggestions
• Build a finite state machine recognizer for
the computability class to easily draw and
hand in their diagrams.
• Build a physics drawing program that
attaches to a design simulator (we have
interactive physics 2005)
• Build a fashion drawing program. You
draw clothes on a person, and it puts them
one the person.
Project Ideas
• Build a robot drawing and simulation
program. You draw the robot and have a
number of gestures to have it do different
things
• Gesture Tetris
Project Suggestions
• Use both rubine and geometrical methods
in recognition
• Develop new ways for editing.
• Build new low level recognizers
Projects!
• Proposal due: Sept 22
• Visualization Contest
– Ivc.tamu.edu/competition
• Smart Boards coming
• IAP (Industrial Affiliates Program) Demo
Projects!
• 2 types:
– Cool application
• Sketch front end to your own research system
• Fun application to go on smart board/vis contest
– Gesture Tetris
– TAMU gesture-based map/directory info
– Computability/Physics/EE/MechEng simulator
– New recognition algorithm
• Significant change to old techniques to make a
new application
Final Project Handin
•
•
•
Implementation… Build it…
In class Demonstration (5-10 minutes)
Previous work
– Find at least 3 relevant papers (not read inc class)
– Assign one for class to read, you lead short discussion
•
Test
–
–
–
–
•
Run your recognition system on data.
Find out what data you need (e.g., UML class diagrams)
Each student in class will supply others data
+ Find 6 more people outside (to give 15 different people)
Paper
–
–
–
–
–
Introduction (why important)
Previous Work
Implementation
Results
Conclusion
Syllabus
• http://www.cs.tamu.edu/faculty/hammond/
courses/SR/2006