US008379078B2
(12) United States Patent
US 8,379,078 B2
(10) Patent N0.:
(45) Date of Patent:
Yamashita et a1.
(54) IMAGING APPARATUS, IMAGING METHOD,
AND PROGRAM
FOREIGN PATENT DOCUMENTS
JP
JP
JP
JP
JP
JP
(75) Inventors: Noriyuki Yamashita, Tokyo (JP);
Toshifumi Fujita, KanagaWa (JP)
(73) Assignee: Sony Corporation, Tokyo (JP)
Feb. 19, 2013
08-223481
2000-305207
2006-217478
3925299
3928222
2008-289095
A
A
A
8/1996
11/2000
8/2006
3/2007
3/2007
11/2008
A
OTHER PUBLICATIONS
(*)
Notice:
Subject to any disclaimer, the term of this
patent is extended or adjusted under 35
U.S.C. 154(b) by 435 days.
an Ultra-High-Speed Shutted Camera and an Optical Correlator”
IEEE Transactions on Instruments and Measurements. vol. 44, No. 3,
Jun. 1995 pp. 755-758.*
(21) App1.No.: 12/588,821
(22) Filed:
Toyoda, H., Kobayashi,Y., Mukohzaka, N.,Yoshida, N., Hara, T., and
Ohno, T., “Frame-Rate Displacement Measurement System Utilizing
Oct. 29, 2009
Japanese Of?ce Action issued Oct. 9, 2012 for corresponding Japa
nese Application No. 2008-312674.
(65)
Prior Publication Data
US 2010/0141735 A1
(30)
* cited by examiner
Jun. 10, 2010
Primary Examiner * David CZekaj
Foreign Application Priority Data
Assistant Examiner * Tyler W Sullivan
Dec. 8, 2008
(JP) ............................... .. 2008-312674
(51)
Int. Cl.
H04N 7/00
(52)
US. Cl. ....................................................... .. 348/36
(58)
Field of Classi?cation Search ...................... .. None
(2006.01)
(57)
See application ?le for complete search history.
(56)
U.S. PATENT DOCUMENTS
5,155,597 A *
10/1992
5905’848 A >1
6,476,861 B1*
5/1999 Yano et a1,
11/2002 Min ....... ..
Lareau et a1. ............... .. 348/311
goga
,,,, ,, 336/225
. 348/208.15
~~~~~~~~~~~~~~~~~ ~~ 348/2181
31113. e
a
2008/0278582 A1 *
A1*
ll/2008 Chung et al
5/2009
Fan
..........
mation of the imaging apparatus; and a controller that pro
cesses information of the position sensor, Wherein the con
troller sends a noti?cation of an appropriate value of a
sweepingangularvelocitybasedonthedetectioninformation
.
'
Zoos/0252717 A1,, 10/2008 Moon et a1‘ ““““““““““ “ 348/36
2008/0259158 A1* 10/2008 Flint et a1.
.. 348/143
2009/0128328
ABSTRACT
An imaging apparatus includes: an imaging device that
images an object image through an optical system; an image
signal processing section having a function of combining a
plurality of imaged images into a combined image piece, the
imaged images being obtained While the imaging apparatus is
sWept; a position sensor capable of obtaining position infor
References Cited
20047605167123:
(74) Attorney, Agent, or Firm * Rader, Fishman & Grauer
PLLC
'
'
.. 348/159
. . . . ..
340/541
7 Claims, 22 Drawing Sheets
16
11
111
J
112
R
J
10
161
2
—
17
IMAGE SIGNAL
PROCESSING
SECTION
12
'
ofthe pos1t1on sensor and outputs a Warmng signal When the
SWeePmg angular veloclty exceeds an aPPFOPFIate range~
13
162
/
1a
ACCELERATION
[.l-COM
141
ANGULAR
VELOCITY SENSOR
UN'T
19
e
SPEAKER
14
UNIT
20
US. Patent
Feb. 19, 2013
Sheet 1 0122
US 8,379,078 B2
zOEm
2x
2|
N9
.523
\
:75
mm g
5
x
w.
mmg;a265 o%296%
W2mwwzm wuog
I~
~
E
N:9
Q
w
28-a
W
\1a:
Qmwémaw
m s8w>a\
zocéwmsg182%
\
3;N3
"1587 2“8.5%
N
8
US. Patent
Feb. 19, 2013
Sheet 2 0f 22
US 8,379,078 B2
‘
ail-unborn
‘
mw
US. Patent
Feb. 19, 2013
Sheet 3 0f 22
US 8,379,078 B2
1611
16%11
161212
16113
FIRST COLOR
CORRECTION
COMBINING
SECOND COLOR
CORRECTION
COMBINING PROCESSING UNIT
WARNING SOUND IS OUTPUT
SWEEPING ANGULAR
VELOCITY
RATIO
100% II
80%
1”“
CASE WHERE
RNG SWEEPING ANGULAR
VELOCITY IS WITHIN
APPROPRIATE RANGE
60%
CASE WHERE
SWEEPING ANGULAR
VELOCITY IS BELOW
APPROPRIATE RANGE
TIME
US. Patent
Feb. 19, 2013
Sheet 4 0f 22
US 8,379,078 B2
ffi
3:
U S. Patent
Feb. 19, 2013
Sheet 5 0f 22
US 8,379,078 B2
US. Patent
Feb. 19, 2013
Sheet 7 0f 22
US 8,379,078 B2
FIG.9
BDR23
i ./
BDROZVA
BLK
.
, BDR13
US. Patent
Feb. 19, 2013
Sheet 8 0f 22
FIG. 10
US 8,379,078 B2
US. Patent
Feb. 19, 2013
Sheet 9 0f 22
FIG. 11
US 8,379,078 B2
US. Patent
Feb. 19, 2013
Sheet 10 0f 22
FIG, 12
US 8,379,078 B2
US. Patent
Feb. 19, 2013
Sheet 11 0122
FIG.13
US 8,379,078 B2
US. Patent
Feb. 19, 2013
Sheet 12 0122
US 8,379,078 B2
FIG. 14A
FIG. 148
I".
I".
I".
I".
I”.
I".
US. Patent
Feb. 19, 2013
Sheet 14 0f 22
US 8,379,078 B2
FIG.16
ZERO CORRECTION OF THE STATIC-STATE SENSOR VALUES
N 141
ACCELERATION
SENSOR
52
“II/‘l /
—~————————-—
REFERENCE VALUE
OF ANGULAR
ANGULAR VELOCITY WM STATIC STATE ~——> VELQC|TY SENSOR
SENSOR
DETECT'ON
142
T
CAMERA
INTER-FRAME
IMAGE
COMPARISON AND
'
AT sTATIO STATE
DETERMINATION
AND RECORD
OF REFERENCE
~> SHIFTAMOUNT
vALuE
DETECTION
CAPTuRED
(
FRAMES
51
2
53
ACHIEVEMENT OF HIGH PRECISION vIA COLLABORATION OF SHIFT INFORMATION
ZERO REFERENCE VALUE
35
ANGULAR
SHIFT AMOUNT
vsEg-gggy
INTEGRATION
/
PREVIOUS
FRAME DATA
54
‘jEEQBACK
142
COLLABORATIVE
mm’ CORRECTION
LoGIc
INTER-FRAME
IMAGE
_, AND sIIIFTAMOuNT
DETECTION
CAPTURED
/
FRAMES
51
l
56
HIGH-PRECISION
RELATIVE SHIFT
INFORMATION
US. Patent
Feb. 19, 2013
Sheet 15 0f 22
NEW IMAGE
IS CAPTURED
A” 5T1
MATCH WITH
PREVIOUS
FRAME
\/ 8T2
US 8,379,078 B2
8T3
MATCHING IS
[YES]
ACHIEVED?
8T4
[NO]
8T8
ST12
MATCIHIN/gJ
SHOWRESULTS
THERE Is NO
[SHIFT
OCCURREDlS
'
SP
[NO SHIFH
PERFORM SAMPLING
OFANGULAR
VELOCITY SENSOR IN
X-AXIS DIRECTION
\l,
[NO SHIFT]
PERFORM SAMPLING
OF ANGULAR
vELOCITY SENSOR IN
Y-AxIs D RECTION
§ST6
INCREMENT
sAMPLE COUNT
\[1
SE9
g ST10
INCREMENT
SAMPLE COUNT
5ST?
CORRECT x-AxIs
DIRECTIONAL zERO
POINT OF ANGULAR
vELoCITY SENSOR
\l,
[NO SHIFT]
PERFORM SAMPLING
OF ANGULAR
VELOCITY SENSOR IN
___Z-AX|S DIRECTION 8T1 4
,
[.1
INCREMENT
sAMPLE COUNT
§ST11
CORRECT Y-AXIS
DIRECTIONAL zERO
POINT OF ANGULAR
VELOCITY sENsOR
ST16
TO CAPTURING
OF NEXT FRAME
S513
,
g ST15
CORRECT Z-AXIS
DIRECTIONAL ZERO
POINT OF ANGULAR
vELOCITY SENSOR
US. Patent
Feb. 19, 2013
Sheet 16 0122
US 8,379,078 B2
FIG. 19
T
ST21 \/\
STZZ w
CAPTURE NEW IMAGE
MATCH WITH PREVIOUS FRAME
ST23
MATCHING IS ACHIEVED?
ST24 w
CORRECT X-AXIS DIRECTIONAL
ANGULAR VELOCITY INTEGRATION
VALUE FROM X-AXIS DIRECTIONAL
MATCHING SHIFT AMOUNT
ST25 w
CORRECT Y-AXIS DIRECTIONAL
ANGULAR VELOCITY INTEGRATION
VALUE FROM Y-AXIS DIRECTIONAL
MATCHING SHIFT AMOUNT
CORRECT Z-AXIS DIRECTIONAL
gm w
ANGuLAR vELOCITY INTEGRATION
VALUE FROM Z-AXIS DIRECTIONAL
MATCHING SHIFT AMOUNT
[
NO
]
US. Patent
Feb. 19, 2013
Sheet 17 0122
CAPTURE
[MAGE
US 8,379,078 B2
}/\, ST31
MATCH WITH PREVIOUS FRAME N ST32
[YES]
MATCHING IS ACHIEVED?
ST34
r.)
OBTAIN RELATIVE SHIFT
AMOUNT FROM PREVIOUS
FRAME BASED
ON MATCHING RESULTS
ST35
[J
OBTAIN RELATIVE SHIFT AMOUNT
FROM PREVIOUS FRAME BASED
ON POSITION SENSOR
INFORMATION
US. Patent
Feb. 19, 2013
Sheet 18 0f 22
US 8,379,078 B2
FIG.21
START
CAPTURING
CAPTURE
NEW FRAME
ST41
/\/ ST42
CALCULATE RELATIVE
ROTATIONAL SH EFT AMOUNT
FROM PREVIOUS FRAME
ROTATE ALL PREVIOUS FRAME BY
RELATIVE ROTATIONAL SHIFT
AMOUNT
w ST44
SET FRONT POSITION a(0, 0, 1.0)
AS CENTER VECTOR OF NEW IMAGE
SET b(0, 0.01, 1.0) SLIGHTLY DISPLACED
FROM FRONT POSITION AS CENTER VECTOR /\/ ST46
OF NEW IMAGE
CAPTURING
COMPLETED?