l|||||||||||||ll||l||||||||l||||||||||||||||||||||||||||l|||||||||||||||||||||||||||||||||
US 20050099407A1
(19) United States
(12) Patent Application Publication (10) Pub. N0.2 US 2005/0099407 A1
Pennington, II et al.
(43) Pub. Date:
(54) TEXT INPUT WINDOW WITH
May 12, 2005
(21) Appl, No;
10/704,127
(22) Filed:
Nov. 10, 2003
AUTO-GROWTH
(75) Inventors: Ernest L. Pennington II, Issaquah, WA
(Us); Adrian James Garside,
Sammamish, WA (US); J e?'rey West
Pettiross, Seattle, WA (US); Shawna
Publication Classi?cation
(51)
Alexander
Julie Davis,Zielinski,
Seattle, Redmond,TObiaSZ
WA
Int. Cl.7 ..................................................... .. G09G 5/00
U-S- Cl- ............................................................ ..
(Us)
(57)
ABSTRACT
Correspondence Address:
BANNER & WITCOEE LTD”
A user input panel dynamically expands to accommodate
ATTORNEYS EOR MICROSOFT
user input, such as handwritten or keyboard input. Expan
1001 G STREET , N.W.
sion may occur in one or tWo out of four possible directions,
ELEVENTH STREET
depending upon the language to be Written or typed. For
WASHINGTON, DC 200014597 (US)
example, When Writing English Words, the input panel may
expand to the right as the user Writes and then doWnWard
(73) Assignee: Microsoft Corporation, Redmond, WA
>
When the input panel has fully expanded rightward.
Check stylus position
Remove hover target if
Hovering over
applicable
input area?
‘
fl
303
Hovered
for timeout
or hover target
selected?
304
No
Display hover target
305
Display input panel
306 I"
V
:
Receive/display input
<
it
307
Input
near right edge of
input panel?
308
Expand input panel down
Input
panel already
Expand input panel right
fl
fully expanded to
right?
1’
310
309
Input
panel already
fully expanded to
bottom?
Patent Application Publication May 12, 2005 Sheet 1 0f 12
US 2005/0099407 A1
Patent Application Publication May 12, 2005 Sheet 2 0f 12
202
201
/
\ \‘
///////////
203
Figure 2
US 2005/0099407 A1
\
Patent Application Publication May 12, 2005 Sheet 3 0f 12
>
Check stylus position
US 2005/0099407 A1
<——
301 W
Remove hover target if
Hovering over
applicable
input area?
303
Hovered
for timeout
.
or hover target
Display hover target
.,
304
selected .
II
305
Display input panel
306 W
l
——->
Receive/display input
4
1
307
Input
near right edge of
input panel?
308
Expand input panel down
Input
.
.
/
panel already
Expand input panel right
fully expanded to
right‘?
/"
309
Input
31o
panel already
fully expanded to
bottom?
311
Figure 3
Patent Application Publication May 12, 2005 Sheet 4 0f 12
‘02:- Y
403
401
Figure 4
US 2005/0099407 A1
Patent Application Publication May 12, 2005 Sheet 5 0f 12
502/“ I alga W501
401
Figure 5
US 2005/0099407 A1
Patent Application Publication May 12, 2005 Sheet 6 0f 12
US 2005/0099407 A1
601
502 \
g- 606
u
'\ 602
604 —-—-L---—
603
Figure 6
1301——--___.
Figure 13
Patent Application Publication May 12, 2005 Sheet 7 0f 12
US 2005/0099407 A1
501
/J
5”? 0/60 JQ! 21*‘ W m P?”
---- "n
401
Figure 7
Mhr
Patent Application Publication May 12, 2005 Sheet 8 0f 12
US 2005/0099407 A1
501
WHOM/WW
@MW 25L;
Mvb
401
Figure 8
Patent Application Publication May 12, 2005 Sheet 9 0f 12
Representatives and direct Taxes shall be apportioned
US 2005/0099407 A1
/
/
among the several States which may be included within this
402
Union, according to their respective Numbers, which shall be
detennined b a
in
e whole Number of tree Persons,
HEW)
including those bound to Sew‘
/ 903
Years, and
-,—-1____-- 501
excluding Indians not taxed, thr
her Persons.
(See Note 2) The actual Enum
nade within
MYears after the first Meeting of the Congress of the
904 /
United States, and within every subsequent Term of ten
Years, in such Manner as they shall by Law direct. The
i
'
\
401
Figure 9
901
Patent Application Publication May 12, 2005 Sheet 10 0f 12
US 2005/0099407 A1
V
r
a
Mvt
A
i
Representatives and direct Taxes shall be apportioned
//
/ 902
among the several States which may be included within thf
402
Union, according to their respective Numbers, which shall be /L/ 903
determined by a mg
whole Number of free Persons,
including those bound to Sew‘
‘
’
‘“
501
//
‘
502 —
904 /
i
.... __>
901
nun
; Mhl
/‘/
---->§ Mhr <---Mvb
\
Q
\
4
401
Figure 10
Patent Application Publication May 12, 2005 Sheet 11 0f 12
F
US 2005/0099407 Al
v
j
Mvt
A
Representatives and direct Taxes shall be apportioned
/
/ 902
among the several States which may be included within this
402
Union, according to their respective Numbers, which shall be ?/ 903
determined by a
mg
e whole Number of free Persons,
including those bound to Service for a Tenn ol Years, and
excluding Indians not taxed, three ?fths of all other Persons.
(See Note 2) The actual Enumeration shall be made within
at least four, but no more than five Years after the first
904 "w Meeting of the Congress of the United States, and within
every subsequent Term of ten Years, in such Manner as
g
"n--.
.
901
._-..
5 Mhl
/‘/
Mhr < ---- -
Mvb
k
\
‘
401
Figure 11
5
J
Patent Application Publication May 12, 2005 Sheet 12 0f 12
US 2005/0099407 A1
1 201
r
i
1 204
//
I]
NM
/
,7
| httpJ/wwwm lcrosofLoom
w/\/12o2
/
1 ,
/
é
_
:
_
UHUU} . "ME/10%
.w/Tn/
Mhr
.... -->
1 205
ML
Mvb
401
Figure 12
<_..-.
May 12, 2005
US 2005/0099407 A1
TEXT INPUT WINDOW WITH AUTO-GROWTH
prevent the user from vieWing a desired destination for neW
data, it is inconvenient for the user to continuously move the
FIELD OF THE INVENTION
stylus back-and-forth betWeen the application and the data
input panel to both control the application and enter data into
[0001] Aspects of the present invention are directed gen
erally to improved user input interfaces in computer sys
tems, and more particularly to providing an input panel that
dynamically expands on an on-demand basis as the user
Writes or types.
BACKGROUND
[0002] User input devices and interfaces have been groW
ing rapidly to meet the demand of neW types of computing
devices. More recently, tablet-based personal computers and
hand-held computers have become popular. These devices
typically have a Writing surface that converts movement of
a stylus across the surface into electronic ink. This ink can
be recognized and converted into text, or can be stored in an
electronic ink format.
[0003] For example, the Microsoft WINDOWS brand XP
Tablet PC Edition operating system provides a data entry
user interface, sometimes referred to as the text input panel,
the Tablet PC input panel or “TIP,” through Which a user can
employ a stylus to enter data into a computer. This type of
user interface falls into the broader category of data entry
graphical user interfaces, and may be referred to as a data
input panel. Some data input panels also may provide a
the application.
SUMMARY OF THE INVENTION
[0006] Aspects of the present invention alloW a user to
easily and efficiently enter user input in a Way that is less
intrusive. An input panel is provided that may be easily
discoverable by the novice user and may require little
manipulation on the part of the user to be effectively used.
The input panel may provide easier access to handWritten
input, keyboard input, and other user input When and Where
needed by the user, While possibly reducing the likelihood of
obscuring certain important displayed elements that may be
of interest to the user.
[0007] Further aspects of the invention are directed to
causing an input panel to appear in a convenient location and
to dynamically expand to accommodate the user input. The
input panel may expand in one or tWo directions, such as
rightWard and doWnWard, Without expanding in the other
tWo directions (upWard and leftWard). This may be particu
larly useful Where a language such as English is Written or
typed into the input panel, for example Where the language
is likeWise Written rightWard in a horiZontal roW and then
doWnWard in succeeding roWs. The direction(s) that the
“soft” keyboard surface that displays the characters of a
conventional keyboard. If the user employs a stylus to tap on
the display of a key With this type of keyboard, then the
computer Will receive the character associated With that key
as input data. Other types of data input panels may have
specialiZed input surfaces. For example, some user inter
faces may provide a plurality of letter-siZed Writing surfaces
for receiving characters in an East Asian alphabet as input.
only up to a certain margin distance aWay from an edge of
a display. This may protect the availability of often useful
user interface elements near the edges of the display.
[0004] While these types of data input panel increase the
usefulness of computers that employ a stylus input device,
[0009] Still further aspects of the invention are directed to
convenient and intuitive summoning and dismissal of the
input panel expands may depend upon the chosen language.
[0008] According to a further aspect of the invention, the
input panel may expand in a Way that respects certain
de?ned margins. For example, the input panel may expand
there are still some inconveniences associated With conven
input panel, While reducing the probability that an uninvited
tional data input panels. For instance, data input panels are
input panel Will appear or that a desired input panel Will
generally speci?cally invoked by a user in a manner that is
inadvertently disappear.
difficult or inconvenient. In order to enter data into an
[0010]
These and other aspects of the invention Will
application using a data input panel, the user manually
activates the data input panel from outside of the application.
become apparent to one of ordinary skill in the art upon a
Many computer users are unfamiliar With data input panels,
claims.
hoWever, and may overlook or not knoW hoW to invoke this
resource, rendering such a resource effectively “hidden”
from the novice user.
[0005] Additionally, because the data input panel is a
graphical user interface invoked separately from the appli
cation, it typically is displayed separately from the applica
tion as Well. In some situations, the user interface may be
overlaid onto the displayed application itself. While this
arrangement alloWs the user to enter data close to its
destination in the application, the user interface may inad
vertently obscure the destination or adjacent destinations.
Even if the user interface does not initially obscure a desired
destination for entered data, hoWever, the user may need to
constantly relocate the user interface as neW data is entered
into the application, to prevent the user interface from
obscuring the neW insertion points for that data. With still
other types of data input panels, the user interface may be
“docked” in a space beloW the application. While position
ing the data input panel in this fashion ensures that it Will not
reading of the folloWing description, draWings, and the
BRIEF DESCRIPTION OF THE DRAWINGS
[0011]
The foregoing summary of the invention, as Well as
the folloWing detailed description of illustrative embodi
ments, is better understood When read in conjunction With
the accompanying draWings, Which are included by Way of
example, and not by Way of limitation With regard to the
claimed invention.
[0012] FIG. 1 is a functional block diagram of an illus
trative computing environment that may be used in accor
dance With at least one aspect of the present invention.
[0013] FIG. 2 is a plan vieW of an illustrative computing
environment that may be used in accordance With at least
one aspect of the present invention.
[0014]
FIG. 3 is a flow chart shoWing illustrative steps
that may be performed in accordance With at least one aspect
of the present invention.
May 12, 2005
US 2005/0099407 A1
FIGS. 4, 5, 7, and 8 are various screen shots
devices (not shoWn) may include a microphone, joystick,
showing hoW an input panel may expand in accordance With
game pad, satellite dish, scanner or the like. These and other
input devices are often connected to the processing unit 110
through a serial port interface 106 that is coupled to the
[0015]
at least one aspect of the present invention.
[0016]
FIGS. 6 and 13 are depictions of illustrative input
panels in accordance With at least one aspect of the present
invention.
[0017]
FIGS. 9 through 11 are various screen shots
system bus, but may be connected by other interfaces, such
as a parallel port, game port or a universal serial bus (USB).
Further still, these devices may be coupled directly to the
system bus 130 via an appropriate interface (not shoWn). A
shoWing hoW an input panel may be used in connection With
monitor 107 or other type of display device is also connected
an input area in a softWare application, in accordance With
at least one aspect of the present invention.
to the system bus 130 via an interface, such as a video
[0018] FIG. 12 is a screen shot shoWing hoW an input
panel may be used in connection With an Internet Web
broWser softWare application, in accordance With at least one
aspect of the present invention.
DETAILED DESCRIPTION OF ILLUSTRATIVE
EMBODIMENTS
adapter 108. In addition to the monitor, personal computers
typically include other peripheral output devices (not
shoWn), such as speakers and printers. In a preferred
embodiment, a pen digitiZer 165 and accompanying pen or
stylus 166 are provided in order to digitally capture freehand
input. Although a direct connection betWeen the pen digi
tiZer 165 and the serial port is shoWn, in practice, the pen
digitiZer 165 may be coupled to the processing unit 110
directly, via a parallel port or other interface and the system
[0019] General Computing Environment
bus 130 as knoWn in the art. Furthermore, although the
[0020]
digitiZer 165 is shoWn apart from the monitor 107, it is
preferred that the usable input area of the digitiZer 165 be
FIG. 1 is a functional block diagram of an example
of a conventional general-purpose digital computing envi
ronment that can be used to implement various aspects of the
present invention. In FIG. 1, a computer 100 includes a
processing unit 110, a system memory 120, and a system bus
130 that couples various system components including the
system memory to the processing unit 110. The system bus
130 may be any of several types of bus structures including
a memory bus or memory controller, a peripheral bus, and a
local bus using any of a variety of bus architectures. The
system memory 120 includes read only memory (ROM) 140
and random access memory (RAM) 150.
[0021] A basic input/output system 160 (BIOS), contain
ing the basic routines that help to transfer information
betWeen elements Within the computer 100, such as during
start-up, is stored in the ROM 140. The computer 100 also
includes a hard disk drive 170 for reading from and Writing
to a hard disk (not shoWn), a magnetic disk drive 180 for
reading from or Writing to a removable magnetic disk 190,
and an optical disk drive 191 for reading from or Writing to
co-extensive With the display area of the monitor 107.
Further still, the digitiZer 165 may be integrated in the
monitor 107, or may exist as a separate device overlaying or
otherWise appended to the monitor 107.
[0023]
The computer 100 can operate in a netWorked
environment using logical connections to one or more
remote computers, such as a remote computer 109. The
remote computer 109 can be a server, a router, a netWork PC,
a peer device or other common netWork node, and typically
includes many or all of the elements described above
relative to the computer 100, although only a memory
storage device 111 has been illustrated in FIG. 1. The logical
connections depicted in FIG. 1 include a local area netWork
(LAN) 112 and a Wide area netWork
113. Such
netWorking environments are commonplace in offices, enter
prise-Wide computer netWorks, intranets and the Internet.
[0024] When used in a LAN netWorking environment, the
computer 100 is connected to the local netWork 112 through
a removable optical disk 192 such as a CD ROM or other
a netWork interface or adapter 114. When used in a WAN
optical media. The hard disk drive 170, magnetic disk drive
netWorking environment, the personal computer 100 typi
180, and optical disk drive 191 are connected to the system
bus 130 by a hard disk drive interface 192, a magnetic disk
drive interface 193, and an optical disk drive interface 194,
cally includes a modem 115 or other means for establishing
respectively. The drives and their associated computer
readable media provide nonvolatile storage of computer
readable instructions, data structures, program modules and
other data for the personal computer 100. It Will be appre
ciated by those skilled in the art that other types of computer
a communications over the Wide area netWork 113, such as
the Internet. The modem 115, Which may be internal or
external, is connected to the system bus 130 via the serial
port interface 106. In a netWorked environment, program
modules depicted relative to the personal computer 100, or
portions thereof, may be stored in the remote memory
storage device.
readable media that can store data that is accessible by a
computer, such as magnetic cassettes, ?ash memory cards,
digital video disks, Bernoulli cartridges, random access
memories (RAMs), read only memories (ROMs), and the
like, may also be used in the example operating environ
ment.
[0022]
A number of program modules can be stored on the
hard disk drive 170, magnetic disk 190, optical disk 192,
ROM 140 or RAM 150, including an operating system 195,
[0025] It Will be appreciated that the netWork connections
shoWn are illustrative and other techniques for establishing
a communications link betWeen the computers can be used.
The existence of any of various Well-knoWn protocols such
as TCP/IP, Ethernet, FTP, HTTP and the like is presumed,
and the system can be operated in a client-server con?gu
ration to permit a user to retrieve Web pages from a remote
Web-based server. Any of various conventional Web broWs
ers can be used to display and manipulate data on Web pages.
one or more application programs 196, other program mod
ules 197, and program data 198. Auser can enter commands
[0026]
and information into the computer 100 through input devices
puter processing system (also referred to as a tablet PC) 201
such as a keyboard 101 and pointing device 102. Other input
that can be used in accordance With various aspects of the
FIG. 2 shoWs an example of a stylus-based com
May 12, 2005
US 2005/0099407 A1
present invention. Any or all of the features, subsystems, and
“system caret” is currently located, or at least a prede?ned
functions in the system of FIG. 1 can be included in the
Zone around and near the system caret. A system caret is the
computer of FIG. 2. Tablet PC 201 includes a large display
insertion point that the user is currently Working With. The
surface 202, e.g., a digitizing ?at panel display, preferably,
a liquid crystal display (LCD) or OLED screen, plasma
insertion point may be indicated to the user via a cursor such
as a blinking text cursor. Another “cursor” may also exist
display and the like, on Which a plurality of WindoWs 203 is
that indicates the location on the display that the stylus 204
(or other input device such as the mouse 102) is interacting
With (e.g., the location on the display over Which the stylus
204 is hovering, or the location on the display that the stylus
204 is touching). The cursor indicating the insertion point is
displayed. Using the tip of a stylus 204 (the tip also being
referred to herein as a “cursor”), a user can select, highlight,
and Write on the digitiZing display area. Examples of suit
able digitiZing display panels include electromagnetic pen
digitiZers, such as the Mutoh or Wacom pen digitiZers. Other
not necessarily the same as the cursor indicating the stylus
types of pen digitiZers, e.g., optical digitiZers, may also be
used. Tablet PC 201 interprets marks made using stylus 204
204 interaction point.
[0031] In step 303, if the stylus 204 is not hovering, then
any invocation target that may be displayed (as discussed
beloW) is removed from being displayed, the computer then
checks again in step 301 Whether the stylus 204 is hovering.
This loop is repeated until it is determined that the stylus 204
in order to manipulate data, enter text, and execute conven
tional computer application tasks such as spreadsheets, Word
processing programs, and the like.
[0027]
The stylus 204 may be equipped With buttons or
other features to augment its selection capabilities. In one
embodiment, a stylus could be implemented as a “pencil” or
“pen”, in Which one end constitutes a Writing portion and the
other end constitutes an “eraser” end, and Which, When
moved across the display, indicates portions of the display
are to be erased. Other types of input devices, such as a
mouse, trackball, or the like could be used. Additionally, a
user’s oWn ?nger could be used for selecting or indicating
portions of the displayed image on a touch-sensitive and/or
proximity-sensitive display. Consequently, the term “user
input device”, as used herein, is intended to have a broad
de?nition and encompasses many variations on Well-knoWn
input devices.
is hovering. Once this occurs, then in step 304 it is deter
mined Whether the stylus 204 has either hovered for a
predetermined timeout period or Whether any displayed
invocation target has been selected. The invocation target
may be selected by, e.g., clicking on the invocation target
such as With a button of the mouse 102, by tapping on the
invocation target With the stylus 204, by hovering over the
invocation target, or by pressing a button on the computer or
the stylus 204. If neither of these determinations occurs, then
in step 305, an invocation target is displayed (or is continued
to be displayed if already displayed), and the computer
continues to check Whether the stylus is hovering in steps
301 and 302. The invocation target may be displayed imme
diately after the stylus has begun hovering over an appro
[0028] Input Panel Expansion Behavior
priate area of the input surface or after a time delay. Step 304
[0029] Referring to FIG. 3, in step 301, a computer (e.g.,
target selection determination, or it may only include one of
these tWo determinations. Although the present illustrative
embodiment is described as displaying the invocation target
computer 100 or 201) may check the position of a user input
device (e.g., stylus 166 or 204) relative to an input surface
(e.g., digitiZer 165 or display surface 202). The user input
device Will be illustratively considered in the folloWing
embodiments to be the stylus 204. HoWever, any user input
device may be used, such as the mouse 102. In step 302, the
computer may determine, based on the position of the stylus
204, Whether the stylus 204 is hovering over the input
surface and Whether the stylus 204 is hovering over a valid
“input area.” The term “hovering” means that the stylus 204
(particularly, for example, the tip of the stylus 204) is
proximate to, but not physically touching, the input surface.
A threshold-based determination may be made. For
example, the computer may determine Whether the tip of the
stylus is Within a predetermined orthogonal distance from
the input surface. If so, then the stylus is hovering. If not
(e.g., the stylus 204 is too far aWay from the input surface
or is touching the input surface), then the stylus 204 is not
hovering.
[0030] The term “input area” means an area of the display
or user input surface that accepts valid user input. At any
given time, there may be one or more input areas, and these
input areas may change, move, appear, and disappear over
time. Which areas are input areas may depend upon Which
applications are running, Which operating system is running,
Which application(s) has focus (i.e., is the application cur
rently and actively being interacted With by the user), and/or
other factors. Input areas typically accept typed text from a
keyboard and/or handWritten ink from a stylus. In some
embodiments, the only input area may be Wherever the
may include both a timeout determination and an invocation
in response to the stylus 204 hovering, the invocation target
may also be displayed in response to other user input, such
as in response to the stylus 204 actually contacting the input
surface or in response to some other gesture performed by
the stylus 204. It Will be understood that hovering as
discussed herein is presented only as an illustrative user
input.
[0032] An “invocation target” is an indication to the user
that an input panel is available and can be invoked. The
invocation target may be an indication that is visible and/or
audible to the user, such as an icon displayed on a display,
an audible sound such as a beep or click, or a modi?ed
cursor. An example of an invocation target 403 is shoWn in
FIG. 4. The invocation target 403 may be displayed at a
predetermined location on a display 401 (Which may be part
of the input surface), or at a location that depends upon the
location 402 on the display 401 over Which the stylus 204 is
hovering or otherWise interacting With the display 401 (the
“interaction point”402). For example, the invocation target
403 may be displayed at a location that is displaced from the
interaction point 402 by a predetermined amount. Alterna
tively, the invocation target 403 may be displayed at a
location that depends upon the location of the insertion point
in the active input area. For example, the invocation target
403 may be displayed at a location that is displaced from the
insertion point by a predetermined amount. In the shoWn
embodiment, the invocation target 403 is displayed at a
location Where the edge of the invocation target 403 nearest
May 12, 2005
US 2005/0099407 Al
the interaction point 402 is displaced on the display 401 by
X distance in the horizontal direction and by Y distance in
the vertical direction, Where X and Y may be the same or
different and may be negative or positive. It should be noted
that in this example, the “X”, “Y”, and broken lines as
shoWn are not actually displayed but are included only for
explanatory purposes. The X and Y distances may be any
amount, such as about 1A of an inch or less, about 1/3 of an
inch, or about 1/2 of an inch or more.
[0033] Although the invocation target 403 as shoWn is a
box With an “X” inside, this is merely an example. The
invocation target 403 may appear to the user in any form,
such as an icon, an animation, an image, text, and/or any
the same as a right visible edge of the input panel 501, or the
right boundary may be a predetermined distance to the left
of the right visible edge. FIG. 6 shoWs an example of a right
boundary 606 that is to the left of the right edge 602 of the
input panel 501. HoWever, the boundary 606 may also be at
the same location as the right edge 602. Also indicated are
the top edge 601, left edge 605, and bottom edge 603 of the
input panel 501.
[0038] FIG. 7 shoWs What may occur in response to the
user input being near or touches a boundary of the input
panel 501. FIG. 7 also shoWs directions “up,”“doWn,”“left,”
and “right.” These illustrative directions Will be used con
other type of visual element(s). Also, although the invoca
sistently throughout the speci?cation to help explain aspects
tion target 403 is shoWn as a rectangle, it may be of any
shape such as a circle, square, triangle, and/or any other
vieW as the user vieWs the display 401. The directional
geometric or non-geometric shape.
[0034] As discussed above, the user’s stylus 204 may
hover for the timeout period, or the user may select the
invocation target 403. The user may select the invocation
target 403 in any of a number of Ways, such as by tapping
of the invention, and are measured from the user’s point of
arroWs and direction text as shoWn are not necessarily
displayed on the display 401, but are merely shoWn for
explanatory purposes. The directions as used herein are also
relative to the display 401 and are not necessarily relevant to
or descriptive of the pull of Earth’s gravity. For example, the
display 401 may lay horiZontally on a table, yet the “up”
on the displayed invocation target 403 With the stylus 204.
direction as shoWn remains the “up” direction relative to the
In response to either of the actions set forth in step 304
display 401.
occurring, an input panel is presented to the user in step 306.
The input panel is able to receive user input, such as
electronic ink input from the stylus 204, typeWritten input
from a keyboard, and/or other input.
[0035] An example of a displayed input panel 501 is
shoWn in FIG. 5. The input panel 501 may replace the
invocation target 403 or it may be displayed in addition to
the invocation target 403. The input panel may include an
anchor point 502, Which may or may not be displayed. The
input panel 501 may be displayed at any location on the
display 401, such as Where its anchor point 502 is at a
predetermined location or at a location that depends upon
the interaction point 402 and/or the location of the invoca
tion target 403. In the example shoWn, the input panel 501
is displayed in the same, or nearly the same, location as the
invocation target 403 that Was previously displayed before
being replaced by the input panel 501.
[0036] Referring back to FIG. 3, in step 307 the input
panel 501 may receive as user input any type of content
desired, such as text, ink, images, audio (e.g., speech), etc.
HoWever, the illustrative input panel 501 in the embodi
ments disclosed Will be considered to be primarily for
receiving text, ink, and/or speech content. The reason for
this is the input panel 501 may, in certain embodiments, be
considered a text input panel for receiving meaningful input
for entry into a currently running application such as a Word
processing application. As can be seen in FIG. 5, the letters
“abc” are Written in handWritten ink as content in the input
[0039] As can be seen, the user input, originally “abc”, has
been changed to noW read “abc def ghi jkl mno pqr”.
Because this user input Would not have ?t in the input panel
501 as shoWn in FIG. 5, the input panel 501 automatically
expands to accommodate the user input. In this embodiment,
responsive to the user input being near or touching a right
boundary of the input panel 501, the input panel 501
dynamically expands its right visible side 602 and/or right
boundary 606 as necessary to ?t the user input. For instance,
if the user input had only been “abc def ghi”, then the input
panel 501 may have expanded rightWard less than shoWn in
FIG. 7. In real time, as the user input moves toWard the
right, the input panel 501 may continue to expand rightWard.
HoWever, the input panel 501 may be limited as to hoW
much it can expand rightWard. Aphysical limitation Would
be the boundary of the display 401. The input panel 501 may
further be arti?cially limited in its rightWard expansion. For
example, the input panel 501 may only be able to expand
rightWard up to a predetermined horiZontal right margin Mhr
from the right boundary of the display 401. The margin Mhr
may be ?xed or variable, and may be set by the user and/or
by an application. Margin Mhr may be measured as absolute
distance or as a percentage of the siZe of the display 401. For
example, Mhr may be about 1A inch, about 1/2 inch, betWeen
about 1A inch and about 1/2 inch, about 3% of the display 401
boundary Width, about 5% of the display 401 boundary
Width, or betWeen about 3% and about 5% of the display 401
boundary Width. HoWever, Mhr may be other values.
panel 501. This may be done by, e.g., using the stylus 204
Throughout the expansion process, the anchor point 502
to Write “abc” on the input surface, and more particularly on
may remain ?xed. This means that the top edge 601 and the
the portion of the input surface corresponding to the input
panel 501.
left edge 605 of the input panel 501 may also remain ?xed
[0037] In FIG. 3, in step 308, the computer may check
[0040] Thus, referring back to FIG. 3, if in step 309 the
input panel 501 has not yet fully expanded rightWard to its
limit, then in step 310 the input panel 501 expands rightWard
Whether the user input is near or touching a boundary of the
input panel 501. Boundaries of the input panel 501 may be
coextensive With the visible edges of the input panel 501,
may be Within the visible representation of the input panel
501, and/or may extend beyond the visible representation of
the input panel 501. For example, a right boundary may be
throughout the expansion process.
and continues to accept further user input in step 307. The
input panel 501 may expand rightWard in increments. The
increments may be of a ?xed siZe or may vary depending
upon the user input. Where the increment siZe is ?xed, the
May 12, 2005
US 2005/0099407 A1
increment may be about 1.5 inches, or about 1.75 inches or
dropping it) to make further room for input panel expansion.
about 2 inches, or more or less than these amounts.
The anchor point 502 may represent the intersection of the
upper and left boundaries of the input panel 501. HoWever,
the anchor point 502 may represent the intersection of other
boundary combinations of the input panel 501, such as the
intersection of the upper and right boundaries of the input
panel 501. Returning to the example Where the anchor point
[0041]
On the other hand, if in step 309 it is determined
that the input panel 501 has already reached its rightWard
limit, then the input panel 501 may expand in another
direction in step 312. In the shoWn embodiment, the other
direction is doWn. The input panel 501 may then continue to
accept further user input in step 307. FIG. 8 illustrates What
may happen When the input panel 501 has reached its
rightWard limit and the user continues to provide user input
to the input panel 501. In this case, the user input has noW
increased to become “abc def ghi jkl mno pqr stu vWx yZ1
234”. This entire user input could not ?t in the input panel
501 as shoWn in FIG. 7, so the input panel 501 has
automatically expanded doWnWard to accommodate the
additional user input. In other Words, the bottom edge 603 of
the input panel 501 has moved doWnWard in response to
rightWard extension of an ink stroke. This rightWard and
doWnWard expansion of the input panel 501 may occur in
real time While the ink is being Written. Note that the right
margin Mhr is still respected. The doWnWard expansion may
begin only after the rightWard expansion has cause the input
panel 501 to expand fully to the right up to margin Mhr.
502 represents the intersection of the upper and left bound
aries, the upper and left boundaries of the input panel 501
may thus remain ?xed along With the anchor point 502.
Therefore, once the input panel 501 has fully expanded in
both the rightWard and doWnWard directions, the input panel
501 cannot expand further. As Will be discussed further,
Which intersection of boundaries the anchor point 502
represents may depend upon Which language is expected to
be Written or typed into the input panel 501.
[0045] Although the input panel 501 may reach a point
Where it can no longer expand in any direction (i.e., Where
the outcome of step 311 is “yes”), the user may continue to
add user input and/or edit existing user input in the existing
area de?ned by the input panel 501. Where the input panel
501 can no longer expand, the input panel 501 may in
response develop one or more scroll bars in the horiZontal
Alternatively, at least some doWnWard expansion may occur
and/or vertical directions for alloWing still further user input
prior the input panel 501 expanding fully to the right, before
to be entered. HoWever, in some embodiments, it may be
it expands to margin Mhr. Moreover, at least some doWn
desirable to keep the interface With the input panel 501 as
Ward expansion may occur simultaneously With rightWard
simple as possible, and so scroll bars Would not become
available in that instance.
expansion (Which may appear to the user as effectively a
diagonal expansion).
[0046]
A horiZontal left margin Mhl and a vertical top
[0042] HoWever, in the present embodiment, the input
panel 501 may expand doWnWard only if it has not yet fully
margin Mvt are also shoWn in FIG. 8. These margins are
expanded in the doWnWard direction. Thus, in step 311, if the
it may expand in an incremental manner. For example, the
501 cannot cross those margins. In some embodiments, the
anchor point 502 can never be located to the left of margin
Mhl, nor can the anchor point 502 ever be located above
input panel 501 may expand by a predetermined increment
margin Mvt. This Would be true regardless of Where the
input panel 501 has not yet fully expanded doWnWard, then
arti?cial margins that may be used such that the input panel
or by an amount depending upon the user input. In one
interaction point 402 is. Any rules de?ning Where the input
embodiment, the doWnWard incremental expansion amount
panel 502 should be displayed should ensure that the input
panel 502 is not located outside of margins Mhl, Mhr, Mvt,
may be a ?xed amount of about 1.5 inches, or about 1.75
inches or about 2 inches, or more or less than these amounts.
and Mvb, both initially and While the input panel 502
In another embodiment, the doWnWard incremental expan
sion amount may depend upon the vertical siZe of the
handWritten or vertical characters that has been previously
expands. If necessary, the initial default location of the input
panel 502 that Would otherWise be used as described previ
Written into the user input panel. The larger the previously
Written text, the larger the doWnWard increment, With the
the input panel 502 does not violate these margins. When
expectation that further user input characters Will be of a
similar siZe.
invocation target and/or the input panel Within the margins,
then the invocation target and/or input panel may be moved
[0043] On the other hand, if the input panel 501 has
default location Without violating the margins.
already expanded to a maximum doWnWard limit as deter
mined in step 311, then the input panel 501 may not expand
further. HoWever, the input panel 501 may continue to
accept user input. The expansion limit in the doWnWard
direction may be limited by the bottom boundary of the
display 401. The doWnWard expansion limit may further be
limited by an arti?cial vertical bottom margin Mvb. Margin
Mvb may be determined in the same manner as margin Mhr.
[0044] At this point, it should be noted that the anchor
point 502 in this embodiment continues to remain in the
same ?xed location on the display 401 throughout FIGS. 5,
7, and 8. This is true even though the input panel 501 may
have fully expanded in the rightWard and/or doWnWard
directions. In some embodiments, the user may be able to
manually move the anchor point (such as by dragging/
ously may be modi?ed by an amount suf?cient to ensure that
ever the default placement rules do not result in placing the
as little as possible so as to be as close as possible to the
[0047] Input Panel Interaction With Input Areas
[0048] Examples of hoW the input panel 501 may be used
is shoWn in FIGS. 9 through 11. In FIG. 9, the display 401
shoWs the graphical user interface representation 901 of an
open softWare application, such as, but not limited to, a Word
processing application, spreadsheet application, Internet
Web broWser, or an email program. The softWare application
may display content 902, Which is text in this example but
may be other content. The content 902 may be interacted
With, such as by editing, deleting, or adding to it. Therefore,
in this particular embodiment, an area 903 on the display
401 (shoWn by broken lines that are not necessarily dis
played) containing the content 902 may be considered an
input area as previously de?ned. The stylus 204 may be
May 12, 2005
US 2005/0099407 A1
hovering over the display 401 in an area indicated by the
interaction point 402 (or over a portion of an input surface
corresponding With the interaction point 402 on the display
401, Where the input surface is not the same as the display
401). Because the interaction point 402 is Within the input
area 903, an invocation target, and eventually the input panel
501, may be displayed as previously described. Had the
interaction point 402 been outside of the input area 903 (and
outside of any input area), then an invocation target, and the
input panel 501, may not have been displayed. Thus,
Whether the invocation target and/or input panel are dis
played may depend upon Where the interaction point 402 is
relative to an input area. Input areas may be de?ned by the
be transferred to the input area associated With the input
panel 501, Which in this example is input area 903. In
particular, the user input may be transferred to a location
Within the input area 903 associated With the interaction
point 402, Which in this case is the insertion point 904
generated by the active softWare application. The user input
may further be converted into another form prior to adding
it to the input area 903. For example, Where the user input
is handWritten ink, the ink may be recogniZed into text, and
the recogniZed text may be added to the input area 903. As
shoWn, the user’s handWriting is recogniZed, and the result
ing text corresponding to the handWriting replaces the
highlighted Word “three” at the insertion point 904, such that
operating system and/or by the application softWare cur
the relevant portion of the content 902 noW reads, “
rently running.
Within at least four, but no more than ?ve, Years . . .”. After
[0049] In this example, the input panel 501 may be asso
ciated With the input area 903 and may initially be empty of
user input and content. The input panel 501 may particularly
be associated With an insertion point in the input area 903.
Typically, the application, or a system hosting the applica
tion (such as the operating system), Will provide an insertion
point Where data created by an input device Will be inserted
into a control such as the input area 903. For example, an
application may display an insertion point to indicate Where
data created by an input device Will be inserted. An insertion
point for text typically may be displayed as a cursor in the
form of a blinking vertical line. The position of the insertion
point Will conventionally be controlled by one or more input
devices. The insertion location may further be a selection of
insertion of the recogniZed user input from the input panel
501, the insertion point 902 is noW located after the inserted
content, and is shoWn in the form of a standard text cursor.
[0053] HandWriting recognition may be performed auto
matically or manually at various times as desired. For
example, handWriting recognition may be invoked only after
the user is done With adding the user input, or automatically
While the user is adding the user input. Where recognition is
performed While the user is adding input, handWriting rec
ognition may be automatically performed after each char
acter is Written or after each Word is Written. The text
resulting from handWriting recognition may be added to the
input area 903 While the user is still adding input Without
Waiting for the user to ?nish, or after the user has ?nished.
replaceable text. When a user Wishes to replace a selection
of text With neW input data, the user may select the entire
text to be replaced. For example, a user may activate a
Recognition may occur in response to the stylus 204 being
lifted off the input surface for at least a certain period.
command button associated With a pointing device, and then
move the pointing device so that a displayed pointer passes
command from the user, in response to sWitching input
modes, or and/or in response to sWitching applications.
over or through the text to be replaced. This process is
sometimes referred to as “blocking” a selection of text.
When a user then enters data into the control, the entirety of
the selected text is replaced With the neWly entered data.
[0050] In this example, the user has blocked the Word
“three” in the content 902, With the intention of editing that
Word. The selected portion may therefore be considered the
insertion point 904 in the input area 903. Any changes that
the user makes to the content 902 using the input panel 501
Would occur at the insertion point 904. As shoWn, the user
adds handWritten user input “at” to the input panel 501. The
insertion point 904 associated With the input panel 501 may
be further highlighted or otherWise modi?ed to indicate that
it is associated With an input panel.
[0051] Referring to FIG. 10, the user continues to Write,
adding to the “at” previously Written, so that the handWritten
content in the input panel 501 is noW “at least four, but no
more than ?ve”. As previously described, the input panel
501 may expand rightWard, and if necessary doWnWard, to
accommodate the user input. Again, the input panel 501 may
expand rightWard and doWnWard in such a Way that it does
not violate the Mhr and Mvb margins and that maintains the
anchor point 502 in a ?xed position. In this example, the
anchor point 502 is the upper left corner of the input panel
501. Also, the input panel 501 may initially appear in such
a location that does not violate the Mhl, Mhr, Mvt, and Mvb
Recognition may alternatively occur in response to a manual
[0054] There may be exceptions to the rule that the anchor
point 502 is ?xed. For example, Where the insertion point
904 is moved under the input panel 501, the input panel 501
may move a sufficient amount to prevent the insertion point
904 from being hidden from the user. The insertion point
may move under the input panel 501 Where, for example, the
user input handWriting in the input panel 501 is recogniZed
and sent to the input area 903 While the user Writes.
[0055] FIG. 12 illustrates another example of using an
input panel in connection With an Internet Web broWser
softWare application. The broWser application may display a
broWser WindoW 1202, With a content box 1203 in the
WindoW for displaying the content of a currently-broWsed
Web page. The broWser application may further display an
Internet address input box 1201, Which may receive user
input Internet addresses such as in the form of universal
resource locators (URLs). In this embodiment, the Internet
address input box 1201 is an input area for purposes of
interacting With an input panel. Another input area 1205 is
shoWn in the content box 1203. The existence and/or loca
tion(s) of any input area(s) in the content box 1203 may be
de?ned by and depend upon the particular Internet Web page
being broWsed at any given time. The displayed Internet Web
page may be considered a document, and in general, any
type of document may de?ne the existence and/or loca
tion(s) of any input area(s) therein.
margins.
As shoWn in FIG. 11, When the user is ?nished
[0056] If the interaction point (not shoWn) remains in or
adding user input to the input panel 501, the user input may
near the Internet address input box 1201, then in response an
[0052]
May 12, 2005
US 2005/0099407 A1
invocation target (not shown) and/or an input panel 1204
that is summoned immediately may be smaller or missing
may appear. In this example, the hover position may be in
the Internet address input box 1201, Which is outside the
Mvt margin. Therefore, placement rules should ensure that
the invocation target and/or the input panel 1204 do not
certain elements (such as keyboard buttons, mode sWitching
user interfaces, etc.). This smaller, simpler state may be
preferable in such a situation because this smaller, simpler
display above the Mvt margin. Depending upon the speci?c
circumstance, the invocation target and/or the initially
displayed input panel 1204 may need to be moved doWn
Ward by a small amount to ensure this. In addition, the
location that the invocation target and/or the input panel
1204 are displayed may be affected by any auto-complete
dropdoWns that are expected to shoW. For example, Where
the Internet address input box 1201 has an auto-complete
feature that drops doWn a list of suggested complete Internet
addresses, the invocation target and/or input panel 1204
initial display position may be adjusted to be displayed
beloW Where the bottom of the expected dropdoWn auto
complete list Would be displayed.
[0057] Also, depending upon hoW far to the right of the
display the interaction point is located, the invocation target
and/or the initially-displayed input panel 1204 may need to
be moved leftWard to ensure that the default initial siZe of
the input panel 1204 does not violate the Mhr margin. The
user may Write, using the stylus 204, a URL such “WWW
.microsoft.com”. This handWritten URL may be recogniZed,
and the resulting text may be added to the Internet address
input box 1201. In response, the Web page associated With
that URL may be broWsed to. Had the interaction point been
associated With the input area 1205 (e.g., inside the input
area 1205), the resulting user input sent to the input area
1205 may interact With the displayed Web page and/or be
sent to a remote server providing the displayed Web page.
[0058] Summoning and Dismissing
[0059] Input panels have been described as being sum
moned by hovering the stylus 204 over an input area.
HoWever, an input panel (such as the input panel 501 or the
input panel 1204) may be summoned in any of a variety of
Ways. For example, an input panel may be explicitly sum
moned in response to a user’s commands, such as a key
board command, menu command, or gesture of the stylus
204. Stylus gestures may include any type of gesture such as
hovering, tapping, and/or dragging of the stylus 204 relative
to the display 401. An input panel may be summoned With
or Without a preceding invocation target.
[0060] Moreover, input panels have been described as
being summoned after an invocation target ?rst appears.
HoWever, the invocation target is not necessary; an input
panel may be summoned Without a prior invocation target.
This may be particularly useful Where the input area is
currently blank (devoid of user input) or is of a certain type.
Thus, in some embodiments, an invocation target may
input panel Would be an educated guess that the user actually
desires to start inputting via the input panel, Whereas shoW
ing the full input panel in such a situation may be a bit
intrusive.
[0061] LikeWise, an input panel (such as the input panel
501 or the input panel 1204) may be dismissed in any of a
variety of Ways. For example, an input panel may be
explicitly dismissed by a user-invoked keyboard command,
menu command, or gesture of the stylus 204. Again, such a
gesture may be any type of gesture. Once closed explicitly,
the invocation target and/or the input panel may not be
automatically displayed again in connection With the same
input area or softWare application. HoWever, if the focus
changes or if the stylus 204 is removed from the input
surface in the currently-focused input area or application,
then automatic invocation of the invocation target and/or
input panel may again be performed in connection With that
same input area.
[0062]
In one example of a gesture-based dismissal, if the
stylus 204 hovers outside the displayed input panel (or at
least a predetermined distance, such as 1/2 inch, outside a
boundary of the input panel), then in response the input
panel may be dismissed. A dialogue WindoW may provide
the user the opportunity to verify that dismissal is desired.
This “hover aWay” dismissal may alWays dismiss the input
panel, or may only dismiss the input panel if the stylus 204
has already entered the area of the input panel. This may
reduce unintended dismissal from occurring. Another
example of gesture-based dismissal may be that the input
panel is dismissed in response to the stylus 204 going out of
range of the input surface. This “out of range” dismissal may
alWays Work or may Work only if user input has already been
provided to the input panel. Again, the latter may help
reduce unintended dismissal. The “hover aWay” dismissal
and the “out of range” dismissal may also each require that
the stylus 204 be aWay from the input panel or out of range
for at least a minimum timeout. The timeout may be any
amount of time, such as about one second or more.
[0063] Yet another Way to dismiss an input panel is to
dismiss the softWare application and/or input area associated
With the input panel. Should this occur, any user input in the
input panel not already sent to the softWare application
and/or input area may be ?ushed and not sent. HoWever,
simply changing the focus to another softWare application or
to the shell may not affect the input panel, Which may remain
open but out of focus until selected again.
[0064]
Input Panel Content Formats
normally ?rst appear folloWed, if summoned, by the input
[0065] Examples of input panels have been described thus
panel, While in certain situations in the same embodiments
far in connection With handWritten user input using a stylus.
HoWever, an input panel may accept a variety of user input
the input panel may be summoned Without the invocation
target. For example, the input panel may be immediately
formats including keyboard input and speech. Depending
summoned in response to the stylus 204 hovering over a
upon the type of user input desired, the input panel may be
displayed in a different form. For example, Where freestyle
handWriting is desired, the input panel may be blank or may
blank input area. Or, the input panel may be immediately
summoned in response to the stylus 204 hovering over or
approaching a blank “To,”“From,” or “Subject” input area in
contain a single line on Which to Write Words. Where
character-by-character recognition is desired, the input panel
an email. The input panel that is summoned immediately
may differ from the input panel that is summoned after an
may display a series of blocks or the like, one for each
invocation target is displayed. For example, the input panel
character, such as is shoWn in FIG. 13. As the input panel