null  null
(12) United States Patent
(10) Patent N0.:
Li et a].
US 7,237,907 B2
(45) Date of Patent:
US. Cl. ....................... ..
Field of Classi?cation Search ................ .. 353/69,
shlgeru Kojlmas Toehlgl Pref (JP);
See application ?le for complete search history.
T0shin0ri Furuhashi, Tochigi Pref.
(JP); Kazuya Arakawa, Tochigi Pref.
References Clted
Ryohichi Yonezawa, Tochigi Pref.
2002/0021418 A1 *
353/70; 353/121; 348/747
353/70, 28, 42, 122, 121, 94; 348/7454747,
(75) Inventors: Baoxin Li, \lllancouver, 'WA (US);
Jul. 3, 2007
sharp Laboratories of America, Inc”
2003/0210381 A1 *
Camels, WA (Us)
2004/0061838 A1 *
Raskar ...................... .. 353/69
11/2003 Itaki .- .... ........... ..
4/2004 MOChlZukl et a1. ......... .. 353/69
* cited by examiner
Subject to any disclaimer, the term of this
patent is extended or adjusted under 35
P1’lmary Exammeriw B Perkey
U_S_C_ 154(1)) by 0 days_
(21) Appl. No.: 11/642,608
Assistant ExamineriAndreW Sever
(74) Attorney, Agent, or Firm4Chernoif, Vilhauer McClung
& StenZel
(22) Filed:
Dec. 19, 2006
Prior Publication D at a
US 2007/0097328 A1
May 3, 2007
A system for adjusting keystoning in a projector may include
sensing using an imaging device at least tWo boundaries
de?ning a projection screen and determining a transforma
tion to adjust the keystoning of an image projected from the
Related U-s- APPheatmn Data
Division of application NO_ 10/630,823, ?led on JUL
29 2003 HOW Pat NO_ 7 175 285
Int_ CL
G03B 21/14
H04N 3/23
projector. The image may be modi?ed from the projector in
accordance With the transformation and projecting the modi
?ed image from the projector, Wherein the imaging device
and the projector are maintained in a ?xed relationship With
respect to each other.
5 Claims, 10 Drawing Sheets
An input line
Median Filter
A 80
Gradient Computation A 82
Zero-crossing/Peak Locator A84
Candidate Pair Matcher
Statistical Inference
(Choosing the Final Pair)
Screen location
U.S. Patent
Jul. 3, 2007
Sheet 1 0f 10
US 7,237,907 B2
3. Height adjustment I
2. Power on
5. Zooming
G’ 4. Focus
1. Cable connection
6-step procedure 1s required to PI‘OJGCt
an image in the right way.
(Steps 3-6 are repeated to make right
adjustment by con?nning the results
of adjustment each time)
2. Power on
zII Iz
1/ ,0’ I
______ "
3. Set button on
1. Cable connection
Setting free model enables
right image projection by
just pressing the power
switch and set button.
FIG. 2
U.S. Patent
Jul. 3, 2007
Sheet 3 0f 10
US 7,237,907 B2
Initiate Keystoning
Keystone Calibration]
Auto Focus I
FIG. 4
U.S. Patent
Jul. 3, 2007
Sheet 4 0f 10
US 7,237,907 B2
[email protected]
:2 5; _-_:_
“63cm owm m
uw g
U.S. Patent
Jul. 3, 2007
Sheet 5 of 10
US 7,237,907 B2
Identify Projection
Screen Boundaries
Adjust Projected
FIG. 9
An input line
Median Filter
A 80
Gradient Computation A 82
Zero-crossing/Peak Locator A 84
Candidate Pair Matcher A 86
Statistical Inference
(Choosing the Final Pair)
Screen location
FIG. 10
U.S. Patent
Jul. 3, 2007
Sheet 6 0f 10
US 7,237,907 B2
.mawe 0 f me S m m anm
the projector's perspective.
The line is assumed to be the
one row that the 1-D sensor
can sense.
140 -
130 i
The luminance values of the
row in (a), illustrating that
120 -
110 -
WdcrowTsm.h1 am on.m m .m mlammyf.nmmmn(wow
(b) 100
thus is potential screen area).
50 100 150 200 250 300 350
0v~rmh ~
_".0 I
:1.m:10:1-x:1 :1.m
I “_
FIG. 11
m f m
U.S. Patent
Jul. 3, 2007
Sheet 7 0f 10
FIG. 12
US 7,237,907 B2
U.S. Patent
Jul. 3, 2007
Sheet 8 0f 10
FIG. 13
US 7,237,907 B2
U.S. Patent
Jul. 3, 2007
Sheet 9 0f 10
US 7,237,907 B2
___________________________ __+___
——————————— ———>
Based on the similarity of the triangles,
the distance L is computed as a function
of the sensor parameters (B and f) and
the disparityAX (difference between the
two images of the same physical point):
FIG. 14
U.S. Patent
Jul. 3, 2007
FIG. 16
Sheet 10 0f 10
US 7,237,907 B2
FIG. 15
US 7,237,907 B2
When the projector is Zoomed and/or focused. This inherent
tages of the invention Will be more readily understood upon
complexity increases the expense of the resulting projector.
The foregoing and other objectives, features, and advan
consideration of the folloWing detailed description of the
invention, taken in conjunction With the accompanying
This application is a division of US. patent application
Ser. No. 10/630,823, ?led Jul. 29, 2003 now US. Pat. No.
This invention relates generally to image projectors, and
more particularly, to modifying projected images.
Portable digital image projectors are common. Such digi
tal image projectors, While connected to a personal computer
or other image/video source, sit on a surface and are directed
at a projection surface to shoW a “slide” presentation or a
video presentation. Many of these projectors use transmis
sive or re?ective liquid crystal displays, and typically only
have a single main lens. Other such projectors use different
1 is an existing projection system.
is a modi?ed projection system.
illustrates a self adjusting projection system.
illustrates keystone correction With autofocus.
a focused image.
an out of focus image.
the spectrum of FIG. 6.
the spectrum of FIG. 5.
FIG. 9 illustrates image adjustment in accordance With
screen siZe.
imaging devices, such as digital micro-mirrors, and may
FIG. 10 illustrates image ?ltering.
include more than one lens. The projectors can display
images one at the time or as a sequence of images, as in the
FIG. 11 illustrates a captured image.
FIG. 12 shoWs an interactive keystone adjustment.
case of video.
FIG. 13 shoWs a keypad.
These digital projectors are typically designed so that
undistorted rectangular images are projected on the projec
tion surface When the projector is placed horizontally on a
level support surface With the projector’s optical axis lined
up perpendicular to the projection surface. HoWever, if the
alignment and orientation is modi?ed, then the resulting
image on the projection surface may be distorted. In many
FIG. 14 illustrates the use of a second imaging source.
cases the image Will appear as a trapeZoid, and in other cases
Referring to FIG. 1, a typical projection system includes
an enclosure housing the electronics and imaging devices.
To set up a projector to display an image properly many
FIG. 15 shoWs projection roll.
FIG. 16 shoWs corrected projection roll.
an arbitrarily shaped quadrilateral. The non-rectangular
shape of the resulting projected image is referred to as
steps are performed Which typically include, (1) connecting
One technique to adjust for keystoning is to tediously
a cable to a laptop or other image source, (2) sWitching a
adjust the physical position of the projector by moving it
poWer sWitch (on/o?‘), (3) adjusting the vertical angle of
projection, (4) focusing the image on the display, (5) adjust
around, tilting and rotating it, until a near rectangular image
is displayed. HoWever, in many situations, it may not be
feasible to suf?ciently physically adjust the position of the
projector. For example, the projector may need to be posi
tioned above or beloW the display surface for proper image
US. Pat. No. 5,548,357, entitled “Keystoning and focus
feWer steps are performed Which typically include, (1)
correction for an overhead projector,” describes a system
Where a test slide is displayed. A user then identi?es line
pairs that appear to be parallel to each other. The user
identi?ed line pair activates a distortion correction program
that uses the oblique angle betWeen the horizontal plane
through the projector and the vieWing screen. This is a
connecting a cable to a laptop or other image source, (2)
sWitching a poWer sWitch (on/o?), and (3) selecting the
auto-set up function. As it may be observed this requires
Referring to FIG. 3, the primary components of a self
adjusting (setting-free) projector include an imaging sensor
system 10, an image pre-processing module 12, projection
image centering and Zooming module 20, projector control
keyboard. The determination and inputting of positional
US. patent Publication 2002/ 00214 1 8 A1, entitled “Auto
system 22, and projector optical and mechanical compo
nents 24.
The imaging sensor system 10 may contain one or more
matic Keystone Correction For Projectors With Arbitrary
imaging sensors, or imaging sensor pairs forming a stereo
pair. Individual imaging sensors may be one-dimensional or
tWo-dimensional sensors. The imaging sensors may also
Orientation”, describes a projection system that includes a
pair of tilt sensors and a camera. The system using data from
the tilt sensors and the camera Warps the projected image to
display an image that is rectangular. Unfortunately, the tilt
sensors tend to be expensive and the system requires com
plex re-calibration When the projector lens moves, e.g.,
screen detection module 14, focus detection and auto-focus
module 16, keystone detection and correction module 18,
user entering positional information into the system via a
information is dif?cult and burdensome for the user.
considerably feWer operations and is less likely to be per
formed in an improper manner.
burdensome task for a user to correctly perform.
US. Pat. No. 5,795,046, entitled “Method for pre-com
pensating an asymmetrical picture in a projection system for
displaying a picture,” describe a system Where the projection
angle, and the trapeZoidal error, is compensated for by the
ing the Zoom of the image, and (6) manually adjusting the
keystone. As it may be observed, this includes many steps
Which typically need to be performed in a suitable order.
Referring to FIG. 2, a modi?ed projection system includes
an enclosure housing the electronics and imaging devices.
To set up the modi?ed projector to display an image properly
include the projector optics, such as the projection lens. In
the case of a pair of sensors, the relative angular deviation
betWeen the surface onto Which the image is projected and
the image plane of the projector may be used as the basis of
US 7,237,907 B2
image modi?cation, such as keystoning. It is noted that a
vertical tilt sensor may be included, if desired.
Orientation”, described above, and determined that occa
right corner, the loWer left comer, and the loWer right comer.
With all four corners identi?ed, or at least a pair of diagonal
corners identi?ed, the aspect ratio of the projection screen
may be determined. Alternatively, the system could detect
the edges of the projection screen and from that determine
the general boundaries of the projection screen. In any case,
sionally the projector system Will fail to properly keystone
compensate. After considering the inability to properly
the identi?cation of one or more aspects of the projection
screen indicating a region that is suitable for displaying an
keystone compensate the present inventors came to the
realiZation that such a potential failure is, at least in part, the
tion screen, Will be referred to as a boundary. In many cases,
The present inventors considered the projector system of
Us. patent Publication 2002/0021418 A1, entitled “Auto
matic Keystone Correction For Projectors With Arbitrary
image on, or otherWise the exterior boundary of the projec
result of the projector being suf?ciently out of focus. In
many cases, the user Will tend to manually focus the
the aspect ratio of the projection screen is similar to the
desired aspect ratio of the projected image. In some cases,
projector prior to keystoning Which Will alleviate the con
only the upper or loWer corners of the projection screen are
cern. HoWever, in other cases the user Will consider the
detected, in Which case, the Width of the projection screen
may be used as the basis to predict a typical aspect ratio of
projector suf?ciently focused for their current needs and Will
attempt to keystone correct the image Without success, thus
the screen, such as a 4:3 aspect ratio. In a similar manner, the
vertical edges of the projection screen may likeWise be used.
In some cases, only the right or left corners of the projection
screen are detected, in Which case, the height of the projec
resulting in frustration to the user. Referring to FIG. 4, to
overcome this previously unconsidered limitation, the sys
tem preferably auto-focuses prior to keystone correction.
While any number of different auto-focus mechanisms may
be used, the system preferably uses an appropriate test
pattern to determine Whether or not the projector is in focus
(see FIG. 5). In the event the system is not properly focused
(see FIG. 6), an iterative process Where the amount of
de-focus is determined, the lens is focused, the amount of
de-focus is determined, the lens is focused, etc. is under
taken. One technique for determining the amount of de
focus is to use the frequency spectrum of the image of the
projected test pattern (see FIG. 7) and compare it With a
reference frequency spectrum (see FIG. 8) of the image of
the test pattern acquired When the projector is properly
tion screen may be used as the basis to predict a typical
aspect ratio of the screen, such as a 4:3 aspect ratio. In a
similar manner, the horiZontal edges of the projection screen
may likeWise be used.
The projected image, such as using a test pattern, is sensed
by the camera and compared With the siZe of the projection
screen. This may be done by comparing the four corners of
the projected test pattern With the four corners (or otherWise)
of the detected projection screen. Then, adjustments to the
projected image may be made to shift the edges of the
projected image in accordance With the projection screen.
The sensing of the projected image by the camera and
focused. Other de-focus (or focus) detection, determination,
resizing the projected image may be repeated until the
and adjustment mechanisms may likeWise be utiliZed, as
desired. After performing an auto-focus of the projection
desired siZe (horizontal and/or vertical) and/or aspect ratio
of the projected image is achieved. For example, if the
system, then the system applies a keystone compensation.
In one embodiment, the keystone compensation may be
manually performed by the user after automatic auto focus.
In another embodiment, the keystone compensation may be
automatically performed by the system after automatic auto
focus. In yet another embodiment, the keystone compensa
tion may be partially manual and partially automatically
right-hand side. In one embodiment, the control parameters
may be the distance betWeen the mass centers of the pro
jection screen and the projected image. The system may use
any suitable relationship betWeen the projected image and
the siZe of the projection screen. For example, the image
performed after automatic auto focus. The auto-focus func
tion should be invoked by the system as a result of activating
or otherWise performing the keystone compensation. The
activation of the auto-focus function, as applied to keystone
projection is to the left of the screen, the screen detection
module may sense such an alignment and output a set of
control parameters to cause the projector to shift toWards the
may be centered on the projection screen and siZed to ?t
compensation, should be in a manner free from separate user
approximately 90% of the projection screen. For example, if
the projection is larger than the detected screen, then the
projector may Zoom in; and if the projection is smaller than
selection of the otherWise existing auto-focus function of the
projector. In this manner, the focus of the system is ensured
the detected screen, then the projector may Zoom out.
prior to completion of the keystone correction.
The present inventors further considered the projector
tion screen is to use a one-dimensional imaging sensor. An
Referring to FIG. 10, one technique to detect the projec
input data line is median-?ltered 80 to remove noise since a
system described in R. Sukthankar et al., “Smarter Presen
typical inexpensive sensor captures data With signi?cant
tations: Exploiting Homography In Camera-Projector Sys
noise level. The median ?lter is preferred due to its property
tems,” IEEE ICCV Conference, 2001, described above, and
of preserving discontinuity. The ?ltered data is then passed
into a gradient computation module 82. It is noted that
determined that While the use of the single camera can be
used to estimate the parameters of the keystone e?fect,
hoWever, there is no guarantee that the projected image after
keystone correction maintains the correct aspect ratio. In
particular, the system needs to compute C. This is performed
by detecting the four screen comers in the image domain, the
four physical screen comers, and then solve for C. Unless
the physical screen happens to have the same aspect ratio as
the screen the system cannot obtain the correct aspect ratio.
To overcome such a limitation the present inventors deter
the detection is preferably performed in the gradient domain.
After the gradients are found, a Zero-crossing/peak locator
module 84 locates the Zero-crossings and peaks in the
gradient array. The Zero-crossing/peaks are presumably the
mined that interactivity With the user and the projection
system can overcome such aspect ratio limitations.
Referring to FIG. 9, the corners of the projection screen
may be identi?ed, such as the upper left comer, the upper
absolute luminance level is not the most reliable cue for
identifying a screen, since there may be luminance varia
tions due to shadoWs/shades as illustrated in FIG. 11. Thus
boundaries of the screen. Since typically there are multiple
Zero-crossings/peaks, and not all of them correspond to valid
screen boundaries, the candidate pair matcher module 86
matches tWo Zero-crossings/peaks to form a pair that is a
plausible candidate of screen. The match is based on mul
tiple criteria such as Width of the screen, average brightness
US 7,237,907 B2
In this implementation, the user ?rst enters the keystone
of the screen, etc. This module Will obtain multiple such
pairs. A statistical inference module 88 uses an inference
adjustment mode by selecting the corresponding option on
algorithm to choose the most plausible pair, based on
the on-screen display menu. The keystone adjustment pat
tern is then projected. Prior to adjustment, the user ?rst
empirical probabilities that re?ect the importance of each
selects one of the four comers that Will be adjusted. There
type of the cues used.
are many possible Ways of making this selection. For
example, the user makes a selection by pressing/clicking on
The basic principles of the above approach can be
extended to the 2-D sensor case Where the input array is
a button on the remote control, or on a soft button on the
2-dimensional. In fact, one can even use the above 1-D
on-screen display (henceforth the SELECT button). As the
approach to process the sensed 2-D array roW by roW, and
user presses the SELECT button, the four-arroW cluster
appears on one particular corner signaling the fact that the
corner is selected for adjustment. The user then selects any
one of the four directions by pressing on the four corre
sponding directional arroWs on the directional pad on the
then column by column. HoWever, a preferred approach Will
also utiliZe the additional 2-D constraints such as that all 1-D
screen boundaries in the horizontal direction should form
tWo lines in the vertical direction. Alternatively, With 2-D
data, one can start With edge detection, then perform line
detection, and then extract a quadrilateral that encloses an
remote control unit (FIG. 13). After the adjustment is
completed, When the user presses/clicks on the SELECT
button, the next corner is selected. This is signaled to the
user by the appearance of the four-arroW cluster at the next
area of relatively uniform color (White) and typically rela
tively brighter than the background. It is noted that the
projection screen may be identi?ed With one or more cam
selected corner. By pressing the SELECT button repeatedly,
After the detection of the projection screen an interactive
fashion. When a comer is selected, the four-arroW cluster
appears on that corner to provide visual con?rmation to the
user. The remaining three comers do not have the four-arroW
technique may be used to permit the user to further adjust the
aspect ratio, and/or keystone, if desired. These techniques
may likeWise be used to over-ride the automatic adjustments
and also provide proper operation When the range of opera
tion for the automatic methods is exceeded (e.g., in a very
interaction for ease of use. A “keystone adjustment pattern”
is projected Which the user interacts With via the laser
With the projector remote control or separate from the
tions on the projector. The user starts and stops intcractivc
imaging device in order to provide effective keystone cor
adjustment on the basis of visual feedback by observing the
dynamically changing shape of the keystone correction
With the projector. Also, after detection of the projection
screen the system may perform auto centering (positioning)
and/or auto Zooming.
rection. Accordingly, the system may use tWo or more
35 imaging devices, such as tWo or more cameras, or one or
Referring to FIG. 12 a rectangular pattern may be used for
interactive adjustment in the preferred embodiment. The
pattern Will appear as Warped rectangle 100 When the
keystone effect is present, and Will appear to be perfectly (or
substantially) rectangular When the keystone effect is cor
of the distance from the projector to the display surface a
stone adjustment mode by selecting the corresponding
arroW, the projection Will be adjusted toWards that direction.
correction mechanism, Which may be in the form of digital
In the preferred implementation, the user interacts With
the arroWs using the remote control unit of the projector, in
directions is depicted in FIG. 13.
In another embodiment the imaging may use the projector
optics (e.g., the projector lens) together With one or more
other imaging devices. The imaging devices may be main
tained in a ?xed location With respect to the projector, and
particular using four-directional (pointing at N, S, W, and E)
placed on a circular pad that can be pressed in 4 different
the system may place a constraint on the detected comers of
more robust.”
integrated into the projector remote control or otherWise be
navigation arroWs on a typical remote control, e.g., those on
a circular directional pad. Such an ensemble of four arroWs
depth-sensing device (e.g., a stereo camera), apart from the
projector optics, may be used. In order to reduce expense,
the depth sensing device may be a one-dimensional imaging
sensor pair, if desired (such as illustrated in FIG. 14). The
depth estimation may likeWise be used to increase the
accuracy of the projection screen detection module. Also,
the projection screen that they are co-planar, Which is
normally the case, in order to make the depth estimation
image pre-processing and/or optical/mechanical control of
the projector imaging system. The laser pointer may be
With the projector optics is acceptable, it is dif?cult to obtain
an accurate estimate of the distance from the projector to the
display surface. In order to obtain a more accurate estimate
option in an on-screen menu, and then uses the laser pointer
to adjust for keystone: Whenever the laser beam falls on an
The camera detects the position of the laser and invokes the
more cameras together With the projector’s optical system.
The pair of optical sensing devices may be used to estimate
the relative horiZontal deviation angle and vertical deviation
angle, typically using a projected test pattern, from Which
the keystone correction parameters may be estimated.
Depth to the projection surface is useful in estimating the
relative projection surface orientation. The relative angles
may be computed from the depth of the test patterns relative
to the projector. While the use of a single sensor coupled
rected. At each one of its comers, the pattern has a cluster of
4 arroWs, pointing north, south, West, and east directions. In
one possible implementation, the user ?rst enters the key
In many cases there is no projection screen or otherWise
a Wall With discernable boundries that is used to display the
image on. In this case, it is problematic to use a single
projector, or alternatively by selecting the appropriate func
pattern in response to user’s actions. In addition, after
detection of the projection screen the system may, if desired,
perform auto focus, auto centering (positioning), auto Zoom
ing, and auto keystoning, all Without further user interaction
A similar paradigm to the one described above can be
used for interactive aspect ratio adjustment. TWo arroWs (up
and doWn or left and right) placed on any appropriate part of
the adjustment pattern are used to stretch and squeeZe the
image to interactively adjust for the correct aspect ratio.
dark room). Such techniques should require minimal user
pointer (or otherWise a pointing device) that is integrated
the user can jump from one comer to the next in clockWise
are preferably integral thereWith. HoWever, it has been
determined that if the projector optics are adjusted in some
manner during use, then the system needs to be re-calibrated
for a neW set of parameters to compensate for the Zooming
and/or focusing and/or lens shifting. To overcome this
US 7,237,907 B2
limitation, the present inventors determined that the system
may pre-calibrate the optics of the projector so that the neW
parameters of the projector optics can be computed from the
lens control parameters, as the lens is moved during Zoom
ing and/or focusing and/or lens shifting. The pre-calibration
may include modeling the variations in the optical param
eters as a function of one or more of the lens control
parameters, so that the on-line re-calibration can be done
automatically. The calibration may include calculations
from pre-calibrations or otherWise a look-up table.
Referring to FIG. 15, after keystone correction the pro
jected image may be rectangular but it is not necessarily
properly oriented With respect to the user, Which typically
desires the top and bottom edges to be horizontal, and the
right and left edges to be vertical. In some embodiments the
detected projection screen, or parts thereof, may be used as
a basis to correct for projection roll so that the projector Will
be aligned With the real World, as illustrated in FIG. 16.
All references cited herein are hereby incorporated by
The terms and expressions Which have been employed in
the foregoing speci?cation are used therein as terms of
description and not of limitation, and there is no intention,
in the use of such terms and expressions, of excluding
equivalents of the features shoWn and described or portions
thereof, it being recogniZed that the scope of the invention
is de?ned and limited only by the claims Which folloW.
We claim:
1. A method for determining the location of a screen by a
projector, comprising:
(a) said projector receiving an image of a potential said
(b) said projector performing a median ?lter operation on
said image;
(c) said projector performing a gradient operation on said
(d) said projector performing a Zero-crossing operation on
said image to determine edge screen candidates;
(e) said projector matching pairs of said edge screen
(f) said projector using statistical inference to select said
matching pairs for said location of said screen.
2. The method of claim 1 Wherein steps (a), (b), (c), (d),
(e), and (f) are performed in the order of steps (a), (b), (c),
(d), (e), and (f)
3. The method of claim 1 Wherein said location is relative
to said projector.
4. The method of claim 1 Wherein said image is received
by an imaging device.
5. The method of claim 4 Wherein said imaging device
25 includes a one-dimensional sensor.
: 7,237,907 B2
Page 1 of 1
APPLICATION NO. : 11/642608
: July 3, 2007
: Baoxin Li et a1.
It is certified that error appears in the above-identi?ed patent and that said Letters Patent is
hereby corrected as shown below:
Column 6, Line 31 “boundries” should be --boundariesColumn 6, Line 59 delete end quote
Signed and Sealed this
Thirteenth Day of November, 2007
m W451i,”
Director ofthe United States Patent and Trademark O?ice
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF