Automatic focus system calibration for image capture systems

Automatic focus system calibration for image capture systems
US008416317B2
(12) Unlted States Patent
(10) Patent No.:
Palum
(54)
(75)
US 8,416,317 B2
(45) Date of Patent:
Apr. 9, 2013
AUTOMATIC FOCUS SYSTEM
6,637,896 B2
CALIBRATION FOR IMAGE CAPTURE
6,796,655 B2
9/2004 Sekl
SYSTEMS
6,854,474 B1
2/2005
Inventor:
Chen _
_
6,984,039 B2
1/2006 Ag0st1nell1
7,961,973 B2 *
6/2011
Russell J. Palum, Rochester, NY (US)
H
.......................... .. 382/274
- ung
(Cont1nued)
(73) Assignee: Apple Inc., Cupertino, CA (US)
( * ) Notice:
10/2003 Li et a1.
FOREIGN PATENT DOCUMENTS
1’???
Subject115522313330?
to any disclaimer,
digsusted
the term
under
of this
35
W0
W0 03/038517
é
5/2003
OTHER PUBLICATIONS
(21) Appl' NO': 12/503’095
(22) Filed:
Jul- 15, 2009
(65)
Light Blue Optics, Ltd., “PVProTM Enabling Personal Video Projec
tors ,WWW.l1ghtblue0pt1cs.c0m.
(Continued)
Prior Publication Data
Us 2009/0273680 A1
Nov. 5, 2009
pr’rf’ary Exam?” * Aung S Moe
Asszslanl Exammer * Mekonnen DagneW
Related US. Application Data
(63)
gtzqzoorgng’lg?igceféiegr
Continuation of application No. 11/496,913, ?led on
Aug. 1, 2006, noW Pat. No. 7,609,958.
(51) Int Cl
H04N ‘9/73
H04N 5/76
(2006 01)
(200601)
G03B 13/00
(200601)
'
_
(57)
* Wong’ Cabeno’ Lutsch’
ABSTRACT
Imaging systems and methods for calibrating imaging sys
P rov1‘de ‘1 . The 1ma
' g’1ng s y stem has a b o dy, a scene
tems are
image capture system that captures images using a taking lens
system that can be set to a plurality of different focus dis
/
_
/
(52)
(58)
US. Cl. ...... .... ...... 348/2231, 348 345, 348 229.1
FIeld of ClassI?catIon Search
348/223.1*225.1,
(56)
See apphcanon ?le for Complete Search hlstory'
References Cited
_ 348/255’ 229'1’ 20812’ 345’ 36_5’366’ 361
U.S. PATENT DOCUMENTS
tances, and a range?nder that is capable of determining a
distance between the imaging system and at least one portion
of a ?eld of View of the talking lens system_ The method
comprises: automatically capturing a ?rst calibration image
of a ?rst ?eld of vieW through the taking lens system With the
taking lens system set to a ?rst focus distance setting; identi
fying a portion of the ?rst calibration image having a prede
termined degree of focus; using the range?nder to determine
a ?rst calibration distance from the imaging device to the
2
gan?tet 11' l
identi?ed portion. A focus correlation is determined based
5’440’369 A
8/1995 Tglgataaei j '
upon the ?rst calibration distance and the ?rst focus distance
5,715,483 A
5,877,809 A
2/1998 Omata et a1.
3/1999 Omata et a1.
Semng
6,067,114 A
5/2000 Omata et a1.
7 Claims, 13 Drawing Sheets
34
27
747
76
AUDIO
22\
23
25
DRIVER
IMA GE
SENSOR
CIRCUITRY
a2
CONTROLLER
26
\
SIGNAL
54
MEMORY
COMMUNICATION
IN TERFA CE
MODULE
24
REMO VA BLE
MEMORY
52
REMOTE
MEMORY
US 8,416,317 B2
Page 2
U.S. PATENT DOCUMENTS
2002/0118151
2002/0159101
2002/0176015
2003/0081184
2005/0024606
2005/0168705
2006/0061676
A1
8/2002 Chen
A1 *
10/2002 Alderson et a1. ............ .. 358/504
A1
A1
A1
A1
A1
11/2002
5/2003
2/2005
8/2005
3/2006
Lichtfuss
Li et al.
Li et al.
Li et al.
Suda
OTHER PUBLICATIONS
Jacqueline Hewett, “Tiny Image Projectors Take a Step Towards the
Shops”, Jul./Aug. 2003, www.optics.org.
Anne Eisenberg, “What’s Next; For Your Viewing Pleasure, a Pro
jector in Your Pocket”, Nov. 4, 2004, www.nytimes.com.
* cited by examiner
US. Patent
Apr. 9, 2013
Sheet 2 0f 13
US 8,416,317 B2
US. Patent
Apr. 9, 2013
Sheet 3 0f 13
US 8,416,317 B2
@
80 *1
ACTIVATE CAMERA
32%
CAPTURE CALIBRATION
IMAGE
I
I
IDENTIFYA PORTION OF THE
34% CALIBRATION IMAGE THAT
HAS A PREFERRED LEVEL OF
FOCUS
I
USE RANGEFINDER TO
MEASURE CALIBRATION
35% DISTANCE FROM THE IMAGE
CAPTURE SYSTEM TO THE
IDENTIFIED PORTION
I
DETERMINE A FOCUS
CORRELATION DA TA SET
88 ,h BASED UPON CALIBRATION
DISTANCE AND THE FOCUS
DISTANCE SETTING USED TO
CAPTURE THE FIRST IMAGE
I
STORE INDICATION OF
90 ’\/
DETERMINED FOCUS
CORRELATION
94 x
USE RANGEFINDER TO
DE TEC T CAPTURE DISTANCE
I
96“
CAPTURE IMAGE USING
INDICATED FOCUS
98
YES
CORRELATION
R
mg’?és
AND CAPTURE DISTANCE TO
DETERMINE FOCUS SETTING
?
FIG. 4
US. Patent
Apr. 9, 2013
Sheet 4 0f 13
US 8,416,317 B2
M3
M
£2
Q
1.1.6
m
/\
M
FIG. 5
US. Patent
Apr. 9, 2013
Sheet 6 0f 13
US 8,416,317 B2
68E
120 ’\/
ACTIVATE CAMERA
122
GO TO
STEP 82
124 A,
USE THROUGH-THE-LENS
FOCUS TO SET CAMERA
FOCUS POSITION FOR AN
IDENTIFIED PORTION OF THE
IMAGE
I
126 “1
CAPTURE ARCHIVAL IMAGE
I
USE RANGEFINDER TO
MEASURE DISTANCE FROM
128 “A
THE IMAGE CAPTURE
SYSTEM To THE IDENTIFIED
PORTION OF THE IMAGE
I
DETERMINE A FOCUS
CORRELATION DATA SET BASED
130 AV UPON CALIBRATION DISTANCE
AND THE FOCUS DISTANCE
SETTING USED TO CAPTURE THE
ARCHIVAL IMAGE
I
STORE INDI CA TI ON OF
132 ’\_»
DETERMINED FOCUS
CORRELA TION
134
136 ,\I
USE RANGEFINDER TO
DE TEC T CAPTURE DISTANCE
*
CAPTURE IMAGE USING
INDICA TED FOCUS
‘'38 “P
140
YES
CORRELATION
$195515
AND CAPTURE DISTANCE TO
9
DETERMINE FOCUS SETTING
'
FIG. 9
US. Patent
Apr. 9, 2013
US 8,416,317 B2
Sheet 7 0f 13
(START)
1
80m
ACTIVATE CAMERA
I
82
CAPTURE CALIBRATION
IMAGE
I
IDENTIFYA PORTION OF THE
84% CALIBRATION IMAGE THAT
HAS A PREFERRED LEVEL OF
FOCUS
I
USE RANGEFINDER TO
MEASURE CALIBRATION
86x DISTANCE FROM THE IMAGE
CAPTURE SYSTEM TO THE
IDENTIFIED PORTION
I
DETERMINEA FOCUS
CORRELA TION DA TA SET
88 "w BASED UPON CALIBRATION
DISTANCE AND THE FOCUS
DISTANCE SETTING USED TO
CAPTURE THE FIRST IMAGE
STORE INDI CA TI ON OF
90%
98
DETERMINED FOCUS
C ORRELA TI ON
MORE
PICTURES
?
YES
92
CONDITION
ETECTED
9
97
YES
SUBJECT
PORTION OF
IMA GE IN
FOCUS
DE TEC T OPERA TING CONDITIONS
9
93
YES
OPERATING
CONDITIONS
96x
RANGEFINDER TO
94x DEUSE
TECT CAPTURE DISTANCE
FIG. 10
CAPTURE IMAGE USING
INDICATED FOCUS
CORRELATION
AND CAPTURE DISTANCE TO
DETERMINE FOCUS SETTING
US. Patent
150 ’\J
Apr. 9, 2013
Sheet 8 0f 13
US 8,416,317 B2
@
ACTIVATE CAMERA
GO TO
STEP 82
USE RANGEFINDER TO
MEASURE DISTANCE FROM
154 ’\J
THE IMAGE CAPTURE
SYSTEM TO THE SUBJECT
AREA PORTION
I
156 Av
758 ' \-
USE PRE-PROGRAMMED
FOCUS CORRELATION TO
DETERMINE FOCUS
DISTANCE
I
I
DETERMINE A FOCUS
CAPTURE ARCHIVAL IMAGE
CORRELATION DATA SET BASED
160 A“ UPON CALIBRATION DISTANCE
AND THE FOCUS DISTANCE
SETTING USED TO CAPTURE THE
ARCHIVAL IMAGE
I
STORE INDICA TION OF
162 m
DETERMINED FOCUS
C ORRELA TI ON
CONDITION
I ETECTED
166 ,\_I
USE RANGEFINDER TO
DE TEC T CAPTURE DISTANCE
i
CAPTURE IMAGE USING
INDICA TED FOCUS
168 “P
CORRELATION
AND CAPTURE DISTANCE TO
DETERMINE FOCUS SETTING
1 70
YE
S
MORE
PI C TgJRES
FIG. 11
US. Patent
Apr. 9, 2013
US 8,416,317 B2
Sheet 9 0f 13
@
.1 80
A C TI VATE CAMERA
182
CAPTUR
CONDITION
E TECTED
GO TO
STEP 82
184m DETECT OPERATING CONDITIOMS)
186%
SELECT FOCUS CORRELATION BASED
ON OPERATING CONDITIONS
I
USE RANGEFINDER TO MEASURE
DISTANCE FROM THE IMAGE
CAPTURE SYSTEM TO THE
SUBJECTAREA PORTION
190'»
CAPTURE ARCHIVAL IMAGE USING
RANGEFINDING DIS TANCE AND
DETERMINED FOCUS CORRELATION
i
USE THROUGH-THE-LENS FOCUS
192% TO DETERMINE SETTINGS FOR
CAPTURING A CALIBRATION IMAGE
i
DETERMINE A FOCUS CORRELATION
DATA SET BASED UPON CALIBRATION
194'» DISTANCE AND THE FOCUS DISTANCE
SETTING USED TO CAPTURE THE
ARCHIVAL IMAGE
i
STORE INDICATION OF
196 ’\I DETERMINED FOCUS CORRELATION
198
CAPTUR
CONDITION
E TEC TED
200 '\,
USE RANGEFINDER TO
DE TEC T CAPTURE DISTANCE
I
CAPTURE IMAGE USING
INDICATED FOCUS
CORRELATION
AND CAPTURE DISTANCE TO
DETERMINE FOCUS SETTING
204
YES
MORE
PICTURES
?
FIG. 12
US. Patent
Apr. 9, 2013
Sheet 11 0113
US 8,416,317 B2
START DETERMINING FOCUS
CORRELATION STEP
DOES
MEASURED
212
7
SELECT FITTING
FOCUS
CORRELATION
214 7
EVALUA TE CALIBRATION IMAGE
TO IDENTIFY EA CH AREA HAVING
_"'_'_"* A FOCUS THATAT OR ABOvEA
PREDETERMINED FOCUS LEVEL
224
/
CAPTURE
ADDITIONAL
CALIBRATION
IMAGE
222
7 I
MOVE LENS SYSTEMA
TAKING LENS POSITION
NOT PREVIOUSLY UsED
TO CAPTUREA
CALIBRATION IMAGE+-
216
7
l
MEASURE DISTANCE TO EACH
IDENTIFIED AREA USING
RA NGEFINDING DATA
218
7DETERMINEI RANGE OF
DISTANCES AND ASSOCIATE
RANGE OF DIsTANCEs WITH
LENS SETTINGS UsED TO
CAPTURE CALIBRATION IMAGE.
DETERMINE FOCUS
CORRELATION USING MEASURED
DISTANCES AND FOCUS
DISTANCE SETTINGS
I
END DETERMINING FOCUS
CORRELATION STEP
FIG. 16
US. Patent
Apr. 9, 2013
Sheet 13 0f 13
310%
PROJECT
IMAGE
I
312%
CA PTURE IMA GE OF
PROJE C TED IMA GE
I
314%
CALCULATE
DIFFERENCE MAP
I
316%
STORE DIFFERENCE
MAP FOR USE
I
318%
DETERMINE
RANGEFINDING
DISTANCES
I
DE TECT VARIATIONS
IN RANGEFINDING
DISTANCES
I
DETERMINE IMAGE
322 ’\J ADJUSTMENTS BASED
UPON RANGEFINDING
DISTANCES
END
FIG. 18
US 8,416,317 B2
US 8,416,317 B2
1
2
AUTOMATIC FOCUS SYSTEM
CALIBRATION FOR IMAGE CAPTURE
SYSTEMS
tured through tWo lens systems that are separated laterally by
a distance and determine the distance to the scene through
triangulation. The focus setting of the adjustable focus lens
system is then determined using a calibrated preprogrammed
curve or look-up table that correlates scene distances With
CROSS-REFERENCE TO RELATED
APPLICATIONS
lens positions that can be used to capture objects at the scene
distance in focus. A Wide variety of range?nder type autofo
cus systems are very Well knoWn in the art.
This Application is a continuation of US. Ser. No. 11/496,
913, ?led Aug. 1, 2006, now US. Pat. No. 7,609,958 issued
Oct. 27, 2009.
Range?nder type autofocus systems have the advantage of
being very fast With some having a response time that can be
in the range of 0.01 -0.05 second. HoWever, the focus quality
produced by some range?nder type autofocus systems can
vary When they are used in different operating conditions. For
example, temperature and humidity can affect the calibration
This application is co-?led With and has related subject
matter to
US.
Patent Application Publication No.
20090273681 published Nov. 5, 2009.
of the distance to focus lens position curve due to ?uctuations
FIELD OF THE INVENTION
in the refractive index and dimensions of both the range?nder
autofocus system components and the taking lens compo
The invention relates to automatic calibration of imaging
systems to improve image quality and enable faster operation.
BACKGROUND OF THE INVENTION
nents.
20
With the lens system positioned at a plurality of different
focus distances. For example, in a contrast based “through
the-lens” autofocus system a plurality of different images
Imaging systems are limited in terms of image quality by
artifacts introduced by the environment that they are operated
in. One Way to avoid capturing or creating images that have
such artifacts is to calibrate the imaging system in an envi
The “through-the-lens” autofocus system determines
focus settings using analysis of a series of images captured
(e.g. 5-20) are captured With the taking lens in different focus
25
lens positions in a so-called hill climb method. The contrast
ronment that is similar to the environment in Which it Will be
present in the captured images is compared and the image
used. Often this is done When the imaging device is manufac
With the greatest contrast is determined to be the image With
the best focus conditions (often the best focus lens position is
further re?ned by interpolating the contrast values betWeen
tured. For example, it is knoWn in the art to use test ?xtures to
calibrate autofocus systems in ?lm cameras While such cam
eras are Within an operating range of environmental condi
30
images). The “through-the-lens” type autofocus system is
tions.
very accurate since it measures focus quality directly from
In particular, one aspect of an imaging system that bene?ts
from calibration is the autofocus system in an imaging sys
images captured With the high quality talking lens.
tem. Many ?lm cameras, digital cameras and scanners cap
systems can be relatively sloW in determining a focus setting.
ture images using an imager and a lens system With an adjust
able focus lens system. Typically, the focus distance of such
an adjustable focus lens system can automatically be set to
one of a plurality of different settings by sensing, control and
drive systems that are adapted to provide optimal focus of
HoWever, conventional “through-the-lens” type autofocus
35
For example, such systems can take as long as 0.5-2.0 seconds
to determine a focus distance. This is because such “through
the-lens” autofocus systems require the capture and process
ing of a number of images.
Accordingly, in some digital cameras, the tWo types of
What is determined to be a subject area in a scene. Lens 40 autofocus systems are used together in a hybrid system in
systems that have automatically adjustable focus settings are
Which the range?nder type autofocus system is used to pro
referred to herein as autofocus systems.
vide a fast estimation of a focus setting that is then folloWed
It Will be appreciated that it is important to properly cali
brate such autofocus systems. In the above example, focus
settings for ?lm cameras are calibrated by using the test
?xture to monitor an image provided by the lens system of
by the use of the “Through-the-lens” autofocus system to
re?ne the focus setting. For example, US. Pat. No. 6,864,474
entitled “Focusing Apparatus forAdjusting Focus of an Opti
cal Instrument”, ?led by MisaWa on Jan. 10, 2003, describes
45
such a ?lm camera and adjusting the lens system until the lens
system reaches a ?rst setting Where a test target located at a
the coordinated use of a range?nder type autofocus system
With a through-the-lens autofocus system. In MisaWa, the
?rst distance from the camera is in focus. The range?nder for
focus position of the taking lens is determined by both the
range?nder based autofocus system and the “through-the
the ?lm camera is then used to measure the distance to the test 50
target and thereafter the range?nder Will position the lens
lens” autofocus system, the difference betWeen the focus
system at the ?rst setting Whenever the range?nder measures
that distance. This process is then repeated for a plurality of
position determined by the range?nder type autofocus system
and the focus position determined by the “through-the-lens”
other test targets, each located at one of a range of additional
distances so that the range?nding measurements are associ
type autofocus system is stored for future reference. In sub
55
sequent image capture episodes, the stored difference infor
ated With each of a plurality or lens focus settings.
Digital cameras typically use one of tWo types of autofocus
mation is used to re?ne the number of images captured and
systems: range?nder type autofocus systems or a “through
the-lens” type autofocus system to automatically determine
the hill climb method to determine the focus lens position
taking lens focus settings. A range?nder autofocus system
analyZed by the “through-the-lens” type autofocus system in
With best focus, thereby reducing the number of images cap
60
tured and processed When the range?nder has been accurate
uses sensors such as optical range?nders or sonic range?nd
and increasing the number of images captured and processed
ers to determine a distance from a camera to one or more
When the range?nder has been inaccurate. HoWever, the
method described by MisaWa assumes that the performance
of the range?nder, adjustable focus lens system, and control
portions of a scene Within a ?eld of vieW of the adjustable lens
system. Common range?nder type autofocus systems include
active and passive systems. In one example of an active
range?nder type system, the range?nder type autofocus sys
system are consistent over time, do not ?uctuate With varia
tions in environmental conditions and do not otherWise
tem compares tWo loW-resolution images that have been cap
change or drift over time.
65
US 8,416,317 B2
4
3
MisaWa also does not eliminate the use of multiple image
FIG. 4 shoWs a block ?oW diagram of one embodiment of
capture and processing used by the “through-the-lens” type
an auto-calibrating range?nder-based autofocus system;
autofocus system so that the hybrid autofocus as described by
MisaWa remains sloW. A further aspect of an imaging system
that Would bene?t from calibration is a projection system in
order to ensure that a projection lens system is properly
focused. There have been efforts to provide automatic feed
back systems to this end. For example, US. Patent Applica
tion Publications US2005/0168705 and US2005/0024606
FIG.
FIG.
FIG.
FIG.
FIG.
5 illustrates one example of a calibration image;
6 illustrates one example of a focus correlation;
7 illustrates one example of a focus correlation;
8 illustrates one example of a focus correlation;
9 shoWs another embodiment of a method for cali
brating an imaging device;
FIG. 10 shoWs another embodiment of a method for cali
both by Li et al., describe projection systems Which include
feedback of a projected image by an imaging sensor system.
brating an imaging device;
In this case Li et al. teaches the use of the imaging sensor
brating an imaging device;
FIG. 11 shoWs another embodiment of a method for cali
FIG. 12 shoWs another embodiment of a method for cali
system to aid in focusing the projector. Li et al. also teaches
brating an imaging device;
the use of an imaging sensor system to enable the projection
FIG.
FIG.
FIG.
FIG.
system to correct for projector-to-surface orientation prob
lems, such as correcting to adjust for keystone in the projected
image, or to ?t the projected image Within the edge of a
projection screen. Thus, Li et al. discloses, essentially, a
“Through-the-lens” focus system With orientation compen
sation. HoWever, here again calibration of such a system is
13 shoWs one example of a focus correlation;
14 shoWs one example of a focus correlation;
15 shoWs one example of a focus correlation;
16 shoWs another embodiment of a method for deter
mining a focus correlation;
20
FIG. 17 shoWs an embodiment of an imaging device With
an associated projection system; and
FIG. 18 shoWs a method for calibrating the projection
typically performed only during manufacturing or during a
manual service procedure.
Therefore the need persists to improve imaging systems
system.
through neW calibration approaches.
25
DETAILED DESCRIPTION OF THE INVENTION
SUMMARY OF THE INVENTION
Before explaining the present embodiments in detail, it is
Image capture systems and methods for calibrating an
imaging system are provided. In one aspect of the invention,
the imaging system has a body, a scene image capture system
that captures images using a taking lens system that can be set
to a plurality of different focus distances, and a range?nder
that is capable of determining a distance betWeen the imaging
to be understood that the embodiments are not limited to the
particular descriptions and that it can be practiced or carried
30
system and at least one portion of a ?eld of vieW of the taking
lens system. The method comprises the steps of: automati
cally capturing a ?rst calibration image of a ?rst ?eld of vieW
through the taking lens system With the taking lens system set
35
image capture system 22 and autofocus system 27 are
mounted. Scene image capture system 22 having a taking lens
?rst calibration image having a predetermined degree of
system 23, a scene image sensor 24, a signal processor 26, an
40
system using the range?nder, and setting the focus distance
setting for the taking lens system for the capture of the archi
val image based upon the determined focus correlation and
the determined capture distance.
have one or more elements.
45
50
stationary element or elements (not shoWn) by lens driver 25
that is motoriZed for automatic movement. Lens driver 25
55
controls both the lens focal length and the lens focus position
of taking lens system 23 and sets a lens focal length and/or
position based upon signals from signal processor 26, an
optional automatic range ?nder system 27, and/or controller
32. It Will be appreciated that in other embodiments, taking
lens system 23 can comprise lenses having shapes that can be
changed to provide an in situ variation of lens focus distance
by modi?cation of lens characteristics such as curvature.
60
FIG. 1 shoWs a block diagram of one embodiment of an
Examples of lens systems of this type include those that use,
for example, liquid lens technology knoWn to those of skill in
the art. In such embodiments, lens driver 25 can provide
signals to cause focus distance changes in the lens Without use
of a motor. Examples of such liquid lenses include lenses sold
image capture system;
FIG. 2 shoWs a back, elevation vieW of the image capture
system of FIG. 1;
FIG. 3 shoWs a front, elevation vieW of the image capture
system of FIG. 1;
Taking lens system 23 is of an automatically adjustable
type. In the embodiment shoWn in FIGS. 1-3, taking lens
system 23 is automatically adjusted to a variety of focus
settings. In the example embodiment shoWn in FIG. 1, taking
lens system 23 is a 6x Zoom lens unit in Which a mobile
element or elements (not shoWn) are driven, relative to a
BRIEF DESCRIPTION OF THE DRAWINGS
In the detailed description of the preferred embodiments
presented beloW, reference is made to the accompanying
draWings, in Which:
optional display driver 28 and a display 30. In operation, light
from a scene is focused by taking lens system 23 to form an
image on scene image sensor 24. Taking lens system 23 can
the scene image capture system is to be used to capture an
archival image of a scene and, in response thereto, performing
the steps of: determining a capture distance from the imaging
system to a portion of the ?eld of vieW of the taking lens
imaging system 10. FIG. 2 shoWs a back, elevation vieW of the
imaging system 10 of FIG. 1, While FIG. 3 shoWs a front
elevation vieW of the imaging system 10 of FIG. 1. As is
shoWn in FIGS. 1-3, imaging system 10 takes the form of a
digital camera 12 comprising a body 20 to Which a scene
to a ?rst focus distance setting; identifying a portion of the
focus; using the range?nder to determine a ?rst calibration
distance from the imaging device to the identi?ed portion of
the ?rst calibration image; determining a focus correlation
based upon the ?rst calibration distance and the ?rst focus
distance setting, said focus correlation associating different
range?nder determined distances With each of the plurality of
focus distance settings With at least one range?nder deter
mined distance; detecting a capture condition indicating that
out in various Ways.
FIG. 1 shoWs a block diagram of an embodiment of an
65
commercially under the name of Fluidfocus by Royal Philips
Electronics, Amsterdam, The Netherlands and other liquid
lens products sold by PGS Precision, Singapore.
US 8,416,317 B2
5
6
In the embodiment of FIG. 1, the focus position of taking
lens system 23 can be automatically selected by a variety of
knoWn strategies. For example, in one embodiment, scene
image sensor 24 is used to provide multi-spot autofocus using
What is called the “Through focus” or “Whole Way scanning”
approach. As described in commonly assigned US. Pat. No.
5,877,809 entitled “Method Of Automatic Object Detection
In An Image”, ?led by Omata et al. on Oct. 15, 1996, the
disclosure of Which is herein incorporated by reference. If the
Signal processor 26 receives image signals from scene
image sensor 24 and transforms the image signals into a
digital image in the form of digital data. The image can
comprise one or more still images, multiple still images and/
or a stream of apparently moving images such as a video
segment. Where the digital image data comprises a stream of
apparently moving images, the digital image data can com
prise image data stored in an interleaved or interlaced image
form, a sequence of still images, and/ or other forms knoWn to
those of skill in the art of digital video.
target object is moving, object tracking may be performed, as
described in commonly assigned US. Pat. No. 6,067,114
Signal processor 26 can apply various image processing
algorithms to the image signals When forming a digital image.
entitled “Detecting Compositional Change in Image” ?led by
These can include but are not limited to color and exposure
Omata et al. on Oct. 26, 1996, the disclosure of Which is
herein incorporated by reference. In an alternative embodi
balancing, interpolation and compression. Where the image
signals are in the form of analog signals, signal processor 26
also converts these analog signals into a digital form. In
certain embodiments, signal processor 26 can be adapted to
ment, the focus values determined by “Whole Way scanning”
are used to set a rough focus position, Which is re?ned using
a ?ne focus mode, as described in commonly assigned US.
process image signals so that the digital image formed
Pat. No. 5,715,483, entitled “Automatic Focusing Apparatus
thereby appears to have been captured at a different Zoom
and Method”, ?led by Omata et al. on Oct. 11, 1998, the
disclosure of Which is herein incorporated by reference.
20
Digital camera 12 has a range?nder 27. Range?nder 27 is
adapted to determine a distance from imaging system 20 to at
least a portion of a ?eld of vieW of taking lens system 23.
Range?nder 27 can determine the distance as an absolute
distance measurement, such as a measurement in feet or
meters, or as a relative distance measurement to the focus
scene image sensor 24 and interpolating the subset of the
image signals to form the digital image. This is knoWn gen
erally in the art as “digital Zoom”. Such digital Zoom can be
25
used to provide electronically controllable Zoom adjusted in
?xed focus, manual focus, and even automatically adjustable
focus systems.
Controller 32 controls the operation of the imaging system
30
scene image capture device 22, display 30 and memory such
setting for taking lens system 23 that is appropriate for the
distance to the subject Without use of it. Range?nder 27 can
operate lens driver 25, directly or as shoWn in FIG. 1, can
10 during imaging operations including, but not limited to,
provide signals to signal processor 26 or controller 32 from
Which signal processor 26 or controller 32 can generate sig
nals that are to be used for image capture. A Wide variety of
suitable multiple sensor range?nders 27 knoWn to those of
skill in the art are suitable for use. For example, US. Pat. No.
as memory 40. Controller 32 causes scene image sensor 24,
signal processor 26, display 30 and memory 40 to capture,
present and store scene images in response to signals received
from a user input system 34, data from signal processor 26
35
5,440,369 entitled “Compact Camera With Automatic Focal
al. on Nov. 30, 1993, the disclosure of Which is herein incor
40
imaging system 10.
45
Controller 32 cooperates With a user input system 34 to
alloW imaging system 10 to interact With a user. User input
system 34 can comprise any form of transducer or other
device capable of receiving an input from a user and convert
ing this input into a form that can be used by controller 32 in
the scene is divided into a grid of areas or spots, and the
optimum focus distance is determined for each spot. One of
the spots is identi?ed as the subject of the image and the focus
distance for that spot is used to set the focus of taking lens
system 23.
A feedback loop is established betWeen lens driver 25 and
controller 32 and/or range?nder 27 so that the focus position
of taking lens system 23 can be rapidly set.
operating imaging system 10. For example, user input system
34 can comprise a touch screen input, a touch pad input, a
4-Way sWitch, a 6-Way sWitch, an 8-Way sWitch, a stylus
50
Taking lens system 23 is also optionally adjustable to pro
vide a variable Zoom. In the embodiment shoWn lens driver 25
automatically adjusts the position of one or more mobile
elements (not shoWn) relative to one or more stationary ele
ments (not shoWn) of taking lens system 23 based upon sig
nals from signal processor 26, an automatic range?nder 27,
55
and/ or controller 32 to provide a Zoom magni?cation. Taking
lens system 23 can be of a ?xed Zoom setting, manually
system, a trackball system, a joystick system, a voice recog
nition system, a gesture recognition system or other such
systems. In the digital camera 12 embodiment of imaging
system 10 shoWn in FIGS. 1 and 2 user input system 34
includes a capture button 60 that sends a capture signal to
controller 32 indicating a desire to capture an archival image.
User input system 34 can also include other buttons including
the joystick 66, mode button 67, and the select it button 68
shoWn in FIG. 2.
Sensors 36 are optional and can include light sensors and
other sensors knoWn in the art that can be used to detect
adjustable and/or can employ other knoWn arrangements for
providing an adjustable Zoom.
and data received from optional sensors 36. Controller 32 can
comprise a microprocessor such as a programmable general
purpose microprocessor, a dedicated micro-processor or
micro-controller, a combination of discrete components or
any other system that can be used to control operation of
Length Dependent Exposure Adjustments” ?led by Tabata et
porated by reference, discloses one such range?nder 27. The
focus determination provided by range?nder 27 can be of the
single-spot or multi-spot type. Preferably, the focus determi
nation uses multiple spots. In multi-spot focus determination,
setting than that actually provided by the optical lens system.
This can be done by using a subset of the image signals from
60
conditions in the environment surrounding imaging system
Light from the scene that is focused by taking lens system
10 and to convert this information into a form that can be used
23 onto scene image sensor 24 is converted into image signals
representing an image of the scene. Scene image sensor 24
by controller 32 in governing operation of imaging system 10.
Sensors 36 can include audio sensors adapted to capture
sounds. Such audio sensors can be of conventional design or
can comprise a charge couple device (CCD), a complimen
tary metal oxide sensor (CMOS), or any other electronic
image sensor knoWn to those of ordinary skill in the art. The
image signals can be in digital or analog form.
65
can be capable of providing controllably focused audio cap
ture such as the audio Zoom system described in US. Pat. No.
4,862,278, entitled “Video Camera Microphone With Zoom
US 8,416,317 B2
7
8
Variable Acoustic Focus”, ?led by Dann et al. on Oct. 14,
1986. Sensors 36 can also include biometric sensors adapted
to detect characteristics of a user for security and affective
as the MPEG-4 (Motion Pictures Export Group) or Apple
QuickTimeTM standard can be used to store digital image data
in a video form. Other image compression and storage forms
imaging purposes. Sensors 36 can also include temperature
can be used.
sensors or humidity sensors to detect the environmental con
The digital images and metadata can be stored in a memory
ditions surrounding the imaging system 10. Where a need for
such as memory 40. Memory 40 can include conventional
additional scene illumination is determined, controller 32 can
memory devices including solid state, magnetic, optical or
cause an optional source of arti?cial illumination 37 such as
other data storage devices. Memory 40 can be ?xed Within
imaging system 10 or it can be removable. In the embodiment
of FIG. 1, imaging system 10 is shoWn having a memory card
a light, strobe, or ?ash system to emit light.
Controller 32 causes an image signal and corresponding
digital image to be formed When a capture condition is
slot 46 that holds a removable memory 48 such as a remov
detected indicating a desire of a user to capture an archival
able memory card and has a removable memory interface 50
image. Typically, the capture condition occurs When a user
for communicating With removable memory 48. The digital
depresses capture button 60, hoWever, controller 32 can deter
images and metadata can also be stored in a remote memory
system 52 that is external to imaging system 10 such as a
mine that a capture condition exists at a particular time, or at
a particular time after capture button 60 is depressed. Alter
personal computer, computer netWork or other imaging sys
natively, controller 32 can determine that a capture condition
exists When optional sensors 36 detect certain environmental
tem.
In the embodiment shoWn in FIGS. 1 and 2, imaging sys
conditions, such as optical or radio frequency signals. Fur
ther, controller 32 can determine that a capture condition
exists based upon affective signals obtained from sensors 36
20
that are adapted to sense the physiology of a user.
Controller 32 can also be used to generate metadata in
association With each image. Metadata is data that is related to
a digital image or a portion of a digital image but that is not
example, an optical, radio frequency or other Wireless circuit
25
necessarily observable in the image itself. In this regard,
controller 32 can receive signals from signal processor 26,
camera user input system 34 and other sensors 36 and, option
ally, generate metadata based upon such signals. The meta
data can include, but is not limited to, information such as the
device. Controller 32 can also receive information and
instructions from signals received by communication module
type of scene image sensor 24, mode setting information,
54 including but not limited to, signals from a remote control
device (not shoWn) such as a remote trigger button (not
integration time information, taking lens system 23 setting
information that characterizes the process used to capture the
imaging system 10 to form the scene image. The metadata can
also include but is not limited to any other information deter
mined by controller 32 or stored in any memory in imaging
system 10 such as information that identi?es imaging system
10, and/ or instructions for rendering or otherWise processing
or transducer that converts image and other data into a form,
such as an optical signal, radio frequency signal or other form
of signal, that can be conveyed to an external device. Com
munication module 54 can also be used to receive a digital
image and other information from a host computer, netWork
(not shoWn), or other digital image capture or image storage
30
time, date and location that the scene image Was captured, the
scene image and processes, methods and algorithms used by
tem 10 has a communication module 54 for communicating
With external devices such as, for example, remote memory
system 52. The communication module 54 can be for
35
shoWn) and can operate imaging system 10 in accordance
With such signals.
Signal processor 26 and/or controller 32 also use image
signals or the digital images to form evaluation images Which
have an appearance that correspond to scene images stored in
40
imaging system 10 and are adapted for presentation on dis
the digital image With Which the metadata is associated. The
play 30. This alloWs users of imaging system 10 to use a
metadata can also comprise an instruction to incorporate a
display such as display 30 to vieW images that correspond to
scene images that are available in imaging system 10. Such
images can include images that have been captured by scene
particular message into a digital image When the digital image
is presented. Such a message can be a text message that is
intended to be shoWn or rendered When the digital image is
presented or rendered. The metadata can also include audio
input capture device and/ or that Were otherWise obtained such
as by Way of communication module 54 and stored in a
signals. The metadata can further include digital image data.
memory such as memory 40 or removable memory 48.
45
Display 30 can comprise, for example, a color liquid crys
In one embodiment of the invention, Where digital Zoom is
used to form the image from a subset of the captured image,
the metadata can include image data from portions of an
image that are not incorporated into the subset of the digital
image that is used to form the digital image. The metadata can
also include any other information entered into imaging sys
tem 10, sensed by imaging system 10 or determined in Whole
or in part by imaging system 10.
tal display (LCD), organic light emitting display (OLED) also
50
knoWn as an organic electro-luminescent display (OELD) or
other type of video display. Display 30 can be external as is
shoWn in FIG. 2, or it can be internal for example used in a
vieW?nder system 38. Alternatively, imaging system 10 can
have more than one display 30 With, for example, one being
55
external and one internal.
A captured digital image and optional metadata, can be
Signal processor 26 and/or controller 32 can also cooperate
stored as an archival image or used for other purposes as
to generate other images such as text, graphics, icons and
other information for presentation on display 30. This can
described herein. A digital image can be stored, for example,
in a compressed form. For example, Where the digital image
comprises a sequence of still images, the still images can be
stored in a compressed form such as by using the JPEG (Joint
alloW interactive communication betWeen controller 32 and a
60
mation to the user of imaging system 10 and the user of
imaging system 10 using user input system 34 to interactively
provide information to imaging system 10. Imaging system
Photographic Experts Group) ISO 10918-1 (ITU-T.81) stan
dard. This JPEG compressed image data is stored using the
so-called “Exif” image format de?ned in the Exchangeable
Image File Format version 2.2 published by the Japan Elec
tronics and Information Technology Industries Association
JEITA CP-3451. Similarly, other compression systems such
user of imaging system 10, With display 30 providing infor
10 can also have other displays such as a segmented LCD or
65
LED display (not shoWn) Which can also permit signal pro
cessor 26 and/or controller 32 to provide information to user.
This capability is used for a variety of purposes such as
US 8,416,317 B2
10
establishing modes of operation, entering control settings,
higher degrees of focus can be located by identifying portions
user preferences, and providing Warnings and instructions to
in the stored image that have a greater amount of high fre
quency data Which in turn is indicative of the level of focus.
a user of imaging system 10.
In the embodiments of FIGS. 1 and 2, imaging system 10
has an optional audio system 70 having an input transducer in
the form of a microphone 72 that receives sonic energy and
generates signals that are provided to audio processing cir
Typically the various portions 102-118 of calibration
image 100 Will depict scene elements With greater and lesser
degrees of focus and thus controller 32 or signal processor 26
Will identify one area (eg area 116) as having the greatest
degree of focus. It Will be appreciated that in other embodi
ments, controller 32 can be adapted to simply identify the ?rst
cuitry 74. Audio processing circuitry 74 is adapted to convert
the signals received from microphone 72 into an electronic
audio signal representing the pattern of sonic energy incident
upon the transducer. Audio processing circuitry 74 is further
portion that has a level of focus that is above a threshold and
to select that portion. Other approaches can also be used.
Range?nder 27 is used to measure a calibration focus dis
tance from the digital camera 12 to selected portion 116 using
adapted to receive signals from controller 32 and to cause
speaker 76 to generate audible sounds. Other systems such as
knoWn circuits, lights and actuators for generating visual
signals, audio signals, vibrations, haptic feedback and other
for example multi-spot range ?nding to measure the distance
forms of signals can also be incorporated into imaging system
10 for use in providing information, feedback and Warnings to
associated With the “spot” (step 86). In one embodiment,
the user of imaging system 10.
controller 32 and/or signal processor 26 divide calibration
from digital camera 12 to a portion of the ?eld of vieW
Typically, display 30 has less imaging resolution than
scene image sensor 24. Accordingly, signal processor 26
reduces the resolution of a captured or stored image signal or
20
HoWever, this is not strictly necessary so long as there exists
a general correspondence betWeen the siZe and location of the
identi?ed portion 116 and the portion of the ?eld of vieW used
digital image When forming evaluation images adapted for
presentation on display 30. DoWn sampling and other con
ventional techniques for reducing the overall imaging reso
lution can be used. For example, resampling techniques such
by range?nder 27 in measuring the distance to the designated
25
as are described in commonly assigned US. Pat. No. 5,164,
831 “Electronic Still Camera Providing Multi-Format Stor
In one embodiment, calibration image 100 is captured at
et al. on Mar. 15, 1990, can be used. The evaluation images
30
evaluation images can be adapted to be provided to an
optional display driver 28 that can be used to drive display 30.
Alternatively, the evaluation images can be converted into
signals that can be transmitted by signal processor 26 in a
form that directly causes display 30 to present the evaluation
images. Where this is done, display driver 28 can be omitted.
made for portions of the ?eld of vieW associated With each
range?nding “spot” in the scene. This reduces the likelihood
that the composition of the scene can change betWeen the time
of image capture and the time of range?nding measurements.
However, it Will be appreciated that minor time variations Will
be acceptable in certain applications.
35
A focus correlation is then determined based upon the
calibration distance and the focus distance setting used to
capture the calibration image (step 88). The focus correlation
can be any type of data, pro grammic, algorithmic, mathemati
cal or logical structure that associates different range?nder
distance values With different focus settings for lens system
FIG. 4 shoWs a block diagram of a ?rst method for auto
calibrating the range?nder based autofocus system in Which a
digital camera 12 is turned ON Without an instruction to
immediately capture an archival image. As is illustrated in
portion to measure a distance to that portion.
about the same time that range?nding measurements are
age Of FullAnd Reduced Resolution Images” ?led by Kuchta
can optionally be stored in a memory such as memory 40. The
image 100 into portions that correspond directly to the por
tions associated With each “spot” used by range?nder 27.
40
23. FIG. 6 shoWs one example of a focus correlation
expressed, in this example, in the form of a tWo-dimensional
FIG. 4, When controller 32 detects that a user has taken an
action to activate digital camera 12 (step 80), controller 32
look-up table (LUT). In this example, taking lens system 23
causes a calibration image 100 of a ?rst ?eld of vieW to be
can be set into one often focus distance settings focusing light
automatically captured (step 82). FIG. 5 illustrates one
example of such a calibration image 100.
45
from a different range of distances onto scene image sensor
24 associated With a range of focus distances. Each of the ten
Controller 32 can determine a taking lens setting for use in
focus distance settings is also associated With a range?nding
capturing calibration image 100 by setting lens system 23 to
distance. In a typical camera of the prior art this correlation set
is ?xed for all images. HoWever, as noted above, such an
a predetermined ?rst focus distance setting Which can be for
example a position at a middle of an adjustable range of
taking lens system 23. HoWever, in other embodiments con
troller 32 can select any of the other focus distance settings.
Alternatively, controller 32 can be adapted to capture an
50
image using Whatever focus distance setting lens system 23 is
plurality of focus correlations available. For this example,
controller 32 can select from tWo other focus correlation
set at the moment that controller 32 detects some condition
indicating that digital camera 12 is to be activated.
Signal processor 26 and/or controller 32 portion the cali
bration image 100 into portions 102-118 and identify one of
the portions 102-118 of calibration images 100 as having a
preferred level of focus as that term is understood in the art
55
(step 84). Such focus level can, for example, be determined by
60
distinctiveness or outline found in the image, or using any
other knoWn metric for analyZing image focus. The preferred
level of focus can be de?ned in relative terms by Way of
frequency domain, portions of calibration image 100 having
LUTs. This plurality is depicted herein as three different
LUTs in FIGS. 6, 7 and 8. Controller 32 selects from among
the available LUTs by looking for the focus correlation that
provides a combination of focus setting and a capture focus
distance that most closely correlates to the focus setting used
to capture the calibration image and the calibration focus
distance determined by range?nder 27 for the portion of the
image identi?ed as having the predetermined degree of focus
at that focus setting. For example, if taking lens system 23 Was
focused at setting 3 during capture of the calibration image
examining any or all of the level of contrast, clarity, detail,
comparison With other portions. In one speci?c example,
Where calibration image 100 is stored by compression in the
arrangement does not permit digital camera 12 to adjust to
variations in operation due to environmental conditions or
mechanical variations.
Accordingly, in this embodiment, digital camera 12 has a
65
and the determined focus calibration distance Was 0.6 meters,
then controller 32 Would select the focus correlation depicted
in FIG. 8.
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement