B. Schunck

B. Schunck
1010
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,
VOL. Il. NO. IO, OCTOBER 1989
Image Flow Segmentation and Estimation by
Constraint Line Clustering
BRIAN G. SCHUNCK,
Abstract-Image
flow is the velocity field in the image plane caused
by the motion of the observer, objects in the scene, or apparent motion.
The image flow velocity field is an important intrinsic image and many
algorithms that use the image flow velocity field have already been described. The image flow velocity field can contain discontinuities due
to object occlusion in the scene. An algorithm that can estimate the
image flow velocity field when there are discontinuities due to occlusion
is described. Experimental
results on a demanding synthetic test case
and a real image are presented. Some error analysis that explains the
performance of the algorithm is provided. The velocity field estimate
can be improved using surface reconstruction
between velocity field
boundaries. A new surface approximation
algorithm that does not blur
sharp boundaries is presented.
Index Terms-Computer
vision, image flow, motion estimation, motion segmentation, optical flow, smoothing, surface approximation,
surface interpolation,
surface reconstruction
I. INTRODUCTION
HIS paper describes a new algorithm called constraint
line clustering for estimating the image flow velocity
field. Particular emphasis is placed on motion estimation
and segmentation in situations such as random dot patterns where motion is the only cue to segmentation. Experimental results and error analysis are presented. A new
smoothing algorithm for improving the velocity field estimate is described. The smoothing algorithm constructs
a smooth estimate of the velocity field by approximating
a surface between step discontinuities.
T
MEMBER,
IEEE
Image flow is modeled by the image flow constraint
equation,
E,u + EYu + E, = 0,
(1)
which models the interaction between the velocity field
(u, v) and the local changes in space and time of the image irradiance E(x, y, t). The image flow constraint equation is discussed by Schunck [2], [ 11.
B. Applications
Potential applications for image flow include structure
from motion, egomotion, object tracking, and image
compression. Schunck [3] provides a summary of image
flow applications. It is possible that the image flow velocity field cannot be estimated with sufficient accuracy
for structure from motion or egomotion. Schunck [4] argues that the image flow constraint equation (1) may not
be the appropriate model for using motion to estimate
scene structure and observer motion. Regardless of how
these concerns are answered, the image flow velocity field
can be used for segmentation since the velocity field is
useful for qualitative purposes [5].
C. Scope of Presentation
The work described in this paper is concerned with developing an algorithm for estimating the image flow velocity field from image input. Since image flow is a shortA. Image Flow
range process, long-range motions that would require corImage flow is the velocity field in the image plane that respondence over large distances are not considered. It is
arises due to the projection of moving patterns in the scene assumed that image flow is an early vision process that
onto the image plane. The motion of patterns in the image precedes (or works in parallel with) feature extracting
plane may be due to the motion of the observer, the mo- processes; hence, image flow algorithms based on feature
tion of objects in the scene, or both. The motion may also matching are not considered. The work presented in this
be apparent motion where a change in the image between paper differs from prior work in the attention given to imframes gives the illusion of motion. Image flow is a short- age flow estimation when there are motion boundaries.
range visual process that is more local than matching Particular emphasis is placed on the performance of the
methods which work with image features, but not as local image flow estimation algorithm in cases where motion is
as difference picture methods which work with pixel-sized the only cue for segmentation and the image sequence
contains motion boundaries. An example of this situation
regions [ 11.
is the test pattern used by Braddick [6], explained in SecManuscript received August 8, 1986; revised March 31, 1989. Rection II-E.
ommended for acceptance by W. B. Thompson. This work was supported
in part by the Advanced Research Projects Agency, Office of Naval Research under Contract N00014-80-C-0505.
The author is with the Artificial Intelligence Laboratory, Department of
Electrical Engineering and Computer Science, University of Michigan, Ann
Arbor, MI 48109.
IEEE Log Number 8929346.
D. Summary
Section II summarizes prior work in image flow estimation. Section III discusses image flow models. The polar form of the image flow constraint equation [2], [l] is
0162-8828/89/1000-1010$01.00 0 1989 IEEE
1011
SCHUNCK: IMAGE FLOW SEGMENTATION AND ESTIMATION
presented and sampling effects are discussed. Section IV
describes the constraint line clustering algorithm for estimating the image flow velocity field and presents experimental results. Section V provides some error analysis
for constraint line clustering. Section VI presents a new
smoothing algorithm, called surface-based smoothing, for
intrinsic images with discontinuities. Surface-based
smoothing must be used to improve the velocity field estimate produced by constraint line clustering. Experimental results with surface-based smoothing applied to the results of constraint line clustering from Section IV are
presented. The paper finishes with a discussion of conclusions in Section VII.
II. BACKGROUND
This section reviews prior work in developing algorithms for estimating the image flow velocity field. Work
in image flow estimation by regularization and correlation, image flow estimation for image compression, and
image flow estimation along contours will be reviewed.
Related work in psychology that suggests a performance
goal for image flow estimation algorithms will be discussed. Image flow research related to motion boundaries
is summarized. A good overview of early work in image
flow is provided by Nagel [7].
A. Image Flow Estimation by Regularization
Regularization has been used to develop many vision
algorithms for early vision processing [8]. Horn and
Schunck [9] formulated an optimization problem for estimating image flow. The optimization measure consisted
of two terms: a penalty on the deviation of the estimated
velocity field from the image flow constraint equation and
a penalty on the deviation of the velocity field components
from smooth surfaces. The smoothness penalty was the
sum of the squares of the magnitude of the gradient of
image flow velocity. Horn and Schunck [9] implemented
iterative equations for solving these equations and presented examples of the application of the iterative equations to several synthetic images. Unfortunately, the algorithm cannot work in cases where there are motion
boundaries in the velocity field since the smoothness constraint leads to iterative equations that blur abrupt changes
in the velocity field.
A test case that demonstrates some of the problems with
image flow estimation algorithms based on regularization
is a textured box translating across a uniform background.
The textured box is any rectangular region with a random
pattern of image u-radiance and the uniform background
has constant image n-radiance. Suppose that an image flow
estimation algorithm based on regularization, such as one
developed by Horn and Schunck [9], is applied to this test
case for many frames until the object is beyond the field
of view. Let a small number (possibly as few as one) iterations of the algorithm be performed per frame of the
test sequence. This provides the best results in most cases
since new motion constraint information is injected into
the algorithm [9, Sect. 171. In the first few frames of the
sequence, an image flow velocity field will be estimated
within the boundaries of the box. The velocity field outside of the box will be zero. As the box moves, the region
of nonzero velocity field does not move along with the
box since there is nothing in the algorithm that forces the
velocity field to be extrapolated in the direction of motion. In fact, this would be the wrong thing to do in many
casessuch as rotation or dilation without translation. Since
the location of the region of nonzero velocity field is not
forced to move along with the box, the translating box
moves out from under the region of nonzero velocity vectors. New nonzero velocity vectors are formed on the surface of the box as it moves into positions. The algorithm
does not eliminate the old region of velocity that trails
behind the box since the uniform background provides no
constraint on the possible velocities. This creates a
“comet tail” of velocity behind the box. In regions of the
image where the gradient is zero, the Horn and Schunck
algorithm degenerates into the Laplace equation which
smooths the velocity field. The region of nonzero velocity
vectors that trails behind the box is smoothed into a consistent (though obviously wrong) velocity field which persists long after the translating box moves beyond the
boundaries of the image. The erroneous velocity field will
never be eliminated; it is just gradually smoothed out into
the uniform background.
B. Motion Estimation by Correlation
Poggio and Reichardt [lo], Reichardt and Poggio [ 111,
and Buckner [12], have studied the visual system of the
fly and concluded that the behavior of the motion detection system can be described as a correlation. Any correlation scheme for motion estimation, whether implemented in a biological organism or in a machine vision
system, cannot produce a velocity field estimate with
sharp boundaries. Correlation cannot achieve the type of
result desired in this work.
The work of Poggio and Reichardt [lo], [ 1 l] was based
on the Volterra series for modeling systems with multiple
inputs [13]. The Volterra series multiplies inputs but provides no special facilities for representing discontinuities.
C. Motion Estimation for Image Compression
Haskell [14], Limb and Murphy [15], [16], Cafforio
and Rocca [ 171, Netravali and Robbins [ 181, Stuller and
Netravali [19], Stuller, Netravali, and Robbins [20], and
Jones and Rashid [21] have studied motion estimation for
image compression. Additional work was reported by
Paquin and Dubois [22]. Image compression has not
forced the development of image flow estimation algorithms that handle discontinuities because image
compression does not require perfect estimation of the
motion and does not require the detection of motion
boundaries. Any discrepancy between frames caused by
inaccurate estimation of the motion is transmitted as a
correction. It may be true that a better motion estimation
algorithm could reduce the image bandwidth further, but
1012
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,
any improvement must be balanced against the added cost
of the image compression implementation [3].
D. Image Flow Estimation Along Contours
Recent work used edge information in the flow computation by combining motion information along edge
contours [23]-[26].
There are problems with this approach: 1) the velocity vector of an edge is a very powerful clue for grouping edges into contours, but this advantage is dismissed when motion estimation proceeds
contour grouping, 2) edge fragments can be combined into
contours that cross motion boundaries and this will lead
to an incorrect motion estimate, and 3) it is not necessary
to restrict motion information to edges since it is possible
to develop one algorithm that simultaneously estimates the
motion at edges and interpolates between edges. The motion estimation algorithm developed by Horn and Schunck
[9] uses a single set of iterative equations that, in effect,
automatically balance the computations performed in regions of high and low gradient. When the gradient is large,
the information provided by the image flow constraint
equation dominates the smoothing process; when the gradient is small, less weight is given to the motion constraint and the iterative equations reduce to the computation of a surface approximation determined by the form
of the smoothness constraint. The surface approximation
computation iteratively fills in the velocity field using the
estimates derived from areas where the gradient is large.
Another problem with image flow estimation algorithms
that follow edge contours is that they do no better in situations where there are velocity field discontinuities than
any of the other algorithms summarized in this section.
E. Results from
Psychology
Although it was not the intent of this work to develop
a model for human motion perception, some results from
psychology focused this project on solving a demanding
problem in motion segmentation with translating objects
[6], [27]-[29]. The work of Braddick [6] provided the key
test case and standard of performance for this research.
Braddick [6] performed apparent motion experiments with
pairs of random-dot images. Pixels were randomly chosen
to be black or white. A central rectangle within one image
was displaced to form the second image. No texture of
image intensity boundaries were present in the test images. The image pairs used by Braddick differed in pixel
size and displacement distance. Subjects were required to
decide whether the rectangle was horizontal or vertical
and rate the clarity of the motion percept. Performance
and clarity decreased with the size of displacement, irregardless of the pixel size. For small displacements of
around 5’ of arc, subjects reported that the motion was
coherent and the boundaries of the rectangle could be
clearly seen. The range over which the clarity of the motion perception deteriorates corresponds closely with the
increasing sizes of the channels in the edge detection theory of Marr and Hildredth [30]. As the displacement increases, it is within the support of fewer channels and the
VOL. II. NO. IO, OCTOBER 1989
ability to perceive coherent motion decreases. Beyond 20’
of arc, coherent motion was not perceived and the bounding contour of the rectangle was not clearly seen. Braddick argued that if human vision used an algorithm that
matched pixels of one type (white or black) to nearby pixels of the same type, then when the size of the displacement was more than one pixel subjects would not see coherent motion. Since this did not occur, Braddick argued
that human vision must incorporate a more sophisticated
algorithm than matching nearby pixels to perceive coherent motion. Subsequent analysis [29, p. 181 argues for a
short-range process based on changes in image intensity
in addition to a long-range process based on feature correspondence.
This research is not concerned with discovering the
mechanisms of biological vision systems, but the results
from psychology on the short-range motion process were
essential in focusing this project. Since the short-range
motion process in human subjects can detect motion
boundaries in scenes where other cues are not present, it
must be possible to develop an algorithm for estimating
the image flow velocity field while retaining the sharp
changes in velocity values that may occur across a motion
boundary that corresponds to occlusion of scene surfaces.
The range of displacement over which coherent motion is
perceived in the short-range motion process is limited by
the size of the largest channel. In machine vision, the
maximum displacement for which an image flow estimation algorithm can be applied is limited by the size of the
smoothing filter applied to the image or the size of the
aperture function if no smoothing is performed. The aperture function is usually approximately equal to the pixel
spacing. If no smoothing filter is applied to the image,
then the displacement between frames must be no greater
than the pixel spacing.
In summary, the illusion of Braddick [6] is a key test
case for image flow estimation because it requires estimation and segmentation based on short-range motion
without other visual cues. Since the subjects in the experiments conducted by Braddick were able to clearly perceive the outline of the moving rectangle, an image flow
estimation algorithm should be able to retain sharp step
changes in the velocity field when given a random dot
image pair as input.
F. Image Flow Estimation and Motion Boundaries
Nakayama and Loomis [31] proposed a function called
the convexity function for detecting motion boundaries in
the image flow field, but did not explain how the velocity
field could be estimated in the vicinity of the motion
boundary. Batali and Ullman [32] tried a local consistency check on the motion information provided by directional selectivity to segment random binary images, but
the algorithm did not estimate the velocity field. Mutch
and Thompson [33] presented a correspondence algorithm
for motion analysis that detected occlusion by looking for
significant regions where matches could not be found.
Thompson, Mutch, and Berzins [34] developed a motion
1013
SCHUNCK: IMAGE FLOW SEGMENTATION AND ESTIMATION
detection operator based on the zero-crossing detector of
Marr and Hildreth [30] and developed an analysis of motion boundaries that allowed the occluding surface to be
differentiated from the occluded surface.
Fennema and Thompson [35] developed an algorithm
for estimating the velocity vector of the most prominent
object in the scene by using histogram analysis to combine image flow constraints. The algorithm could only determine the motion of the single most prominent object,
could not handle rotation or translation in depth, and assumed that the background was stationary. Since the local
structure was lost, the algorithm could not compute an
image flow velocity field and this meant that the algorithm
was not useful for scenes containing multiple objects. For
example, it is not possible to differentiate between 1) the
situation where one large object is translating and 2) the
situation where two objects, each half the size of the object in situation (1)) but separated by some significant distance, are translating. Nevertheless, the algorithm was
significant because the histogram technique was immune
to the incorrect motion constraints generated along motion boundaries [l] and was not confused by multiple
moving objects in the scene as long as one moving object
was significantly larger than the others. The Fennema and
Thompson algorithm inspired the cluster analysis approach presented in this paper.
G. Summary of Prior Work
Image flow algorithms have been developed with regularization, but the algorithms are not suitable for realworld scenes because the algorithms cannot handle velocity field discontinuities. Correlation and image flow estimation along contours have the same problem. Much work
on image flow estimation has been done for image
compression, but the problem of estimating the image flow
velocity field with discontinuities has not been addressed.
Results from psychology strongly suggest that the human
vision system can estimate the image flow velocity field
without blurring motion boundaries. This implies that a
machine vision system should be capable of estimating
the image flow velocity field with little distortion in the
velocity field discontinuities. The intent of this work is to
develop an image flow velocity field estimation algorithm
with performance comparable to that of the human vision
system; the algorithm should be able to estimate the velocity field in images that contain motion boundaries
without blurring the boundaries or allowing the presence
of the boundaries to distort the velocity field estimate. It
is not the intent of this work to propose the image flow
estimation algorithm presented in this paper as a model
for the human vision system, since it would require more
experimentation and comparison with psychophysical results; but the algorithm presented in this paper may suggest a model for image flow estimation in the human vision system that could be subjected to the required tests.
III.
IMAGE FLOW MODELS
The image flow constraint equation (1) models the relationship between the velocity field and the changes in
the image it-radiance in space and time. The polar form of
the equation is the basis for the algorithm development in
this paper. As stated by Schunck [ 11, [4], two conditions
are required to ensure the validity of the image flow constraint equation: 1) the perceived change in image irradiance at each point in the image plane must be entirely
due to motion of the image pattern, as opposed to changes
in the pattern due to reflectance effects, and 2) the image
must be smooth except at a finite number of discontinuities. The first condition is satisfied by the image of a
translating, diffuse object with distant light sources; the
second condition is a technical point. In other words, the
perceived change in image irradiance at each point in the
image plane must be entirely due to shifts of the image
pattern, as opposed to changes in the pattern. These conditions are probably too restrictive to allow image flow to
be useful for structure from motion, but the image flow
equation is obeyed in practice with sufficient accuracy for
tasks such as segmentation.
A. Polar Form of the Constraint Equation
The image flow constraint equation defines a line in velocity space as shown in Fig. 1. The line in velocity space
that is the locus of possible velocities specified by the image flow constraint equation is called the constraint line.
The constraint line is uniquely defined by d, the distance
of the line from the origin along the perpendicular bisector and the angle cr with respect to the u axis. The displacement and angle of the constraint line are given by
PI
FI
d= qq=IVEI;
CY=
arctan (E,, Ey)
if E, 1 0;
arctan ( -E,,
otherwise.
-Ey)
(2)
(3)
To derive the constraint equation in polar coordinates,
note that the image flow equation contains the dot product
of the image irradiance gradient with the velocity vector
E,u + E,v + E, = VE * (u, v) + E,.
(4)
If p is the speed of motion, CYis the angle of the constraint
line, and fi is the direction of motion, then the dot product
in (4) is
plVE(
cos (a - 6).
(5)
Dividing through by the magnitude of the image gradient
yields the polar form of the image flow constraint equation
d=pcos(cr-0).
(6)
Note that fi is constrained to be between (Y - a/2 and CY
+ 7r/2. Since the displacement d of the constraint line
from the origin must always be nonnegative, the orientation Q is reflected when E, > 0. The displacement d is
the projection of the motion vector onto the line of the
gradient of image ii-radiance and is independent of the
magnitude or polarity of the gradient.
The polar form of the image flow constraint equation
1014
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. II, NO. IO, OCTOBER 1989
rithm must be resistant to grossly incorrect measurements. The errors along motion boundaries are too severe
to ignore.
In this work, any motion constraint that deviates greatly
from the true motion constraint is called a grossly incorrect constraint. Experimental observations [2], [l] indicate that grossly incorrect constraints almost always occur
along motion boundaries. Since motion estimation in the
presence of motion boundaries is central to this work,
grossly incorrect constraints must be handled.
Fig. 1. The locus of points satisfied by the image flow constraint equation
is shown. The equation defines a line of points in velocity space with
velocities that satisfy the constraint. The distance d from the origin to
the constraint line L is called the displacement of the constraint line: the
angle (Y is called the orientation of the constraint line.
should be used in the analysis of algorithms for image
flow estimation with motion boundaries since the polar
form will not contain &functions at step discontinuities.
In other words, if an example contains an ideal step discontinuity in image irradiance, then the coefficients in the
conventional form of the image flow constraint equation
(1) will be &functions. The polar form will not contain
b-functions since the displacement d is the ratio of the
magnitude of the change in time of the image irradiance
to the magnitude of the gradient of the image irradiance
and this ratio remains finite in the limit as the image irradiance becomes an ideal step discontinuity.
B. Sampled Image Flow Models
The time-varying image E(x, y, t) is sampled in space
and time to produce an image sequence. The spatial sampling distance must be smaller than the scale of image
texture. This concept is most easily explained by refering
to Fig. 9. Note how the shift in the image irradiance function is small relative to the wavelength of the image irradiance. The temporal sampling period must be much
shorter than the scale of time over which the velocity field
changes and sufficiently short, relative to the velocity, so
that the local displacement is smaller than the local scale
of the image texture. In other words, the image ii-radiance
pattern in the image sequence must evidence a phase shift
that is large enough to be measured accurately and small
enough to avoid phase ambiguity.
The constraints are suflicient to allow the sampled image sequence to accurately portray the local motion data
between motion boundaries, but are not sufficient to capture the motion information at motion boundaries. The
motion constraints computed along motion boundaries are
extremely sensitive to the effects of occlusion boundaries
and are usually severely in error. No practical sampling
period can provide useful motion information along a motion boundary [2], [l]. An image flow estimation algorithm must be able to work in situations where there are
motion boundaries in the image; consequently, the algo-
IV. CONSTRAINT LINE CLUSTERING
An algorithm called constraint line clustering [36] that
estimates the image flow field when the image irradiance
pattern or the velocity field contains discontinuities will
be presented.
A. Problem Statement
The constraint line clustering algorithm uses the polar
form of the image flow constraint equation
d=pcos(a!--P)
(7)
where p (x, y) and /3(x, y) are the speed and direction of
motion, respectively. The velocity vector ( p, 0) for the
motion at any point in the image must lie along the line
in velocity space defined by the image flow constraint
equation (7) shown in Fig. 1. The constraint line is
uniquely defined by the displacement d of the constraint
line from the origin and the orientation (Yof the constraint
line. The constraint line displacement has the dimensions
of speed; it is the minimum speed consistent with the motion constraint.
Assume that the image intensity E(x, y, t) already incorporates any spatial or temporal filtering performed on
the image. For example, the image could be smoothed
with a Gaussian filter. The image is sampled in space and
time to produce an image sequence E (xi, yj , tk). The d
and a intrinsic image arrays d (Xi, yj ) and (Y(Xi, Yj ) are
computed from the image sequence using the formulas in
(3). The partial derivatives are computed by first differences over a cube in space and time [9, sect. 71. The image flow estimation problem is to compute the intrinsic
image arrays p (Xi, yj ) and /3(xi, yj ) for the speed and
direction of motion from the motion measurements in the
d and a arrays. Prior knowledge of the image flow velocity field is not available. By assumption, the image flow
equation (7) describes the changes in the image sequence;
hence, the restrictions discussed by Schunck [l] apply.
B. Constraint Line Clustering
The constraint line clustering algorithm uses a clever
form of cluster analysis to extract the motion estimate
from contradictory data. What could be a multidimensional cluster analysis problem is transformed into a trivial cluster analysis problem in one dimension. Suppose
that for each d and cy measurement a set of measurements
{ di, (Yi } is taken from some spatial neighborhood of the
point that generated d and a. Compute the set of intersec-
SCHUNCK: IMAGE FLOW SEGMENTATION AND ESTIMATION
1015
tions of each of the neighboring measurements di and pi
with the given d and CYmeasurement. All of these intersections lie along the line defined by d and cr. Any constraint lines in { di, (yi } that are part of the same region
of motion as d and CYwill tend to intersect the line defined
by d and cx in a tight cluster around the true velocity. Any
constraint lines in { di, (Yi } that are from regions of different motion will intersect the line defined by d and 01
over a broad range of positions. The grossly incorrect
constraint lines generated along a motion boundary will
not intersect the line defined by d and (Y at a consistent
point. The velocity estimate for the neighborhood of a
given d and CYmeasurement can be computed by one-dimensional cluster analysis of intersections along a constraint line.
1) Computing Constraint Line Intersections: The formula for the position of the intersection along a constraint
line is easily derived by first rotating the coordinate system so that the constraint line defined by d and (Y is parallel to the vertical axis in velocity space. Let the angle
between the two constraint lines be denoted by 4 = (Y’CX,as shown in Fig. 2. The distance of the intersection
along the constraint line defined by d and cx is given by
b=C
03)
tan+’
The distance c is related to d, d’, and 4 by
(d + c) cos 4 = d’,
(9)
so using (8) to eliminate c from (9) yields
dcos+
+ bsin4
= d’.
(10)
Solving for b and substituting the definition for 4 yields
b=d’-dcos4
sin 4
=
d’-dcos(a’-cr)
sin (CY’- c~)
*
(11)
This formula provides the position along a constraint line
defined by d and CYof the intersection of the constraint
line with another constraint line defined by d’ and CX’.
2) One-Dimensional Cluster Analysis: Given a set
{ bi } of intersections of the constraint lines within a
neighborhood with a given constraint line at the center of
the neighborhood, the set of intersections must be analyzed to determine the most consistent subset of intersections that cluster about the likely velocity. The cluster
analysis criterion is motivated and explained by the following example. Imagine that a motion boundary passes
almost vertically through a neighborhood just to the left
of the center element. At any reasonable pixel resolution,
the boundary will most likely pass smoothly through the
neighborhood dividing the neighborhood almost in half.
The region on the left corresponds to one surface in the
scene and the region on the right including the center element corresponds to a different surface. In the extreme
case where the boundary passes very close to the center
of the neighborhood, almost half of the intersections of
neighborhood constraint lines with the constraint line from
V
t
t
d
I
\
Fig. 2. The method for deriving the formula for the position of the intersection of two constraint lines along one of the constraint lines is
shown.
the center of the neighborhood will be with constraints
from the left surface or the motion boundary itself. These
intersections will most likely not be close to the velocity
of the right surface from which the center constraint line
was obtained. Almost half of the intersections may be
useless in the extreme case; but at least half of the intersections will correspond to the same region of motion.
Since rejecting correct intersections is less harmful than
accepting incorrect intersections, a conservative stand is
taken: the algorithm looks for the tightest cluster that contains roughly half of the intersections.
In one dimension, cluster analysis is easy since there is
a total ordering of points along a line and cluster analysis
reduces to interval analysis. The constraint line clustering
algorithm sorts the set of n intersections { bi } and then
looks for the tightest interval that contains half of the intersections by examining successive pairs of intersections
that are Ln/2 1 intersections apart. The algorithm
chooses the pair of intersections that are closest together
as the estimate of the majority cluster of intersections.
The test assumes that there is at most one motion boundary within the spatial neighborhood of the d and CYmeasurement and the motion detectors are sufficiently finely
spaced that the motion boundary within the neighborhood
does not pass erratically through the neighborhood. The
assumption excludes cases where the center d and CYmeasurement in a neighborhood is from one side of a motion
boundary while the majority of d and (Y measurements in
the neighborhood are from the opposite side of the motion
boundary. In practice, the constraint line clustering algorithm continues to work when the assumption is violated for reasons explained in Section IV-C 1).
A formula for the exact fraction of constraint lines that
should be in the majority when the motion boundary
passes cleanly through the neighborhood is easily derived. Suppose that a straight, vertical motion boundary
passes through a neighborhood to one side of the center
element. Let the neighborhood size be n by n where n =
2k + 1 for some positive integer k. The number of constraint lines in the compatible majority is (2k + l)( k +
1016
IEEE
TRANSACTIONS
ON PATTERN
ANALYSIS
1)) the number of incompatible minority constraint lines
is (2 k + 1) (k - 1)) and the number of erroneous constraint lines is 2k + 1. The number of compatible constraint line intersections forming a relatively tight cluster
will be
(2k + l)(k
+ 1) -
1 = 2k2 + 3k = k(2k + 3),
(12)
and the fraction of constraint line intersections that should
form a tight cluster is
k(2k + 3)
2k + 3
= kP
+ 3)
(2k+ I)*1
4k2+4k
=4k’
03)
The constraint line clustering algorithm determines the
tightest cluster of constraint line intersections that contains this fraction of the total number of constraint line
intersections. There may be fewer than 4k (k + 1) intersections, since some of the constraint lines may be parallel to the constraint line at the center of the neighborhood. The location of the center of the shortest interval
that contains the fraction of intersections given by the formula above is used as an estimate for the most consistent
velocity within the neighborhood. Note that the formula
for the fraction of the constraint lines on the majority side
of the boundary asymptotically goes to l/2 as k + 00.
3) Estimating the Velocity Vector: The center position
6 of the tightest cluster along the constraint line is the
midpoint of the smallest interval that contains roughly half
of the intersections. The estimated speed of motion is
given by
h=XLFX
(14)
and the estimated direction of motion is given by
6 = (Y + tan(i/d).
(15)
The estimated speed and direction of motion ( B, ,8 ) computed with constraint line clustering from the motion parameters { di } and { ai } satisfy the problem statement in
Section IV-A.
C. Potential Problems with Constraint Line Clustering
Some concerns that may be raised about the performance of constraint line clustering will be addressed in
this section. Implementation details of.he constraint line
clustering algorithm will also be explained.
1) Motion Boundary Corners: The constraint line
clustering algorithm is very robust and works even if the
basic assumption that at least half of the intersections must
be from the same region of motion is violated. For example, if the center constraint is from just inside a right
angle comer so that only about l/4 of the intersections
are from the same region of motion, the algorithm still
works because the intersections from constraints outside
the comer are almost uniformly distributed along the constraint line and do not bias the estimate of the tightest
cluster [3]. The error analysis in Section V attempts to
explain this in detail.
AND
MACHINE
INTELLIGENCE,
VOL.
II.
NO.
IO, OCTOBER
1989
2) The Aperture Problem: Marr and Ullman [37] discuss the aperture problem for image flow estimation. The
problem occurs in any situation where an estimate must
be obtained from data in a local neighborhood in an image
that may contain a boundary. If an algorithm contains an
assumption that the data are consistent over the neighborhood and the assumption is violated, then the image flow
estimate will be incorrect. On the other hand, it is necessary to use measurements in a neighborhood large
enough to provide sufficient variation in gradient orientation. The constraint line clustering algorithm does not
suffer from an aperture problem, because the one-dimensional cluster analysis computation provides a statistical
check that prevents the aperture problem from occurring
within a neighborhood of any size. This means that the
size of the neighborhood can be increased to include more
good constraint lines and improve the local estimate of
the velocity field. This property is particularly useful
given the problem that erroneous constraint lines will occur along a motion boundary. Since the aperture problem
is not a concern, the relative number of erroneous constraint lines in a neighborhood can always be controlled
by varying the size of the neighborhood. Constraint lines
that occur along the curve of a motion boundary are
grossly incorrect due to sampling effects [2], [l]. If the
neighborhood size w is defined to be the width of a neighborhood, then the number of grossly incorrect constraint
lines generated along a motion boundary within a neighborhood increases at most linearly with neighborhood
size. The number of constraint lines that are not grossly
incorrect always increases as the square of neighborhood
size. The ratio of grossly incorrect to reasonably accurate
constraint lines will always decrease at least as fast as
l/w where w is the neighborhood width. W ithin the
neighborhood of any good constraint line, there will always be enough reasonably accurate constraint lines to
form a tight cluster.
3) Numerical Condition: Constraint line intersections
are not always numerically well conditioned. W h e n two
constraint lines are close in orientation there can be a large
error in the intersection position. W h e n the constraint lines
have the same orientation and different displacements, the
lines will not intersect. In spite of these objections, the
constraint line clustering algorithm is robust by virtue of
the fact that only the tightest group of points are retained
to compute the motion estimate. W h e n an intersection position is computed for two constraint lines that are close
in orientation and displacement, then the position of the
intersection may be distant from the true position. Nearly
half of the interaction positions will be discarded. Any
grossly incorrect intersections will most probably be discarded since they lie outside the tightest cluster.
4) Selecting the Neighborhood Size: The algorithm is
relatively insensitive to the choice of neighborhood size.
Good results have been obtained even for the minimum
neighborhood size of 3 by 3. Larger neighborhood sizes
produce only slightly better motion estimates. There are
two reasons for this. The first reason is that the accuracy
1017
SCHUNCK: IMAGE FLOW SEGMENTATION AND ESTIMATION
of the motion estimate is limited by the accuracy of the
constraint line along which the motion estimate is obtained. The second reason is that the number of points in
the neighborhood may have little impact on the quality of
the motion estimate; it is the spatial extent of the neighborhood relative to the gradient variation that determines
the quality of the motion estimate.
Increasing the spatial extent of the neighborhood, without necessarily increasing the number of points in the
neighborhood, may bring new information (new constraints) into the calculation of the velocity estimate if the
larger neighborhood provides more gradient variation.
Consider a region in the image where the gradient is nearly
constant. The motion constraints derived in this region
will be similar and will not yield good intersection measurements because the intersection computations are illconditioned. In order to get good intersection measurements, motion constraints must be used from beyond the
region of similar gradient. The motion estimate can be
greatly improved by increasing the spatial coverage of the
neighborhood. It is not necessary to increase the number
of motion constraints in the neighborhood since the motion constraints close to the center of the neighborhood
are redundant. The amount of spatial variation in the motion constraint measurements determines the minimum
spatial extent of the neighborhood.
The constraint line clustering algorithm has a wide tolerance to the neighborhood size: the neighborhood must
be large enough to provide sufficient variation in image
gradient direction as explained in Section IV-C6), but
small enough so that constraint line clustering does not
average away variations in the velocity field as explained
in Section IV-(X). The wide tolerance in the neighborhood size made it easy to select the neighborhood size for
all of the experiments in this work: a neighborhood size
of 5 by 5 possibly with some downsampling as explained
in Sections IV-C7) and IV-D) was sufficient.
5) variation in the Velocity Field: The amount of slow
spatial variation in the velocity field may limit the maximum neighborhood size due to an averaging effect. Suppose that a neighborhood does not contain a motion
boundary. The intersection measurements will be closely
grouped together, relative to the situation where the
neighborhood extends across a motion boundary. If there
is some variation in the velocity field within the neighborhood, then the spread of the intersection positions will
be greater than if there were no variation. There is actually no point where constraint line clustering will fail to
yield a reasonable estimate of the center position of the
group of intersections along the constraint line from the
center of the neighborhood. Using the center position of
the intersections as the velocity estimate is like averaging
or low-pass filtering the velocities in the neighborhood.
The averaging will reduce the spatial resolution of the velocity field estimate. The neighborhood size should be
chosen so that the spatial extent of the neighborhood is at
least as large as the scale of the variation in the image
gradient, but not so large that the resolution in the veloc-
ity field estimate would be excessively reduced due to averaging .
6) ZnsuJicient Variation in Image Gradient: Any algorithm for estimating the image flow velocity field depends on the assumption that there will be sufficient constraints on the motion. If there are insufficient constraints,
then the algorithm cannot solve the problem; and if the
constraints conspire to yield an incorrect solution, then
the algorithm will not compute the correct solution. For
example, if all of the constraint lines in the neighborhood
of a given constraint line are nearly parallel except for the
constraint line at the center of the neighborhood, then the
bundle of constraint lines may not intersect the center
constraint line near the correct velocity. The point is that
any algorithm including regularization over regions [9] or
regularization along contours [24], [25] will have trouble
in cases where there is insufficient or misleading motion
constraints because fundamentally any algorithm will have
trouble if the data is ill-conditioned.
In other words, any
algorithm including constraint line clustering must have
sufficient variety of information. Since constraint line orientation is gradient direction or its opposite, there must
be sufficient variety of gradient directions in the neighborhood .
In practical examples, there were spurious problems in
stationary areas of the test images. Some points in the
velocity estimate that should have had velocities near zero
had large velocities. The constraint line clustering algorithm was not at fault. The constraint line displacements
at the points corresponding to these erroneous velocity estimates were large when they should have been very small.
Closer inspection revealed that the magnitude of the image gradient was very small at the points where the large
d values occurred. The errors in the constraint line displacements were caused by numerical ill-conditioning.
The constraint line displacement is computed by
(E,I
d = (VE(
(16)
and the calculation is most sensitive when 1VE ) is small.
To eliminate this problem, the constraint line displacement is forced to zero whenever the magnitude of the image gradient is small. A small gradient indicates locally
uniform image irradiance and the velocity estimate in a
region of uniform irradiance should be zero. Changing the
constraint line displacement from an erroneously large
value to zero allows constraint line clustering to produce
a small velocity if the majority of the constraint lines in
the neighborhood also have small displacements.
7) Downsampling: If the width of the neighborhood
over which the constraint line clustering is performed is
not large enough to provide a set of constraint lines of
sufficient variety to allow the constraint line clustering algorithm to form a well-conditioned
set of intersections,
then the size of the neighborhood over which constraint
line clustering is performed can be increased. Because of
the statistical nature of constraint line clustering, there will
1018
UXE
TRANSACTIONS
ON PATTERN
ANALYSIS
be no problem if the larger neighborhood extends across
a motion boundary. In practice, a large neighborhood may
not be desirable since the amount of computation that must
be performed is proportional to the area of the neighborhood. Excessive computation may be avoided by downsampling the arrays of d and CYvalues prior to applying
the constraint line clustering algorithm. Downsampling by
a factor of k increases the effective area of the constraint
line clustering neighborhood by a factor of k2 without increasing the amount of computation that must be performed for each neighborhood. The total amount of computation decreases by a factor of k2 since the constraint
line clustering is performed over fewer neighborhoods in
the d and a! arrays.
An alternative to downsampling the d and Q! arrays is
to use overlapping sparse neighborhoods. The constraint
line clustering neighborhoods could be extended to cover
the same pattern of points as in downsampling, but neighborhoods would be overlapped into the points that were
left unused in pure downsampling. The amount of computation performed per neighborhood would be the same
and the effective area of the neighborhood would increase
in the same manner as for downsampling. But the neighborhoods would be overlapped and the number and density of velocity measurements obtained would be about
the same as the size and density of the original d and (Y
arrays.
8) Incorrect Center Constraint Line: One apparent
weakness with constraint line clustering is that it depends
on the accuracy of the constraint line at the center of the
neighborhood. In fact, this is not a problem with the constraint line clustering algorithm itself, but is an instance
of a more fundamental problem with assigning a motion
estimate to a point in an image. If the motion constraint
information is grossly incorrect, then there is a fundamental ambiguity in assigning a motion estimate to
the corresponding image location. For example, if the
grossly incorrect motion constraint is caused by a motion
boundary (which is likely [2], [l]), then it is unclear which
motion present in the neighborhood should be assigned to
that location. If the grossly incorrect motion constraint is
not caused by a motion boundary, then it is still caused
by some effect in the image and the ambiguity is fundamental and legitimate. Stated briefly, if the motion constraint is grossly incorrect, a correct motion estimate cannot be assigned to the location. Another way of saying
this is that d and CYcan be computed accurately when the
gradient is large and there is no motion boundary; otherwise, no motion estimate is reasonable.
9) Relationship to Other Methods: The information in
the image flow constraints could be combined by least
squared methods [38]-[41], if it were not for the focus of
this research on segmentation in situations where there are
several motion boundaries. The image flow constraint
equation is only one constraint with two unknowns and
the problems in combining image flow constraints near
motion boundaries are so severe that least-squares algorithms cannot be used without modifications to select a
AND
MACHINE
INTELLIGENCE,
VOL.
II,
NO.
IO, OCTOBER
1989
consistent set of motion constraints [2], [ 11. Fischler and
Bolles [42] present an example where the least-squares fit
of a model to inconsistent data leads to a wrong estimate
that cannot be fixed by eliminating the outlier. In this
problem, the constraints that are far from the true velocity
are the outliers. Robust statistics provides some methods
for fitting models to inconsistent data [43], [44]. These
methods were not used in this work because the special
structure of the problem leads to a more efficient solution.
Given the assumption on motion boundaries used in constraint line clustering, the test for outliers is simple. Further experiments should be conducted to compare robust
estimates with constraint line clustering.
There are several kinds of robust statistical measures.
In typical robust statistical procedures, data points are
sorted into ascending order and statistical measures are
derived from the ordering of the data points. Two measures that may be related to this work are M-statistics and
L-statistics. An M-statistic is a maximum likelihood estimate derived from the order of a set of measurements [44,
pp. 43-551. An L-statistic is a linear combination of the
ordered data points [44, pp. 55-611. An example of an Lstatistic is the range of a set of data points [43, p. 331.
Constraint line clustering informally combines several
ideas from robust statistics. The selection of the tightest
interval that contains half of the data points resembles an
M-statistic; but there is no formal proof that the tightest
interval containing half the data points is the most likely
choice. Once the tightest interval has been determined,
the location of the velocity estimate is computed with an
L-statistic. Constraint line clustering also resembles the
trimmed mean for computing the mean while eliminating
the influence of outliers [45, p. 2471. Order statistics [43]
presumes that the data points can be ranked into ascending
order. In constraint line clustering, the data points that are
ranked are the locations along the center constraint line of
the intersections with neighboring constraint lines. Although constraint line clustering achieves good results,
the intersections may not be the best choice as a starting
point for the application of formal statistical methods.
Further work should be done with robust regression for
determining the velocity field estimates directly from the
constraint line coefficients.
The random sample consensus (RANSAC) algorithm
[42] developed by Fischler and Bolles handles outliers by
computing an estimate from a consistent subset of inconsistent data points. The algorithm randomly selects a set
S of the minimum number of points required to fit a model
from the set of data points P. The set S is used to compute
a model estimate. The consensus set S* of points in P that
are within an error tolerance r of the model is determined.
If the number of points in S* is greater than some threshold, then the model is recomputed using the data points
in S*; otherwise, a new set S is randomly selected from
the data points P and the consensus procedure is repeated.
After a predetermined number of iterations, the algorithm
either fails or settles for the model obtained from the largest consensus set. The RANSAC algorithm has three pa-
SCHUNCK: IMAGE FLOW SEGMENTATION AND ESTIMATION
1019
ple to a pixel, the pixel was compared to the original pixel
range of [0, 256). If the pixel was negative, then it was
changed to zero; otherwise if the pixel was greater than
255, then it was reduced to 255. This procedure models
saturation in black and white. Subsequent work with images obtained from an RCA vidicon camera indicated that
a noise level at 5 percent of the maximum pixel amplitude
simulated typical error magnitudes. Since the foreground
and background textures had identical statistics, the square
could not be seen in either frame alone. Only motion between frames allowed the square to be differentiated from
the background. The two frames of this test case are displayed in Fig. 3.
The results of applying constraint line clustering with
5 by 5, 9 by 9, and 13 by 13 neighborhoods to the displaced square test case with 5 percent uniform noise is
displayed in Figs. 4, 5, and 6, respectively. The important point to note is that the estimated velocity field is not
improved by increasing the size of the neighborhood in
this test case. This means that the quality of the motion
estimate is limited by the quality of the d and CYmeasurements, not by the performance of the constraint line clustering algorithm. This is an attractive characteristic from
the stand point of this research since it indicates that the
algorithm will work well with a given accuracy for the d
and LYmeasurements. Better algorithms for measuring the
d and a data will lead to better estimates of the image
flow velocity field, but constraint line clustering works
very well with the primitive methods for computing d and
CYused in this work.
Constraint line clustering was run on a real image pair
shown in Fig. 7. The scene was taken with a camera looking down on a laboratory table. A sheet of circuit board
was lying on the table and a phonebook with an outdoor
scene was placed on the circuit board. Scene illumination
was provided by typical fluorescent room lighting. The
camera was an inexpensive RCA Vidicon camera with automatic gain control that precluded any adjustment of the
lens aperture. The phonebook was displaced by hand
D. Experimental Results
slightly between frames. The displacement was smaller
A version of the test stimulus used by Braddick [6] was than the wavelength of the texture on the cover of the
the key test case for developing the constraint line clus- phonebook as shown by the slices through the original
tering algorithm for image flow estimation. A synthetic image pair in Fig. 8. The test images were very noisy as
image was generated with a textured square on a textured can be seen from the plots in Fig. 8. Each image was
background. The texture of the foreground square and the smoothed with a 16 by 16 point Gaussian filter. Three
background were produced by independent sampling from slices after smoothing are shown in Fig. 9. The displacea uniform random number generator in the range [0, 256). ment between the slices shows the degree of displacement
The foreground image was overlayed on the background appropriate to a motion estimation algorithm based on the
image at a position roughly in the center of the back- image flow constraint equation. Even after Gaussian filground for the first frame of the test case and at the po- tering, the quality of the motion constraint data was very
sition one pixel to the right for the second frame of the poor as can be seen by looking at the scatter plot of contest case. Uniform noise at various amplitudes was added straint line displacement versus angle in Fig. 10. Ideally,
to these test frames to simulate realistic and greater than the data should form a cosine pulse where the speed of
realistic noise levels. The additive noise was obtained by motion is the height of the pulse and the direction of moindependent sampling from a uniform random number tion corresponds to the center of the pulse; but errors due
generator in the range [ - ‘X, % 1, where the noise ampli- to- noise and occlusion are very severe. To reduce comtude 32 was some specified percentage of the maximum putation and increase the spatial extent of the constraint
pixel amplitude which was 255. After adding a noise sam- line clustering neighborhood, the d and (II arrays were
rameters that must be chosen by the user: the error tolerance, the minimum acceptable size of the consensus set,
and the number of trials. In constraint line clustering, the
consensus set is the set of intersections in the tightest interval. There is no error tolerance that must be chosen by
the user and the size of the consensus set is fixed at half
of the intersections. The fixed limit on the size of the consensus set is justified by the worst case geometry of a motion boundary in a neighborhood which in turn is justified
by reasonable assumptions on pixel spacing as described
in Section IV-B2). The user does not have to choose the
limit on the number of trials in constraint line clustering
becausethe algorithm is not iterative. Although constraint
line clustering shares many important similarities with
random sample consensus, constraint line clustering
avoids the choice of parameters and is faster since it it not
iterative. Constraint line clustering is like a one step
RANSAC algorithm that eliminates parameter choices because it is tailored to the specific problem of estimating
the image flow velocity field in situations that satisfy the
restrictions of the short-range motion process.
In principle, a random sample consensus algorithm
could be developed for this problem by randomly choosing pairs of constraint lines until an intersection is found
that is consistent with at least half of the constraint lines;
but an iterative algorithm would be more computationally
demanding than constraint line clustering and it would be
very hard to choose the error tolerance for the consensus
set. Experimental observations made during this work indicate that the width of the tightest interval in constraint
line clustering can vary greatly. In the early stages of testing the constraint line clustering algorithm, it was thought
that the width of the tightest interval could be used as a
measure to detect motion boundaries; but it was found
that correct velocity field estimates could have intervals
that were very wide and it was not possible to choose a
threshold on interval width that would identify boundary
points.
1020
IEEE
TRANSACTIONS
O N PATTERN
ANALYSIS
AND
MACHINE
INTELLIGENCE.
VOL.
II,
NO.
IO. OCTOBER
1989
Fig. 3. The two frames of a test case used frequently in this work are displayed. The test frames contain a central square that is displaced one
pixel to the right between FRAME-l and FRAME-2. This frame pair
does not contain added noise.
.\
\.
------..-s-v-.,.
,
I
--.----v-\---d--
,
I
--------
I..
.
.
.
.
~.
--
-
+
---_-_---w-d&--.
.
.
L
1,
,
-----\------
.
.-----------
.
.
’
.
.
__
---\-
w--w
.
*
*
--A-
N
.
.
,
Fig. 4. The velocity field after applying constraint line clustering to the
test case of the displaced square with 5 percent uniform noise added is
displayed. The neighborhood size was 5 by 5.
..,.,I,.
.
.
,
. .
- .
.-.
. \ *
_
.
.
I
.
.
.
.
.
.
..e..
.
.
.
,
,
,
t
--------
*
I
-----\
----
,,--
,
.
---.
N-.--c.
----:
.
.
\.
\
-e-d
.
..,.,
,
.
-.
.
.
;.-
.\
I
.
.
.
.
.
.
.
.
.
.
.
_
.
.
.
.I.,.
Fig. 6. The velocity field after applying constraint line clustering to the
test case of the displaced square with 5 percent uniform noise added is
displayed. The neighborhood size was 13 by 13. Note that there is no
significant improvement over the field displayed in either Fig. 4 or Fig.
5. The constraint line clustering algorithm performed very well; the
quality of the motion estimate was limited by the d and 01measurements.
..,...,....
* . *
. .
,...
.
.
---v
--
---
--w--4---------.
-.
,
.
.
-b-F
.
,
.,
.
..-
.
.
.
-------------\--.e.
.
.
.
-c-.-d..
-------------_.
.
\
‘.
I
r.4,
.
.
-
.
.
.
.
downsampled by a factor of 4. The result of constraint
line clustering with a 5 by 5 neighborhood is shown in
Fig. 11. Since the motion data was downsampled by a
factor of 4, the effective neighborhood size was 20 by 20.
The original images were 256 by 256 pixels before filtering and computing the motion constraints. An idea of the
relative size of the constraint line clustering neighborhood
can be found by dividing the width of the original images
by the effective width of the constraint line clustering
neighborhood: the width of the neighborhood for constraint line clustering was almost 8 percent of the image
width. Smoothed versions of the velocity field are provided in the experimental results in Section VI-D.
,.......
Fig. 5. The velocity field after applying constraint line clustering to the
test case of the displaced square with 5 percent uniform noise added is
displayed. The neighborhood size was 9 by 9. Note that there is no significant improvement over the field displayed in Fig. 4.
E. Summary
The constraint line clustering algorithm estimates the
velocity vector in successive neighborhoods by intersecting the constraint lines in a neighborhood with the con-
1021
SCHUNCK: IMAGE FLOW SEGMENTATION AND ESTIMATION
Fig. 7. A pair of real images that contains an object displaced slightly
between frames. The scene cants lined a phonebook lying on a sheet of
circuit board. The cover of the pt tonebook had an outdoor scene.
d
Fig. 8. Three pairs of horizontal slices through the images of the phonebook test case from Fig. 7 are plotted. The plots were created by overlaying the plots from the same position in the two frames of the image
pair. The images are very noisy with considerable variation in the left
hand portion of the plots which corresponds to the stationary part of the
scene.
Fig. 10. A scatter plot of constraint line displacement d versus angle IX for
the motion constraints obtained from the phonebook test case is shown.
This illustrates the poor quality of the motion data due to noise and the
effects of occlusion.
Fig. 9. Three pairs of horizontal slices through the images of the phonebook test case after smoothing with a 16 by 16 point Gaussian filter are
plotted. The displacement between frames is clearly seen as the phase
shift in the plots. This illustrates the degree of displacement appropriate
to algorithms based on the image flow constraint equation.
straint line from the center of the neighborhood and
choosing the midpoint of the tightest cluster. The tightest
cluster is the interval of smallest width along the constraint line that contains half of the intersections. This test
is motivated by the characteristics of motion boundaries.
If the motion boundary is smooth, then at least half of the
d and CYmeasurements in the neighborhood belong to the
same motion as the d and CYmeasurement at the center of
the neighborhood. It is harmless to assume the remaining
measurements are not from the same region of motion;
rejecting a helpful intersection is harmless, but accepting
a bad one can ruin the motion estimate.
V. ERROR ANALYSIS
This section provides some analysis of constraint line
clustering that explains some of the characteristics of the
algorithm that have been seen in experimental results. The
analysis assumes that the distribution of constraint line
orientations is uniform. This analysis may seem too op-
1022
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. Il. NO. IO, OCTOBER 1989
\---
.
“’
-\
.
.
,
. .
;i
_
ea.
..--\
.-_...
-7,
.
-,
\
--
\’
.c
..-
..---
\ .$;=<
I.
.C
./
\
’ ‘H
--.
\
\ .-
.
--.-.
-\,
\
.
-.
*--/---.-
\-----L--z
--
/-,
-
’
-
-L-.
I /
I
. . . NV,
--.\-.,.
-/P’r.
, .
.
-,:--<:,
?c
_.
-
.
.
:
/,...
.t
-
:
\
--
-
.
,--c.\----
. -/.
* -.-.
-
..-----,.A----.-
-
Fig. 12. A line L is displaced from a point P by distance d. Another line,
passing through P with an angle 4 between the line and the perpendicular
bisector of L, intersects line L. The distance from the perpendicular bisector of L passing through P to the point of intersection is b. The probability distribution for b is derived in the text and is part of the error
analysis of constraint line clustering.
--....
,.-.,.w,\,-
1,
,“‘-
,-.--
\
.-c-.-l.
L,.
--A-
.-
Fig. 11. The velocity field obtained by constraint line clustering with a 5
by 5 neighborhood is shown. The d and a arrays were downsampled by
a factor of 4 to increase the effective neighborhood size to 20 by 20
without increasing computation.
timistic, but the neighborhood size for constraint line
clustering can and should be large enough to provide sufficient variation in constraint line orientation for the assumption to be reasonable. Certainly the assumption is
justified for the random dot images that are the key test
case in this work. The important point addressed by the
analysis is to offer an explanation of why constraint line
clustering works in situations such as comers that violate
the assumptions of the algorithm.
surrounding the center are accurate, but the center constraint line is not.
The spread in intersection positions that will occur when
multiple valid constraint lines intersect constraint line L
must be determined. To obtain this statistic, the probability density function for the distance b along line L from
the perpendicular bisector of L, passing through P, to the
point of intersection must be determined. Assume that the
angle between the valid constraint line and the perpendicular bisector of L is 4. The distance b is given by
b = dtan+.
A. Constraint Line Intersections
The analysis begins by deriving an interesting formula
for a geometric problem that will lead us to approximate
formulas for the general case. Suppose that there are one
or more lines intersecting a given line L and that these
lines all pass through a point P, as depicted in Fig. 12.
The line L is displaced from point P by distance d along
the perpendicular bisector of L passing through P. The
probability density function for the intersection positions
will be determined. This problem corresponds to the situation where the one-dimensional cluster analysis algorithm is applied along a constraint line that is associated
with a motion, but due to errors in the constraint line coefficients the line passes a distance d from the true velocity
at point P and all constraint lines in the neighborhood of
L are error free and pass through point P. This is an important case since experimental results indicate that the
accuracy of constraint line clustering is determined predominantly by the accuracy of the estimate of the motion
constraint coefficients. If the constraint line at the center
of the neighborhood is accurate, but some of the constraint lines in the neighborhood are not, then constraint
line clustering will filter out the inaccurate intersections.
This analysis addresses the case where the constraint lines
(17)
A probability density function for b can be derived from
a probability density function for 4 by equating equal
probability measures and applying change of variables.
For any value of r$o in the range [ - 3r/2, 7r/2 1,
40
Prob [ 4 < 40] =
s
pn,2~d+)& = Problb < &I
(18)
where b = d tan +o, and
bo
Prob [b < b,] =
s
--mpb(x) dx.
(19)
By change of variables
=shl
’
--m
~&an-’
(x/d))
1 + (x/d)*
1 dx.
dd
(20)
So the probability density function for b, in terms of the
probability density function for 4 is
I%(X)= -&-p&an-l
b/d)).
(21)
To continue the analysis, assume that 4 is uniformly distributed between - 7r/2 and n/2. This assumption should
typically be valid since the size of a neighborhood should
SCHUNCK: IMAGE FLOW SEGMENTATION AND ESTIMATION
1023
be large enough to guarantee that a diverse range of con- distribution of intersections scales in proportion to the
straint line orientations lies in the neighborhood. The as- distance of the motion from the constraint line, this is an
sumption of uniform distribution in 4 leads to a Cauchy unlikely situation.
The assumptions used to formulate the test for the tightdistribution for the intersection position:
est cluster of intersections along a constraint are not valid
Pb(X)= x2
-ALL
(22) at the comers of sharp motion boundaries, but the con+ d*’
straint line clustering algorithm works well at comers.
This distribution has an unbounded second moment, so This is explained by the statistical results presented above.
the variance will not be a useful measure of spread. The At a comer, the statistics of constraint line intersections
percentage of the distribution that lies in the interval [ - 1, are given by (28). The intersections caused by constraint
+1 ] can be computed. The value of 1 that contains some lines from inside the comer form a tight cluster with a
fixed percentage (such as half) of the intersection posi- Cauchy distribution while the intersections caused by
tions can be used as a measure of the spread. The per- constraint lines from outside the comer form a broad cluscentage of intersections in the interval [ -I, + 1 ] is given ter with a Cauchy distribution. The broader Cauchy distribution trails off slowly from its central peak and apby
1
1
proximates a uniform distribution over the interval that
(23) overlaps the tight cluster of the narrower Cauchy distri-I P&) dx = 2 s o~d4 dx
s
bution. So in the vicinity of the small cluster of intersec1
1
tions from the constraint lines inside the comer there are
~
dx
= 2d/n
(24) additional intersections that are approximately uniformly
s od2 +x2
distributed and these additional intersections do not bias
(25) the location of the tightest peak in the interval. The constraint line clustering algorithm will, under the most probable circumstances, choose the location of the center of
= 2/7r tan-’ (l/d).
(26) the small, tight cluster of intersections from the constraint
The width s of the interval that contains a fraction P of lines inside the comer as the motion estimate over some
the intersections is
other location near the much broader spread of intersections from the constraint lines outside the comer.
s = 2d tan (P7r/2).
(27)
IMAGE FLOWS
VI. SMOOTHING DISCONTINUOUS
Note that the spread is proportional to d and the constant
of proportionality is 2 tan (Pr/2). This constant is about
An algorithm called surface-based smoothing for im1 for P around 30 percent and is exactly 2 for P = 50 proving the velocity field estimate of an image flow vector
percent. Above 50 percent, the constant of proportionality field is described in this section. The algorithm iteratively
increases rapidly.
computes the motion boundaries in the velocity field and
smooths the velocity field estimate between the motion
B. Multiple Motions
boundaries. An excellent velocity field estimate with sharp
When there are two motions near a constraint line, the motion boundaries can be derived from a noisy initial veintersections with the constraint line will have a distri- locity field estimate. The algorithm is based on a fundabution that is the sum of two Cauchy distributions, mental property of all intrinsic images and can be used to
weighted according to the fraction of the intersections that improve the estimate of any intrinsic image.
arise from each motion. The combined distribution is
Surface-based smoothing is able to improve the velocity estimate produced by constraint line clustering bep&Id,,
4) = rlC(bjd,) + (1 - a>W~dd.
(28) cause the velocity estimate is good enough to allow the
This result gives us an insight into the effect of interfer- motion boundaries to be detected, so smoothing can be
ence between two motions. The spread of the intersec- restricted to regions between motion boundaries. The altions from each motion will be proportional to the dis- gorithm relies on the fact that the true velocity function is
tance of the motion from the constraint line. The smooth between motion boundaries because the moving
constraint line clustering algorithm chooses the tightest surfaces in the scene that generated the image flow field
cluster of intersections that contains roughly half of the are smooth between step changes in depth.
intersections. This tends to guarantee that the cluster of
intersections that corresponds to the majority of intersec- A. Motion Boundary Detection
tions (that is, the set of intersections from the same side
The velocity field estimated by constraint line clusterof the motion boundary from which the constraint line was ing is good enough to allow any of the usual edge detectaken) will be chosen over the minority cluster. Severe tion algorithms to be used to detect the motion boundary,
interference can only take place if the majority cluster has but better performance can be obtained by using a better
a very large spread and overlaps the minority cluster suf- edge detection method. Thompson, Mutch and Berzins
ficiently to add enough intersections to the minority clus- [34] presented an algorithm for motion boundary detecter to make it seem to be the majority cluster. Since the tion that was an extension of the edge detection algorithm
1024
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. VOL. II, NO. 10, OCTOBER 1989
of Marr and Hildreth [30] and presented insights on occlusion analysis. In this work, the gradient of the Gaussian followed by nonmaxima suppression is used to detect
boundaries in the components of the image flow field [46].
After each velocity field component is smoothed with a
Gaussian filter, the Prewitt operator is used to compute
the x and y partial derivatives of each motion estimate
[47, p. 1061. The Prewitt masks are applied to the
smoothed u and z, motion data to produce four arrays of
boundary data: the x and y partial derivative approximations for the each of the two velocity field components.
The partial derivative approximations are used to compute
the gradient magnitude and angle for each velocity field
component.
Nonmaxima suppression reduces the broad ridges and
isolated mounds of edge detector output to thin curves and
edge fragments. The gradient orientation is reduced to one
of four sectors, then the magnitude of the gradient at each
point is compared to the magnitude of its two neighbors
along the line of the sector. If either of the neighbors of
the center element have large magnitudes, then the magnitude of the center element is set to zero. The total number of edge detector output values that are above some
threshold is decreased. This makes more of the velocity
field data available to the smoothing algorithm. Nonmaxima suppression also produces paths through weak boundaries that allow the velocity field estimates on opposite
sides of the boundaries to mix.
ity field estimate from the original velocity field estimate
is
(u - ii)* + (u - 8)‘.
The cost assigned to lack of smoothness is the sum of the
squares of the magnitudes of the gradients of the x and y
velocity field components,
ii; + ii; + a; + z;,
(30)
as used by Horn and Schunck [9]. The combined optimization criterion is
ss
K2(U,2 + ii; + 8; + z2Y)
+ (u - iQ2 + (u - a)* dx dy
(31)
where ~~ weights the degree of smoothing versus the degree of deviation from the original velocity field estimate.
This optimization problem is easily solved by the calculus
of variations [60, pp. 191-1923. The result is a pair of
linear partial differential equations that are decoupled:
K2v2ii
-
U =
0
(32)
K2v2ij
-
V
0.
(33)
=
The equations can be solved iteratively by approximating
the Laplacian with a nine point mask [61, p. 1281,
i 1
4 1
1
4
B. Velocity Field Smoothing
The smoothing problem is formulated as a minimization problem with two components: 1) ,a cost for the lack
of smoothness in the velocity field between motion
boundaries and 2) a cost on the deviation of the smoothed
velocity field from the velocity field estimated by constraint line clustering. This approach was motivated by
the work of Horn and Schunck [9] and Grimson [48]-[50].
Other methods for implementing the smoothing equations
are discussed by Terzopoulos [51] and an interesting discussion of viewpoint invariant surface interpolation is
provided by Blake [52]. The mathematics of smoothing
operators is presented by Brady and Horn [53]. Surface
interpolation at multiple scales of resolution is thoroughly
discussed by Terzopoulos [5 11, [54]. Markov random field
methods for surface reconstruction have been discussed
by Marroquin [55] and Geman and Geman [56]. The relationship of problems in surface reconstruction to regularization has been described by Poggio and Torre 1571.
An analog network model for the implementation of surface reconstruction algorithms based on Markov random
fields was presented by Hutchinson and Koch [58]. Blake
and Zisserman [59] present a thorough discussion of algorithms for the reconstruction of surfaces in vision.
Let u and z1 denote the x and y components of the velocity field estimate obtained by constraint line clustering,
let U and V denote the smoothed velocity field estimate.
The cost assigned to the deviation of the smoothed veloc-
(29)
v* z
Using this approximation,
ily obtained
--nfl
u
=
-n+l
=
V
-20
4 .
(34)
4 1
1
the iterative equations are eas-
Un
+
[email protected]“)]
(35)
Vn
+
K2s(an)],
(36)
where the function S ( * ) represents the computation of the
eight point surround of the approximation to the Laplacian.
These smoothing equations should only be used over
the interior of a region of slowly varying velocity. The
equations should never be applied across a motion boundary since to do so would blur the boundary. Natural
boundary conditions should be used when the support of
the approximation to the Laplacian is next to a motion
boundary. Since the shape of the motion boundary is arbitrary, it is difficult to develop and apply the approximations to the Laplacian that should be used at a motion
boundary. A simple scheme was used to approximate natural boundary conditions near a motion boundary: the velocity field value at the center of the Laplacian is used in
place of the velocity field value at any point in the support
of the Laplacian that is a boundary point. This scheme is
simple to implement and works for boundaries of arbitrary
shape.
1025
SCHUNCK: IMAGE FLOW SEGMENTATION AND ESTIMATION
C. Interleaved Detection and Smoothing
The algorithm for detecting motion boundaries and the
algorithm for smoothing the velocity field estimate between motion boundaries are interleaved so that each
computation can benefit from the improved information
provided by the other. The computation begins with the
original velocity field estimate computed by constraint line
clustering. The x and y components of the velocity field
are contained in separate arrays. The motion edge detector is applied to each velocity component to obtain separate arrays of edge information for each velocity component. One iteration of the smoothing operator is applied
in place to each component of the velocity field, using the
boundary information for that component, to obtain the
new smoothed version of each velocity field component.
The border of the arrays is filled using data from the rows
and columns just inside the border to approximate natural
boundary conditions along the perimeter of the field of
view. Subsequent iterations begin by applying the motion
boundary detection operator to the smoothed velocity field
estimate. The smoothing that was performed between motion boundaries over the interior of surfaces will have reduced the variation in the velocity field. Since the variation is less, the output of the boundary detection operator
in these regions will be weaker. Near a motion boundary,
the local variation in the velocity field will be more apparent: the velocity values on either side of the boundary
will be more consistently the same and the velocity values
on opposite sides of the boundary will be more consistently different. The velocity values in the interior of the
surfaces corresponding to objects and the background will
not wander far from the original values estimated by constraint line clustering since there is a cost on the deviation
of the smoothed velocity estimates from the original velocity estimates. This anchors the velocity estimates to
the original motion data. As successive iterations are performed, the height of the boundary operator output will
increase along a motion boundary, decrease over the interior of objects and background points between boundaries, and the smoothing of the velocity estimate will be
confined to a well within which a smooth velocity field
estimate will be achieved independent of the velocity field
values at points on the other side of the motion boundary.
D. Experimental Results
The boundary estimation and surface smoothing computation was applied to the results of constraint line clustering for the random box test case shown in Fig. 4. The
results of surface-based smoothing are displayed in Figs.
13-15. The plot in Fig. 15 shows that the maximum deflection of the estimated velocity field from the true velocity field is around 3 percent. Maximum deflection is a
better statistic than mean-squared error since extreme errors are significant. The results show that the interleaved
iterations of boundary estimation and surface smoothing
can improve the velocity field estimate and produce a
clean estimate of the motion boundary.
\
.----------4----.......
. . . I
..
......
.
---------------..
----------c----
......
---------------
.......
Fig. 13. The velocity field computed after 16 smoothing iterations have
been performed on the velocity field shown in Fig. 4 is displayed. The
motion edges were computed using the Prewitt x and y partial derivative
approximations applied to the motion components after smoothing with
an 8 by 8 Gaussian filter. The motion edge threshold was 1000 and 16.0
was used for the smoothing weight.
. .. . . . . . . ,.****................:*
:
:
.:
:
:
:
:
:
l ..
:
:
:
:
:
:
:
:.
i.
:.
:
:
:
:
:
:. . . . ,
.. :.
.
.
.
.
.
.
.
.
..
.. . . ..
......
...
Fig. 14. The plot is the boundary of the u component of the velocity field
shown in Fig. 13. Note that there are no false boundary points.
I
I
Fig. 15. Three horizontal slices through the a component of the smoothed
velocity field from Fig. 13 are graphed. The deflection of the estimated
value from the true value is no greater than 3 percent.
Surface-based smoothing was also applied to the output
of constraint line clustering for the phonebook example
shown in Fig. 11. The results of surface-based smoothing
for the phonebook test case are shown in Figs. 16 and 17.
VII. SUMMARY AND CONCLUSIONS
This paper presented work on velocity field estimation
with emphasis on image flows that may contain step discontinuities in the image irradiance or velocity field. A
1026
IEEE TRANSACTIONS ON PATTERN ANALYSlS AND MACHINE INTELLIGENCE, VOL. II, NO. 10, OCTOBER 1989
.
.
..
..
..
..
. .. .
.
.
.
.
.
.
.
.
..,
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. -Nw-w~w*-------N
.
.
.
.
.
.
.
.
.
.
-
. ---+NL--------N\
-N--N-w---------
Fig. 16. The smoothed velocity field after 16 iterations of surface-based
smoothing applied to the output of constraint line clustering from Fig.
11 for the phonebook example is shown. The smoothing parameter was
16.0 and the threshold on the motion edge operator output was 2500.
The u and v velocity components had to be scaled down by a factor of
5 to prevent integer overflow in the iterative calculations.
one-dimensional cluster analysis algorithm for the estimation of discontinuous image flows was described and a
partial analysis of errors was presented. The paper concluded with a presentation of a smoothing algorithm that
improves velocity field estimates without blurring the motion boundaries.
Two contributions important to vision research were
presented: 1) the constraint line clustering algorithm that
uses a clever statistical test to estimate the image flow
velocity field in the presence of step discontinuities and
2) the surface-based smoothing algorithm that improves
velocity field estimates without blurring motion boundaries. The constraint line clustering algorithm shows that
it is possible to formulate an algorithm with statistical
techniques that estimates an intrinsic image without losing the boundary information that is important to perception. The surface-based smoothing algorithm shows that
an improved approximation to an intrinsic image can be
obtained without blurring boundaries in the estimate of
the intrinsic image. These two results have implications
for vision research beyond image flow estimation.
ACKNOWLEDGMENT
Many thanks to my thesis committee for their support
during this work, especially B. Horn and M. Brady for
their immense technical skill and patience; R. Jain for his
encouragement and his efforts to provide an excellent research environment in the Artificial Intelligence Laboratorv at the Universitv of Michigan: and the reviewers for
.
.
.
.
Fig. 17. The boundary in the u component of the velocity field for the
phonebook example after 16 iterations is shown. The boundary is not
dense due to poor choice of threshold, but the boundary estimate is good
enough for surface-based smoothing without excessive leakage of motion data from one region to another as shown in the smoothed velocity
field in Fig. 16.
their many helpful comments, especially the references to
related work in robust statistics.
REFERENCES
B. G. Schunck, “The image flow constraint equation,” Camp. Vision, Graph. Image Proc., vol. 35, pp. 20-46, 1986.
“The motion constraint equation for optical flow,” in Proc. Int.
PI Join; Con& Pattern Recogn., 1984, pp. 20-22.
“Motion segmentation and estimation,” Ph.D. thesis, Dep.
t31 Elec’. Eng. Comput. Sci., M.I.T., Cambridge, MA, 1983.
“Image
flow continuity equations for motion and density,” in
t41
Pro;. Workshop Motion, 1986.
[51 A. Verri and T. Poggio, “Qualitative information in the optical flow,”
in Proc. DARPA Image Understanding Workshop, 1987, pp. 825834.
1610. Braddick, “A short-range process in apparent motion,” Vision
Res., vol. 14, pp. 519-527, 1974.
[71 H. Nagel, “From digital picture processing to image analysis,” in
Proc. Int. Con5 Image. Anal. Processing, 1980.
181T. Poggio, “Early vision: From computational structure to algorithms and parallel hardware,” Comp. Vision, Graph. Image Proc.,
vol. 31, pp. 139-155, 1985.
[91 B. K. P. Horn and B. G. Schunck, “Determining optical flow,” Artificial Intelligence, vol. 17, pp. 185-203, 1981.
IlO1 T. Poggio and W. Reichardt, “Visual control of orientation behavior
in the fly,” Quart. Rev. Biophys., vol. 9, pp. 377-438, 1976.
t111 W. Reichardt and T. Poggio, “Figure-ground discrimination by relative movement in the visual system of the fly,” Bio. Cybern., vol.
35, pp. 81-100, 1979.
tw E. Buckner, “Elementary movement detectors in an insect visual system,” Bio. Cybern., vol. 24, pp. 85-101, 1976.
[II
SCHUNCK: IMAGE FLOW SEGMENTATION AND ESTIMATION
[13] M. Schetzen, The Volterra and Wiener Theories of Nonlinear Systems. New York: Wiley, 1980.
coding of television pictures using
[14] B. G. Haskell, “Frame-to-frame
two-dimensional Fourier transforms,” IEEE Trans. Inform. Theory,
vol. 20, pp. 119-120, 1974.
[15] J. 0. Limb and J. A. Murphy, “Measuring the speed of moving objects from television signals,” IEEE Trans. Commun., vol. 23, pp.
474-478, 1975.
[16] -,
“Estimating the velocity of moving images in television signals,” Camp. Graph. Image Proc., vol. 4, pp. 311-327, 1975.
[17] C. Cafforio and F. Rocca, “Methods for measuring small displacements of television images,” IEEE Trans. Inform. Theory, vol. 22,
pp.573-79,
1976.
television
[I81 A. N. Netravali and J. D. Robbins, “Motion-compensated
coding: Part I,” Bell Syst. Tech. .I., vol. 58, pp. 631-670, 1979.
[19] J. A. Stuller and A. N. Netravali, “Transform domain motion estimation,” Bell Syst. Tech. J., vol. 58, pp. 1673-1702, 1979.
1201 J. A. Stuller, A. N. Netravali, and J. D. Robbins, “Interframe television coding using gain and displacement compensation,” Bell Syst.
Tech. J., vol. 59, pp. 1227-1240, 1980.
[21] R. A. Jones and H. Rashid, “Residual recursive displacement estimation,” in Proc. Conf. Pattern Recog. Zmage Processing, 1981, pp.
508-509.
[22] R. Paquin and E. Dubois, “A spatio-temporal gradient method for
estimating the displacement field in time-varying imagery,” Camp.
Vision, Graph. Image Proc., vol. 21, pp. 205-221, 1983.
[23] L. S. Davis, Z. Wu, and H. Sun, “Contour-based motion estimation,” Tech. Rep., Comput. Vision Lab., Univ. Maryland, 1981.
[24] E. C. Hildreth, “The computation of the velocity field,” Proc. Roy.
Sot. Lond. B, vol. 221, pp. 189-220, 1984.
[25] E. C. Hildreth, The Measurement of Visual Motion.
Cambridge,
MA: M.I.T. Press, 1984.
[26] E. C. Hildreth and S. Ullman, “The measurement of visual motion,”
Memo 699, Artificial Intelligence Lab., M.I.T., Cambridge, MA,
1982.
[27] S. M. Anstis, “Phi movement as a subtraction process,” Vision Res.,
vol. 10, pp. 1411-1430, 1970.
[28] S. M. Anstis and B. J. Rogers, “Illusory reversal of visual depth and
movement during changes of contrast,” Vision Res., vol. 15, pp. 957961, 1975.
[29] S. Ullman, The Interpretation of Visual Motion.
Cambridge, MA:
M.I.T. Press, 1979.
[30] D. Marr and E. C. Hildreth, “Theory of edge detection,” Proc. Roy.
Sot. Lond. B, vol. 207, pp. 187-217, 1980.
[31] K. Nakayama and J. Loomis, “Optical velocity patterns, velocitysensitive neurons, and space perception: A hypothesis,” Perception,
vol. 3, pp. 63-80, 1974.
[32] J. Batali and S. Ullman, “Motion detection and analysis,” in Proc.
DARPA Image Understand. Workshop, Nov. 1979.
[33] K. M. Mutch and W. B. Thompson, “Analysis of accretion and deletion at boundaries in dynamic scenes,” IEEE Trans. Pattern Anal.
Machine Intel., vol. 7, pp. 133-138, Mar. 1985.
[34] W. B. Thompson, K. M. Mutch, and V. A. Berzins, “Dynamic occlusion analysis in optical flow fields,” IEEE Trans. Pattern Anal.
Machine Intel., vol. 7, pp. 374-383, July 1985.
[35] C. L. Fennema and W. B. Thompson, “Velocity determination in
scenes containing several moving objects,” Comp. Graph Image
Proc., vol. 9, pp. 301-315, 1979.
[36] B. G. Schunck, “Motion segmentation and estimation by constraint
line clustering, ” in Proc. Workshop Comput. Vision, 1984, pp. 5862.
[37] D. Marr and S. Ullman, “Directional selectivity and its use in early
visual processing,” Proc. Roy. Sot. Lond. B, vol. 211, pp. 151-180,
1981.
[38] B. Noble and J. W. Daniel, Applied Linear Algebra.
Englewood
Cliffs, NJ: Prentice-Hall, 1977, 2nd ed.
[39] A. Ben-Israel and T. N. E. Greville, Generalized Inverses: Theory
and Applications.
New York: Wiley-Interscience,
1974.
New
[40] T. L. Boullion and P. L. Odell, Generalized Inverse Matrices.
York: Wiley-Interscience,
1971.
1027
[41] C. L. Lawson and R. J. Hanson, Solving Least Squares Problems.
Englewood Cliffs, NJ: Prentice-Hall, 1974.
[42] M. A. Fischler and R. C. Belles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, ” Commun. Assoc. Camp. Mach., vol. 24, pp.
381-395, June 1981.
1431 H. A. David, Order Statistics.
New York: Wiley, 1981, 2nd ed.
[44] P. J. Huber, Robust Statistics.
New York: Wiley, 1981.
[45] R. J. Settling, Approximation Theorems of Mathematical Statistics.
New York: Wiley, 1980.
1461 J. F. Canny, “A computational approach to edge detection,” IEEE
Trans. Pattern Anal. Machine Intel., vol. 8, pp. 679-698, 1986.
1471 R. Nevatia, Machine Perception.
Englewood Cliffs, NJ: PrenticeHall, 1982.
[48] W. E. L. Grimson, From Images to Surfaces: A Computational Study
of the Human Early Vision System. Cambridge, MA: M.I.T. Press,
1981.
[49] W. E. L. Grimson, “A computational theory of visual surface interpolation,” Memo 613, Artificial Intelligence Lab., M.I.T., 1981.
[SO] W. E. L. Grimson, “The implicit constraints of the primal sketch,”
Memo 663, Artificial Intelligence Lab., M.I.T., 1981.
[51] D. Terzopoulos, “Multilevel
reconstruction of visual surfaces: Variational principles and finite element representations,” Memo 671,
Artificial Intelligence Lab., M.I.T., 1982.
[52] A. Blake, “Reconstructing a visible surface,” in Proc. NUZ. Cant
Artificial Intelligence, 1984, pp. 23-26.
1531 M. Brady and B. K. P. Horn, “Rotationally symmetric operators for
surface interpolation,”
Memo 654, Artificial Intelligence Lab.,
M.I.T., Nov. 1981.
computational processes for visual sur[54] D. Terzopoulos, “Multilevel
face reconstruction,” Comp. Vision, Graph. Image Proc., vol. 24,
pp. 52-96, 1983.
[55] J. L. Marroquin, “Surface reconstruction preserving discontinuities,” Memo 792, Artificial Intelligence Lab., M.I.T., 1984.
1561 S. Geman and D. Geman, “Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images,” IEEE Trans. Pattern
Anal. Machine Intel., vol. 6, pp. 721-741, 1984.
1571 T. Poggio and V. Torre, “Ill-posed problems and regularization analysis in early vision,” Memo 773, Artificial Intelligence Lab., M.I.T.,
Apr. 1984.
[58] J. M. Hutchinson and C. Koch, “Simple analog and hybrid networks
for surface interpolation, ” in Neural Networks for Computing, J. S.
Denker, Ed. New York: Amer. Inst. Phys., 1986, pp. 235-240.
[59] A. Blake and A. Zisserman, Visual Reconstruction.
Cambridge,
MA: M.I.T. Press, 1987.
[60] R. Courant and D. Hilbert, Methods ofMathematical Physics. New
York: Wiley, 1937, vol. 1.
1611 W. F. Ames, Numerical Methods for Partial Differential Equations.
New York: Academic, 1977, 2nd ed.
Brian G. Schunck (S’75-M’83) has worked for several years on the development of systems for machine vision and industrial automation. He
was educated in computer science at the University of California, Irvine.
where he received the B.S. magna cum laude in 1976. He studied electrical
engineering, systems theory, and artificial intelligence at M.I.T. where he
received the Master’s and E.E. degrees in 1979 for work on the control of
robotic manipulators and the doctorate in 1983 for research on image flow.
He has worked for several companies on projects in machine vision,
robotics, control systems and electronics, and computer architecture. He is
currently an Assistant Professor in the Department of Electrical Engineering and Computer Science and a member of the Artificial Intelligence Laboratory at the University of Michigan, Ann Arbor. His interests include
statistical methods for machine vision and industrial inspection, qualitative
reasoning and constraint propagation in engineering design, electronic media, and marine navigation.
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement