A super depth of field height measurement based on local disparity

A super depth of field height measurement based on local disparity
Optica Applicata, Vol. XLV, No. 2, 2015
DOI: 10.5277/oa150207
A super depth of field height measurement
based on local disparity
LI CUI, YI LIU, MEI YU*, GANGYI JIANG, SHENGLI FAN, YIGANG WANG
Faculty of Information Science and Engineering, Ningbo University,
Ningbo 315211, China
*
Corresponding author: yumei@nbu.edu.cn
A super depth of field height measurement method is proposed to measure the object height with
the optical stereoscopic microscope. The quasi-Euclidean epipolar rectification algorithm is
utilized on the original stereoimage to obtain rectified stereoimages and calibrate two camera
parameters. Then, feature points are obtained by the SURF (speed up robust feature) algorithm
and their corresponding disparities are calculated. The disparity-depth of field curve is fitted by
combining the step height values of a stepper motor. Moreover, through local disparity value got
from feature points on the object, the relative shift height is calculated through regression analysis.
Finally, according to binocular vision geometry, the thickness of the object can be calculated.
Experimental results show that the measurement error in Z direction is from 1.51% to 7.71%, which
indicates that the proposed method is able to measure the height of a microobject beyond depth of
field within a tolerant error.
Keywords: microscopes, binocular vision, local disparity map, depth of field (DOF), height measurement.
1. Introduction
Computer vision systems simulate human visual system for acquiring, processing,
analyzing, and understanding images, and are widely used in many applications such
as automation of detection, object measurement, scene reconstruction and so on. Capturing an object from different points of view in different space directions will cause
disparity, which is an indispensable part of machine vision, especially for human visual
perception of object shape and distance information, and is widely used in binocular
vision, multiple-view vision and the motion of monocular vision.
Spurred by improvements in the technology for manufacturing microlenses (smaller
than 1 mm), researchers have proposed to use arrays of microlenses for computer vision applied in the domain of biomedical [1, 2], industrial inspection [3], micropositioning operations [4], etc. Multiple viewpoints can be collected in a single image using
a plenoptic camera[5], in which a major step towards understanding the relations between triangulation and depth of field (DOF) has been taken. LEVIN et al. achieved the
206
LI CUI et al.
recovery of an all-focus image and depth map using a simple modification to the camera’s aperture [6]. Further, based on image fusion algorithms, some methods extended
the DOF of images, and are used in photogrammetric process [7]. However, this summation does not provide an accurate depth map [8]. Besides, these methods are not
well applied to solve the limitation of DOF in the optical microscope. There is a lot of
literature about estimating focal fuzzy [9–11], but these publications do not guarantee
the accuracy of the measurements, or the equipment platform is too expensive, and not
universal. Hence, a super DOF height measurement method is proposed to measure
object height with a binocular optical microscope with shallow focus (small DOF).
The proposed method uses the principle of binocular vision to construct a measurement
model, which enables the super DOF measurement of the thickness of the printed circuit board (PCB), and effectively solves the problem of shallow focus.
2. Super DOF measurement method
based on the binocular optical microscope
DOF is the distance between the nearest and farthest objects in a scene that appears
acceptably sharp in an image. A shallow focus problem exists in the camera and optical
microscope, but for the optical microscope, the problem is more serious. If an object
is within the DOF, it is sharp in the captured image, otherwise, it will be unsharp. It
is no doubt that this is a disadvantage of the height measurement of the object. As illustrated in Fig. 1, the top surface of an object to be measured is within the DOF, but
the working stage at the same plane of the undersurface of the object is out of the DOF,
thus the height of the object is hard to be measured due to the blurred working stage
in the image.
In this paper, a super DOF height measurement method is proposed to measure
the object height with the binocular optical microscope with shallow focus. Firstly,
the sequence of original stereoscopic images taken at different lifting height is selected, the calibration error of epipolar rectification of each stereoscopic image is calculated, and two local optimal image pairs with a relatively lowest calibration error are
determined and selected; additionally, the corresponding homographic matrices Hl
Camera lens
DOF
Object
Working stage
Fig. 1. Microscope platform.
A super depth of field height measurement...
207
and Hr of the two local optimal image pairs are calculated. Then, the selected sequence
of original stereoscopic images is epipolar rectified with the Hl and Hr, and the disparity-depth of field (D-DOF) curve is fitted. Finally, the optimal epipolar rectification
is implemented on images to be measured, and according to the twice measured local
disparity values of the rectified image pair, the distance from the top surface or the
undersurface of the object to the camera lens with respect to the rectified image pair
can be obtained. When it is combined with the step height values of a stepper motor,
the thickness of the object can be calculated.
2.1. Epipolar rectification
Image matching is important for a stereoscopic image analysis. However, match points
may fall near epipolar due to the noise, which is harmful to a subsequent series of image
processing. According to epipolar geometric relationships, the epipolar constraint is
the only constraint that is not dependent on the scene. Therefore, the epipolar constraint
is used to rectify the error, estimate and improve the matching accuracy, and finally
obtain the match points.
On the basis of the Euclidean geometry and the horizontal epipolar, looking for
a fundamental matrix between the original images F = H rT [ u 1 ] x H l , with a pair of
–1
–1
homographic matrices H r = K nr R r K or
, H l = K nl R l K ol
, where (Kol, Kor) and
(Knl, Knr) are the old intrinsic parameters and new intrinsic parameters, respectively,
Kol and Kor are defined as
K ol = K or =
fx
0
w/2
0
fy
h/2
0
0
1
(1)
where fx and fy are horizontal and vertical focal length, w and h are the width and height
of the image, the rectified image pair can be obtained with the squared Sampson error
which is defined as follows for the j-th correspondence
T
2
Ej
2
( m rj F m lj )
= -------------------------------------------------------------------------------------------------------T
T
2
2
( F m lj )21 + ( F m lj )22 + ( m rj F ) 1 + ( m rj F ) 2
(2)
where (∗)i is the i-th component of the normalized vector. The detail can be found in [12].
As an example, Fig. 2 shows the rectification error with respect to a sequence of
stereoscopic images captured with a binocular optical microscope. The lifting step unit
is 60 μm. The 0-th stereoscopic image corresponds to the relatively sharpest stereoscopic image, and the images between the –8th and the 8th are sharp enough, while
the others are unsharp images. From the figure, it is clear that the –7th and the 5th ste-
208
LI CUI et al.
3.0
Rectify error
2.5
2.0
1.5
1.0
0.5
0.0
–20
–10
0
10
20
Picture sequence
Fig. 2. Rectification error of image sequence.
reoscopic images marked with black points are the two ones with the relatively lowest
rectification errors among the sharp images. The squared Sampson errors of the two
images are 0.285676 and 0.263863, respectively. Thus, the rectification parameters of
these two stereoscopic images, that is, the pair of homographic matrices Hr and Hl of
these two stereoscopic images will finally be utilized to implement epipolar rectification on the sequence of the original stereoscopic images.
2.2. D-DOF curve fitting
The purpose of using a pair of homographic matrices Hr and Hl with respect to the two
stereoscopic images with relatively lowest squared Sampson errors to rectify the sequence of the original stereoscopic images is to reduce the vertical disparity of stereoscopic images, so as to get the equivalent images taken with the parallel binocular
imaging system. Then, the feature points of the rectified stereoscopic images can be
obtained by using the speed up robust feature (SURF) algorithm [13], which uses
an integer approximation to the determinant of a Hessian blob detector, and can be
computed extremely quickly with an integral image. After that, using the fast library
for approximate nearest neighbors (FLANN) algorithm to match feature points [14],
the random sample consensus (RANSAC) algorithm is used to eliminate some false
matching points [15]. Finally, according to the rest matched local feature points,
the average disparity can be calculated. Moreover, the D-DOF curve can be fitted according to the average disparities with respect to stereoscopic images taken at different
lifting height.
As an example, Fig. 3 shows the local matched feature points in stereoscopic
images of PCB to be measured. In Fig. 3, the feature points are determined with
the SURF algorithm [13], which uses the determinant of the Hessian for selecting
the location and the scale. A non-maximum suppression in a 3×3×3 neighborhood is
A super depth of field height measurement...
209
Fig. 3. ROI feature points matching.
120
Disparity value
100
Mean 1
Optimized 1
Mean 2
Optimized 2
80
60
40
–20
–10
0
10
20
Image sequence
Fig. 4. D-DOF folding lines.
applied to localize interest points in the image and over scales, and the maxima of
the determinant of the Hessian matrix are then interpolated in the scale and image
space. For the left pairs of stereoscopic images in Fig. 3, the focus is on the top surface
of the PCB, while for the right ones, the bottom of the hole, that is, the undersurface
of the PCB is sharp but the top surface is blurred. Figure 4 gives the folding line of
the D-DOF. The x-coordinate “image sequence” is associated with the lifting height,
that is, the depth, which is controlled with a stepper motor of the microscope. In the
figure, mean 1 and mean 2 are D-DOF folding lines with respect to two groups of
stereoscopic images rectified with Hl and Hr obtained from the two stereoscopic images
with two relatively lowest squared Sampson errors. Optimized 1 and optimized 2
are D-DOF folding lines of the same two groups of rectified stereoscopic images but
after eliminating some false matching points with the RANSAC algorithm. Since
the undulation of surface or even the thickness of the PCB are quite small compared
210
LI CUI et al.
with the focal length and baseline distance of the microscope, the disparities of points
of the PCB are almost the same. Thus, the so-called false matching points are the points
whose estimated disparity departs quite a lot from that of the others due to inaccurate
disparity estimation. According to the binocular vision geometry technique, the optimized curve with better linearity is finally selected as the D-DOF measuring curve.
Then, the slope k and intercept b of the straight line D = kZ + b can be got through fitting, where Z and D are depth and disparity, respectively.
2.3. Height measurement
The thickness of the PCB can be measured through double measurements, as shown
in Fig. 5. Figures 5a and 5b illustrate two possible situations with respect to the stepper
motor lifting down and up, respectively. In Fig. 5a, the step 1 is used to determine
the working stage, that is, the undersurface of the object, thus it should guarantee that
the working stage is within the DOF, and the corresponding image is the sharpest one
for the working stage. Similarly, the step 2 is for the top surface of the object and the
corresponding image has the sharpest top surface of the object. In Fig. 5b, the situation
is the opposite. The two measurements are used to measure the distance from the top
surface of the object to the lens and the distance from the working stage to the lens,
Step 1
a
Step 2
Z2
Z1
H
Step 1
h
b
Step 2
Z2
Z1
h
H
Fig. 5. Principle diagram of height calculation. Stepper motor lifts down (a) and lifts up (b).
A super depth of field height measurement...
211
respectively, and these two distances are denoted as Z1 or Z2, as illustrated in Fig. 5.
As the stepper motor can lift up and down, the measuring distance formula is derived
by the geometric relationship shown in Fig. 5 as follows:
 h + Z1 – Z2 ,
H = 
 h + Z2 – Z1 ,
if stepper motor lifts down
if stepper motor lifts up
(3)
where H is PCB thickness to be measured, h is the lifting distance of the stepper motor,
Z1 and Z2 can be calculated by the fitted curve of D-DOF Z = (D – b)/k.
3. Experimental results
The stereo- or stereoscopic microscope is an optical microscope variant designed for
low magnification observation of a sample, typically using light reflected from the surface of an object rather than transmitted through it. The instrument uses two separate
optical paths with two objectives and eyepieces to provide slightly different viewing
angles to the left and right eye.
A
B
Fig. 6. Experimental platform. A – a dual camera with
an independent light path, B – a lifting platform,
controlled by a stepper motor with the stepper accuracy
of 2 μm auto-adjusted
In this paper, the experimental platform uses a stereomicroscope of ZOOM460N,
as shown in Fig. 6. The lights of the observed object first go through a large lens, then
go into two separate light paths, go across the objective lens, and then the lights are
captured on both sides of the relative camera. The used cameras are 902B, produced
by the Japanese company Watec. The cameras are directly connected to the eyepiece
of a stereomicroscope through an interface. The resolution of the captured stereoscopic
image is 704×576.
The statistical analysis is under the condition of the fixed focus. As shown in Fig. 4,
the “optimized 1” folding line, which is obtained by using the corresponding Hl and
Hr of the 5th stereoscopic image to rectify the sequence of original stereoscopic images,
achieves the best linearity. According to the relationship between the disparity and
212
LI CUI et al.
a
b
c
d
Fig. 7. Rectified PCB images out of DOF. Top surface (a, d) and undersurface (b, c) of PCB.
the DOF, fitting parameters k = 0.101899 and b = 325.859 can be obtained, and the
corresponding Hl and Hr are as follows:
Hl =
0.9913676381
– 0.1188631207
18.63250542
0.117786698
0.9930397272
– 19.73069954
– 8.741740203 × 10
Hr =
–6
1.048118179 × 10
–6
1.001370072
0.9822682738
– 0.1271616966
21.25362015
0.1137051731
0.9886323214
– 3.925105333
– 6.22121733 × 10
–5
– 2.484930337 × 10
–5
(4)
(5)
1.013513684
Figure 7 shows that the images of a PCB whose thickness is 1700 μm, exceed
the DOF of the microscope. In Figs. 7a and 7d, the top surface of the PCB to be measured is sharp, while in Figs. 7b and 7c the undersurface of the PCB (the bottom of
the holes) is sharp. According to the stereoscopic images of the PCB in Fig. 7, local
T a b l e 1. Parameters of ROI feature points.
xl
xr
yl
yr
283.981
405.271
43.602
318.262
348.720
123.546
332.661
202.440
465.359
583.052
217.880
509.010
343.781
113.655
325.177
194.649
d
187.727
190.498
183.901
200.765
(see Fig. 7a)
(see Fig. 7b)
(see Fig. 7c)
(see Fig. 7d)
Note: xl and xr are the average x-coordinate values of local matching points in the left and right view
images; yl and yr are the average y-coordinate values of local matching points in left and right view images;
d is the disparity value.
A super depth of field height measurement...
213
T a b l e 2. Results of measurement.
Z1 [μm]
10080.33
10005.23
Z2 [μm]
10134.72
10336.23
h [μm]
1620.00
1500.00
H [μm]
1674.39
1831.00
Ed
–4.71%
11.76%
Ealg
–1.51%
7.71%
Note: Z1 and Z2 are the distance from the top surface or undersurface of the object to the lens, which can
be obtained from the fitted D-DOF curve; h is the lifting height of stepper motor; H is the height calculated
by the proposed method; Ed is measurement error of h if it is directly regarded as the thickness of PCB;
Ealg is the measurement error with respect to the proposed method.
disparity can be obtained through binocular stereo-matching. Then, the average coordinates and corresponding disparities of local matching points can be calculated, as
shown in Table 1. The final results of measurements, according to the proposed method, are shown in Table 2 which also gives the thicknesses directly measured according
to the lifting height of the stepper motor. The thickness of the PCB shown in Figs. 7a
and 7b, estimated as the lifting height, is (6080 – 5270) × 2 μm = 1620 μm, while
the thickness of the PCB in Figs. 7c and 7d is (6080 – 5330) × 2 μm = 1500 μm, in
which the numbers 6080, 5270 and 5330 are the corresponding step heights. Obviously,
these two values deviate from the actual thickness of the PCB 1700 μm quite a lot.
In the experiments, the stereoscopic images were taken with the fixed focus. As
can be seen from Table 2, if directly regarding the lifting height of the stepper motor
as the thickness of the PCB, the measurement error is unacceptable, because it is hard
to determine the sharpest plane manually even within the DOF. By contrast, the measured thickness of the PCB for Figs. 7a and 7b obtained with the proposed method is
1674.39 μm, and the value is 1831.00 μm for Figs. 7c and 7d, and the corresponding
measurement errors are –1.51% and 7.71%, respectively. The accuracy of the measurement, compared with the method directly using the lifting height of the stepper
motor, is improved by 3.20% and 4.06% in the proposed method, respectively.
4. Conclusion
In this paper, based on local disparity, a super depth of the field measurement method
is proposed to measure the height of the object exceeding the DOF of a microscope.
The proposed method can improve the accuracy of measurement effectively. However,
there are still 1.51%–7.71% or so measurement errors. These errors may be caused by
the matching accuracy, which influences the accuracy of the initial calibration curve.
Even though, the proposed super DOF height measurement by using the regression
analysis of local disparity can be used in non-contact height measurement of tiny artifacts within the tolerance error.
Acknowledgements – This work is supported by the Natural Science Foundation of China (Grant Nos.
61271270, 61311140262, U1301257), Natural Science Foundation of Zhejiang (Y1101240), and Natural
Science Foundation of Ningbo (2011A610200, 2011A610197).
214
LI CUI et al.
References
[1] ZHENXING HU, HUIYANG LUO, YINGJIE DU, HONGBING LU, Fluorescent stereo microscopy for
3D surface profilometry and deformation mapping, Optics Express 21(10), 2013, pp. 11808–11818.
[2] OANCEA R., VASILE L., MARCHESE C., SAVA-ROSIANU R., Stereomicroscopic study of the human tooth
caries - clinical and morphological correlations, Proceedings of SPIE 8427, 2012, article 842740.
[3] LEE M.P., GIBSON G.M., PHILLIPS D., PADGETT M.J., TASSIERI M., Dynamic stereo microscopy for
studying particle sedimentation, Optics Express 22(4), 2014, pp. 4671–4677.
[4] JUNG HYUN KIM, Visually guided 3D micro positioning and alignment system, International Journal
of Precision Engineering and Manufacturing 12(5), 2011, pp. 797–803.
[5] ADELSON E.H., WANG J.Y.A., Single lens stereo with a plenoptic camera, IEEE Transactions on
Pattern Analysis and Machine Intelligence 14(2), 1992, pp. 99–106.
[6] LEVIN A., FERGUS R., DURAND F., FREEMAN W.T., Image and depth from a conventional camera with
a coded aperture, ACM Transactions on Graphics (TOG) 26(3), 2007, article 70.
[7] GALLO A., MUZZUPAPPA M., BRUNO F., 3D reconstruction of small sized objects from a sequence of
multi-focused images, Journal of Cultural Heritage 15(2), 2014, pp. 173–182.
[8] IKHYUN LEE, MUHAMMAD TARIQ MAHMOOD, TAE-SUN CHOI, Adaptive window selection for 3D shape
recovery from image focus, Optics and Laser Technology 45, 2013, pp. 21–31.
[9] SHAOJIE ZHUO, SIM T., Defocus map estimation from a single image, Pattern Recognition 44(9), 2011,
pp. 1852–1858.
[10] TARKAN AYDIN, YUSUF SINAN AKGUL, A New Adaptive Focus Measure for Shape From Focus,
BMVC, 2008.
[11] YIBIN TIAN, Autofocus using image phase congruency, Optics Express 19(1), 2011, pp. 261–270.
[12] FUSIELLO A., IRSARA L., Quasi-Euclidean epipolar rectification of uncalibrated images, Machine
Vision and Applications 22(4), 2011, pp. 663–670.
[13] BAY H., ESS A., TUYTELAARS T., VAN GOOL L., SURF: speeded up robust features, Computer Vision
and Image Understanding (CVIU) 110(3), 2008, pp. 346–359.
[14] MUJA M., LOWE D.G., Fast approximate nearest neighbors with automatic algorithm configuration,
[In] International Conference on Computer Vision Theory and Applications (VISAPP ’09), 2009.
[15] FISCHLER M.A., BOLLES R.C., Random sample consensus: a paradigm for model fitting with
applications to image analysis and automated cartography, Communications of the ACM 24(6),
1981, pp. 381–395.
Received November 2, 2014
in revised form January 7, 2015
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertising