Optica Applicata, Vol. XLV, No. 2, 2015 DOI: 10.5277/oa150207 A super depth of field height measurement based on local disparity LI CUI, YI LIU, MEI YU*, GANGYI JIANG, SHENGLI FAN, YIGANG WANG Faculty of Information Science and Engineering, Ningbo University, Ningbo 315211, China * Corresponding author: yumei@nbu.edu.cn A super depth of field height measurement method is proposed to measure the object height with the optical stereoscopic microscope. The quasi-Euclidean epipolar rectification algorithm is utilized on the original stereoimage to obtain rectified stereoimages and calibrate two camera parameters. Then, feature points are obtained by the SURF (speed up robust feature) algorithm and their corresponding disparities are calculated. The disparity-depth of field curve is fitted by combining the step height values of a stepper motor. Moreover, through local disparity value got from feature points on the object, the relative shift height is calculated through regression analysis. Finally, according to binocular vision geometry, the thickness of the object can be calculated. Experimental results show that the measurement error in Z direction is from 1.51% to 7.71%, which indicates that the proposed method is able to measure the height of a microobject beyond depth of field within a tolerant error. Keywords: microscopes, binocular vision, local disparity map, depth of field (DOF), height measurement. 1. Introduction Computer vision systems simulate human visual system for acquiring, processing, analyzing, and understanding images, and are widely used in many applications such as automation of detection, object measurement, scene reconstruction and so on. Capturing an object from different points of view in different space directions will cause disparity, which is an indispensable part of machine vision, especially for human visual perception of object shape and distance information, and is widely used in binocular vision, multiple-view vision and the motion of monocular vision. Spurred by improvements in the technology for manufacturing microlenses (smaller than 1 mm), researchers have proposed to use arrays of microlenses for computer vision applied in the domain of biomedical [1, 2], industrial inspection [3], micropositioning operations [4], etc. Multiple viewpoints can be collected in a single image using a plenoptic camera[5], in which a major step towards understanding the relations between triangulation and depth of field (DOF) has been taken. LEVIN et al. achieved the 206 LI CUI et al. recovery of an all-focus image and depth map using a simple modification to the camera’s aperture [6]. Further, based on image fusion algorithms, some methods extended the DOF of images, and are used in photogrammetric process [7]. However, this summation does not provide an accurate depth map [8]. Besides, these methods are not well applied to solve the limitation of DOF in the optical microscope. There is a lot of literature about estimating focal fuzzy [9–11], but these publications do not guarantee the accuracy of the measurements, or the equipment platform is too expensive, and not universal. Hence, a super DOF height measurement method is proposed to measure object height with a binocular optical microscope with shallow focus (small DOF). The proposed method uses the principle of binocular vision to construct a measurement model, which enables the super DOF measurement of the thickness of the printed circuit board (PCB), and effectively solves the problem of shallow focus. 2. Super DOF measurement method based on the binocular optical microscope DOF is the distance between the nearest and farthest objects in a scene that appears acceptably sharp in an image. A shallow focus problem exists in the camera and optical microscope, but for the optical microscope, the problem is more serious. If an object is within the DOF, it is sharp in the captured image, otherwise, it will be unsharp. It is no doubt that this is a disadvantage of the height measurement of the object. As illustrated in Fig. 1, the top surface of an object to be measured is within the DOF, but the working stage at the same plane of the undersurface of the object is out of the DOF, thus the height of the object is hard to be measured due to the blurred working stage in the image. In this paper, a super DOF height measurement method is proposed to measure the object height with the binocular optical microscope with shallow focus. Firstly, the sequence of original stereoscopic images taken at different lifting height is selected, the calibration error of epipolar rectification of each stereoscopic image is calculated, and two local optimal image pairs with a relatively lowest calibration error are determined and selected; additionally, the corresponding homographic matrices Hl Camera lens DOF Object Working stage Fig. 1. Microscope platform. A super depth of field height measurement... 207 and Hr of the two local optimal image pairs are calculated. Then, the selected sequence of original stereoscopic images is epipolar rectified with the Hl and Hr, and the disparity-depth of field (D-DOF) curve is fitted. Finally, the optimal epipolar rectification is implemented on images to be measured, and according to the twice measured local disparity values of the rectified image pair, the distance from the top surface or the undersurface of the object to the camera lens with respect to the rectified image pair can be obtained. When it is combined with the step height values of a stepper motor, the thickness of the object can be calculated. 2.1. Epipolar rectification Image matching is important for a stereoscopic image analysis. However, match points may fall near epipolar due to the noise, which is harmful to a subsequent series of image processing. According to epipolar geometric relationships, the epipolar constraint is the only constraint that is not dependent on the scene. Therefore, the epipolar constraint is used to rectify the error, estimate and improve the matching accuracy, and finally obtain the match points. On the basis of the Euclidean geometry and the horizontal epipolar, looking for a fundamental matrix between the original images F = H rT [ u 1 ] x H l , with a pair of –1 –1 homographic matrices H r = K nr R r K or , H l = K nl R l K ol , where (Kol, Kor) and (Knl, Knr) are the old intrinsic parameters and new intrinsic parameters, respectively, Kol and Kor are defined as K ol = K or = fx 0 w/2 0 fy h/2 0 0 1 (1) where fx and fy are horizontal and vertical focal length, w and h are the width and height of the image, the rectified image pair can be obtained with the squared Sampson error which is defined as follows for the j-th correspondence T 2 Ej 2 ( m rj F m lj ) = -------------------------------------------------------------------------------------------------------T T 2 2 ( F m lj )21 + ( F m lj )22 + ( m rj F ) 1 + ( m rj F ) 2 (2) where (∗)i is the i-th component of the normalized vector. The detail can be found in [12]. As an example, Fig. 2 shows the rectification error with respect to a sequence of stereoscopic images captured with a binocular optical microscope. The lifting step unit is 60 μm. The 0-th stereoscopic image corresponds to the relatively sharpest stereoscopic image, and the images between the –8th and the 8th are sharp enough, while the others are unsharp images. From the figure, it is clear that the –7th and the 5th ste- 208 LI CUI et al. 3.0 Rectify error 2.5 2.0 1.5 1.0 0.5 0.0 –20 –10 0 10 20 Picture sequence Fig. 2. Rectification error of image sequence. reoscopic images marked with black points are the two ones with the relatively lowest rectification errors among the sharp images. The squared Sampson errors of the two images are 0.285676 and 0.263863, respectively. Thus, the rectification parameters of these two stereoscopic images, that is, the pair of homographic matrices Hr and Hl of these two stereoscopic images will finally be utilized to implement epipolar rectification on the sequence of the original stereoscopic images. 2.2. D-DOF curve fitting The purpose of using a pair of homographic matrices Hr and Hl with respect to the two stereoscopic images with relatively lowest squared Sampson errors to rectify the sequence of the original stereoscopic images is to reduce the vertical disparity of stereoscopic images, so as to get the equivalent images taken with the parallel binocular imaging system. Then, the feature points of the rectified stereoscopic images can be obtained by using the speed up robust feature (SURF) algorithm [13], which uses an integer approximation to the determinant of a Hessian blob detector, and can be computed extremely quickly with an integral image. After that, using the fast library for approximate nearest neighbors (FLANN) algorithm to match feature points [14], the random sample consensus (RANSAC) algorithm is used to eliminate some false matching points [15]. Finally, according to the rest matched local feature points, the average disparity can be calculated. Moreover, the D-DOF curve can be fitted according to the average disparities with respect to stereoscopic images taken at different lifting height. As an example, Fig. 3 shows the local matched feature points in stereoscopic images of PCB to be measured. In Fig. 3, the feature points are determined with the SURF algorithm [13], which uses the determinant of the Hessian for selecting the location and the scale. A non-maximum suppression in a 3×3×3 neighborhood is A super depth of field height measurement... 209 Fig. 3. ROI feature points matching. 120 Disparity value 100 Mean 1 Optimized 1 Mean 2 Optimized 2 80 60 40 –20 –10 0 10 20 Image sequence Fig. 4. D-DOF folding lines. applied to localize interest points in the image and over scales, and the maxima of the determinant of the Hessian matrix are then interpolated in the scale and image space. For the left pairs of stereoscopic images in Fig. 3, the focus is on the top surface of the PCB, while for the right ones, the bottom of the hole, that is, the undersurface of the PCB is sharp but the top surface is blurred. Figure 4 gives the folding line of the D-DOF. The x-coordinate “image sequence” is associated with the lifting height, that is, the depth, which is controlled with a stepper motor of the microscope. In the figure, mean 1 and mean 2 are D-DOF folding lines with respect to two groups of stereoscopic images rectified with Hl and Hr obtained from the two stereoscopic images with two relatively lowest squared Sampson errors. Optimized 1 and optimized 2 are D-DOF folding lines of the same two groups of rectified stereoscopic images but after eliminating some false matching points with the RANSAC algorithm. Since the undulation of surface or even the thickness of the PCB are quite small compared 210 LI CUI et al. with the focal length and baseline distance of the microscope, the disparities of points of the PCB are almost the same. Thus, the so-called false matching points are the points whose estimated disparity departs quite a lot from that of the others due to inaccurate disparity estimation. According to the binocular vision geometry technique, the optimized curve with better linearity is finally selected as the D-DOF measuring curve. Then, the slope k and intercept b of the straight line D = kZ + b can be got through fitting, where Z and D are depth and disparity, respectively. 2.3. Height measurement The thickness of the PCB can be measured through double measurements, as shown in Fig. 5. Figures 5a and 5b illustrate two possible situations with respect to the stepper motor lifting down and up, respectively. In Fig. 5a, the step 1 is used to determine the working stage, that is, the undersurface of the object, thus it should guarantee that the working stage is within the DOF, and the corresponding image is the sharpest one for the working stage. Similarly, the step 2 is for the top surface of the object and the corresponding image has the sharpest top surface of the object. In Fig. 5b, the situation is the opposite. The two measurements are used to measure the distance from the top surface of the object to the lens and the distance from the working stage to the lens, Step 1 a Step 2 Z2 Z1 H Step 1 h b Step 2 Z2 Z1 h H Fig. 5. Principle diagram of height calculation. Stepper motor lifts down (a) and lifts up (b). A super depth of field height measurement... 211 respectively, and these two distances are denoted as Z1 or Z2, as illustrated in Fig. 5. As the stepper motor can lift up and down, the measuring distance formula is derived by the geometric relationship shown in Fig. 5 as follows: h + Z1 – Z2 , H = h + Z2 – Z1 , if stepper motor lifts down if stepper motor lifts up (3) where H is PCB thickness to be measured, h is the lifting distance of the stepper motor, Z1 and Z2 can be calculated by the fitted curve of D-DOF Z = (D – b)/k. 3. Experimental results The stereo- or stereoscopic microscope is an optical microscope variant designed for low magnification observation of a sample, typically using light reflected from the surface of an object rather than transmitted through it. The instrument uses two separate optical paths with two objectives and eyepieces to provide slightly different viewing angles to the left and right eye. A B Fig. 6. Experimental platform. A – a dual camera with an independent light path, B – a lifting platform, controlled by a stepper motor with the stepper accuracy of 2 μm auto-adjusted In this paper, the experimental platform uses a stereomicroscope of ZOOM460N, as shown in Fig. 6. The lights of the observed object first go through a large lens, then go into two separate light paths, go across the objective lens, and then the lights are captured on both sides of the relative camera. The used cameras are 902B, produced by the Japanese company Watec. The cameras are directly connected to the eyepiece of a stereomicroscope through an interface. The resolution of the captured stereoscopic image is 704×576. The statistical analysis is under the condition of the fixed focus. As shown in Fig. 4, the “optimized 1” folding line, which is obtained by using the corresponding Hl and Hr of the 5th stereoscopic image to rectify the sequence of original stereoscopic images, achieves the best linearity. According to the relationship between the disparity and 212 LI CUI et al. a b c d Fig. 7. Rectified PCB images out of DOF. Top surface (a, d) and undersurface (b, c) of PCB. the DOF, fitting parameters k = 0.101899 and b = 325.859 can be obtained, and the corresponding Hl and Hr are as follows: Hl = 0.9913676381 – 0.1188631207 18.63250542 0.117786698 0.9930397272 – 19.73069954 – 8.741740203 × 10 Hr = –6 1.048118179 × 10 –6 1.001370072 0.9822682738 – 0.1271616966 21.25362015 0.1137051731 0.9886323214 – 3.925105333 – 6.22121733 × 10 –5 – 2.484930337 × 10 –5 (4) (5) 1.013513684 Figure 7 shows that the images of a PCB whose thickness is 1700 μm, exceed the DOF of the microscope. In Figs. 7a and 7d, the top surface of the PCB to be measured is sharp, while in Figs. 7b and 7c the undersurface of the PCB (the bottom of the holes) is sharp. According to the stereoscopic images of the PCB in Fig. 7, local T a b l e 1. Parameters of ROI feature points. xl xr yl yr 283.981 405.271 43.602 318.262 348.720 123.546 332.661 202.440 465.359 583.052 217.880 509.010 343.781 113.655 325.177 194.649 d 187.727 190.498 183.901 200.765 (see Fig. 7a) (see Fig. 7b) (see Fig. 7c) (see Fig. 7d) Note: xl and xr are the average x-coordinate values of local matching points in the left and right view images; yl and yr are the average y-coordinate values of local matching points in left and right view images; d is the disparity value. A super depth of field height measurement... 213 T a b l e 2. Results of measurement. Z1 [μm] 10080.33 10005.23 Z2 [μm] 10134.72 10336.23 h [μm] 1620.00 1500.00 H [μm] 1674.39 1831.00 Ed –4.71% 11.76% Ealg –1.51% 7.71% Note: Z1 and Z2 are the distance from the top surface or undersurface of the object to the lens, which can be obtained from the fitted D-DOF curve; h is the lifting height of stepper motor; H is the height calculated by the proposed method; Ed is measurement error of h if it is directly regarded as the thickness of PCB; Ealg is the measurement error with respect to the proposed method. disparity can be obtained through binocular stereo-matching. Then, the average coordinates and corresponding disparities of local matching points can be calculated, as shown in Table 1. The final results of measurements, according to the proposed method, are shown in Table 2 which also gives the thicknesses directly measured according to the lifting height of the stepper motor. The thickness of the PCB shown in Figs. 7a and 7b, estimated as the lifting height, is (6080 – 5270) × 2 μm = 1620 μm, while the thickness of the PCB in Figs. 7c and 7d is (6080 – 5330) × 2 μm = 1500 μm, in which the numbers 6080, 5270 and 5330 are the corresponding step heights. Obviously, these two values deviate from the actual thickness of the PCB 1700 μm quite a lot. In the experiments, the stereoscopic images were taken with the fixed focus. As can be seen from Table 2, if directly regarding the lifting height of the stepper motor as the thickness of the PCB, the measurement error is unacceptable, because it is hard to determine the sharpest plane manually even within the DOF. By contrast, the measured thickness of the PCB for Figs. 7a and 7b obtained with the proposed method is 1674.39 μm, and the value is 1831.00 μm for Figs. 7c and 7d, and the corresponding measurement errors are –1.51% and 7.71%, respectively. The accuracy of the measurement, compared with the method directly using the lifting height of the stepper motor, is improved by 3.20% and 4.06% in the proposed method, respectively. 4. Conclusion In this paper, based on local disparity, a super depth of the field measurement method is proposed to measure the height of the object exceeding the DOF of a microscope. The proposed method can improve the accuracy of measurement effectively. However, there are still 1.51%–7.71% or so measurement errors. These errors may be caused by the matching accuracy, which influences the accuracy of the initial calibration curve. Even though, the proposed super DOF height measurement by using the regression analysis of local disparity can be used in non-contact height measurement of tiny artifacts within the tolerance error. Acknowledgements – This work is supported by the Natural Science Foundation of China (Grant Nos. 61271270, 61311140262, U1301257), Natural Science Foundation of Zhejiang (Y1101240), and Natural Science Foundation of Ningbo (2011A610200, 2011A610197). 214 LI CUI et al. References [1] ZHENXING HU, HUIYANG LUO, YINGJIE DU, HONGBING LU, Fluorescent stereo microscopy for 3D surface profilometry and deformation mapping, Optics Express 21(10), 2013, pp. 11808–11818. [2] OANCEA R., VASILE L., MARCHESE C., SAVA-ROSIANU R., Stereomicroscopic study of the human tooth caries - clinical and morphological correlations, Proceedings of SPIE 8427, 2012, article 842740. [3] LEE M.P., GIBSON G.M., PHILLIPS D., PADGETT M.J., TASSIERI M., Dynamic stereo microscopy for studying particle sedimentation, Optics Express 22(4), 2014, pp. 4671–4677. [4] JUNG HYUN KIM, Visually guided 3D micro positioning and alignment system, International Journal of Precision Engineering and Manufacturing 12(5), 2011, pp. 797–803. [5] ADELSON E.H., WANG J.Y.A., Single lens stereo with a plenoptic camera, IEEE Transactions on Pattern Analysis and Machine Intelligence 14(2), 1992, pp. 99–106. [6] LEVIN A., FERGUS R., DURAND F., FREEMAN W.T., Image and depth from a conventional camera with a coded aperture, ACM Transactions on Graphics (TOG) 26(3), 2007, article 70. [7] GALLO A., MUZZUPAPPA M., BRUNO F., 3D reconstruction of small sized objects from a sequence of multi-focused images, Journal of Cultural Heritage 15(2), 2014, pp. 173–182. [8] IKHYUN LEE, MUHAMMAD TARIQ MAHMOOD, TAE-SUN CHOI, Adaptive window selection for 3D shape recovery from image focus, Optics and Laser Technology 45, 2013, pp. 21–31. [9] SHAOJIE ZHUO, SIM T., Defocus map estimation from a single image, Pattern Recognition 44(9), 2011, pp. 1852–1858. [10] TARKAN AYDIN, YUSUF SINAN AKGUL, A New Adaptive Focus Measure for Shape From Focus, BMVC, 2008. [11] YIBIN TIAN, Autofocus using image phase congruency, Optics Express 19(1), 2011, pp. 261–270. [12] FUSIELLO A., IRSARA L., Quasi-Euclidean epipolar rectification of uncalibrated images, Machine Vision and Applications 22(4), 2011, pp. 663–670. [13] BAY H., ESS A., TUYTELAARS T., VAN GOOL L., SURF: speeded up robust features, Computer Vision and Image Understanding (CVIU) 110(3), 2008, pp. 346–359. [14] MUJA M., LOWE D.G., Fast approximate nearest neighbors with automatic algorithm configuration, [In] International Conference on Computer Vision Theory and Applications (VISAPP ’09), 2009. [15] FISCHLER M.A., BOLLES R.C., Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography, Communications of the ACM 24(6), 1981, pp. 381–395. Received November 2, 2014 in revised form January 7, 2015

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertising