SubmittedtoPLOSOneJournalonNovember3,2016 Feature Enhancement in Visually Impaired Images 1 Madhuri Suthar1*, Mohammad Asghari1 and Bahram Jalali1,2,3,4 Department of Electrical Engineering, University of California Los Angeles, Los Angeles, California, USA 2 3 4 California NanoSystems Institute, Los Angeles, California, USA Department of Bioengineering, University of California Los Angeles, Los Angeles, California, USA Department of Surgery, David Geffen School of Medicine, University of California Los Angeles, Los Angeles, California, USA * Corresponding author email: [email protected] Abstract One of the major open problems in computer vision is detection of features in visually impaired images. In this paper, we describe a potential solution using Phase Stretch Transform, a new computational approach for image analysis, edge detection and resolution enhancement that is inspired by the physics of the photonic time stretch technique. We mathematically derive the intrinsic nonlinear transfer function and demonstrate how it leads to (1) superior performance at low contrast levels and (2) a reconfigurable operator for hyper-dimensional classification. We prove that the Phase Stretch Transform equalizes the input image brightness across the range of intensities resulting in a high dynamic range in visually impaired images. We also show further improvement in the dynamic range by combining our method with the conventional techniques. Finally, our results show a method for computation of mathematical derivatives via group delay dispersion operations. Introduction Feature detection in images plays a critical role in the field of computer vision for solving problems associated with object recognition, image registration, content-based image retrieval and deep learning [1-3]. Prior works for improving the feature detection in images have focused on use of grey level statistics of the image [1] and on application of edge detection methods [4]. Color distinctiveness and color models [5, 6] and scale selections [7] in images have also been 1 SubmittedtoPLOSOneJournalonNovember3,2016 exploited for enhancing the feature detection. The main goal of feature detection is to classify objects more accurately and at the same time be robust to varying viewing conditions that include changes in illumination or environment conditions, object orientation as well as zoom factor of the camera. Environmental conditions can severely impair detection and localization of objects in images. For instance, under foggy condition, acquired images suffer from visual impairments that reduce contrast, cause blur and noise leading to lower resolution and lower contrast [8, 9]. This constitutes a major bottleneck for many computer vision applications, in particular, for autonomous vehicles. While the emerging imaging technologies such as High Dynamic Range (HDR) hold promise to solve feature detection problems in the computer vision but their slow frame rate pose a challenge in self-driven cars, autonomous robotics and other real-time applications. The Phase Stretch Transform (PST) was recently introduced as a computational approach to signal and image processing [10,11]. PST is a physics-based algorithm that has its roots in photonic time stretch technique [12-15], a method for real-time measurements of ultra-fast events and one that has enabled the discovery of optical rogue waves [16], observation of relativistic electron structure [17], label-free cancer cell detection with record accuracy [18] and optical data compression [19]. The algorithm mimics the propagation of electromagnetic waves through a diffractive medium with engineered 3D dispersive property (refractive index) [10,11]. PST can be applied to both digital images as well as time series data [20] and has been used for edge detection in biomedical images [11, 21-22] and Synthetic Aperture Radar (SAR) images [23]. PST has also been applied for resolution enhancement in super-resolution localization microscopy for imaging of single molecule [24]. The algorithm has been open sourced on GitHub and Matlab Central File Exchange [25]. We first show that PST has an inherent equalization ability that gives a response ideal for feature detection in low contrast regimes of visually impaired images caused by the fog. To do this, we apply the algorithm on two road traffic images under fog as shown in Fig 1. It shows how the algorithm could significantly improve the feature detection by outperforming the conventional edge detection methods based on derivative of the image. The method based on derivative is unable to capture details with small contrast variations in the brighter low resolution areas where 2 SubmittedtoPLOSOneJournalonNovember3,2016 as our technique clearly shows the contrast changes in these visually impaired regions. The warp and strength parameters of PST kernel as described in [10, 11] for the feature detection in these images are 22 and 500, respectively. As we will show in our mathematical formulations, this property emerges because PST’s transfer function is a reconfigurable operator. Later, we examine the superior performance of PST at low light levels and its application to HDR images. Fig 1. Comparison of feature detection using conventional derivative operator to the case of feature detection using PST in case of visually impaired images under fog conditions. Original images are shown in (A). Results of feature detection using conventional derivative operator and PST operator are shown in (B) and (C), respectively. Note that the conventional derivative operator misses to efficiently visualize the small contrast changes in visually impaired regions of the images (shown in green dashed boxes). However, PST can clearly show the contrast changes in the low resolution regions (shown in green dashed boxes) due to its reconfigurable mechanism that enables highlighting features over a wide dynamic range. The strength of features detected using PST over both low and high resolution regions of the images is consistent unlike derivative operator. 3 SubmittedtoPLOSOneJournalonNovember3,2016 Mathematical foundations of Phase Stretch Transform We prove the superior performance of Phase Stretch Transform in the low contrast regime by deriving closed-form analytical expressions for its transfer function. Mathematical results reveal that the transform has an intrinsic intensity equalization property leading to high dynamic range performance. Analytical results are supported by numerical simulation confirming the dynamic range enhancement. We define the stretch operator 𝕊 E# x, y = 𝕊 E( x, y as follows ≜ IFFT2 K u, v ⋅ L u, v ⋅ FFT2 E( x, y (1) and E# x, y is a complex quantity defined as, E# x, y = E# x, y e56 7,8 (2) In the above equations, E( x, y is the input image, x and y are the spatial variables, FFT2 is the two dimensional Fast Fourier Transform, IFFT2 is the 2-dimensional Inverse Fast Fourier Transform and u and v are spatial frequency variables. The function K u, v is called the warped phase kernel and the function L u, v is a localization kernel implemented in frequency domain. Here for simplicity, we assumeL u, v = 1. PST operator is defined as the phase of the Warped Stretch Transform output as follows [10], PST E( x, y ≜ ∡ 𝕊 E( x, y (3) where ∡〈·〉 is the angle operator. Without the loss of generality and in order to keep the notations manageable in what follows, we consider operation of PST on 1D data, i.e., PST E( x = ∡ E# x = ∡ IFFT K u ∙ FFT E( x (4) The warped phase kernel K u is described by a phase function with nonlinear dependence on frequency, u, K u = e5∙> ? (5) By using the Taylor expansion for the phase term in the kernel K u we have, K u =e @ A A 5 C ? ADE A! 4 (6) SubmittedtoPLOSOneJournalonNovember3,2016 where φ G is the mth-order discrete derivative of the phase φ u evaluated for u=0 and values of m are even numbers. PST phase term φ u only contains even-order terms in its Taylor expansion due to even symmetry requirement for the phase term φ u for proper operation of PST [20]. Using the warped phase kernel described in Eq. (6), output complex-field, E# x , can be calculated as follows, E# x = IFFT E( u ×K u = IFFT E( u ×e I A A 5 C ? ADE A! (6) where EJ u is the discrete Fourier transform of the input. Simulation show that PST works best when the applied phase is small. Under these conditions, using small value approximation, the exponential term in Eq. (7) can be simplified to, E# x = IFFT E( u × 1 + j → E# x ≈ 1×E( x + j where E( x G A M N GOP G! ST A/E M A N GOP G! PV A uG E( x G (8) (9) is the mth-order discrete derivative of the input E( x . Since the input is a real quantity, the output phase can be calculated as, PST E( x = ∡ E# x Z[ A/E I A A C ADE A! E\ A ]^ 7 ST ≈ tan ]^ 7 (10) Finally, since the phase is restricted to small values, Eq. (10) can be simplified to, PST E( x ≈ Z[ A/E I A A C ADE A ]^ 7 A! E\ ]^ 7 (11) We see that the transfer function consists of summation of even order derivatives in the numerator divided by the amplitude (brightness) in the denominator. The numerator extracts a hyper dimensional set of features that corresponding to different measures of the curvature of the edge. The denominator renders the response nonlinear in such a way that low light levels are enhanced. These results were obtained for a general phase kernel and for small values of phase in the PST kernel. We now consider two additional scenarios revealing further insight into the unique properties of our transform. 5 SubmittedtoPLOSOneJournalonNovember3,2016 Case 1: We consider the Phase Kernel K u = uP as a quadratic function of frequency variable u. Under this condition, using small phase approximation as used before, the exponential term in Eq. (8) can be simplified to, E# x = IFFT E( u × 1 + j uP → E# x ≈ 1×E( x − j T PV E ∗ aE ]^ 7 a7E (12) (13) Finally, for calculating the phase of the complex output we assume it to be restricted to small values. Therefore, phase of the output in Eq. (13) can be simplified to, PST E( x = ∡E# x ≈ Z[ dE e^ f ∗ (E\)E dfE ]^ 7 (14) Case 2: We consider here, the same Phase Kernel K u = uP as a quadratic function of frequency variable u as discussed in case 1. However, we remove the use of small phase approximation. The exponential term in Eq. (8) now leads to, E# x = IFFT E( u × cos uP + jsin uP (15) Expanding the sine and cosine terms using Euler expansion up to third order and then applying small value approximation to the complex output of Eq. (12) will result in the complex PST as shown below PST E( x = ∡E# x ≈ Z[ dE e^ f ∗ E\ E dfE ]^ 7 dm e^ f d[o e^ f [ [ ∗ S ∗ l! E\ m dfm n! E\ [o df[o dp e^ f dq e^ f [ [ S ∗ k ∗ E! E\ p dfp p! E\ q dfq k (16) The closed-form expression presented in Eq. (11) relates the output to the input in the case of arbitrary phase kernel with small phase approximation. To give an example, the core functionality of the PST as a feature detector can be understood by closed-form expression shown in Eq. (11). The output of the PST operator is related directly to the even-order derivatives of the input with weighting factors of ST A/E M A G! PV A . Each derivatives detects a different feature in the input. Thus, the weighting factors can be designed to select features of 6 SubmittedtoPLOSOneJournalonNovember3,2016 interest. In other words, our transform is a reconfigurable operator that can be tuned to emphasize different features in an input image. The important observation from Eq. (11) is that, the output is inversely related to the input brightness level and this is valid for small phase approximation. Therefore, for the same contrast level, the output is larger in darker low light level parts of the image. This crucial property inherent in PST equalizes the input brightness and allows for more sensitive feature detection and enhancement. Brightness level equalization is a well-studied method to improve feature detection algorithms in High Dynamic Range (HDR) images (see [26] for an example). One technique for brightness level equalization in images is a log function applied to the input before feature detection. The log function has a higher gain for lower brightness input which equalizes the brightness and results in more efficient feature detection. Fortunately, PST operator has a built-in logarithmic behavior which gives it excellent dynamic range. However, this does not completely describe the transform. As observed in Eq. (16), our transform outputs a hyperdimensional feature set for classification. These results also show a method for computation of mathematical derivatives via group delay dispersion operations. Simulation results In this section we present some simulation results that confirm the closed-form expression derived in previous section. We also show examples of operation of PST on HDR images, supporting the new theory developed above. In the first example, we simulate the output for a given 1D data and compare it to the output estimated by the mathematical expression derived in Eq. (11). The phase kernel φ f7 designed for the PST operator is shown in Fig 2A. The warp and strength parameters of the kernel as described in [10,11] are 12.5 and 4000, respectively. The phase and its derivative are shown in Fig 2A and Fig 2B respectively. The input is shown in Fig 2C. Numerically simulated output is compared to the output estimated by Eq. (11) in Fig 2D using red-solid and blue-dotted lines, respectively. It is evident that the simulations confirm the accuracy of the closed-form analytical model of our algorithm. 7 SubmittedtoPLOSOneJournalonNovember3,2016 Fig 2. Comparison of numerically simulated output of PST algorithm with the output given by the closed-form analytical expression derived in Eq. (11). The phase kernel and the corresponding phase derivative profile are shown in (A) and (B), respectively. The input 1D brightness data is shown in (C). Numerically calculated output data is compared to the output data estimated by Eq. (11) in (D) using red-solid and blue-dotted lines, respectively. Simulation results confirm the accuracy of the closed-form analytical model in Eq. (11). In the next example, we evaluate the effect of PST on features with different contrast levels at a fixed brightness level and compare it to the case of using the conventional technique of differentiation to detect features in the same input. The warp and strength parameters used for the 8 SubmittedtoPLOSOneJournalonNovember3,2016 PST operator are 12.15 and 0.48, respectively. The input was designed to have different contrast level change at fixed brightness level, see Fig 3A. Numerically simulated output a using PST is compared to the output using differentiation in Fig 3B. As expected, the output of the differentiator is linearly related to the contrast level and is insensitive to the brightness level. However, the relation of the PST output to contrast level at fixed brightness level is nonlinear. This effect is due to the brightness level equalization feature of PST described in Eq. (11). Fig 3. Effect of PST on features with different contrast levels at fixed brightness level compared to the output by differentiation. The input was designed to have different contrast level change at fixed brightness level, shown in (A). Numerically simulated output using PST is compared to the output using differentiation in (B) using red-solid and blue-dotted lines, respectively. The output of the differentiator is linearly related to the contrast level and is insensitive to the brightness. However, the dependence of the PST output to contrast at fixed brightness is nonlinear. This effect is due to the equalization mechanism of PST described in Eq. (11). Fig 4 shows an example of using PST for feature enhancement in 14 bits HDR image. The image has sharp features in the dark regions, as shown in the dashed box. The image also has features in 9 SubmittedtoPLOSOneJournalonNovember3,2016 a region that the brightness slowly decreases, as shown in the sloid box. Here we compare the feature detection using the derivative operator with that using PST. The derivative operator was implemented from native smooth derivative function. For fair comparison, both methods use the same localization kernel with sigma factor of 2. The warp and strength parameters used for the PST operator are 12.15 and 0.48, respectively. Results of feature detection using smooth derivative operator and PST operator are shown in Fig 4B and Fig 4C, respectively. The derivative operator is unable to unveil the small contrast changes in dark areas of the image, see dashed box in Fig 4B . However, PST extracts more information on the contrast changes in dark areas due to its natural equalization mechanism, see dashed box in Fig 4C. It also can be observed that the intensity of detected edges in the case of smooth derivative is related linearly to the brightness level of the original image, compare solid box areas in Fig 4A and Fig 4B. In contrast, PST has automatically equalized the brightness level in the solid box region in the image and outputs relatively fixed features intensity for that region, see solid box in Fig 4C. We note that PST has missed to visualize features in very bright areas in the image. This is because of the inverse dependence on brightness as shown in Eq. (11). This issue can be mitigated by setting a maximum threshold for detected features or by equalizing the image brightness before passing through PST operator. 10 SubmittedtoPLOSOneJournalonNovember3,2016 Fig 4. Comparison of feature detection using smooth derivative operator to the case of feature detection using PST. Original image is shown in (A). Smooth derivative operator is unable to efficiently visualize the small contrast changes in dark areas of the image. However, PST can clearly show the contrast changes in dark areas due to its natural equalization property. 11 SubmittedtoPLOSOneJournalonNovember3,2016 To better explain, the role of PST for feature detection in low-light-level and high-light-level regions, we use a single line scan of the HDR image shown previously in Fig 4. The blue box in the Fig 5 demonstrates the response of PST to low-light-level regions where it outperforms conventional derivative operator. Similarly, for high-light-level regions of the image (shown in green and purple box in the Fig 5), PST outperforms when the contrast is low (shown in green box in the Fig 5). In contrast, the conventional derivative operator response is dominating only under high contrast (shown in purple box in the Fig 5). Fig 5. A line scan corresponding to Row 524 from the image previously shown in Fig 4 comparing feature detection using conventional derivative operator and PST operator under low-light-level and high-light-level conditions. 12 SubmittedtoPLOSOneJournalonNovember3,2016 Original input line scan is shown in (A). Feature detection of this line scan using derivative and PST operators are shown in (B). The blue box highlights that the feature detection response of PST is much larger than the derivative operator under low-light-level conditions. The green and purple box demonstrates the response of PST and derivative operator for feature detection under high-light-level conditions. While PST enhances features under low contrast also under highlight-levels (see green box) unlike derivative operator which identifies features only under high contrast (see blue box). To further enhance the dynamic range, we introduce a hybrid system that selects the best edge response from both the PST and the conventional edge detection filter. As shown in Fig , the hybrid system combines the edge response from PST and conventional derivative operator, providing edge detection in both low-light and high-light levels under low and high contrast variations. Fig 6. Hybrid system for feature detection using derivative operator and PST. The Original image is shown in (A). Results of feature detection using smooth derivative operator and PST operator are shown in (B) and (C), respectively. And the output of hybrid system is shown in (D). Note that in (D), the strength of the detected features in both the highlight-level and low-light-level regions is same. The hybrid system selects the detected features in the darker regions using PST operator and in the brighter regions using smooth derivative operator providing a wider range of dynamic operation. 13 SubmittedtoPLOSOneJournalonNovember3,2016 Conclusions In this paper, we presented a new method for edge detection in visually impaired images using a new mathematical transform inspired by the physics of photonic time stretch. We showed via analytical derivation and numerical simulations that the so called Phase Stretch Transform equalizes the input brightness resulting in a high dynamic range in feature detection and acts as a hyper-dimensional feature set classifier. Furthermore, our results show a method for computation of mathematical derivatives via group delay dispersion operations. References: 1. Manjunath BS, Shekhar C, Chellappa R. A new approach to image feature detection with applications. Pattern Recognition. 1996 Apr 1;29(4):627-40. 2. Zitova B, Flusser J. Image registration methods: a survey. Image and vision computing. 2003 Oct 31;21(11):977-1000. 3. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015 May 28;521(7553):436-44. 4. Medioni G, Nevatia R. Matching images using linear features. IEEE Transactions on Pattern Analysis and Machine Intelligence. 1984 Nov(6):675-85. 5. Stokman H, Gevers T. Selection and fusion of color models for image feature detection. IEEE transactions on pattern analysis and machine intelligence. 2007 Mar;29(3). 6. Van de Weijer J, Gevers T, Bagdanov AD. Boosting color saliency in image feature detection. IEEE transactions on pattern analysis and machine intelligence. 2006 Jan;28(1):150-6. 7. Lindeberg T. Feature detection with automatic scale selection. International journal of computer vision. 1998 Nov 1;30(2):79-116. 8. Pavlić M, Belzner H, Rigoll G, Ilić S. Image based fog detection in vehicles. InIntelligent Vehicles Symposium (IV), 2012 IEEE 2012 Jun 3 (pp. 1132-1137). IEEE. 9. Sivaraman S, Trivedi MM. Looking at vehicles on the road: A survey of vision-based vehicle detection, tracking, and behavior analysis. IEEE Transactions on Intelligent Transportation Systems. 2013 Dec;14(4):1773-95. 10. Asghari MH, Jalali B. Physics-inspired image edge detection. InSignal and Information Processing (GlobalSIP), 2014 IEEE Global Conference on 2014 Dec 3 (pp. 293-296). IEEE. 11. Asghari MH, Jalali B. Edge detection in digital images using dispersive phase stretch transform. Journal of Biomedical Imaging. 2015 Jan 1;2015:6. 12. Bhushan AS, Coppinger F, Jalali B. Time-stretched analogue-to-digital conversion. Electronics Letters. 1998 May 28;34(11):1081-2. 14 SubmittedtoPLOSOneJournalonNovember3,2016 13. Han Y, Jalali B. Photonic time-stretched analog-to-digital converter: fundamental concepts and practical considerations. Journal of Lightwave Technology. 2003 Dec 1;21(12):3085. 14. Ng W, Rockwood T, Reamon A. Demonstration of channel-stitched photonic time-stretch analog-to-digital converter with ENOB≥ 8 for a 10 GHz signal bandwidth. InProceedings of the Government Microcircuit Applications & Critical Technology Conference (GOMACTech’14) 2014 Mar. 15. Mahjoubfar A, Chen CL, Jalali B. Design of warped stretch transform. Scientific reports. 2015;5. 16. Solli DR, Ropers C, Koonath P, Jalali B. Optical rogue waves. Nature. 2007 Dec 13;450(7172):1054-7. 17. Szwaj C, Evain C, Le Parquier M, Bielawski S, Roussel E, Manceron L, Brubach JB, Tordeux MA, Ricaud JP, Cassinari L, Labat M. Unveiling the complex shapes of relativistic electrons bunches, using photonic timestretch electro-optic sampling. InPhotonics Society Summer Topical Meeting Series (SUM), 2016 IEEE 2016 Jul 11 (pp. 136-137). IEEE. 18. Chen CL, Mahjoubfar A, Tai LC, Blaby IK, Huang A, Niazi KR, et al. Deep Learning in Label-free Cell Classification. Scientific reports. 2016;6. 19. Chen CL, Mahjoubfar A, Jalali B. Optical data compression in time stretch imaging. PloS one. 2015 Apr 23;10(4):e0125106. 20. Jalali B, Mahjoubfar A. Tailoring wideband signals with a photonic hardware accelerator. Proceedings of the IEEE. 2015 Jul;103(7):1071-86. 21. Suthar M, Mahjoubfar A, Seals K, Lee EW, Jalaii B. Diagnostic tool for pneumothorax. InPhotonics Society Summer Topical Meeting Series (SUM), 2016 IEEE 2016 Jul 11 (pp. 218-219). IEEE. 22. Suthar M. Decision Support Systems for Radiologists based on Phase Stretch Transform, electronic, scholarly journal. M.S. Thesis, University of California, Los Angeles 2015, Available from: http://escholarship.org/uc/item/39p0h9jp. 23. Ilioudis CV, Clemente C, Asghari MH, Jalali B, Soraghan JJ. Edge detection in SAR images using phase stretch transform. 24. Ilovitsh T, Jalali B, Asghari MH, Zalevsky Z. Phase stretch transform for super-resolution localization microscopy. Biomedical Optics Express. 2016 Oct 1;7(10):4198-209. 25. JalaliLabUCLA/Image-feature-detection-using-Phase-Stretch-Transform, Available from: https://github.com/JalaliLabUCLA/Image-feature-detection-using-Phase-Stretch-Transform. 26. Hameed Z, Wang C. Edge detection using histogram equalization and multi-filtering process. In2011 IEEE International Symposium of Circuits and Systems (ISCAS) 2011 May 15 (pp. 1077-1080). IEEE. 15

Download PDF

advertisement