User manual | Feature Enhancement in Visually Impaired Images

SubmittedtoPLOSOneJournalonNovember3,2016
Feature Enhancement in Visually Impaired Images
1
Madhuri Suthar1*, Mohammad Asghari1 and Bahram Jalali1,2,3,4
Department of Electrical Engineering, University of California Los Angeles, Los Angeles, California, USA
2
3
4
California NanoSystems Institute, Los Angeles, California, USA
Department of Bioengineering, University of California Los Angeles, Los Angeles, California, USA
Department of Surgery, David Geffen School of Medicine, University of California Los Angeles, Los Angeles,
California, USA
*
Corresponding author email: [email protected]
Abstract
One of the major open problems in computer vision is detection of features in visually impaired
images. In this paper, we describe a potential solution using Phase Stretch Transform, a new
computational approach for image analysis, edge detection and resolution enhancement that is
inspired by the physics of the photonic time stretch technique. We mathematically derive the
intrinsic nonlinear transfer function and demonstrate how it leads to (1) superior performance at
low contrast levels and (2) a reconfigurable operator for hyper-dimensional classification. We
prove that the Phase Stretch Transform equalizes the input image brightness across the range of
intensities resulting in a high dynamic range in visually impaired images. We also show further
improvement in the dynamic range by combining our method with the conventional techniques.
Finally, our results show a method for computation of mathematical derivatives via group delay
dispersion operations.
Introduction
Feature detection in images plays a critical role in the field of computer vision for solving
problems associated with object recognition, image registration, content-based image retrieval
and deep learning [1-3]. Prior works for improving the feature detection in images have focused
on use of grey level statistics of the image [1] and on application of edge detection methods [4].
Color distinctiveness and color models [5, 6] and scale selections [7] in images have also been
1
SubmittedtoPLOSOneJournalonNovember3,2016
exploited for enhancing the feature detection. The main goal of feature detection is to classify
objects more accurately and at the same time be robust to varying viewing conditions that
include changes in illumination or environment conditions, object orientation as well as zoom
factor of the camera. Environmental conditions can severely impair detection and localization of
objects in images. For instance, under foggy condition, acquired images suffer from visual
impairments that reduce contrast, cause blur and noise leading to lower resolution and lower
contrast [8, 9]. This constitutes a major bottleneck for many computer vision applications, in
particular, for autonomous vehicles. While the emerging imaging technologies such as High
Dynamic Range (HDR) hold promise to solve feature detection problems in the computer vision
but their slow frame rate pose a challenge in self-driven cars, autonomous robotics and other
real-time applications.
The Phase Stretch Transform (PST) was recently introduced as a computational approach to
signal and image processing [10,11]. PST is a physics-based algorithm that has its roots in
photonic time stretch technique [12-15], a method for real-time measurements of ultra-fast
events and one that has enabled the discovery of optical rogue waves [16], observation of
relativistic electron structure [17], label-free cancer cell detection with record accuracy [18] and
optical data compression [19]. The algorithm mimics the propagation of electromagnetic waves
through a diffractive medium with engineered 3D dispersive property (refractive index) [10,11].
PST can be applied to both digital images as well as time series data [20] and has been used for
edge detection in biomedical images [11, 21-22] and Synthetic Aperture Radar (SAR) images
[23]. PST has also been applied for resolution enhancement in super-resolution localization
microscopy for imaging of single molecule [24]. The algorithm has been open sourced on
GitHub and Matlab Central File Exchange [25].
We first show that PST has an inherent equalization ability that gives a response ideal for feature
detection in low contrast regimes of visually impaired images caused by the fog. To do this, we
apply the algorithm on two road traffic images under fog as shown in Fig 1. It shows how the
algorithm could significantly improve the feature detection by outperforming the conventional
edge detection methods based on derivative of the image. The method based on derivative is
unable to capture details with small contrast variations in the brighter low resolution areas where
2
SubmittedtoPLOSOneJournalonNovember3,2016
as our technique clearly shows the contrast changes in these visually impaired regions. The warp
and strength parameters of PST kernel as described in [10, 11] for the feature detection in these
images are 22 and 500, respectively. As we will show in our mathematical formulations, this
property emerges because PST’s transfer function is a reconfigurable operator. Later, we
examine the superior performance of PST at low light levels and its application to HDR images.
Fig 1. Comparison of feature detection using conventional derivative operator to the case of
feature detection using PST in case of visually impaired images under fog conditions.
Original images are shown in (A). Results of feature detection using conventional derivative
operator and PST operator are shown in (B) and (C), respectively. Note that the conventional
derivative operator misses to efficiently visualize the small contrast changes in visually impaired
regions of the images (shown in green dashed boxes). However, PST can clearly show the
contrast changes in the low resolution regions (shown in green dashed boxes) due to its
reconfigurable mechanism that enables highlighting features over a wide dynamic range. The
strength of features detected using PST over both low and high resolution regions of the images is
consistent unlike derivative operator.
3
SubmittedtoPLOSOneJournalonNovember3,2016
Mathematical foundations of Phase Stretch Transform
We prove the superior performance of Phase Stretch Transform in the low contrast regime by
deriving closed-form analytical expressions for its transfer function. Mathematical results reveal
that the transform has an intrinsic intensity equalization property leading to high dynamic range
performance. Analytical results are supported by numerical simulation confirming the dynamic
range enhancement. We define the stretch operator 𝕊
E# x, y = 𝕊 E( x, y
as follows
≜ IFFT2 K u, v ⋅ L u, v ⋅ FFT2 E( x, y
(1)
and E# x, y is a complex quantity defined as,
E# x, y = E# x, y e56 7,8 (2)
In the above equations, E( x, y is the input image, x and y are the spatial variables, FFT2 is the
two dimensional Fast Fourier Transform, IFFT2 is the 2-dimensional Inverse Fast Fourier
Transform and u and v are spatial frequency variables. The function K u, v is called the warped
phase kernel and the function L u, v is a localization kernel implemented in frequency domain.
Here for simplicity, we assumeL u, v = 1. PST operator is defined as the phase of the Warped
Stretch Transform output as follows [10],
PST E( x, y
≜ ∡ 𝕊 E( x, y
(3)
where ∡〈·〉 is the angle operator.
Without the loss of generality and in order to keep the notations manageable in what follows, we
consider operation of PST on 1D data, i.e.,
PST E( x
= ∡ E# x
= ∡ IFFT K u ∙ FFT E( x
(4)
The warped phase kernel K u is described by a phase function with nonlinear dependence on
frequency, u,
K u = e5∙> ?
(5)
By using the Taylor expansion for the phase term in the kernel K u we have,
K u =e
@ A A
5 C
?
ADE
A!
4
(6)
SubmittedtoPLOSOneJournalonNovember3,2016
where φ G is the mth-order discrete derivative of the phase φ u evaluated for u=0 and values of
m are even numbers. PST phase term φ u only contains even-order terms in its Taylor
expansion due to even symmetry requirement for the phase term φ u for proper operation of
PST [20]. Using the warped phase kernel described in Eq. (6), output complex-field, E# x , can
be calculated as follows,
E# x = IFFT E( u ×K u
= IFFT E( u ×e
I A A
5 C
?
ADE
A!
(6)
where EJ u is the discrete Fourier transform of the input. Simulation show that PST works best
when the applied phase is small. Under these conditions, using small value approximation, the
exponential term in Eq. (7) can be simplified to,
E# x = IFFT E( u × 1 + j
→ E# x ≈ 1×E( x + j
where E( x
G
A
M
N
GOP G!
ST A/E M A
N
GOP G! PV A
uG E( x
G
(8)
(9)
is the mth-order discrete derivative of the input E( x . Since the input is a real
quantity, the output phase can be calculated as,
PST E( x
= ∡ E# x
Z[ A/E I A
A
C
ADE A! E\ A ]^ 7
ST
≈ tan
]^ 7 (10)
Finally, since the phase is restricted to small values, Eq. (10) can be simplified to,
PST E( x ≈ Z[ A/E I A
A
C
ADE
A ]^ 7
A! E\
]^ 7 (11)
We see that the transfer function consists of summation of even order derivatives in the
numerator divided by the amplitude (brightness) in the denominator. The numerator extracts a
hyper dimensional set of features that corresponding to different measures of the curvature of the
edge. The denominator renders the response nonlinear in such a way that low light levels are
enhanced. These results were obtained for a general phase kernel and for small values of phase in
the PST kernel. We now consider two additional scenarios revealing further insight into the
unique properties of our transform.
5
SubmittedtoPLOSOneJournalonNovember3,2016
Case 1: We consider the Phase Kernel K u = uP as a quadratic function of frequency variable
u. Under this condition, using small phase approximation as used before, the exponential term in
Eq. (8) can be simplified to,
E# x = IFFT E( u × 1 + j uP → E# x ≈ 1×E( x − j
T
PV
E
∗
aE ]^ 7
a7E
(12)
(13)
Finally, for calculating the phase of the complex output we assume it to be restricted to small
values. Therefore, phase of the output in Eq. (13) can be simplified to,
PST E( x
= ∡E# x ≈ Z[ dE e^ f
∗
(E\)E
dfE
]^ 7
(14)
Case 2: We consider here, the same Phase Kernel K u = uP as a quadratic function of
frequency variable u as discussed in case 1. However, we remove the use of small phase
approximation. The exponential term in Eq. (8) now leads to,
E# x = IFFT E( u × cos uP + jsin uP (15)
Expanding the sine and cosine terms using Euler expansion up to third order and then applying
small value approximation to the complex output of Eq. (12) will result in the complex PST as
shown below
PST E( x
= ∡E# x ≈ Z[ dE e^ f
∗
E\ E
dfE
]^ 7
dm e^ f
d[o e^ f
[
[
∗
S
∗
l! E\ m
dfm
n! E\ [o
df[o
dp e^ f
dq e^ f
[
[
S
∗
k
∗
E! E\ p
dfp
p! E\ q
dfq
k
(16)
The closed-form expression presented in Eq. (11) relates the output to the input in the case of
arbitrary phase kernel with small phase approximation. To give an example, the core
functionality of the PST as a feature detector can be understood by closed-form expression
shown in Eq. (11). The output of the PST operator is related directly to the even-order
derivatives of the input with weighting factors of
ST A/E M A
G! PV A
. Each derivatives detects a
different feature in the input. Thus, the weighting factors can be designed to select features of
6
SubmittedtoPLOSOneJournalonNovember3,2016
interest. In other words, our transform is a reconfigurable operator that can be tuned to
emphasize different features in an input image.
The important observation from Eq. (11) is that, the output is inversely related to the input
brightness level and this is valid for small phase approximation. Therefore, for the same contrast
level, the output is larger in darker low light level parts of the image. This crucial property
inherent in PST equalizes the input brightness and allows for more sensitive feature detection
and enhancement. Brightness level equalization is a well-studied method to improve feature
detection algorithms in High Dynamic Range (HDR) images (see [26] for an example). One
technique for brightness level equalization in images is a log function applied to the input before
feature detection. The log function has a higher gain for lower brightness input which equalizes
the brightness and results in more efficient feature detection. Fortunately, PST operator has a
built-in logarithmic behavior which gives it excellent dynamic range. However, this does not
completely describe the transform. As observed in Eq. (16), our transform outputs a hyperdimensional feature set for classification. These results also show a method for computation of
mathematical derivatives via group delay dispersion operations.
Simulation results
In this section we present some simulation results that confirm the closed-form expression
derived in previous section. We also show examples of operation of PST on HDR images,
supporting the new theory developed above. In the first example, we simulate the output for a
given 1D data and compare it to the output estimated by the mathematical expression derived in
Eq. (11). The phase kernel φ f7 designed for the PST operator is shown in Fig 2A. The warp
and strength parameters of the kernel as described in [10,11] are 12.5 and 4000, respectively.
The phase and its derivative are shown in Fig 2A and Fig 2B respectively. The input is shown in
Fig 2C. Numerically simulated output is compared to the output estimated by Eq. (11) in Fig 2D
using red-solid and blue-dotted lines, respectively. It is evident that the simulations confirm the
accuracy of the closed-form analytical model of our algorithm.
7
SubmittedtoPLOSOneJournalonNovember3,2016
Fig 2. Comparison of numerically simulated output of PST algorithm with the output
given by the closed-form analytical expression derived in Eq. (11).
The phase kernel and the corresponding phase derivative profile are shown in (A) and (B),
respectively. The input 1D brightness data is shown in (C). Numerically calculated output data is
compared to the output data estimated by Eq. (11) in (D) using red-solid and blue-dotted lines,
respectively. Simulation results confirm the accuracy of the closed-form analytical model in Eq.
(11).
In the next example, we evaluate the effect of PST on features with different contrast levels at a
fixed brightness level and compare it to the case of using the conventional technique of
differentiation to detect features in the same input. The warp and strength parameters used for the
8
SubmittedtoPLOSOneJournalonNovember3,2016
PST operator are 12.15 and 0.48, respectively. The input was designed to have different contrast
level change at fixed brightness level, see Fig 3A. Numerically simulated output a using PST is
compared to the output using differentiation in Fig 3B. As expected, the output of the
differentiator is linearly related to the contrast level and is insensitive to the brightness level.
However, the relation of the PST output to contrast level at fixed brightness level is nonlinear.
This effect is due to the brightness level equalization feature of PST described in Eq. (11).
Fig 3. Effect of PST on features with different contrast levels at fixed brightness level
compared to the output by differentiation.
The input was designed to have different contrast level change at fixed brightness level, shown in
(A). Numerically simulated output using PST is compared to the output using differentiation in
(B) using red-solid and blue-dotted lines, respectively. The output of the differentiator is linearly
related to the contrast level and is insensitive to the brightness. However, the dependence of the
PST output to contrast at fixed brightness is nonlinear. This effect is due to the equalization
mechanism of PST described in Eq. (11).
Fig 4 shows an example of using PST for feature enhancement in 14 bits HDR image. The image
has sharp features in the dark regions, as shown in the dashed box. The image also has features in
9
SubmittedtoPLOSOneJournalonNovember3,2016
a region that the brightness slowly decreases, as shown in the sloid box. Here we compare the
feature detection using the derivative operator with that using PST. The derivative operator was
implemented from native smooth derivative function. For fair comparison, both methods use the
same localization kernel with sigma factor of 2. The warp and strength parameters used for the
PST operator are 12.15 and 0.48, respectively. Results of feature detection using smooth
derivative operator and PST operator are shown in Fig 4B and Fig 4C, respectively. The
derivative operator is unable to unveil the small contrast changes in dark areas of the image, see
dashed box in Fig 4B . However, PST extracts more information on the contrast changes in dark
areas due to its natural equalization mechanism, see dashed box in Fig 4C. It also can be
observed that the intensity of detected edges in the case of smooth derivative is related linearly to
the brightness level of the original image, compare solid box areas in Fig 4A and Fig 4B. In
contrast, PST has automatically equalized the brightness level in the solid box region in the
image and outputs relatively fixed features intensity for that region, see solid box in Fig 4C. We
note that PST has missed to visualize features in very bright areas in the image. This is because
of the inverse dependence on brightness as shown in Eq. (11). This issue can be mitigated by
setting a maximum threshold for detected features or by equalizing the image brightness before
passing through PST operator.
10
SubmittedtoPLOSOneJournalonNovember3,2016
Fig 4. Comparison of feature detection using smooth derivative operator to the case of
feature detection using PST.
Original image is shown in (A). Smooth derivative operator is unable to efficiently visualize the
small contrast changes in dark areas of the image. However, PST can clearly show the contrast
changes in dark areas due to its natural equalization property.
11
SubmittedtoPLOSOneJournalonNovember3,2016
To better explain, the role of PST for feature detection in low-light-level and high-light-level
regions, we use a single line scan of the HDR image shown previously in Fig 4. The blue box in
the Fig 5 demonstrates the response of PST to low-light-level regions where it outperforms
conventional derivative operator. Similarly, for high-light-level regions of the image (shown in
green and purple box in the Fig 5), PST outperforms when the contrast is low (shown in green
box in the Fig 5). In contrast, the conventional derivative operator response is dominating only
under high contrast (shown in purple box in the Fig 5).
Fig 5. A line scan corresponding to Row 524 from the image previously shown in Fig 4
comparing feature detection using conventional derivative operator and PST operator
under low-light-level and high-light-level conditions.
12
SubmittedtoPLOSOneJournalonNovember3,2016
Original input line scan is shown in (A). Feature detection of this line scan using derivative and
PST operators are shown in (B). The blue box highlights that the feature detection response of
PST is much larger than the derivative operator under low-light-level conditions. The green and
purple box demonstrates the response of PST and derivative operator for feature detection under
high-light-level conditions. While PST enhances features under low contrast also under highlight-levels (see green box) unlike derivative operator which identifies features only under high
contrast (see blue box).
To further enhance the dynamic range, we introduce a hybrid system that selects the best edge
response from both the PST and the conventional edge detection filter. As shown in Fig , the
hybrid system combines the edge response from PST and conventional derivative operator,
providing edge detection in both low-light and high-light levels under low and high contrast
variations.
Fig 6. Hybrid system for feature detection using derivative operator and PST.
The Original image is shown in (A). Results of feature detection using smooth derivative
operator and PST operator are shown in (B) and (C), respectively. And the output of hybrid
system is shown in (D). Note that in (D), the strength of the detected features in both the highlight-level and low-light-level regions is same. The hybrid system selects the detected features in
the darker regions using PST operator and in the brighter regions using smooth derivative
operator providing a wider range of dynamic operation.
13
SubmittedtoPLOSOneJournalonNovember3,2016
Conclusions
In this paper, we presented a new method for edge detection in visually impaired images using a
new mathematical transform inspired by the physics of photonic time stretch. We showed via
analytical derivation and numerical simulations that the so called Phase Stretch Transform
equalizes the input brightness resulting in a high dynamic range in feature detection and acts as a
hyper-dimensional feature set classifier. Furthermore, our results show a method for computation
of mathematical derivatives via group delay dispersion operations.
References:
1.
Manjunath BS, Shekhar C, Chellappa R. A new approach to image feature detection with applications. Pattern
Recognition. 1996 Apr 1;29(4):627-40.
2.
Zitova B, Flusser J. Image registration methods: a survey. Image and vision computing. 2003 Oct
31;21(11):977-1000.
3.
LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015 May 28;521(7553):436-44.
4.
Medioni G, Nevatia R. Matching images using linear features. IEEE Transactions on Pattern Analysis and
Machine Intelligence. 1984 Nov(6):675-85.
5.
Stokman H, Gevers T. Selection and fusion of color models for image feature detection. IEEE transactions on
pattern analysis and machine intelligence. 2007 Mar;29(3).
6.
Van de Weijer J, Gevers T, Bagdanov AD. Boosting color saliency in image feature detection. IEEE
transactions on pattern analysis and machine intelligence. 2006 Jan;28(1):150-6.
7.
Lindeberg T. Feature detection with automatic scale selection. International journal of computer vision. 1998
Nov 1;30(2):79-116.
8.
Pavlić M, Belzner H, Rigoll G, Ilić S. Image based fog detection in vehicles. InIntelligent Vehicles
Symposium (IV), 2012 IEEE 2012 Jun 3 (pp. 1132-1137). IEEE.
9.
Sivaraman S, Trivedi MM. Looking at vehicles on the road: A survey of vision-based vehicle detection,
tracking, and behavior analysis. IEEE Transactions on Intelligent Transportation Systems. 2013
Dec;14(4):1773-95.
10.
Asghari MH, Jalali B. Physics-inspired image edge detection. InSignal and Information Processing
(GlobalSIP), 2014 IEEE Global Conference on 2014 Dec 3 (pp. 293-296). IEEE.
11.
Asghari MH, Jalali B. Edge detection in digital images using dispersive phase stretch transform. Journal of
Biomedical Imaging. 2015 Jan 1;2015:6.
12.
Bhushan AS, Coppinger F, Jalali B. Time-stretched analogue-to-digital conversion. Electronics Letters. 1998
May 28;34(11):1081-2.
14
SubmittedtoPLOSOneJournalonNovember3,2016
13.
Han Y, Jalali B. Photonic time-stretched analog-to-digital converter: fundamental concepts and practical
considerations. Journal of Lightwave Technology. 2003 Dec 1;21(12):3085.
14.
Ng W, Rockwood T, Reamon A. Demonstration of channel-stitched photonic time-stretch analog-to-digital
converter with ENOB≥ 8 for a 10 GHz signal bandwidth. InProceedings of the Government Microcircuit
Applications & Critical Technology Conference (GOMACTech’14) 2014 Mar.
15.
Mahjoubfar A, Chen CL, Jalali B. Design of warped stretch transform. Scientific reports. 2015;5.
16.
Solli DR, Ropers C, Koonath P, Jalali B. Optical rogue waves. Nature. 2007 Dec 13;450(7172):1054-7.
17.
Szwaj C, Evain C, Le Parquier M, Bielawski S, Roussel E, Manceron L, Brubach JB, Tordeux MA, Ricaud JP,
Cassinari L, Labat M. Unveiling the complex shapes of relativistic electrons bunches, using photonic timestretch electro-optic sampling. InPhotonics Society Summer Topical Meeting Series (SUM), 2016 IEEE 2016
Jul 11 (pp. 136-137). IEEE.
18.
Chen CL, Mahjoubfar A, Tai LC, Blaby IK, Huang A, Niazi KR, et al. Deep Learning in Label-free Cell
Classification. Scientific reports. 2016;6.
19.
Chen CL, Mahjoubfar A, Jalali B. Optical data compression in time stretch imaging. PloS one. 2015 Apr
23;10(4):e0125106.
20.
Jalali B, Mahjoubfar A. Tailoring wideband signals with a photonic hardware accelerator. Proceedings of the
IEEE. 2015 Jul;103(7):1071-86.
21.
Suthar M, Mahjoubfar A, Seals K, Lee EW, Jalaii B. Diagnostic tool for pneumothorax. InPhotonics Society
Summer Topical Meeting Series (SUM), 2016 IEEE 2016 Jul 11 (pp. 218-219). IEEE.
22.
Suthar M. Decision Support Systems for Radiologists based on Phase Stretch Transform, electronic, scholarly
journal.
M.S.
Thesis,
University
of
California,
Los
Angeles
2015,
Available
from:
http://escholarship.org/uc/item/39p0h9jp.
23.
Ilioudis CV, Clemente C, Asghari MH, Jalali B, Soraghan JJ. Edge detection in SAR images using phase
stretch transform.
24.
Ilovitsh T, Jalali B, Asghari MH, Zalevsky Z. Phase stretch transform for super-resolution localization
microscopy. Biomedical Optics Express. 2016 Oct 1;7(10):4198-209.
25.
JalaliLabUCLA/Image-feature-detection-using-Phase-Stretch-Transform,
Available
from:
https://github.com/JalaliLabUCLA/Image-feature-detection-using-Phase-Stretch-Transform.
26.
Hameed Z, Wang C. Edge detection using histogram equalization and multi-filtering process. In2011 IEEE
International Symposium of Circuits and Systems (ISCAS) 2011 May 15 (pp. 1077-1080). IEEE.
15

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project