DEVELOPMENT AND IMPLEMENTATION OF IMAGE FUSION ALGORITHMS BASED ON WAVELETS A Thesis Submitted in Partial Fulfilment of the Requirements for the Award of the Degree of Master of Technology in Electronics and Instrumentation Engineering by PRIYA RANJAN MUDULI Roll No: 211EC3315 Department of Electronics & Communication Engineering National Institute of Technology, Rourkela Odisha- 769008, India May 2013 DEVELOPMENT AND IMPLEMENTATION OF IMAGE FUSION ALGORITHMS BASED ON WAVELETS A Thesis Submitted in Partial Fulfilment of the Requirements for the Award of the Degree of Master of Technology in Electronics and Instrumentation Engineering by PRIYA RANJAN MUDULI Roll No: 211EC3315 Under the Supervision of Prof. Umesh Chandra Pati Department of Electronics & Communication Engineering National Institute of Technology, Rourkela Odisha- 769008, India May 2013 Department of Electronics & Communication Engineering National Institute of Technology, Rourkela CERTIFICATE This is to certify that the thesis report entitled “DEVELOPMENT AND IMPLEMENTATION OF IMAGE FUSION ALGORITHMS BASED ON WAVELETS ”Submitted by Mr PRIYA RANJAN MUDULI bearing roll no. 211EC3315 in partial fulfilment of the requirements for the award of Master of Technology in Electronics and Communication Engineering with specialization in “Electronics and Instrumentation Engineering” during session 2011-2013 at National Institute of Technology, Rourkela is an authentic work carried out by him under my supervision and guidance. To the best of my knowledge, the matter embodied in the thesis has not been submitted to any other University / Institute for the award of any Degree or Diploma. Prof. Umesh Chandra Pati Place: Associate Professor Date: Dept. of Electronics and Comm. Engineering National Institute of Technology Rourkela-769008 Dedicated to My Family & Teachers ACKNOWLEDGEMENTS First of all, I would like to express my deep sense of respect and gratitude towards my advisor and guide Prof. U.C. Pati, who has been the guiding force behind this work. I am greatly indebted to him for his constant encouragement, invaluable advice and for propelling me further in every aspect of my academic life. His presence and optimism have provided an invaluable influence on my career and outlook for the future. I consider it my good fortune to have an opportunity to work with such a wonderful person. Next, I want to express my respects to Prof. T. K. Dan, Prof. S. K. Patra, Prof. K. K. Mahapatra, Prof. S. Meher, Prof. A. Swain, Prof. Poonam Singh and Prof. L. P. Roy for teaching me and helping me how to learn. They have been great sources of inspiration to me and I thank them from the bottom of my heart. I also extend my thanks to all faculty members and staff of the Department of Electronics and Communication Engineering, National Institute of Technology, Rourkela who have encouraged me throughout the course of Master’s Degree. I would like to thank all my friends and especially my classmates for all the thoughtful and mind stimulating discussions we had, which prompted us to think beyond the obvious. I have enjoyed their companionship so much during my stay at NIT, Rourkela. I am especially indebted to my parents for their love, sacrifice, and support. They are my first teachers after I came to this world and have set great examples for me about how to live, study, and work. Date: Roll No: 211EC3315 Place: Dept. of ECE NIT, Rourkela i ABSTRACT Image fusion is a process of blending the complementary as well as the common features of a set of images, to generate a resultant image with superior information content in terms of subjective as well as objective analysis point of view. The objective of this research work is to develop some novel image fusion algorithms and their applications in various fields such as crack detection, multi spectra sensor image fusion, medical image fusion and edge detection of multi-focus images etc. The first part of this research work deals with a novel crack detection technique based on Non-Destructive Testing (NDT) for cracks in walls suppressing the diversity and complexity of wall images. It follows different edge tracking algorithms such as Hyperbolic Tangent (HBT) filtering and canny edge detection algorithm. The fusion of detector responses are performed using Haar Discrete Wavelet Transform (HDWT) and maximumapproximation with mean-detail image fusion algorithm to get more prominent detection of crack edges. The proposed system gives improved edge detection in images with superior edge localization and higher PSNR. . The second part of this research work deals with a novel edge detection approach for multi-focused images by means of complex wavelets based image fusion. An illumination invariant hyperbolic tangent filter (HBT) is applied followed by an adaptive thresholding to get the real edges. The shift invariance and directionally selective diagonal filtering as well as the ease of implementation of Dual-Tree Complex Wavelet Transform (DT-CWT) ensure robust sub band fusion. It helps in avoiding the ringing artefacts that are more pronounced in Discrete Wavelet Transform (DWT). The fusion using DT-CWT also solves the problem of low contrast and blocking effects. To fulfil the symmetry of sub-sampling structure and biorthogonal property, a Q-shift dual tree CWT is implemented here. The adaptive thresholding varies the threshold value smartly over the image. This helps to combat with a potent illumination gradient, shadowing and multi focus blurring of an image. In the third part, an improved DT-CWT based image fusion technique has been developed to compose a resultant image with better perceptual as well as quantitative image quality indices. A bilateral sharpness based weighting scheme has been implemented for the high frequency coefficients taking both gradient and its phase coherence in account. A normalized maximum gradient weighting scheme is implemented for low frequency wavelet components. The proposed technique shows superior result as compared to DWT and traditional DT-CWT based image fusion algorithms. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page ii TABLE OF CONTENTS Page No. ACKNOWLEDGEMENTS ...………………………………………………………………i ABSTRACT …………………………………………………………………………….… ii TABLE OF CONTENT ……………………………………………………………….….. iii LIST OF FIGURES ……………………………………………………………………. .v LIST OF ABBREVIATIONS ……………………………………………………………...vii Chapter 1 INTRODUCTION TO IMAGE FUSION …………………………………1 1.1 Overview ……………………………………………………………….....2 1.2 Single Sensor Image Fusion System……………………………………....5 1.3 Multi Sensor Image Fusion System……………………………….....…....5 1.4 Image Pre-processing………………………………………………….…..7 1.5 Image Fusion Techniques ………………………………………….….….7 1.6 Motivation…………………………… ……………………………...…...8 1.7 Objectives.…………………………………………………………….......9 1.8 Thesis Organisation.……………………………………………….…..….9 Chapter 2 LITERATURE REVIEW………………..…………………………….......12 2.1 Multiresolution Pyramidal Image Fusion....……………………………..14 2.2 Wavelet Transform based Image Fusion Algorithms…….……………...20 2.2.1 Discrete wavelet transform...……………………….………….…..23 Chapter 3 IMAGE FUSION AND EDGE DETECTION…..……………………….. 29 3.1 Crack Detection using Image Fusion ………..……………………....….30 3.1.1 Proposed crack detection technique………………………………..31 3.1.2 Wavelet decomposition and fusion...……………………….……...35 3.1.3 Result and Discussion……………………………………..….........37 Submitted by PRIYA RANJAN MUDULI [211EC3315] Page iii 3.1.4 Summary ………………………………………………………...........42 3.2 Edge Detection for Multi-focus Images using Image Fusion …………....43 3.2.1 Proposed technique ………………………………………….…….44 3.2.2 DT-CWT based image fusion ………………………………..........44 3.2.3 Edge detection using HBT filter……………………………….......46 3.2.4 Results and Discussion…………………………………………….49 3.2.5 Summary............................................................................................53 Chapter 4 IMAGE FUSION BASED ON BILATERAL SHARPNESS CRITERION IN DT-CWT DOMAIN…………………………………....54 4.1 Dual-Tree Complex Wavelet Transform.……………………………......56 4. 2 Proposed Image Fusion using DT-CWT..………………….…….…......58 4.2.1 Fusion rules for low frequency coefficients..……………………...58 4.2.2 Gradient based sharpness criterion…………………………...........58 4.2.3 Fusion rules for high frequency coefficients………….……...........60 4.3 Simulation Results and Discussions.……………………………............61 4.3.1 Quantitative evaluation ……………………………….…….........64 4.4 Summary …………………………………………………………...….....67 Chapter 5 CONCLUSIONS……………………………………………………….........68 5.1 Conclusions ………………………………………………………..........69 5.2 Future Work ……………………….………………………………........70 BIBLIOGRAPHY…………………………………………………….………………........72 DISSEMINATION OF THIS RESEARCH WORK ……………………………….…...77 Submitted by PRIYA RANJAN MUDULI [211EC3315] Page iv LIST OF FIGURES Figure No. Page No. Fig.1.1: Fundamental information fusion system block diagram……………………............4 Fig.1.2: The level classification of the various popular image fusion methods…………......4 Fig.1.3: Single sensor image fusion system …………………………………………….......5 Fig.1.4: Multi sensor image fusion system ……………………………………….…………6 Fig.2.1: Pyramid transform description with an example ………………………….....…...15 Fig.2.2: Wavelet families representation …………………………………………….….....24 Fig.2.3: Two channel wavelet filter bank ………………………………………….……....25 Fig.2.4: Filter bank structure of the DWT analysis. …………………………………….....26 Fig.2.5: Filter bank structure of the reverse DWT synthesis…..……………………….......27 Fig.2.6: Image decomposition with natural orientation of sub bands ……………….…......27 Fig.3.1: Proposed crack detection algorithm ……………………………………..……..…32 Fig.3.2: Discrete Wavelet Transform filter banks ……………………………..………..…36 Fig.3.3: Original wall image showing a hairline cracks………………..…………………..38 Fig.3.4: Canny edge detector response ……………………………………..…………...…38 Fig.3.5: HBT filter response with sigma = 0.48, Totalminerror = 0.168, gamma = 0.0208, Threshold = 0.83 ………………………………………….…39 Fig.3.6: Second largest PCA Eigen values in spatial domain for wall image ……….........39 Fig.3.7: Third largest PCA Eigen values in spatial domain for wall image ………..….…..39 Fig.3.8: Total minimum error Vs. w plot ……………………………..…………...….…..40 Fig.3.9: GUI for DWT based Image Fusion ……………………………..……...……..…..40 Fig.3.10: Fusion with 3 level Haar DWT decomposition using GUI …………..….....…....41 Fig.3.11: Image fusion response using GUI …………………………..…………………...41 Fig.3.12: Flow chart for proposed edge detection technique …………………….…….….44 Fig.3.13: Dual tree Q-shift CWT………………………………………………………..….45 Fig.3.14: Edge detection result of multi-focus clock images ……………………….……..49 Fig.3.15: Edge detection result of multi-focus Pepsi-Can images………………………....50 Fig.3.16: Total min-error Vs sigma plot for Clock image showing total min-error of 0.1627 at 0.55 …………………………………………….…………..…51 Submitted by PRIYA RANJAN MUDULI [211EC3315] Page v Figure No. Page No. Fig.3.17: Total min-error Vs sigma plot for Pepsi Can image showing total min-error of 0.3626 at 0.68 …………………………………………………………...51 Fig.3.18: PCA Eigen value e2 and e3 for fused Clock Image.…………………………….52 Fig.3.19: PCA Eigen value e2 and e3 for fused Pepsi Can Image..……………………..... 52 Fig.4.1: Dual tree of real filters for the Q-shift wavelet transform...………………….…...57 Fig.4.2: Dual Tree Complex Wavelet Transform (DT-CWT) fusion ……………………..58 Fig.4.3: LLTV and FLIR sensor image fusion responses using proposed method ………. 61 Fig.4.4: Multispectral sensor image fusion responses using proposed method …………...62 Fig.4.5: CT and MRI image fusion responses using proposed method.…………………...63 Submitted by PRIYA RANJAN MUDULI [211EC3315] Page vi LIST OF ABBREVIATIONS CT: Computerized Tomography HR: High Resolution LR: Low Resolution MR/MRI: Magnetic Resonance (Imaging) PSNR: Peak Signal to Noise Ratio NDE: Non-Destructive Evaluation DWT: Discrete Wavelet Transform DT-CWT: Dual Tree Complex Wavelet Transform HBT: Hyperbolic Tangent Filter GUI: Graphical User Interface HSV: Hue Saturation Value color representation IHS: Intensity Hue Saturation color space MRA: Multi Resolution Approach PCA: Principal Component Analysis SAR: Synthetic Aperture Radar GUI: Graphical User Interface LLTV: Low Light Television FLIR: Forward-Looking-Infrared Submitted by PRIYA RANJAN MUDULI [211EC3315] Page vii CHAPTER 1 Introduction to Image Fusion Overview Single Sensor Image Fusion System Multi-Sensor Image Fusion System Image Fusion Techniques Motivation for Image Fusion Research Objectives Thesis Organisation Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 1 Introduction 1. INTRODUCTION 1.1 Overview Image fusion is the technique of merging several images from multi-modal sources with respective complementary information to form a new image, which carries all the common as well as complementary features of individual images. With the recent rapid developments in the domain of imaging technologies, multisensory systems have become a reality in wide fields such as remote sensing, medical imaging, machine vision and the military applications. Image fusion provides an effective way of reducing thisincreasing volume of information by extracting all the usefulinformation from the source images. Image fusion provides an effective method to enable comparison andanalysis of Multi-sensor data having complementary information about the concerned region. Image fusion creates new images that are more suitable for the purposes of human/machine perception, and for further imageprocessing tasks such assegmentation, object detection or target recognition in applications such as remotesensing and medical imaging. Images from multiple sensors usually have different geometric representations, which have to be transformed to a common representation for fusion. This representation should retain the best resolution of either sensor. The alignment of multi-sensor images is also one of the most important preprocessing steps in image fusion. Multi-sensor registration is also affected by the differences in the sensor images. However, image fusion does not necessarily imply multi-sensor sources. There can be single-sensor or multi-sensor image fusion, whichhas been vividly described in this report. Analogous to other forms of information fusion, image fusion is usually performed at one of the three different processing levels: signal, feature and decision. Signal level image fusion, also known as pixel-level image fusion, represents fusion at the lowest level, where a number of raw input image signals are combined to produce a single fused image signal. Object level image fusion, also called feature level image fusion, fuses feature and object labels and property descriptor information that have already been extracted from individual input images. Finally, the highest level, decision or symbol level image fusion represents fusion of probabilistic decision information obtained by local decisionmakers operating on the results of feature level processing on image data produced from individual sensors. Figure 1.1 instances a system using image fusion at all three levels of processing. This general structure could be used as a basis for any image processing system, for example an Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 2 Introduction automatic target detection/recognition system using two imaging sensors such as visible and infrared cameras. The main objective is to detect and correctly classify objects in a presented scene. The two sensors (1 and 2) survey the scene and register their observations in the form of image signals. Two images are then fused atpixel-level to produce a third fused image and are also passed independently to local feature extraction processes. The fused imagecan be directly displayed for a human operator to aid better scene understanding or used in a further local feature extractor. Feature extractors act as simple automatic target detection systems, including processing elements such as segmentation, region characterization, morphological processing and even neural networks to locate regions of interest in the scene. The product of this process is a list of vectors describing the main characteristics of identified regions of interest. Feature level fusion is then implemented on the feature sets produced from the individual sensor outputs and the fused image. This process increases the robustness of the feature extraction process and forms a more accurate feature set by reducing the amount of redundant information and combining the complimentary information available in different individual feature sets. Feature level fusion may also produce an increase in the dimensionality of the feature property vectors. The final processing stage in an ATD system is the classification stage. Individual sensor and fused feature property vectors are input to local decision makers which represent object classifiers, assigning each detected object to a particular class with proper decision. Decision level fusion is performed on the decisions reached by the local classifiers, on the basis of the relative reliability of individual sensor outputs and the fused feature set. Fusion is achieved using statistical methods such as Bayesian inference and the Dempster-Schafer [1], [2], [3] method with the aim of maximizing the probability of correct classification for each object of interest. The output of the whole system is a set of classification decisions associated to the objects found in the observed scene. The classification of some of the most popular image fusion algorithms based on the computation source is illustrated in Figure.1.2. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 3 Introduction Fig. 1.1 An information fusion system at all three processing Fig. 1.2 Level classification of the various popular image fusion methods based on the computation source. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 4 Introduction 1.2 Single Sensor Image Fusion System The basic single sensor image fusionscheme has beenpresented in Figure 1.3. The sensor shown could be visible-band sensors or some matching band sensors. This sensor captures the real world as a sequence of images. The sequence of images are then fused together to generate anew image with optimum information content. For example in illumination variant and noisy environment, a human operator may not be able to detect objects of his interest which can be highlighted in the resultant fused image. Fig. 1.3 Single Sensor Image Fusion System The shortcoming of this type of systems lies behind the limitations of the imaging sensor that is being used. The conditions under which the system can operate, the dynamic range, resolution, etc. are all restricted by the competency of the sensor. For example, a visible-band sensor such as the digital camera is appropriate for a brightly illuminated environment such as daylight scenes but is not suitable for poorly illuminated situations found during night, or under adverse conditions such as in fog or rain. 1.3 Multi-Sensor Image Fusion System A multi-sensor image fusion scheme overcomes the limitations of a single sensor image fusion by merging the images from several sensors to form a composite image. Figure 1.4illustrates a multi-sensor image fusion system. Here, an infrared camera is accompanying the digital camera and their individual images are merged to obtain a fused image. This approach overcomes the issues referred to before. The digital camera is suitable for daylight scenes; the infrared camera is appropriate in poorly illuminated environments. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 5 Introduction Fig.1.4 Multisensory Image Fusion System The benefits of multi-sensor image fusion include [4]: i. Improved reliability – The fusion of multiple measurements can reduce noise and therefore improve the reliability of the measured quantity. ii. Robust system performance – Redundancy in multiple measurements can help in systems robustness. In case one or more sensors fail or the performance of a particular sensor deteriorates, the system can depend on the other sensors iii. Compact representation of information – Fusion leads to compact representations. For example, in remote sensing, instead of storing imagery from several spectral bands, it is comparatively more efficient to store the fused information. iv. Extended range of operation – Multiple sensors that operate under different operating conditions can be deployed to extend the effective range of operation. For example, different sensors can be used for day/night operation. v. Extended spatial and temporal coverage – Joint information from sensors that differ in spatial resolution can increase the spatial coverage. The same is true for the temporal dimension. vi. Reduced uncertainty – Joint information from multiple sensors can reduce the uncertainty associated with the sensing or decision process. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 6 Introduction 1.4 Image Preprocessing Analogous to signal processing, there are very often some concerns that have to be normalized before the final image fusion. Most of the time the images are geometrically misaligned. Registration is the techniqueto establish a spatial correspondence between the sensor images and to determine a spatial geometric transformation. The misalignment of image features is induced by various factors including the geometries of the sensors, different spatial positionsand temporal capture rates of the sensors and the inherent misalignment of the sensing elements. Registration techniques align the images by exploiting the similarities between sensor images. The mismatch of image features in multisensor images reduces the similarities between the images and makes it difficult to establish the correspondence between the images. The second issue is the difference in spatial resolution between the images developed by different sensors. There are several techniques to overcome this issue such as the Superresolution techniques [5],[6]. Another methodology is to use multi-resolution image representations so that the lower resolution imagery does not adversely affect the higher resolution imagery. 1.5 Image Fusion Techniques The most essential dispute concerning image fusion is to decide how to merge the sensor images. In recent years, a number of image fusion methods have been projected [7]. One of the primitive fusion schemes is pixel-by-pixel gray level average of the source images. This simplistic method often has severe side effects such as dropping the contrast. Some more refined approaches began to develop with the launching of pyramid transform in mid-80s. Improved results were obtained with image fusion, performed in the transform domain. The pyramid transform solves this purpose in the transformed domain. The basic idea is to perform a multiresolution decomposition on each source image, then integrate all these decompositions to develop a composite depiction and finally reconstruct the fused image by performing an inverse multi-resolution transform. A number of pyramidal decomposition techniques have beendeveloped for image fusion, such as, Laplacian Pyramid, Ratio-of-lowpass Pyramid, Morphological Pyramid, and Gradient Pyramid. Most recently, with the evolutionof wavelet based multi resolution analysis concepts, the multi-scale wavelet decomposition has begun to take the place of pyramid decomposition for image fusion. Actually, the wavelet transform can be considered one special type of pyramid Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 7 Introduction decompositions. It retains most of the advantages for image fusion but has much more complete theoretical support.The real Discrete Wavelet Transform (DWT) has the property of good compression of signal energy. Perfect reconstruction is possible using short support filters. The unique feature of DWT is the absence of redundancy and very low computation. Therefore, DWT has been used extensively for Multi Resolution Analysis (MRA) based image fusion. The Discrete Wavelet Transform primarily suffers from the various problems (Ivan, W. Selesnick, Richard G. Baraniuk, and Kingsbury, N., 2005) such as oscillations, aliasing, shift variance and lack of directionality. The ringing artefacts introduced by DWT are also completely eliminated by the implementation of Dual Tree Complex Wavelet (DTCWT) based image fusion methods. The research work proposed in this thesis deals with the development and implementation of some novel Discrete Wavelet Transform based image fusion techniques. A novel image fusion approach based on bilateral sharpness measure by the help of Dual-Tree Complex Wavelet Transform has been proposed in the later part of the thesis.For all the image fusion work demonstrated in this thesis, it has been assumed that the input images must be of the same scene, i.e. the fields of view of the sensors must contain a spatial overlap. Again, the input images are assumed to be spatially registered and of equal size as well as equal spatial resolution. 1.6 Motivation The motivation for image fusion research is mainly due to the contemporary developments in the fields of multi-spectral, high resolution, robust and cost effective image sensor design technology. Since last few decades, with the introduction of these multisensory imaging techniques, image fusion has been an emerging field of research in remote sensing, medical imaging, night vision, military and civilian avionics, autonomous vehicle navigation, remote sensing, concealed weapons detection, various security and surveillance systems applications.There has been a lot of improvement in dedicated real time imaging systems with the high spatial, spectral resolution as well as faster sensor technology. The solution for information overloading can be met by a corresponding increase in the number of processing units, using faster Digital Signal Processing (DSP) and larger memory devices. This solution however, can be quite expensive. Pixel-level image fusion algorithms represent an efficient solution to this problem of operator related information overload. Pixel Level Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 8 Introduction fusion effectively reduces the amount of data that needs to be processed without any significant loss of useful information and also integrates information from multi-spectral sensors. Explicit inspiration for the research work has come from the necessity to develop some competent image fusion techniques along with the enhancement of existing fusion technologies. Furthermore, aNon-Destructive Testing (NDT) has been a popular analysis technique used in industrial product evaluation and for troubleshooting in research work without causing damage which can also save both money and time. There has always been the requirement of some novel edge detection techniques based on NDT for detection of faults in industrial products suppressing the diversity and complexity of measuring environment.Using the wavelet based Multiresolution analysis techniques and some efficient edge detection technique, it is possible to accomplish distortion less fusion which results in a reduced loss of input information. The proposed novel fusion methods in this research work also exhibit improvement with respect to objective as well as subjective evaluation point of view as compared to some of the existing image fusion techniques. 1.7 Objectives The objectives of the thesis are as follows. i. Development of a novel crack detection technique using discrete wavelet transform based image fusion suppressing the diversity and complexity of imaging environment. ii. Development of an effective edge detection technique for multi-focus images using Dual-Tree Complex Wavelet Transform (DT-CWT) based image fusion technique. iii. Development and implementation of an improved image fusion technique based on Bilateral Sharpness Criterion in DT-CWT Domain. 1.8 Thesis Organisation Including the introductory chapter, the thesis is divided into 5 chapters. The organization of the thesis is presented below. Chapter-2 Literature Review This chapter illustrates the chronological evolution of some competitive image fusion algorithms from various publications both in the fields of pixel-level fusion and performance evaluation. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 9 Introduction Chapters-3 Image Fusion and Edge Detection This chapter is devoted to the first and second objectives. In the first part of this chapter, the complete methodology and illustration of crack detection technique for non-destructive evaluation in civil structures has been performed using Discrete Wavelet Transform (DWT) based image fusion. It also reveals the detail exploration of two competitive edge detectors, i.e. Canny edge detector and Hyperbolic Tangent (HBT) based edge detector. The second part of this chapter proposes a novel edge detection technique for multi-focus images using complex wavelet based image fusion algorithm. Chapter – 4 Image Fusion based on Bilateral Sharpness Criterion in DT-CWT Domain In this chapter, an improved DT-CWT based image fusion technique has been developed to generate a resultant image with better perceptual as well as quantitative image quality indices. The competency of the proposed technique is properly justified by comparing its response with traditional DWT as well as Complex Wavelet based image fusion. Chapter – 5 Conclusions The overall conclusion of the thesis is presented in this chapter. It also contains some future research areas, which need Submitted by PRIYA RANJAN MUDULI [211EC3315] attention and further investigation. Page 10 Literature Review CHAPTER 2 Literature Review Multiresolution Pyramidal Image Fusion Wavelet Transform based Image Fusion Algorithms Discrete Wavelet Transform Classification of Wavelets Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 11 Literature Review 2 LITERATURE REVIEW Since last few decades, an extensive number of approaches to fuse visual image information. These techniques vary in their complexity, robustness and sophistication. Remote sensing is perhaps one of the leading image fusion applications with a large number of dedicated publications. The main principle of some of the popular image fusion algorithms have been discussed below. Fusion using Principle Component Analysis (PCA): The PCA image fusion method [8] basically uses the pixel values of all source images at each pixel location, adds a weight factor to each pixel value, and takes an average of the weighted pixel values to produce the result for the fused image at the same pixel location. The optimal weighted factors are determined by the PCA technique. The PCA image fusion method reduces the redundancy of the image data. Super-resolution image reconstruction: Super-resolution (SR) reconstruction [9] is a branch of image fusion for bandwidth extrapolation beyond the limits of a traditional electronic image system. Katartzis and Petrou describe the main principles of SR reconstruction and provide an overview of the most representative methodologies in the domain. The general strategy that characterizes super-resolution comprises three major processing steps which are low resolution image acquisition, image registration/motion compensation, and high resolution image reconstruction. Katartzis and Petrou presented a promising new approach based on Normalized Convolution and a robust Bayesian estimation, and perform quantitative and qualitative comparisons using real video sequences.. Image fusion schemes using ICA bases: Mitianoudis and Stathaki demonstrate the efficiency of a transform constructed using Independent Component Analysis (ICA) and Topographic Independent Component Analysis based for image fusion in this study [10]. The bases are trained offline using images of similar context to the observed scene. The images are fused in the transform domain using novel pixel-based or region-based rules. An unsupervised adaption ICA-based fusion scheme is also introduced. The proposed schemes feature improved performance when compared to approaches based on the wavelet transform and a slightly increased computational complexity. The authors introduced the use of ICA and topographical ICA based for image fusion applications. These bases seem to construct very efficient tools, which can complement common Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 12 Literature Review techniques used in image fusion, such as the Dual-Tree Wavelet Transform. The proposed method can outperform the wavelet approaches. The Topographical ICA based method offers a more accurate directional selectivity, thus capturing the salient features of the image more accurately. Region-based multi-focus image fusion: Li and Yang first describe the principle of region-based image fusion in the spatial domain [11]. Then two region-based fusion methods are introduced. They proposed a spatial domain region-based fusion method using fixed-size blocks. Experimental results from the proposed methods are encouraging. More specifically, in spite of the crudeness of the segmentation methods used, the results obtained from the proposed fusionprocesses, which consider specific feature information regarding the source images, are excellentin terms of visual perception. The presented algorithm, spatial domain region-based fusion method using fixed-size blocks, is computationally simple and can be applied in real time. It is also valuable in practical applications. Although the results obtained from a number of experiments are promising, there are more parameters to be considered as compared to an MR-based type of method, such as the wavelet method. Adaptive methods for choosing those parameters should be researched further. In addition, further investigations are necessary for selecting more effective clarity measures. Image fusion techniques for non-destructive testing and remote sensing application: Theauthors present several algorithms of fusion based on multi-scale Kalman filtering and computational intelligence methodologies [12]. The proposed algorithms areapplied to two kinds of problems: a remote sensing segmentation, classification, and object detection application performed on real data available from experiments and a nondestructive testing/evaluation problem of flaw detection using electro-magnetic and ultrasound recordings. In both problems, the fusion techniques are shown to achieve a modest superior performance with respect to the single-sensor image modality. The joint use of the eddy current and ultrasonic measurements is suggested because of the poor results that are obtained by processing each single recorded type of signal alone. Therefore, both measurements are jointly processed, and the information used to perform the classification has been extracted at three different levels: pixel, feature, and symbol. The numerical performance of these techniques has been compared by using the probability of detection and probability of false alarm. Experiments performed on real data confirmed the effectiveness of the proposed SL based approach, by maximizing the Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 13 Literature Review probability of detection and achieving an acceptable probability of false alarm with respect to the PL and FL fusion techniques. 2.1 Multi-resolution Pyramidal Image Fusion Hierarchical multiscale and multiresolution image processing techniques, as mentioned previously, are the basis for the majority of sophisticated image fusion algorithms. The usefulness of such approaches to image processing was initially established by Burt and Adelson [13, 14]. Multiresolution processing methods enable an image fusion system to fuse image information in a suitable pyramid format. Image pyramids are made up of a series of Sub-band signals, organized into pyramid levels, of decreasing resolution each representing a portion of the original image spectrum. Information contained within the individual sub-band signals corresponds to a particular scale range, i.e. each sub-band contains features of a certain size. Coarse resolution pyramid levels contain large scale information while those of higher resolution contain finer detail from the original image signal. Fusing images in their pyramid representation therefore, enables the fusion system to consider image features of different scales separately even when they overlap in the original image. By fusing information in the pyramid domain, superposition of features from different input images is achieved with a much smaller loss of information than in the case of single resolution processing where cut and paste or arithmetic combination methods are used. Furthermore, this scale reparability also limits damage of sub-optimal fusion decisions, made during the feature selection process, to a small portion of the spectrum. These properties make multiresolution fusion algorithms potentially more robust than other fusion approaches. Multiresolution image processing was first applied to pixel-level image fusion using derivatives of the Gaussian pyramid representation [13] in which the information from the original image signal is represented through a series of (coarser) low-pass approximations of decreasing resolution. The pyramid is formed by iterative applicationof low-pass filtering, usually with a 5x5 pixel Gaussian template, followed by subsampling with a factor 2, a process also known as reduction. All multiresolution image fusion systems based on this general approach exhibit a very similar structure which is shown in the block diagram of Figure 2.1. Input images obtained from different sensors are first decomposed into their Gaussian pyramid representations. Gaussian pyramids are then used as a basis for another type of high pass pyramids, such as the Laplacian, which contain, at each level, only Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 14 Literature Review information exclusive to the corresponding level of the Gaussian pyramid. HP pyramids represent a suitable representation for image fusion. Important features from the input images are identified as significant coefficients in the high pass pyramids and they are transferred (fused) into the fused image by producing a new, fused, high pass pyramid from the coefficients of the input pyramids. The process of selecting significant information from the input pyramids is usually referred to as feature selectionand the whole process of forming a new composite pyramid is known as pyramid fusion. The fused pyramid is transformed into the fused image using a multiresolution reconstruction process. This process is dual to the decomposition andinvolves iterative expansion (up-sampling) of the successive levels of the fused Gaussian pyramid and combination (addition in the case of Laplacian pyramids) with the corresponding levels of the fused high pass pyramid, known as expandoperation. Fig. 2.1 Pyramid Transform description with an example Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 15 Literature Review The first multiresolution image fusion research work done using derivatives of the Gaussian pyramid was at the TNO Institute for perception in the Netherlands. Toet et. al. presented an algorithm based on the contrast or Ratio of Low Pass (RoLP) pyramid [15]. In this representation each level of the RoLP pyramid is formed as the ratio of the corresponding level of the Gaussian pyramid and the expanded version of its low-pass approximation (the next level). The coefficients of the RoLP pyramid, reduced by unity, represent an approximation of the local luminance contrast, C, as defined by Weber: C L 1 Lb (2.1) Where,L is the local luminance given by the signal value at the current level and Lb is the background luminance approximated by its low-pass approximation. RoLP pyramid fusion is achieved as the maximization of the local luminance contrast at each position and scale, by choosing and transferring the input pyramid coefficient corresponding to the greatest local contrast into the fused pyramid. Finally, the fused image is obtained from the fused RoLP pyramid by recursively expanding the lowest level of the Gaussian pyramid and multiplying by the corresponding levels of the fused RoLP pyramid until all the levels of the fused pyramid are used up. Further to this fusion system, the same author presented a multiscale contrast enhancement technique that increases the performance of the RoLP fusion process [16]. Contrast enhancement results in fusion performance that is independent of changes in lighting and gray-level gradients, and is achieved through non-linear multiplication of successive layers of the RoLP pyramid. The usefulness of this technique was demonstrated on fusion of degraded visual and infrared images. The contrast pyramid [16] was also used in another interesting fusion approach presented by Cui et. al. [17]. In their case, the fused pyramid was obtained by multiplying the corresponding levels of the input contrast pyramids. The main advantage of using this pyramid approach is that by avoiding the selection process, an efficient implementation can be obtained. Indeed, the authors reported real time operation at input image resolution level of 256x256 pixels and quasi real time at 512x512, when implemented on high-speed DSP devices. Generally however, the RoLP (contrast) pyramid fusion suffers from instability due to the multiplication/division operations used in the decomposition and reconstruction which often leads to the introduction of false edges in the fused image and amplification of noise that might be present in the inputs. The quality of the fused image is clearly good and reconstruction artifacts are not easily noticeable, however false edges are also obvious, such as on the roofs in the top right of the image. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 16 Literature Review An alternative multiresolution pyramid representation derived from the Gaussian and used for pixel-level image fusion is the Laplacian pyramid [18, 19]. Similarly to the RoLP pyramid used by Toet, each level of the Laplacian pyramid is formed as a difference between the corresponding level of the Gaussian and the expanded version of its low-pass approximation. Although the coefficients (pixels) of the Laplacian pyramid are not direct representations of the local contrast like those of the RoLP pyramid, the value of these coefficients is still proportional to the saliency of the high frequency detail at a given location. Saliency in the context of information fusion signifies perceptual importance of visual information in an image. Pavel et. al. [18] used the Laplacian pyramid approach to fuse simulated passive millimeter wave (PMMW) images with synthetic images formed from the information obtained from terrain databases. They use arithmetic pyramid fusion, where the fused pyramid coefficients take the value of a weighted sum of the input coefficients. The corresponding equation is DlF (n, m) K lA (n, m) DlA (n, m) K lB (n, m) DlB (n, m) (2.2) Where DlF (n, m) , DlA (n, m) and DlB (n, m) represent coefficients of the fused and input pyramids, at level l and position (n, m), respectively. Weighting coefficients K lA (n, m) , K lB (n, m) determine the relative influence of each input on the fused pyramid at thatposition and scale. In the system by Pavel et. al., the size of the weighting coefficientsdepends on the local uncertainty of the PMMW image, measured through variance, andthe level of correlation between the input pyramid coefficients [18]. The pyramid fusion method used by Akerman [19] employs a coefficient selection approach. It is based on a pixel by pixel selection but the selection rule was left to be flexible and application dependent. The most common coefficient selection is the pixel-based select max approach where the fused coefficient takes the value of the input with the largest absolute value, as expressed by Equation 2.3 as. DlF (n, m) = DlA (n, m) if DlA (n, m) DlB (n, m) (2.3) DlB (n, m) Otherwise Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 17 Literature Review The usefulness of Laplacian pyramid fusion in remote sensing applications was further demonstrated in the work by Aiazzi et. al. [20]. They used a generalized Laplacian pyramid (GLP) approach to solve the most common problem in remote sensing image fusion that of increasing the resolution of multi-spectral (color) images with high resolution panchromatic (monochrome) images. By replacing the reduce and expand operations of the Gaussian pyramid multiresolution decomposition reconstructionprocesses with reduce{expand{ }} and expand{reduce{ }},respectively, using low pass filters with appropriate cut-off frequencies and corresponding decimationinterpolation factors (p and q), the GLP approach allows a reduction in resolution by a rational scale factor, p : q. In this way, images whose resolution ratios are not powers of 2 can be fused without having to be resampled. Fusion resolution enhancement is then achieved by simple level replacement in the pyramid domain when the highest resolution level of the panchromatic Laplacian pyramid becomes the missing highest resolution level for each channel pyramid of the multi-spectral image. This scheme is significant in its applicability to a wide range of remotely sensed data in addition to slightly superior performance compared with the wavelet based approach. The gradient pyramid fusion presented by Burt and Kolczynski is another important pixel-level image fusion method based on the Gaussian pyramid approach [21]. Their work represents an extension of the Laplacian pyramid representation in that visual information, in the gradient pyramid, is separated into sub-bands according to direction as well as scale. Gradient pyramid is derived from the filter-subtract-decimate (FSD) Laplacian pyramid by applying four directionally sensitive filters. When applied at all levels of scale, each filter removes all the information that does not fall within a well-defined orientation range, which results in four oriented Laplacian pyramids which are then fused independently. That the four directional filters are complementary means that the original Laplacian pyramid is obtained by a direct summation of the four oriented pyramids. Indeed, the final fused image is obtained by conventional Laplacian pyramid reconstruction from the fused pyramid produced in this way. More recently, a multi-scale image fusion system for visual display was proposed by Peli et. al.[22]. Multi-scale image analysis is based on a series of oriented octave band-pass filters which separate the original input spectra into a series of sub-bands according to scale and orientation. Sub-band signals of different input images are fused by a simple pixel by pixel selection using a criterion based on the local contrast evaluation. There is also an improvement using a different number of orientations in multi-scale from two to four different orientations. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 18 Literature Review There are various types of pyramid transforms. Some of these are as the follows: Filter Subtract Decimate Pyramid Gradient Pyramid Laplacian Pyramid Ratio Pyramid Morphological Pyramid The concise multi-resolution analysis based pyramidal image fusion methodology can be illustrated with the three major phases: Decomposition Formation of the initial image for decomposition. Recomposition Decomposition is the process where a pyramid is generated successively at each level of the fusion. The depth of fusion or number of levels of fusion is pre decided. The number of levels offusion is decided based on the size of the input image. The recomposition process, in turn, formsthe finally fused image, level wise, by merging the pyramids formed at each level to the decimated input images. Decomposition phase basically consists of the following steps. These steps are performed number of times till the levels to which the fusion will be performed. The different pyramidal methods have a predefined filter with which the input images convolved/filtered. Formation of the pyramid for the level from the filtered input images usingBurt’s method or Li’s Method. The input images are decimated to half their size, which would act as the input imagematrices for the next level of decomposition. Merging the input images is performed after the decomposition process. This resultant image Matrix would act as the initial input to the recomposition process. The finally decimated input pair of images is worked upon the decimated input image by means of suitable fusion rules. The recomposition is the process wherein, the resultant image is finally developed from the pyramids formed at each level of decomposition. The various steps involved in the recomposition phase are discussed below. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 19 Literature Review The input image to the level of recomposition is undecimated The undecimated matrix is convolved/filtered with the transpose of the filter vector used in the decomposition process The filtered matrix is then merged, by the process of pixel intensity value addition, with the pyramid formed at the respective level of decomposition. The newly formed image matrix would act as the input to the next level of recomposition. The merged image at the final level of recomposition will be the resultant fused image. The flow of the pyramid based image fusion can be explained by the following an example of multi-focus image as depicted in Fig.2.1 2.2 Wavelet Transform based Image Fusion Algorithms The Discrete Wavelet Transform (DWT) was successfully employed in the field of image processing with the introduction of Mallat’s algorithm [25]. It enabled the application of twodimensional DWT using one dimensional filter banks. DWT based multiresolution approach has been implemented successfully in chapter 3. Its general structure, briefly describe here, is very similar to that of the Gaussian pyramid based approach. Input signals are transformed using the wavelet decomposition process into the wavelet pyramid representation. Contrary to Gaussian pyramid based methods, high pass information is also separated into different subband signals according to orientation as well as scale. The scale structure remains logarithmic, i.e. for every new pyramid level the scale is reduced by a factor of 2 in both directions. The wavelet pyramid representation has three different sub-band signals containing information in the horizontal, vertical and diagonal orientation at each pyramid level. The size of the pyramid coefficients corresponds to “contrast” at that particular scale in the original signal, and can therefore, beused directly as a representation of saliency. In addition, wavelet representation is compact, i.e. the overall size of all sub-band signals in the pyramid is the same as the size of the original image The size difference, as well as the lack of expansion operations during wavelet decomposition makes the wavelet approach much more efficient in terms of the processing required to fuse two images. Advantages of these properties in fusion applications were demonstrated by the considerable number of publications on the subject of wavelet image fusion in the last five years. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 20 Literature Review One of the first wavelet based fusion systems was presented by Li et. al.[24]. It uses Mallat's technique to decompose the input images and an area based feature selection for pyramid fusion. In the proposed system, Li et. al. use a 3x3 or a 5x5 neighborhood to evaluate a local activity measure associated with the center pixel. It is given as the largest absolute coefficient size within the neighborhood. In case of coefficients from the two input pyramids exhibiting dissimilar values, the coefficientwith the largest activity associated with it is chosen for the fused pyramid. Otherwise, similar coefficients are simply averaged to get the fused value. Finally, after the selection process, a majority filter is applied to the binary decision map to remove bad selection decisions caused by noise “hot-spots”. This fusion technique works well at lower pyramid levels, but for coarser resolution levels, the area selection and majority filtering, especially with larger neighborhood sizes, can significantly bias feature selection towards one of the inputs. Almost contemporarily with the formermethod, wavelets in image fusion were also considered by Chipman et. al. [25]. The algorithm was basically deals with the general aspects of wavelet fusion. A comparison was exercised between the conventional isotropic and more exotic tensor wavelet pyramid representation, in which decomposition is performed in one direction only. The inference was that isotropic representation produces better fusion results. For pyramid fusion methods they advised flexibility, suggesting that an “optimal solution” should be sought for each application independently. More importantly, they considered problems associated with wavelet image fusion. Miss-registration of the inputs and the loss of coefficients were deemed as having the worst effects on the fused image quality. These effects produce what is known as "ringing" artifacts – shadowing and rippling effects, especially around strong edges. Finally, the authors also considered noise removal incorporated in the fusion process. They suggested hard thresholding of wavelet coefficients at lower pyramid levels as a possible solution. Another significant contribution to the field of wavelet image fusion was given by Yocky [26]. He investigated wavelet image fusion to increase the resolution of multi-spectral satellite images with high resolution panchromatic data. The basic principle is that of pyramid enlargement, i.e. higher detail levels of the panchromatic pyramid are appended to the multispectral pyramids to provide the missing detail information. The number of levels added depends on the final resolution requirement or the maximum resolution available in the panchromatic pyramid. Wavelet pyramid extension, to increase resolution of multi-spectral low resolution satellite images, was also proposed by Garguet-Duport et. al. [27] in a system very similar to that proposed by Yocky [26]. Concealed weapon detection (CWD) is another Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 21 Literature Review application which has benefited from the use of multiresolution wavelet based image fusion techniques. The CWD system proposed by Ramac et. al. [28] uses the same image fusion method based on the wavelet decomposition followed by Burt and Kolczynski's feature selection algorithm. This time however, image fusion is applied to low level processed multisensory images obtained from infrared and millimeter wave (MMW) cameras. Image fusion is applied after morphological filtering, but prior to feature extraction. Their results again show that fusion improves detection and that morphological filtering removes some unwanted noise artifacts that degrade the fused result. Wang et. al. [29] also proposed a wavelet based image fusion algorithm for fusion of low light dual spectrum (visual and infrared) images. The system uses conventional wavelet decomposition technique and a target contrastmaximization mechanism to fuse input pyramids. Target contrast is evaluated in according to the ratio of the wavelet coefficient and local brightness evaluated over a 5x5 template. Chibani and Houacine [30] examined the effects of multiscale versus multiresolution wavelet approaches to image fusion. Multiscale wavelet approach corresponds to the redundant wavelet pyramid representation where all sub-band signals remain at the same resolution, i.e. there is no sub-sampling. The multiresolution approach is the isotropic decomposition obtained by applying Mallat's algorithm. The authors report that fusion using the redundant representation exhibits better results in terms of preserving the consistency of dominant features and the fidelity of finer details when fusing images with different focus points. The reason for this is the reduction in the reconstruction error (ringing artifacts) caused by the reduced sensitivity of the over complete multiscale wavelet fusion to discontinuities introduced in the pyramid fusion process. A mechanism for wavelet fusion of image sequences has been also proposed by Rockinger and Fechner [31]. To achieve temporal stability and consistency in the fused sequence, the system uses a shift invariant extension of the two dimensional discrete wavelet transform (SIDWT). SIDWT is a multiscale, redundant wavelet representation that does not decimate the filtered signals. Instead, analysis filters are interpolated by inserting zeros between impulse response coefficients to change the pass-band cut-off. Pyramid fusion of input sequences is implemented through selective fusion of sub-band coefficients and modified averaging fusion of the low-pass residuals. The SIDWT based fusion is reported to produce significantly better results in terms of the temporal stability in fused multisensor sequences compared to conventional multiresolution DWT fusion. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 22 Literature Review Finally, Zhang [32] presented a wide-ranging exploration of multiresolution pixel-level image fusion. A number of different multiresolution and pyramid fusion approaches were verified. Laplacian and both isotropic and shift invariant wavelet representations were tested with matching pyramid fusion mechanisms. In terms of pyramid fusion, different vertical and horizontal integration/grouping methods, area and pixel based selection mechanisms and selection consistency verification strategies were combined to obtain “optimal” fusion. According to the results presented, the shift invariant wavelet representation fusion using a rank-filter-based activity measurement, evaluated in a window of coefficients as criterion for a choose-max selection with multiscale selection grouping and followed by region based consistency verification, produced the best results. Further to the problem of image fusion, this work also considers a number of other issues connected to image fusion such as multisensor image registration and fusion performance in the presence of sensor noise. 2.2.1 Discrete wavelet transform The Wavelet Transform provides a time-frequency representation of the signal. It was developed to overcome the shortcoming of the Short Time Fourier Transform (STFT), which can also be used to analyze non-stationary signals. While STFT gives a constant resolution at all frequencies, the Wavelet Transform uses multi-resolution technique by which different frequencies are analyzed with different resolutions. Classification of wavelets We can classify wavelets into two fundamental classes: (a) orthogonal and (b) biorthogonal. (a)Features of orthogonal wavelet filter banks The coefficients of orthogonal filters are real numbers. The filters are of the same length and are not symmetric. The low pass filter, G and the high pass filter, H are related to each other 0 H 0 ( z ) Z N G0 (Z 1 ) by 0 (2.4) The two filters are alternated flip of each other. The alternating flip automatically gives double-shift orthogonality between the low pass and high pass filters, i.e., the scalar product of the filters, for a shift by two is zero. i.e., ΣG[k]H[k-2l] = 0, where k,lЄ Z . Perfect reconstruction is possible with alternating flip. Orthogonal filters offer a high number of vanishing moments. This property is useful in many signal and image processing applications. They have regular structure which leads to easy implementation and scalable architecture. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 23 Literature Review (b)Features of biorthogonal wavelet filter banks In the case of the biorthogonal wavelet filters, the low pass and the high pass filters do not have the same length. The low pass filter is always symmetric, while the high pass filter could be either symmetric or anti-symmetric. The coefficients of the filters are either real numbers or integers. For perfect reconstruction, biorthogonal filter bank has all odd length or all even length filters. The two analysis filters can be symmetric with odd length or one symmetric and the other antisymmetric with even length. Also, the two sets of analysis and synthesis filters must be dual. Wavelet families There are a number of basis functions that can be used as the mother wavelet for Wavelet Transformation. Since the mother wavelet produces all wavelet functions used in the transformation through translation and scaling, it determines the characteristics of the resulting Wavelet Transform. Therefore, the details of the particular application should be taken into account and the appropriate mother wavelet should be chosen in order to use the Wavelet Transform effectively. Figure 2.2 illustrates some of the commonly used wavelet functions. Fig. 2.2 Wavelet families (a) Haar (b) Daubechies4 (c) Coiflet1 (d) Symlet2 (e) Meyer (f) Morlet (g) Mexican Hat. Haar wavelet is one of the oldest and simplest wavelet. Daubechies wavelets are the most popular wavelets. They represent the foundations of wavelet signal processing and are used in numerous applications. There exists another type of wavelets called Maxflat wavelets. Here, Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 24 Literature Review the frequency responses have maximum flatness at frequencies 0 and π. This is a very desirable property in some applications. The Haar, Daubechies, Symlets and Coiflets are compactly supported orthogonal wavelets. These wavelets along with Meyer wavelets are capable of perfect reconstruction. The Meyer, Morlet and Mexican Hat wavelets are symmetric in shape. The wavelets are chosen based on their shape and their ability to analyse the signal in a particular application. The wavelet transform provides a multi-resolution decomposition of an image in a biorthogonal basis and results in a non-redundant image representation. This basis is called wavelets, and they are functions generated from one single function, called mother wavelet, by dilations and translations. Although this is not a new idea, whatmakes this transformation more suitable than other transformations such as the Fourier Transform or the Discrete Cosine Transform, is the ability of representing signal features in both time and frequency domain. Figure 2.3 shows an implementation of the discrete wavelet transform. In this filter bank, the input signal goes through two one-dimensional digital filters. One of them, H0, performs a high pass filtering operation and the other H1 low pass one. Each filtering operation is followed by subsampling by a factor of 2. Then, the signal is reconstructed by first up sampling, then filtering and summing the sub bands. Fig. 2.3 Two channel wavelet filter bank The synthesis filters F0and F1 must be specially adapted to the analysis filters H0 and H1 to achieve perfect reconstruction. By considering the z-transfer function of the 2-chanel filter bank shown in Figure 2.3, it is easy to obtain the relationship that those filters need to satisfy. After analysis, the two subbands are: 1 H 0 ( z 1 / 2 ) X ( z 1 / 2 ) H 0 ( z 1 / 2 ) X ( z 1 / 2 ) 2 Submitted by PRIYA RANJAN MUDULI [211EC3315] (2.5) Page 25 Literature Review 1 H 1 ( z 1 / 2 ) X ( z 1 / 2 ) H 1 ( z 1 / 2 ) X ( z 1 / 2 ) 2 (2.6) The combined filter bank in z-domain is given by ^ X ( z) 1 F0 ( z) H 0 ( z) F1 ( z) H1 ( z)X ( z) 1 F0 ( z) H 0 ( z) F1 ( z) H1 ( z)X ( z) (2.7) 2 2 In order to eliminate the problems of aliasing and distortion, the following conditions must be satisfied: F0 ( z) H1 ( z) F1 ( z ) H 0 ( z ) The final filtering equation with the delay term by Smith and Barnwell is presented as: ^ X ( z) 1 N z H 0 ( z ) H 0 ( z 1 ) H 0 ( z ) H 0 ( z 1 ) X ( z ) 2 (2.8) The multiscale pyramid decomposition and reconstruction of an image with high and low pass filtering is shown below. Fig. 2.4 Filter bank structure of the DWT Analysis. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 26 Literature Review Fig. 2.5 Filter bank structure of the reverse DWT Synthesis Successive application of this decomposition to the LL sub band gives rise to apyramid decomposition where the sub images correspond to different resolution levels and orientations. Fig. 2.6 Image decomposition. Each subband has a natural orientation. After one level of decomposition, there will be four frequency bands, namely Low-Low (LL), Low-High (LH), High-Low (HL) and High-High (HH). The next level decomposition is just applied to the LL band of the current decomposition stage, which forms a recursive decomposition procedure. Thus, an N-level decomposition will finally have 3N+1 different frequency bands, which include 3N high frequency bands and just one LL frequency band. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 27 Image Fusion & Edge Detection CHAPTER 3 Image Fusion & Edge Detection Crack Detection using Image Fusion Edge Detection Technique for Multi-focus Images using Image Fusion Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 29 Image Fusion & Edge Detection 3. IMAGE FUSION AND EDGE DETECTION Analysis of image contents has been a substantial target for computer vision as well as image processing researchers since last few decades. An image carries variety of information regarding contour, colour, as well as orientation. The first step for contour extraction begins with the detection of edges. This realism exposes the real significance of edge detection techniques in image processing field. Edge detection has a wide range of applications in image compression, enhancement of images, watermarking, morphological operations, and restoration process and so on. The most important advantage of edge detection is that it reduces the bulky data in an image, upholding the structural attributes for further processing. The introduction of multi-sensory image fusion techniques have pointed towards a new dimension of research work in edge detection process. This chapter of the research work illustrates the development and implementation of some novel edge detection techniques using image fusion algorithms. 3.1 Crack Detection Using Image Fusion Since last few decades, Non Destructive Technique (NDT) has been concerning field of research for quality evaluation of civil structures, aerospace engineering and industrial products. In civil structures, the typical foundation crack will run vertically or at an angle. Although human operator based crack detection methods have successfully illustrated that by manually tracking the start and end of a crack, one can use pixel-based algorithms to deﬁne the crack characteristics. Many literatures concerning tracking of defects in civil structures are unable to identify the crack edges accurately due to poor contrast, uneven illumination and noisy environment. Complications due to the inherent noise in the scanning process, irregularly shaped cracks, as well as wide range of background patterns are also challenges for error free detection in camouflaged environment. Therefore a new crack detection technique is required which is based on Non Destructive Evaluation (NDE) along with some efficient edge detection algorithms and an efficient image fusion technique to combat contrast, noise sensitivity and uneven illumination. Since more than 25 years, so many systems have been developed which basically deals with detection of linear features on optic imaging [33]. Basically it has been tried to combine a local criterion using evaluating radiometry and a global criterion using wide scale knowledge for edges to be detected. In many cases local criterion are insufficient in detecting very fine crack edges. Some classical gradient-magnitude (GM) methods [34], [35] are usually dependent on edge strength; hence, Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 30 Image Fusion & Edge Detection weaker edges due to texture may not be detected. An alternative method for detecting edges regardless of their magnitude is being proposed [36]. It is based on the computation of the cosine of the projection angles between neighbourhoods and predefined edge filters. So it is otherwise known as an angle-based (AN) method. But this technique is very sensitive to noise and uneven illumination. Local thresholding of image gradients are also sensitive to uneven illumination since they inhibit low luminance regions. The improved method based on phase congruency described the frequency domain image representation [37]. Since an edge exists near points of maximum phase congruency, such methods are invariant towards uneven illumination and low contrast. Due to the use of the log polar Gabor filter, they produce poorer edge localization in those false edges and are detected in the vicinity of sharp transitions. A contrast invariant edge detection method [38] based on the Helmholtz principle describes edges as geometric structures with large deviations from randomness; but, sensitive to the window size and edge localization. The other filter projected by Marr and Hildreth suffers from the problems affined to zero-crossing approach [34]. This approach is basically undeviating in edge localization, provided these are properly separated when the SNR in the image is high. Again the localization of the real edge dislodges for a bounded width staircase steps. The secondary issue is related to the identification of false edges. Laplacian of Gaussian filter also can’t deal with the missing edges. However, merging Laplacian of Gaussian filtering and zero crossing approach is a unmanageable job. Because, an edge does not cope with a zero crossing for very confined number of steps. A robust edge detection algorithm [39] produces superior result than the methods discussed above. This method basically emphasizes on optimizing two of Canny’s criteria- accurate edge detection and localization, without explicitly including the third criterion i.e. minimal response. 3.1.1 Proposed crack detection technique We have proposed a model for an efficient and reliable crack detection, which combines the best features of canny edge detection algorithm and Hyperbolic Tangent filtering technique using an efficient Max-Mean image fusion rule. Here the detection architecture consists of the some major steps as follows: 1. Acquisition of concerned wall image. 2. Crack detection using two efficient algorithms. 3. Wavelet decomposition and Fusion. The proposed algorithm is shown in Fig.3.1 Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 31 Image Fusion & Edge Detection Concerned Wall Image Canny Edge Detector HBT Edge Detector HAAR DWT Max-Mean Fusion Inverse Wavelet Transform Cracks Detected Fig .3.1 Proposed crack detection algorithm Acquisition of concerned wall image Since the quality of detection result dominantly depend on the quality of the acquisition process, the choice of acquisition system must be done carefully. Normally image acquisition by means of 2D sensors needs image processing technique. In this experimental work , the cracked wall image sample is acquired by means of a camera with focal length of 4mm, exposure time: 0.002 sec, max aperture: 3.5. The lighting system should be designed in order to preserve the crack edges which may not well contrast and negligible as compared to wall image .The illumination problem can be solved by means of a stereoscopic system. Crack Detection The edge detection algorithm [39] is based on the actual profiles of image edges and it optimizes only two of Canny’s criteria i.e. accurate edge detection and localization. It doesn’t include the third criterion- minimal response i.e. a given edge in the image should only be marked once, and where possible, image noise should not create false edges. So we have selected canny detector to fulfil the third criterion. Again, from the spatial and frequency properties of HBT filter, It is clearly observed that the family of FIR HBT filters has a narrow bandwidth, indicating better noise reduction compared to Canny’s Gaussian first derivative. Hence, our proposed edge detection architecture gives superior result by the fusion of Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 32 Image Fusion & Edge Detection common as well as complementary features of Canny and HBT based edge detection techniques. a) Canny edge detector Canny considered three criteria desired for any edge detector such as good detection, good localization, and minimal response. The technique is basically known as feature synthesis. The image is smoothed using Gaussian convolution followed by a 2D first derivative operator. Then, non-maximal suppression technique is applied using two thresholds. Usually for good result, the upper tracking threshold can be set quite high and lower threshold quite low [42]. A wide Gaussian kernel reduces the sensitivity of detector. The edge detected by canny operator are much more smooth and hence more tolerance to noise. So in this research work we have considered canny detector. b) HBT Filter based edge detector An edge similarity measurement based algorithm by Saravana Kumar [39] gives superior result than GM and AN method. This technique is more rugged irrespective of diversity in illumination, contrast and noise level. The filtering technique basically highlights the edge similarities between image adjacency and directional finite impulse response by means of hyperbolic tangent figuration. This edge detection technique based on similarity measurement results an optimal identification of image edges by using principal component analysis. The Principal Component Analysis is applied to a set of local neighbourhoods which can be expressed as 2 bi m nj 1 uij e j (3.1.1) Where u ij is the projection of bi –m onto the jth Eigen vector e j and bi is of size n x n .The PCAgenerates n² eigenvectors each of size n x n and { ei , 1 ≤ i ≤ n² }. The average value of all local neighbourhoods (bi) is m 1 N N i 1 i b (3.1.2) The Eigen values are in decreasing order of magnitude and eigenvector has similar characteristics as a low pass filter. The eigenvector pairs are orthogonal to each other. The PCA scheme is primarily based on successive approximation criterion. So, it gives an idea to Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 33 Image Fusion & Edge Detection use e j values so as to minimize the approximation error. Starting with eigenvector pairs (e2, e3) and at higher ranges, it is found that, there is an accession of zero crossing points, which signifies that the lower range eigenvectors contains more clues regarding the local neighbourhoods at high frequency ranges. . Since the Eigen vectors e2, e3 assists in the accurate approximation process of local neighbourhoods gray level alteration, our proposed model deals with eigenvectors e2 and e3 for crack edge identification. From Fig.3.6 and Fig.3.7, it is clear that e2 and e3 have blurred step edge profiles. Their approximated values are as follows: ^ e2 21h1 22h2 And ^ e3 31h1 32 h2 (3.1.3) Where { α } are weights and HBT filter pairs h1 , h2 are determined from a set of four 2D HBT filters oriented along 0 degrees, 45 degrees, 90 degrees, and 135 degrees . The HBT profile h1 w.r.t h2 is expressed as 1 e x y Gw For |x|, |y| ≤ W and 0 otherwise. 1 e x y (3.1.4) The region of support for Gw is confined within a window size W to guarantee edge identification. By the sampling process of Gw at integer positions in a period [-w, w], h1 and h2 filters are determined. The parameter w determines the steepness of the profile at zero crossing. w is determined for a given filter width W so as to best approximate the natural step edges in an image by means of the HBT filter pair correspondence to smallest total 2 3 . The weights ij are determined by projecting both Eigen values e2 and e3 onto orthogonal HBT filter pairs h1 and h2, i.e. ij ei , h j hj , hj (3.1.5) The approximation error is given as i ei e j ej Submitted by PRIYA RANJAN MUDULI [211EC3315] (3.1.6) Page 34 Image Fusion & Edge Detection Here the extra sensitivity to illumination can be mitigated by modifying their edgesimilarity measure with Ri with regularization parameter γ, an empirical constant c. The Ri is expressed as, Ri bi bi c , g (3.1.7) bi bi c . g An estimate , obtained by mean absolute deviation is given by median(Yi : 1 i N ) 0.6745 Where Yi 2 1 n2 b j b n2 j 1 2 i i (3.1.8) (3.1.9) i=1, 2… N. Applying Ri to the sample wall image to compute four similarity maps where each map corresponds to one of the four HBT filters equivalent similarity map is determined. By means of suitable threshold value on local maxima, the edge pixels are determined. 3.1.2 Wavelet Decomposition and Fusion After detection of crack edges by the different detectors, the next key issue is the type and level at which the image fusion takes place. The wavelets-based approach is appropriate for performing fusion tasks due to its multiresolution characteristics to deal with images at varying resolution as described by Pajares et al. [41].The discrete wavelets transform (DWT) performs theimage decomposition in different kinds of coefficients preserving the original information of an image. The iterative decomposition helps in increasing the frequency resolution. The approximation coefficients are then disintegrated through high and low pass filters along with the down sampling operation as shown in fig.3.2. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 35 Image Fusion & Edge Detection 1 2 HH 1 ORIGINAL WALL IMAGE 2 2 1 2 HL 1 2 LH 1 2 LL 1 Fig.3.2 Discrete wavelet filter banks In the comparative study of image fusion algorithms by S. Krishnamoorthy et al.[42], Haar Discrete Wavelet Transform based fusion method was evaluated as the outstanding method in terms of subjective analysis. Like all wavelet transforms, the Haar transform decomposes a discrete signal into two sub signals of half its length. One sub signal is a running average or trend; the other sub signal is a running difference or fluctuation. Some of the exclusive aspects of Haar wavelet transform are its efficient processing speed, simplicity, memory management and reversibility. The Haar wavelet's mother wavelet function t can be described as t = 1 0 t 0.5 -1 0.5 t 1 0 (3.1.10) otherwise Its scaling function t can be described as t = 1 0 0 t 0.5 (3.1.11) otherwise The 2×2 Haar matrix associated with the Haar wavelet is 1 1 H2 1 1 (3.1.12) The coefficients derived from input images can be suitably integrated to acquire new coefficients; retaining crude information of the original images. Once the coefficients are merged, then fused image is achieved through the inverse discrete wavelets transform Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 36 Image Fusion & Edge Detection (IDWT), where the information in the merged coefficients is also preserved. The key step in image fusion based on wavelets is to merge coefficients in an appropriate way in order to obtain the best quality in the fused image. The hybrid fusion algorithm [43] combines the advantages of both pixel and region based fusion by selecting maximum approximations, averaging of detail coefficients. 3.1.3 Results & Discussion In this edge detection technique the concerned wall image acquisition is done by means of a camera with focal length of 4mm, exposure time: 0.002 sec, max aperture: 3.5. The image is resized to 256X256 for ease of processing and is shown in Fig.3.3. Our proposed method is based on crack detection process using two efficient and reliable edge detection algorithms such as canny edge detection and HBT filtering. Canny detector response fulfils the three criteria desired for any edge detector such as good detection, good localization, and minimal response as shown in Fig.3.4; whereas HBT filtering technique highlights the edge similarities between image adjacency and directional finite impulse response as shown in Fig.3.5. In the Fig.3.6, 3.7, the second and third largest PCA Eigen values are plotted in spatial domain respectively. The Total error vs. w plot in Fig.3.8 shows that w value of 0.48 best approximates the natural step edges in an image by means of the HBT filter pair correspondence to smallest approximate error value of 0.1680. We have considered three levels Haar DWT decomposition technique shown in Fig.3.9 using Graphical User Interface (GUI). In the next stage, the decomposition results of individual cracked images are fused using an efficient fusion rule. Here we have selected maximum-approximation and meandetail fusion selection algorithm. The high pass filter mask enhances the edges whereas averaging filter mask helps in removing noise by taking mean of gray values surrounding the center pixel of the window. Finally by the application of Inverse DWT, the synthesized fused image is recovered to identify the cracks more accurately. The proposed crack detection technique produces a crack detected image with improved PSNR, Entropy, Normalized Absolute Error value and the Feature Similarity Index (FSIM), which is based on phase congruency. The performance metrics is shown in Table. I. Here the overlapped crack edges of both detectors highlight the genuine crack locations in the original image avoiding the false edges, which is shown in Fig.3.11. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 37 Image Fusion & Edge Detection Fig.3.3 Original Wall image showing a hairline crack Fig.3.4 Canny Edge detector response Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 38 Image Fusion & Edge Detection Fig.3.5 HBT filter response with sigma = 0.48 ,Total minimum error = 0.168, gamma = 0.0208, Threshold = 0.83 Fig.3.6 Second largest PCA Eigen values in spatial domain for wall image Fig. 3.7 Third largest PCA Eigen values in spatial domain for wall image Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 39 Image Fusion & Edge Detection Fig.3.8 Total minimum error as a function of w Fig.3.9 GUI for Image Fusion Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 40 Image Fusion & Edge Detection Fig.3.10 Fusion with 3 level Haar DWT decomposition using GUI Fig.3.11 Image Fusion Response using GUI Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 41 Image Fusion & Edge Detection Table I Performance Metrics Quality Indices PSNR JOINT NORMALIZED (dB) ENTROPY ABSOLUTE FSIM Methods ERROR Canny Edge Detector 10.5183 5.9816 0.3947 0.6429 10.8215 6.1161 0.3875 0.6319 11.1523 6.4054 0.3699 0.6602 HBTEdge Detector Proposed Technique 3.1.4 Summary In this research work, we proposed novel crack detection technique based on two efficient crack detection algorithms along with an efficient image fusion by means of Haar discrete wavelet transform. HBT filtering method emphasizes on optimization of two of Canny’s criteria- accurate edge detection and localization, without explicitly including the minimal response criterion and Canny Edge detector avoids the false edge detection. In our proposed technique for crack detection, both Canny and HBT based filter responses are fused together resulting an optimized edge detection technique. Here, we have chosen maximumapproximation and mean-detail fusion selection algorithm. The high pass filter mask enhances the edges whereas averaging filter mask helps in removing noise by taking mean of gray values surrounding the centre pixel of the window. Here the image fusion response is having higher values of PSNR, Entropy and Feature Similarity Index as compared to canny as well as HBT edge detector responses. The Normalized Absolute Error also gets reduced. Finally, the smoothness parameter should be taken relatively high value to decrease the slope of the filter function reducing the oscillations of the filter response function in the time domain. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 42 Image Fusion & Edge Detection 3.2 Edge Detection for Multi-Focus Images using Image Fusion Several researchers have proposed many edge detection techniques since three decades. The classical edge detector such as Sobel, Prewitt, Kirsch, Robinson and Frei-Chen [36] are preferred due to their simplicity in detecting the edges and their orientations. However, due to the absence of smoothing filter, their performance degrades appending noise and inaccuracy. The zero crossing algorithms such as Laplacian, second directional derivative have fixed characteristics in all directions. However, these are having low noise performance. Laplacian of Gaussian (LoG) by Marr-Hildreth [34] uses laplacian filter. Therefore, it malfunctions at the corners, curves and fails to find the real edge orientation. The Gaussian algorithms such as Canny and Shen-Castan [35] are having complex computations, delusive zero crossing and time consuming. Lacroix [44] proposed an algorithm base on canny’s method that voids the issue of splitting edges. But, it introduces localization error. Jeong and Kim [45] proposed an automatic optimum determination of scales for each pixel. However, this technique has a low execution speed. The angle-based technique is tender to noise and uneven light. The improved method [37] based on phase congruency are invariant towards uneven illumination and low contrast. It is found that the edge detection techniques, which are robust towards contrast variation, are prone to be stirred by noise. A robust edge detection algorithm [39] produces superior performance compared to methods discussed above. Focusing onto the image fusion techniques, several algorithms have been proposed. Spatial domain is simple and computationally efficient for image fusion. Low performances as well as blocking effects [46] are the main constraint for spatial domain. Again, the multi resolution pyramid based algorithms are redundant and bad orientation selectivity [19]. The widely used DWT based MRA [47, 48, 49] technique have the advantage of good time-frequency analysis and nonredundancy. The DWT suffers from the following hazards: shift variance, non-directionality, oscillations and aliasing [50]. The Dual-Tree Complex Wavelet Transform is having the property of shift invariance and directional selectivity. Therefore, it takes the advantage over DWT. In this research work, a multi-focus image fusion algorithm with Dual-Tree Complex Wavelet Transform (DT-CWT) [51] is implemented followed by Hyperbolic Tangent based edge detection technique followed by an adaptive thresholding method [53]. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 43 Image Fusion & Edge Detection 3.2.1 Proposed technique We have proposed a new approach for efficient and reliable edge detection in multi-focus images, which is a challenging task due to blurring effect. It uses the Dual-Tree Complex Wavelet Transform (DT-CWT) based image fusion followed by a Hyperbolic Tangent filtering technique. Here image-I, II are the multi-focus image. The edge detection architecture is shown in Fig.3.12. Image-I Image- II Image Fusion using DT-CWT Fused Image Edge Detection using HBT Filter with Adaptive Thresholding Detected Edges Fig. 3.12 Flow chart for proposed Edge Detection Technique 3.2.2 DT-CWT based image fusion Nick Kingsbury was the first pioneer of DT-CWT technique in 1998. It possesses the competitive attributes such as approximate shift invariance and improved directional selectivity [50]. The DT-CWT employs two real Discrete Wavelet Transform by splitting the real and imaginary parts of transform into two trees as shown in Fig.3.13. The technique uses delayed samples between the real part and its correspondence imaginary part in each level in combination with the alternate odd length and even length linear phase filters. To avoid the issue regarding the non-symmetrical and non bi-orthogonal property of DT-CWT, a Q-shift dual tree has been proposed [51]. Here the first level filters are even length. A quarter (q) group delays are implemented followed by 2q delay and so on as shown in Fig.3.13. This upholds the shift-invariance as well as directional selectivity attributes. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 44 Image Fusion & Edge Detection Fig.3.13 Dual Tree Q-shift CWT Fusion Rules Following an optimum rule [54] is one of the most important parts of image fusion based on multi resolution analysis. The low frequency coefficients represent the texture information of background image. So the rule for low frequency components is based on neighbourhood due to high correlation among the pixels. Since we are concerned for the multi-focus image fusion to have an improvement of blurred regions, the weighted scheme based neighbourhood rule is implemented for low frequency coefficients. Here the neighbourhood maximum selectivity (NGMS) rule is enforced. The neighbourhood gradient is computed as follows. Gi (m, n) H (m' , n' )L (m m' , n n' ) m' , n ' s i (3.2.1) Where i =A or B, are the input images. Li (m, n) is the coefficient at (m, n) and ‘S’, ‘H’ represents the neighbourhood and the corresponding mask. The laplacian mask used here is 1 1 1 H 1 8 1 1 1 1 Submitted by PRIYA RANJAN MUDULI [211EC3315] (3.2.2) Page 45 Image Fusion & Edge Detection The edges and detail information is presented by means of high frequency coefficients. In general imaging techniques, the in-focus image system functions are wider as compared to the out of focus images. Due to the high correlation pixel property, an absolute value maximum selection (AVMS) technique is preferred for high frequency components. The actual fusion rule is as follows S Aj , i (m, n) if S Aj , i (m, n) ≥ S Bj , i (m, n) S fj , i (m, n) = (3.2.3) S Bj ,i (m, n) if S Aj ,i (m, n) ≤ S Bj ,i (m, n) Where j=1, 2…N, represents the level of the decomposition, i=1, 2…6, represents the level wise direction of high frequency coefficients. 3.2.3 Edge detection using HBT filter An edge similarity measurement based algorithm by Saravana Kumar [39] gives superior result than GM and Angle based method. This technique is more rugged irrespective of diversity in illumination, contrast and noise level. The filtering technique highlights the edge similarities between image adjacency and directional finite impulse response by means of hyperbolic tangent figuration. This edge detection technique based on similarity measurement results an optimal identification of image edges by using principal component analysis. The Principal Component Analysis (PCA) is implemented onto a set of local neighbourhoods, which can be expressed as 2 bi m nj 1 uij e j (3.2.4) Where u ij is the projection of bi –m onto the jth Eigen vector e j and bi is of size n x n .The PCA generates n² eigenvectors each of size n x n and { ei , 1 ≤ i ≤ n² }. The average value of all local neighbourhoods (bi) is m 1 N N i 1 i b Submitted by PRIYA RANJAN MUDULI [211EC3315] (3.2.5) Page 46 Image Fusion & Edge Detection The Eigen values are in decreasing order of magnitude and eigenvector has similar characteristics as a low pass filter. The PCA scheme is primarily based on successive approximation criterion. Therefore, it gives an idea to use e j values to minimize the approximation error. Starting with eigenvector pairs (e2, e3) and at higher ranges, it is found that the lower range eigenvectors contains more clues regarding the local neighbourhoods at high frequency ranges. . Since the Eigen vectors e2, e3 assists in the accurate approximation process of local neighbourhoods gray level alteration, our proposed model deals with eigenvectors e2 and e3 for crack edge identification. It is observed that e2 and e3 have blurred step edge profiles. Their approximated values are as follows: ^ ^ e2 21h1 22h2 And e3 31h1 32 h2 (3.2.6) Where { α } are weights and HBT filter pairs h1 , h2 are determined from a set of four 2D HBT filters oriented along 0 degrees, 45 degrees, 90 degrees, and 135 degrees . The HBT profile h1 w.r.t h2 is expressed as Gw 1 e x y 1 e x y 0 For |x|, |y| ≤ W (3.2.7) otherwise. The region of support for Gw is confined within a window size W to guarantee edge identification. By the sampling process of Gw at integer positions in a period [-w, w], h1 and h2 filters are determined. The parameter w determines the steepness of the profile at zero crossing. w is determined for a given filter width W so as to best approximate the natural step edges in an image by means of the HBT filter pair correspondence to smallest total 2 3 . The weights ij are determined by projecting both Eigen values e2 and e3 onto orthogonal HBT filter pairs h1 and h2, i.e. ij ei , h j (3.2.8) hj , hj The approximation error to be minimized is given as i ei e j (3.2.9) ej Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 47 Image Fusion & Edge Detection Here the extra sensitivity to illumination is mitigated by modifying Ri with a regularization parameter γ, an empirical constant c. Now Ri is represented by, Ri bi bi c , g bi bi c . g (3.2.10) An estimate , obtained by mean absolute deviation is given by Where Yi 2 1 2 n median (Yi : 1 i N ) 0.6745 (3.2.11) b j b n2 j 1 2 i i Where i=1, 2… N. Ri is applied to the fused image to compute four similarity maps where each map corresponds to one of the four HBT profile orientations. Equivalent similarity map is determined. A suitable threshold value is applied to have the edge pixels. Adaptive thresholding Thresholding is one of the important processes followed by image segmentation. It is an effective way for separating the object from its background. The Chow and Kaneko approach [52] fails due to its computational expensiveness. So it is not suitable for real time applications. In this research work the adaptive thresholding method [53] is incorporated followed by the edge detection using hyperbolic tangent profile. The adaptive thresholding technique uses the local information and boundary characteristics of an image. Due to the small size of intensity range and the resemblance of mean value with the central pixel value, simple mean does not perform well. So the mean-c concept is implemented, where the threshold value changes dynamically over the image, which ensures that it is best, suited for the illumination variant imaging. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 48 Image Fusion & Edge Detection 3.2.4 Results & Discussion In this edge detection technique, the simulation is done using multi-focus test images as shown in Fig.3.14 and Fig.3.15. The multi-focus image-I and image-II are fused by means of Dual-Tree Complex Wavelet Transform based image fusion to get the proper image with focused features. Fig.3.14 (a, b) represents the multi-focus images of Clock and the fused image is shown in Fig.3.14(c). Similarly the multi focus Coke-Can images are shown in Fig.3.15 (a, b) and the fusion result is shown in Fig.3.15(c). The image fusion result is treated through an HBT filter and the edges are tracked as depicted in Fig. 3.14(d) and Fig.3.15 (d). The total min-error Vs. sigma plots for multifocal clock and multifocal Pepsi-Can image using HBT filter optimization are shown in fig.3.16 and Fig.3.17 respectively. For clock image, the edge is detected with 0.55 and total minimum error of 0.1627. Again, the Coke Can image is also treated by HBT profile with 0.68 and total min-error of 0.3626. Here, the proposed multi focus image edge detection technique performs well with the adaptive thresholding. The performance chart is shown in Table. I. (a) Submitted by PRIYA RANJAN MUDULI [211EC3315] (b) Page 49 Image Fusion & Edge Detection (c) (d) Fig.3.14 (a) Multi-Focus Image-I, (b) Multi-Focus Image-II, (c) Image Fusion result, (d) Edge Detection result (a) (b) (c) (d) Fig.3.15 (a) Multi-Focus Image-I, (b) Multi-Focus Image-II, (c) Image Fusion result, (d) Edge Detection result Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 50 Image Fusion & Edge Detection Fig.3.16 Total min-error Vs sigma plot for Clock image showing total min-error of 0.1627 at 0.55 Fig.3.17 Total min-error Vs sigma plot for Pepsi Can image showing total min-error of 0.3626 at 0.68 Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 51 Image Fusion & Edge Detection Fig.3.18 (a) PCA Eigen value e2 for Fused Clock Image (b) PCA Eigen value e3 for Fused Clock Image Fig.3.19 (a) PCA Eigen value e2 for Fused Pepsi-Can Image, (b) PCA Eigen value e3 for Fused Pepsi-Can Image Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 52 Image Fusion & Edge Detection Table I Comparison Chart PARAMETERS IMAGES PROPOSE CANNY SOBEL METHOD PSNR Clock 11.4986 11.1355 11.1869 in dB Can 12.0113 11.6394 11.9688 Total Clock 74.4641 66.4037 37.2234 STD Can 77.1899 70.2991 52.3905 Entropy Clock 4.1011 2.5368 1.2697 Can 2.1793 1.5774 1.2510 3.2.5 Summary Edge detection in multi-focus images has been one of the challenging tasks due to severe blurring effects. In this research work, we have proposed a novel edge detection architecture, which combines the individual advantages of Q-shift DT-CWT based image fusion and HBT filtering based edge detection technique. The Q-shift DT-CWT removes the blocking effect, ringing artefacts during fusion and improves the directional selectivity. The use of HBT profile makes the edge detection technique more robust towards uneven illumination, contrast variation and noise. The proposed technique performs superior as compared to classical sobel method as well as canny algorithm in terms of PSNR, total standard deviation and Entropy. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 53 Image Fusion based on Bilateral Sharpness Criterion CHAPTER 4 Image Fusion based on Bilateral Sharpness in DT-CWT Domain Introduction Dual Tree Complex Wavelet Transform Proposed Image Fusion Technique Simulation Results and Discussion Summary Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 54 Image Fusion based on Bilateral Sharpness Criterion 4. IMAGE FUSION BASED ON BILATERAL SHARPNESS CRITERION IN DT-CWT DOMAIN Image fusion is basically the technique of merging several images from multi-modal sources with respective complementary information to form a new image, which carries all the common as well as complementary features of individual images. Image fusion has an extensive area of application such as, multi-spectral remote sensing, target detection, military surveillance systems medical imaging and so on. Various algorithms have been proposed for effective fusion of multi-source images such as simple averaging, maximum and minimum fusion rules. Afterwards, the fusion performance is improved by the introduction of Principal Component Analysis (PCA) and Morphological processing algorithms [42]. Again, the fusion algorithms such as Brovey technique, PCA and Intensity-HueSaturation fail due to the characteristic spectral losses and color deformation. Data fusion by means of pyramidal decomposition results in growth of redundancy and orientation deficiency [19]. Further, with the development of Discrete Wavelet Transform (DWT) based image fusion techniques, the perceptual quality has been enhanced upholding the spectral information contents. The real DWT has the property of good compression of signal energy. Perfect reconstruction is possible using short support filters. The unique feature of DWT is the absence of redundancy and very low computation [49]. Therefore, DWT has been used extensively for Multi Resolution Analysis (MRA) based image fusion. The Discrete Wavelet Transform primarily suffers from the various problems [55] such as oscillations, aliasing, shift variance and lack of directionality. The ringing artefacts introduced by DWT are also completely eliminated by the implementation of Dual Tree Complex Wavelet (DT-CWT) based image fusion methods. In this research work, an improved version of Dual Tree Wavelet Transform based image fusion algorithm is proposed. The fusion process is implemented using efficient fusion rules for high frequency coefficients as well as low frequency coefficients depending on their characteristics. The robustness of the proposed method is verified from Low Light Television (LLTV) and Forward-Looking-Infrared (FLIR) image fusion, multi-spectral satellite image fusion and CT-MR image fusion. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 55 Image Fusion based on Bilateral Sharpness Criterion 4.1 Dual-Tree Complex Wavelet Transforms The Dual-Tree Complex Wavelet Transform (DT-CWT) was first introduced by Nick Kingsbury in the year 1998. The DT-CWT has been one of the most popular transform domain techniques due to some of the unique features such as, good shift invariance, good directional selectivity in 2-Dimensons as well as 3-Dimensons, perfect reconstruction using short support filters, limited redundancy and low computation. The DT-CWT basically utilizes two real DWTs. The first one yields the real part of the transform while the second one yields the imaginary part. In order to accomplish the perfect reconstruction, it is required to process signals with the help of wavelets. The DT-CWT [55] accomplishes this by using two filter banks and thus two bases. With the help of two filter banks { h0 (n) , h1 (n) } and { g 0 (n) , g1 (n) }, four DWTs, Fhh , Fgg , Fgh and Fhg are generated. The Fgh component is extracted from the filters g i (n) along the rows and filters hi (n) along columns. Since at each decomposition level of DWT, three sub-bands are produced, there can be twelve sub-bands to generate six directionally selective complex sub- bands, which are approximately analytic. These complex sub-bands are basically oriented at ±15°, ±45°, and ±75°. Mathematically the two dimensional DT-CWT decomposition of an image I(x, y) can be expressed by means of the complex shifted and dilated mother wavelet (x) and scaling function (x) as I ( x, y) S j0 ,l j0 ,l ( x, y) C j ,l j ,l ( x, y) (4.1) j j0 lz 2 lz 2 The mother wavelet is expressed as j ,l ( x) jr ,l ( x) 1 ij ( x) 0 0 (4.2) The scaling wavelet function is j ,l ( x) rj ,l ( x) 1 ij ,l ( x) (4.3) Where ‘z’ is the natural number set. S j0 ,l represents the scaling coefficient with shifting of ‘j’ and dilation of ‘l’. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 56 Image Fusion based on Bilateral Sharpness Criterion C j ,l Shows the complex wavelet coefficients r, i are the indexing for real and imaginary parts respectively. The directionality of the six complex sub bands generated is guided along {15 ,45 ,75 } . Thus, the two dimensional Dual Tree Complex Wavelet Transform gives rise to one real low-pass image along with six complex high-pass sub-images at each decomposition level. To obtain the final DT-CWT outcome, the difference between the filters in the two trees is estimated. It is basically implemented by a delayed version of one sample between the level-1 filters of the real as well as imaginary decomposition trees. The outcomes of subsequent levels are obtained by the help of alternate odd-length and even-length linearphase filters. Unluckily the odd and even filtering techniques often suffer from the problems like: unsymmetrical sub-sampling structure, frequency responses variation between decomposition trees [50]. To beat these issues, a Q-shift dual tree [51] is implemented for this research work. This novel algorithm proposes that, all the filters beyond level 1 has to be of even length without any rigid linear phase condition with a group delay of nearly a quarter samples (q). Likewise, other filters beyond level-1 are imitated from the orthonormal prototype set. A symmetric sub-sampling arrangement takes care of the shift invariance as well as the directional selectivity property of DT-CWT. The dual tree filters for Q-shift wavelet decomposition is shown in Fig.3.2.2. Fig.4.1 Dual tree of real filters for the Q-shift Wavelet Transform Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 57 Image Fusion based on Bilateral Sharpness Criterion 4.2 Proposed Image Fusion using DT-CWT At every decomposition level of DT-CWT, six directional high frequency wavelet coefficients are generated along with two low frequency coefficients. Generally, the wavelet coefficients of each band are blended using some suitable fusion rules. A new fused image is reconstructed using inverse DT-CWT. The complete fusion process is shown in Fig.4.2. Fig.4.2 DT-CWT fusion 4.2.1 Fusion rule for Low frequency coefficients In an image, the background texture information is primarily highlighted by the low frequency components. Appropriate fusion selection rule for low frequency components is one of the important criterions during the fusion process. The rules can be classified as maximum selection rule or weighted rule. In this research work, we have implemented the weighted fusion scheme for low frequency components. The normalized weighting factor selection for this research work is inspired by the maximum gradient based sharpness of the neighbourhood pixels [57]. 4.2.2 Gradient-based sharpness criterion Sharpness has always been considered as one of the prime norm for image quality measurement. The sharpness and information content of an image primarily depends on the strength measures. Image fusion algorithms based on simple normalized aggregation of input images fails since, the concerned high frequency regions are also weighted equally along with the unimportant regions. To overcome such issues, an improved version of the novel weighting criterion [57] has been implemented in this research work. Here, to measure the structural contents of an image I(r, c) effectively, a gradient covariance matrix of a region is specified by means of a local window of size MxN [58]. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 58 Image Fusion based on Bilateral Sharpness Criterion The gradient covariance matrix is given by w I r2 (r , c) C I r (r , c) I c (r , c) w I (r , c) I c (r , c) w I c2 (r, c) w r (4.4) Where I r (r , c) and I c (r , c) represent the row-gradient and column-gradient of the image respectively. The gradient covariance matrix can be represented as C VDV (v1 T 1 0 v1T T v2 ) 0 2 v2 (4.5) Where the 2x2 matrix V consists of the eigen vectors v1 and v 2 along its columns. The 1 2x2 matrix D 0 0 . Where 1 and 2 represents the Eigen values of the gradient 2 covariance matrix [59]. The image gradient strength is represented as G(r , c) 1 2 The maximum gradient strengths of input images can (4.6) be calculated as Gmax max(max( G(r, c))) The maximum gradient strengths are computed for individual image-A and image-B using the above formula. Now, the normalized weighting factors W A for image-A is laid out by WA Gmax A Gmax A Gmax B (4.7) Similarly, the normalized weighting factor for image-B can be represented as WB Gmax B Gmax A Gmax B (4.8) The weights W A and WB corresponding to the image-A and image-B respectively, are applied to the low frequency wavelet coefficients for the fusion process. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 59 Image Fusion based on Bilateral Sharpness Criterion 4.2.3 Fusion rule for High frequency coefficients Recently, most of the fusion rules meant for high frequency coefficients are based on neighbourhood characteristics such as neighbourhood energy, variance, etc. The high frequency components basically describe the detail information of an image. In this research work, the fusion of high frequency components is implemented by means of an effective weighting scheme proposed by Jing Tian [57]. For high frequency wavelet component fusion process, the bilateral gradient-based sharpness weighting method is implemented. This bilateral sharpness measurement comprises of both gradient strength criterion from Eq. (4.6) and corresponding phase coherence measurement. The local phase coherence criterion plays a vital role with respect to human visual perception towards the fusion response [38]. The phase criterion is also robust towards noise and gradient based illumination variation. The phase coherence for image gradient can be presented as P(r , c) cos( (r , c) (r , c)) (4.9) Where (r , c) evaluated from principal vector v1 , is the phase information at coordinates (r, c). (r , c) is the average phase of neighbouring pixels. The maximum phase coherence value corresponds to the edge pixels. The bilateral sharpness criterion is developed using the gradient sharpness criterion in Eq. (4.6) and its corresponding phase coherence in Eq. (4.9). This can be expressed as S G ( r , c) P ( r , c) (4.10) The factors and can be tuned to some suitable values to maximize the contribution of suitable sharpness criterions. In this experimental work, and values are set to 1 and 0.5 respectively confined in an window size of w=5 [57]. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 60 Image Fusion based on Bilateral Sharpness Criterion 4.3 Simulation Results and Discussion The Bilateral sharpness criterion based fusion in Dual Tree Complex Wavelet domain has been performed using various multi-sensor images from a standard image database of Dr. Oliver Rockinger. The robustness of the proposed fusion technique is verified successfully with some multi-sensor images such as: LLTV sensor image, FLIR sensor image, multispectral remote sensing images and medical images such as CT, MR images. The original input images and their corresponding fusion results using the proposed technique are depicted in Fig.4.3, Fig.4.4 and Fig.4.5. (a) (b) Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 61 Image Fusion based on Bilateral Sharpness Criterion (c) Fig.4.3 (a) LLTV sensor image, (b) FLIR sensor image (c) Fused image using proposed method (a) Fig. 4.4(b) Multispectral sensor image-B (b) Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 62 Image Fusion based on Bilateral Sharpness Criterion Fig. 4.4 (a) Multispectral sensor image-A, (b) Multispectral sensor image-B, (c) Fused image using proposed method (a) (b) Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 63 Image Fusion based on Bilateral Sharpness Criterion (c) Fig. 4.5 (a) CT image, (b) MRI image, (c) Fused image using proposed method 4.3.1 Quantitative evaluation The quantitative evaluation of this research work using various multi-sensor images has been shown in Table-I, Table-II and Table-III. The performance comparison of the proposed method is accomplished with Discrete Wavelet Transform (DWT) and Dual Tree Complex Wavelet Transform (DT-CWT) in terms of some non-referential image quality measures such as entropy, average gradient, edge intensity and standard deviation. The superiority as well as robustness of the proposed image fusion technique is evidently justified from the fused image quality assessment tables. Some of the major non-referential image quality measures are discussed below. Entropy (E) Entropy is considered as one of the vital image quality index to evaluate the information content in an image. N It is formulated as E p( xi ) log p( xi ) (4.11) i 0 Where x i is the gray level value at i th pixel with corresponding probability ‘p’. The entropy value is larger for images containing more the information. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 64 Image Fusion based on Bilateral Sharpness Criterion Average gradient The detail contrast and texture variations in an image is usually indexed by means of average gradient values and is given as f f x y 2 2 g ( M 1)( N 1) 1 ( M 1)( N 1) i 1 2 (4.12) Edge intensity The measurement of sharp discontinuities in an image can be considered as one of the image quality assessment parameters. It can be easily accomplished using the Sobel edge detection algorithm. It uses horizontal differentiation kernel g x and a vertical differentiation kernel g y , which are presented as: 1 gx 2 1 1 2 1 gy 0 0 0 1 2 1 1 2 , 1 0 0 0 (4.13) For an image I, the edge intensity values are given as: S G 2 x G y2 (4.14) Where, Gx I * g x , G y I * g y ` Standard deviation The Standard deviation is considered as one of the best metrics for contrast value measurement for an image. High contrast level of an image can be make out from high standard deviation value. It can be formulated as j 1 N N (x i 1 ji j )2 Where, the mean pixel value j (4.15) 1 N N x i 1 ji Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 65 Image Fusion based on Bilateral Sharpness Criterion Table. I Quantitative assessment for fusion of navigation images (LLTV and FLIR) Quality Entropy Average Gradient Edge Intensity Standard Deviation DWT 5.4799 3.8137 39.5306 80.7748 DT-CWT 5.4147 3.8473 39.7661 84.1700 PROPOSED METHOD 5.7038 3.9373 40.7322 87.4002 I Indices Methods Table..II Quantitative assessment for fusion of multispectral remote sensing images Quality Indices Entropy Average Gradient Edge Intensity Standard Deviation DWT 5.4926 7.5051 74.6469 78.4850 DT-CWT 5.7928 7.6045 76.8336 80.2671 PROPOSED METHOD 6.0335 7.7230 78.2545 81.9917 Methods Table.III Quantitative assessment for fusion of medical images (CT and MR) Quality Indices Entropy Average Gradient Edge Intensity Standard Deviation DWT 3.8010 3.9580 41.9148 91.7107 DT-CWT 3.9944 4.1174 43.7937 96.1673 PROPOSED METHOD 4.1129 4.2091 44.8069 100.7343 Methods Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 66 Image Fusion based on Bilateral Sharpness Criterion 4.4 Summary An enhanced fusion scheme proposed in this research work implements a bilateral gradient-based sharpness-weighting criterion in Dual-Tree Complex Wavelet Transform. The proposed fusion technique compensates all the shortcomings of Discrete Wavelet Transform by the implementation of Q-shift DT-CWT. It also removes the ringing artefacts introduced in the fused image by assigning suitable weighting schemes to high pass wavelet coefficients and low pass coefficients independently. The normalized maximum gradient-based sharpness criterion for low frequency coefficients enhances the background texture information as well as improves the quality of the blurred regions in the fusion result. The most vital information contents concealed in the high frequency coefficients are also boosted up by the implementation of bilateral sharpness criterion. From the image quality assessment tables, it is clear that the proposed fusion technique outperforms other methods in terms of Entropy, Average Gradient, Edge Intensity Submitted by PRIYA RANJAN MUDULI [211EC3315] and Standard deviation. Page 67 Conclusions CHAPTER 5 Conclusion Conclusions Suggestions for Future Work Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 68 Conclusions 5. CONCLUSIONS This chapter describes the conclusion about the research work and gives the suggestions for future work. 5.1 Conclusions In this research work, attention was drawn towards the current trend of the use of multiresolution image fusion techniques, especially approaches based on discrete wavelet transforms and Dual Tree Complex Wavelet Transforms. The work started with the review of several image fusion algorithms and their implementation. The significance of image fusion in edge detection has been illustrated with some proposed techniques in chapter-3. A novel crack detection technique has been proposed here, which is based on two efficient crack detection algorithms along with an efficient image fusion by means of Haar discrete wavelet transform. HBT filtering method emphasizes on optimization of two of Canny’s criteriaaccurate edge detection and localization, without explicitly including the minimal response criterion and Canny Edge detector avoids the false edge detection. In our proposed technique for crack detection, both Canny and HBT based filter responses are fused together resulting an optimized edge detection technique. A maximum-approximation and mean-detail fusion selection rule has been implemented. The high pass filter mask enhances the edges whereas averaging filter mask helps in removing noise by taking mean of gray values surrounding the centre pixel of the window. The response of image fusion is found to have higher values of PSNR, Entropy and Feature Similarity Index as compared to canny as well as HBT edge detector responses. The Normalized Absolute Error also gets reduced. Finally, the smoothness parameter should be taken relatively high value to decrease the slope of the filter function reducing the oscillations of the filter response function in the time domain. Edge detection in multi-focus images has been one of the challenging tasks due to severe blurring effects. In this research work, we have proposed a novel edge detection architecture, which combines the individual advantages of Q-shift DT-CWT based image fusion and HBT filtering based edge detection technique. The Q-shift DT-CWT removes the blocking effect, ringing artefacts during fusion and improves the directional selectivity. The use of HBT profile makes the edge detection technique more robust towards uneven illumination, contrast Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 69 Conclusions variation and noise. The proposed technique performs superior as compared to classical sobel method as well as Canny algorithm in terms of PSNR, total standard deviation and Entropy. An enhanced fusion scheme proposed in chapter-4 implements a bilateral gradient-based sharpness-weighting criterion in Dual-Tree Complex Wavelet Transform. The proposed fusion technique compensates all the shortcomings of Discrete Wavelet Transform by the implementation of Q-shift DT-CWT. It also removes the ringing artefacts introduced in the fused image by assigning suitable weighting schemes to high pass wavelet coefficients and low pass coefficients independently. The normalized maximum gradient based sharpness criterion for low frequency coefficients enhances the background texture information as well as improves the quality of the blurred regions in the fusion result. The most vital information contents concealed in the high frequency coefficients are also boosted up by the implementation of bilateral sharpness criterion. From the image quality assessment tables, it is clear that the proposed fusion technique outperforms other methods in terms of Entropy, Average Gradient, Edge Intensity and Standard deviation. 5.2 Suggestions for Future Work Image Registration has significant contribution towards the enhancement of image fusion quality. Image Registration has not been incorporated in this research work. By the Implementation of suitable image registration techniques, the competitiveness of the proposed image fusion methods can be properly justified with some more set of sample test/perfect images. The number of decomposition levels in the Multiresolution analysis has a great impact on image fusion performance. However, using more decomposition levels do not necessarily implies better results. Therefore methods for selection ofoptimized number of decomposition levels can be explored. A learning algorithm like neural networks and more specifically Support Vector Machine could be devised for assigning weightage to the image quality metrics so as to assess them. A more extensive number of image sets could be considered initiating a learning process using SVM, based on which the metrics could be provided with weighted ranks. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 70 Future Work The final aspect in future development and improvement is how to estimate and evaluate the quality of afused image. As we have discussed in the previous chapter, depending on the applications, some fusion system might not have a perfect ground truth reference image for objective evaluation. Therefore, access methods without reference image are important for our concern in multi-camera imaging system. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 71 Bibliography Bibliography [1]. L.A. Klein, "Sensor and Data Fusion Concepts and Applications," SPIE Optical Engineering Press, Tutorial Texts, Vol. 14, pp.132-139, 1993. [2]. D. Hall and J. Llinas, "An introduction to multisensor data fusion", Proceedings IEEE, Vol. 85(1), pp. 6-23, 1997. [3]. J Aggarwal, “Multisensor Fusion for Computer Vision”, Springer Publisher-Verlag, pp. 456-466, berlin, 1993. [4]. R. K. Sharma, “Probabilistic Model-based Multisensor Image Fusion”,PhD thesis, Oregon Graduate Institute of Science and Technology, Portland, Oregon, 1999. [5]. D. Rajan and S. Chaudhuri, “Generalized Interpolation and Its Application in SuperResolution Imaging”, Image and Vision Computing, Vol.19, Issue 13, pp. 957-969, 1 November 2001. [6]. M. C. Chiang and T. E. Boult, “Efficient Super-Resolution Via Image Warping”, Image and Vision Computing, Vol.18, Issue 10, pp.761-771, July 2000. [7]. www.eecs.lehigh.edu/SPCRL/IF/image_fusion.htm. [8]. Pohl et al. “Multi-sensor image fusion in remote sensing: concepts, methods and applications” Int. J. of Remote sensing, Vol. 19, No. 5, pp.823-854, 1998. [9]. A. Katartzis and M. Petrou,“Robust Bayesian estimation and normalized convolution for super-resolution image reconstruction”, Workshop on Image Registration and Fusion, Computer Vision and Pattern Recognition, CVPR’07, Minneapolis, USA, pp.17, June 2007. [10]. N.Mitianoudis and T. Stathaki “Image fusion schemes using ICA bases”, Information fusion 8, pp.131-142, 2007. [11]. B.Yang and S. Li, “Multi Focus Image Fusion using Watershed Transform and Morphological Wavelets clarity measure”, Int. J. of Innovative Computing, Information and Control, Vol.7, No.5, May 2011. [12]. F.C.Morabito, G.Simone and M.Cacciola “Image fusion techniques for nondestructive testing and remote sensing application”, Image Fusion: Algorithms and Applications, Academic Press, pp.367-392, 2008. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 72 Bibliography [13]. P. Burt, "The pyramid as structure for efficient computation", in Multiresolution Image Processing and Analysis, Springer-Verlag, pp. 6-35, 1984. [14]. P. Burt and E.Adelson, "The Laplacian pyramid as a compact image code", IEEE Transactions Communications, Vol. 31(4), pp 532-540, 1983. [15]. A.Toet, "Hierarchical Image Fusion", Machine Vision and Applications, Vol. 3, pp 311,1990 [16]. A.Toet, "Multiscale contrast enhancement with applications to image fusion”, Optical Engineering, Vol. 31(5), pp 1026-1031, 1992. [17]. Y Cui, T Pu, G Ni, Y Zhong, X Li, "Image fusion with high-speed DSP", Proc. SPIE, Vol. 3561, pp.286-292, 1998. [18]. M Pavel, J Larimer and AAhumada, "Sensor Fusion for Synthetic Vision", Proc. AIAA Conf. on Computing and Aerospace, Baltimore, pp. 164-173, 1991. [19]. A Akerman, "Pyramid techniques for multisensor fusion", Proc. SPIE, Vol.1828, pp 124-131, 1992. [20]. B Aiazzi, L Alparone, S Boronti, R Carla and L Mortelli, "Pyramid-based multisensory image data fusion", Proc. SPIE, Vol. 3169, pp. 224-235, 1997. [21]. P Burt and R Kolczynski, "Enhanced Image Capture through Fusion", Proc. 4th International Conference on Computer Vision, Berlin, pp. 173-182, 1993. [22]. T Peli, E Peli, K Ellis and R Stahl, "Multi-Spectral Image Fusion for Visual Display", Proc. SPIE, Vol. 3719, pp. 359-368, 1999. [23]. S Mallat, "A Theory for Multiresolution Signal Decomposition: The Wavelet Representation", IEEE Transactions Pattern Analysis and Machine Intelligence, Vol. 11(7), pp. 674-693, 1989. [24]. H Li, B Munjanath and S Mitra, "Multisensor Image Fusion Using the Wavelet Transform", Graphical Models and Image Processing, Vol. 57(3), pp. 235-245, 1995. [25]. L Chipman, T Orr and L Graham, "Wavelets and Image Fusion", Proc. SPIE, Vol.2569, pp. 208-219, 1995. [26]. D Yocky, "Multiresolution Wavelet Decomposition Image Merger of Landsat Thematic Mapper and SPOT Panchromatic Data", Photogrammetric Engineering and Remote Sensing, Vol. 62(9), pp. 1067-1074), 1996. [27]. B Garguet-Duport, J Girel, J Chassery and G Pautou, "The Use of Multiresolution Analysis and Wavelets Transform for Merging SPOT Panchromatic and Multispectral Image Data", Photogrammetric Engineering & Remote Sensing, Vol. 62(9), pp. 10571066, 1996. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 73 Bibliography [28]. L Ramac, M Uner, P Varshney, M Alford and D Ferris, "Morphological Filters and Wavelet Based Image Fusion for Concealed Weapons Detection", Proc. SPIE, Vol. 3376, pp. 110-119, 1998. [29]. L Wang, B Liu, J Hu and B Zhang, "Low light level dual spectrum images fusion based on wavelet transform", Proc. SPIE, Vol. 3561, pp. 393-397, 1998. [30]. Y Chibani and A Houacine, "Multiscale versus Multiresolution Analysis for Multisensor Image Fusion", Proc. Eusipco98, Rhodes, pp.235-245, 1998. [31]. Rockinger and T Fechner, "Pixel-Level Image Fusion: The Case of Image Sequences", Proc. SPIE, Vol. 3374, pp. 378-388, 1998. [32]. Z Zhang and R Blum, “A Categorization of Multiscale-Decomposition-Based Image Fusion Schemes with a Performance Study for a Digital Camera Application”, Proceedings of the IEEE, Vol. 87(8), pp1315-1326, 1999. [33]. N. Merlet, and J. Zerubia, “New Prospects in Line Detection by dynamic Programming”, IEEE Trans, Pattern Anal. Machine Intell, vol.18 (4), pp.426-431, 1996. [34]. D. Marr and E. Hildreth, “Theory of Edge Detection,” Proc. Royal Soc. London B, Vol. 207, pp. 187-217, 1980. [35]. J. Canny. “A Computational Approach to Edge Detection”, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 8(6), pp.679-698, 1986. [36]. W. Frei and C.C. Chen, “Fast Boundary Detection: A Generalization and a New Algorithm,” IEEE Trans. Computers, Vol. 26(10), pp. 988-998, 1977. [37]. P. Kovesi, “Phase Congruency: A Low Level Image Invariant”, Psychological Research, Vol.64, pp. 136-148, 2000. [38]. Desolneux, L. Moisan, and J. M. Morel, “Edge Detection by Helmholtz Principle,” Math. Imaging and Vision, Vol. 14, pp. 271-284, 2001. [39]. S. Kumar, S. H. Ong, Surendra Ranganath and Fook Tim Chew, “A Luminance and Contrast-Invariant Edge-Similarity Measure”, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.28(12), pp. 2042-2048, 2006. [40]. U. Qidwai and C.H. Chen, “Digital Image Processing-An Algorithmic Approach with MATLAB”, 1st Ed. CRC Press, Taylor & Francis Group, 2009. [41]. G. Pajares and J. Cruz, “A wavelet-based image fusion tutorial”, Elsevier Pattern Recognition. Vol. 37, pp. 1855-1872, 2004. [42]. S. Krishnamoorthy and K. Soman, “Implementation and Comparative Study of Image Fusion Algorithms”, Intl. J. of Computer Applications, Vol-9, pp. 25-34, 2010. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 74 Bibliography [43]. S. Vekkot and P. Shukla, “A Novel Architecture for Wavelet based Image fusion”, World Academy of Science, Engineering and Technology, Vol. 57, pp.372-377, 2009. [44]. Lacroix, “The primary raster: A multi resolution image description”, Proc. 10th Int. Conf. Pattern Recognition, pp. 903–907, 1990. [45]. H. Jeong and C. I. Kim, “Adaptive determination of filter scales for edge-detection”, IEEE Trans. Pattern Anal. Machine Intell. Vol. 14, pp. 579–585, May 1992. [46]. Paul Hill, Nishan Canagarajah and Dave Bull, “Image fusion using complex wavelets”, British Machine Vision Association, pp. 487-496, 2002. [47]. Li Huihui, Guo lei and Liu Hang, “Research on Image Fusion Using Wavelet Transform Based on Gradient Selective Rule”, Computer Engineering and Applications. Vol.12, pp.76-78, 2005. [48]. Chu Heng, Li Jie and Zhu Wei-le, “Multi-focus image fusion scheme with wavelet transform”, Opto-Electronic Engineering, Vol.32, pp.59-63, 2005. [49]. C. Rui, K. Zhang and Y. Li, “An Image Fusion Algorithm Using Wavelet Transform”, ACTA ELECTRONICA SINICA, Vol.32, pp.750-75, 2004. [50]. N. Kingsbury, “Shift Invariant properties of the dual-tree complex wavelet transform acoustics, speech, and signal processing”, ICASSP 99. Proceedings, pp.1221-1224, 1999. [51]. N. Kingsbury, “A Dual-Tree Complex Wavelet Transform with Improved Orthogonally and Symmetry Properties”, International Conference on Image Processing, Vol. 2, pp. 375-378, 2000. [52]. C.K. Chow and T. Kaneko, “Automatic Boundary Detection of the Left Ventricle from Cineangiograms, Comp. Biomed”, Vol.5, pp. 388-410, 1972. [53]. M. Ahmad and T. Choi,”Local Threshold and Boolean Function based Edge Detection”, IEEE Transactions on Consumer Electronics, Vol. 45, pp. 674 – 679, Aug 1999. [54]. W. Sun and K.Wang,”A Multi-Focus Image Fusion Algorithm with DT-CWT”, Proc. Int. Conf. Conf. on Computational Intelligence and Security, pp.147-151, 2007. [55]. Ivan, W. Selesnick, Richard G. Baraniuk, and Kingsbury, N., “The Dual-Tree Complex Wavelet Transform”, IEEE Signal Processing Magazine, Vol. 151, pp. 123-151, 2005. [56]. R. Yu, "Theory of Dual-Tree Complex Wavelets", IEEE Transactions on Signal Processing, Vol. S6, no.9, pp.4263-4273, 2008. [57]. J. Tian, et al. “Multi-focus image fusion using a bilateral gradient-based sharpness criterion”, Journal of Optics Communications, Elsevier, Vol.284, pp.80-87, 2011. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 75 Bibliography [58]. A. Agrawal, R. Raskar and R. Chellappa, “Edge suppression by gradient field transformation using cross-projection tensors”, Proc. IEEE Int. Conf. on Computer Vision and Pattern Recognition, Vol.8, pp. 2301-2308, 2006. [59]. C.Y. Wee, and R. Paramesran, “Measure of image sharpness using eigenvalues”, Info. Sciences, Vol.177, pp.2533-2552, 2007. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 76 Dissemination of the Research Work Dissemination of the Research Work [1]. P.R. Muduli and U.C. Pati, “A Novel Technique for Wall Crack Detection Using Image Fusion”, Proc. of IEEE International Conference on Computer Communication and Informatics (ICCCI-2013), Jan-2013, Coimbatore, India. [2]. P. R. Muduli and U.C. Pati, “A Novel Edge Detection Technique for Multi-Focus Images Using Image Fusion”, Advances in Intelligent and Soft Computing, Springer Publication, ISSN: 1867-5670, 2013. (Accepted) [3]. P. R. Muduli and U.C. Pati, “Image Fusion based on Bilateral Sharpness Criterion in DTCWT Domain“, International Journal of Computational Vision and Robotics, INDERSCIENCE Publishers, ISSN: 1752-9131, 2013. (Communicated) [4]. V. S. Bind, P. R. Muduli and U. C. Pati, “A Robust Technique for Feature-based Image Mosaicing using Image Fusion”, International journal of Advanced Computer Research (IJACR), Vol.3, Issue-8, pp.263-268, Mar 2013. Submitted by PRIYA RANJAN MUDULI [211EC3315] Page 77

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement