Time-recursive de-interlacing for high-quality television receivers G. de Haan and P.W.A.C. Biezen. Philips Research Laboratories, Television Systems Group, Prof. Holstlaan 4, WO01, 5656 AA Eindhoven, The Netherlands Abstract Abstract - A new de-interlacing algorithm is proposed, which is suitable for high quality flicker-free display of television images and (as a basis) for line-rate conversions (e.g. vertical zoom). The algorithm includes motion estimation and compensation techniques to achieve a high performance sequential scan conversion for moving and stationary signals. The paper provides the algorithms and a novel evaluation method that shows the improvement in an objective score. 1. Introduction De-interlacing is a basic operation required for most video scanning format conversions. Vertical and temporal interpolation of image data cause practical and fundamental difficulties, as the conditions of the sampling theorem are generally not met in video signals. Linear methods, based on sampling rate conversion theory, therefore negatively influence the resolution and/or the motion portrayal. The more advanced algorithms can be characterized by their common attempt to interpolate the 3-D image data in the direction with the highest correlation. To this end, they have either an explicit or an implicit detector to find this direction. In case of (1D) temporal interpolation the explicit detector is usually called a motion detector. For 2D spatial interpolation it is called an edge detector. The most advanced device estimating the optimal spatio-temporal (3D) interpolation direction is a motion estimator. The interpolation filter can be either recursive or transversal, but the number of taps in the temporal direction is preferably small, because of cost constraints. Recently, papers have been published proposing a recursive scheme for motion compensated sequential scan conversion [1,2]. Experiments suggest that the recursiveness significantly improves the motion compensated median filter , on which these algorithms are based, if good quality motion vectors are available. In this paper an improved recursive sequential scan conversion algorithm is introduced that suppresses remaining artifacts of the prior art further. The proposal is not limited to a combination with a time-recursive deinterlacing algorithm. It can in fact be used to improve any known de-interlacing method. The paper provides the algorithms and a novel evaluation method that shows the improvement in an objective score. For high quality, sub-pixel accurate, motion estimation the algorithm of  is used. 2. De-interlacing techniques In general the samples required for the motion compensated de-interlacing do not exist in the time discrete input signal, e.g. due to non-integer velocities. In the horizontal domain this problem can be solved with linear Sampling Rate Conversion (SRC) theory, see , but not in the vertical domain, as the constraints of the sampling theorem are not met. Three solutions have been proposed recently: 1) A straight extension of the motion vector into earlier pictures until it points to an existing pixel  2) The application of a generalized sampling theorem (GST)  3) Recursive de-interlacing of the signal [1,2] Solution 1) is valid only if we assume the velocity constant over a larger temporal interval. This assumption is often violated which makes the method practically useless. The implication of GST, in method 2), is the possibility to perfectly reconstruct a signal sampled at 1/n times the Nyquist rate with n independent sets of samples that describe the signal. For the de-interlacing problem n = 2, and the required two sets are the current field and the motion compensated previous field. If the two do not coincide, i.e. the object does not have an odd vertical motion vector component, the independency constraint is fulfilled, and the problem can be solved. Practical problems are: a) A perfect reconstruction requires pixels for which the velocity need not be constant b) To apply GST in the motion estimator, three successive fields are required, and the velocity has to be assumed constant over these three fields, which is a drawback in accelerating scene parts c) It is unclear how to cope with unreliable motion vectors, e.g. at picture parts where the motion model is invalid Solution 3) is based on the assumption that it is possible at some time to have a perfectly de-interlaced picture in a memory. Once this is true, this picture is used to de-interlace the next input field. With motion compensation this solution can be perfect as the de-interlaced picture in the memory allows the use of SRCtheory in the vertical domain. If this new de-interlaced field is written in the memory it can be used to deinterlace the next incoming field etc... Limitations of this method are: I) Propagation of errors due to motion vector inaccuracy and interpolation inaccuracies. The proposed protection measures  again introduce alias in the output signal II) Even a perfectly de-interlaced picture can contain alias in the vertical frequency domain, assuming the common case of a camera without an optical prefilter It seems that recursive de-interlacing and de-interlacing based on GST are the best, though still imperfect, methods presently known. In this paper an algorithm is presented that can be applied in combination with both methods to suppress the remaining artifacts in the de-interlaced output signal. In fact this algorithm can be used to improve any de-interlacing algorithm. To show the advantages, we combined the algorithm with the recursive de-interlacing algorithm of , as the implementation requires only minor adaptations to accommodate the improvement. The algorithm of  also served as a reference in our evaluation. 3. Description of the applied algorithms In  a time-recursive de-interlacing algorithm is proposed in which the interpolated pixels are found by motion compensating the previously found de-interlaced output frame: F( x, n ) , Fout( x, n ) ( original pixels ) Fout (x D( x, n ), n 1 ) , ( interpolated pixels ) (1) where F(x, t) is the interlaced input signal, Fout(x, n) the de-interlaced output, n the field number, D(x, n) the displacement or motion vector, and x is the spatial position. To prevent errors from propagating, several measures are described in  to protect the interpolated pixels. Particularly, the median filter is recommended to realize this protection, which modifies eq. (1) to: F( x, n ) , ( original pixels ) 0 , n), 1 Fout( x, n ) median , ( interpolated pixels ) 0 F( x , n) , 1 Fout( x D( x, n ), n 1 ) F( x (2) Although refinements are available in , we use this algorithm as the basis for our comparison. As the quality of the resulting algorithm depends heavily on the performance of the motion estimator, we applied the high quality motion estimation method of . This algorithm, yields a quarter pel accuracy, and a close to truemotion vector field, which is relevant for scan rate conversion. Rather than calculating all possible candidate vectors, the recursive search block-matcher takes spatial and/or temporal "prediction vectors" from a 3-D neighbourhood, and a single updated prediction vector. This implicitly assumes spatial and/or temporal consistency. The updating process involves updates added to either of the spatial predictions. We applied a candidate set CS(X, n), from which the block-matcher selects its result vector D, defined by: CS( X, n ) X X , n) U a( X, n ) C D( X , Y Y X X , n) , D( X , n) , C D( X A D( X Y Y C CS max C D( X (3) where the update vectors Ua(X, n) and Ub(X, n) are alternatingly (on block basis) available, and are taken from a limited fixed integer update vector set US, in our case: USi 0 , 0 0 , 1 0 , 1 0 , 2 0 , 2 1 , 0 1 3 , , 0 0 3 ! 0 (4) To realize sub-pixel accuracy, the update set of eq. (4) is extended with fractional update values. We realized a quarter pel resolution by adding the following fractional vectors to the update set: USf 0 , 0.25 0 , 0.25 0.25 , 0 0.25 ! 0 (5) Because of the small number of candidate vectors, the method is very efficient. Furthermore, due to the inherent smoothness constraint, it yields very coherent vector fields that closely correspond to the true-motion of objects. This particularly, makes it suitable for scanning format conversion. 4. Improved recursive de-interlacing algorithm The main imperfection of the recursive de-interlacing algorithm is remaining alias in the output signal. Although this imperfection is worse for most alternative methods, further improvement seems attractive. Difficulty with this defect is, that it is hardly visible in single images but mainly in moving sequences. This makes illustration in a paper difficult, while also quantitative measures to show the improvement seem to lack. A common method to evaluate the de-interlacing quality, e.g. applied in , compares an original sequentially scanned image with a de-interlaced result using, what we will call here, an MSEs -criterion: MSEs( n ) 1/N M x field Forig( x, n ) Fout( x, n ) 2 (6) This criterion is not exclusively sensitive for alias, as it sums differences without discriminating for different origin, e.g. resolution losses, noise, vector errors, etc. An additional inconvenience is that the criterion cannot be used to check the performance of the algorithm on originally interlaced source signals. A perfectly de-interlaced sequence is stationary along the motion trajectory in parts where the motion model is valid. Applying this characteristic, we suggested an alternative in ; we measured how well the current interlaced input field n was predicted by the motion compensated previously de-interlaced picture: MSEi( n ) 1/N M x field F( x, n ) Fout( x D(x, n ), n 1 ) 2 (7) This measure has the advantage that it can be applied to judge the performance in absence of an original sequentially scanned sequence. However, it has limited value in case of critical velocities, as in that case the quality of the interpolated pixels is not reflected in the figure. In an attempt to improve on this aspect, we suggest here to measure the "Motion Trajectory Inconsistency", MTI(n), for all output lines in field n, defined as: MTI( n ) 1/N M x field Fout( x, n ) Fout( x D( x, n ), n 1 ) 2 (8) A problem with this measure is that a good score is a necessary but insufficient constraint. It is possible, e.g. applying a strong temporal filter, to force the MTI(n) to very low values, while the picture quality is seriously degraded. However, a lower MTI(n) coupled to a hardly increased MSEi(n) score is a strong indication for quality improvement. There is a clear analogy with the motion vector smoothness constraint , where motion estimation techniques have been improved by adding a smoothness term in the match criterion which yields a significant consistency improvement accepting a slight MSE-degradation. Similarly we introduce a Cost-figure here defined as: Cost( n ) [ MSEi( n ) MTI( n ) ] 1/N M x field F( x, n ) Fout( x D( x, n ), n 1 ) 2 /N M x field Fout( x, n ) (9) The parameter allows tuning of the cost function to match the subjective impression. After quantifying "remaining alias" applying eq. (9), evidently an improvement of the MTI figure could be realized by suppressing A nonstationarities along the motion trajectory. As long as this does not seriously degrade the MSE-score, it brings an improved Cost-score and, with tuned correctly, also an improved subjective performance. Baseband spectrum Repeat spectrum A Original lines As nonstationarities reside on interpolated as well as on original lines, the implication is that both have to be A temporally filtered. That filtering of original pixels, which is somewhat contra-intuitive, helps to suppress alias, can also B be understood from the frequency spectrum of the (signal Interpolated lines on) original and interpolated lines. In the original recursive de-interlacing algorithm1, the pixels existing in the input F Fs /2 field are directly fed to the output and never modified. As Figure 1 As the spectrum of the interpolated the first repeat spectrum of the interpolated lines will usually lines is degraded (B) and that of the original suffer from inaccuracies in the motion vector estimates and lines not (A), alias will remain in the result. The the protection features, this spectrum cannot fully repeat spectra of A and B are in opposite phase compensate for the (anti-phase) repeat spectrum resulting due to the interlace. from the original lines. Figure 1 illustrates this matter. F Fs /2 Consequently, the recursive de-interlacing method that we propose interpolates not only the new lines, but, in an attempt to minimize the motion trajectory inconsistency on both type of lines, also the lines existing in the interlaced signal: Fout( x, n ) ( 1 k( R ) ) . ( Fout( x D( x, n ), n 1 ) k( R ) . ( Fi( x, n ) (10) where k(R) is a parameter that reflects the local reliability R of the motion vectors. The initial result Fi(x, n) can equal Fout (x, n) as it results from eq. (2) but also, with little disadvantage as we will show later, using a simpler intra-field interpolation: F( x, n ) Fi( x, n ) F( x ,( original pixels ) 0 0 , n ) F( x , n ) / 2, ,( interpolated pixels ) 1 1 (11) The filtering of pixels on existing lines of the input signal is not necessarily the same as that of the interpolated ones. To reflect that more clearly we modify eq. (10) to: Fout( x, n ) ( 1 k1 ) . ( Fout( x D( x, n ), n 1 ) k1 . ( Fi( x, n ) ) , interpolated ( 1 k2 ) . ( Fout( x D( x, n ), n 1 ) k2 . ( Fi( x, n ) ) (12) , Figure 2 shows the resulting architecture of the proposed de-interlacing algorithm. Using a limiter function "clip" defined as: 1 In fact, in all alternative de-interlacing algorithms. 0, (a < 0) clip( 0, 1, a ) 1, (a > 1) (13) a, ( 0 a 1 ) k2 is calculated as: k2 clip 0, 1, C2 . Diff (14) with: Diff( x, n ) Fout( x D( x, n ), n 1 ) Fi( x, n ) (15) Consequently, k2 depends on the prediction error, Diff, in the motion compensated recursive loop. As Diff is related to the MSE(n) of eq. (7) it is possible by appropriately choosing C2, to tune the relative importance of MSE(n) and MTI(n), as defined by eqs. (7, 8). . Compress & multiplex 1-k 1 Interlaced input Initial Interpolated k1 lines sequential scan Original convertor lines k2 De-interlaced output Field memory Motion compensation stage Field 1-k 2 memory Motion estimator Figure 2 Architecture for proposed sequential scan conversion algorithm. Parameters k are controlled by the vector reliability. As the reliability of interpolated pixels, resulting from the initial de-interlacing method, is less than that of the original pixels, the filtering of these pixels should be stronger. As furthermore, Diff(x, n), on interpolated lines, has little value to determine the quality of the motion compensated prediction we propose to derive k1 from the quality of the motion compensated prediction at the vertically neighbouring original pixels using: k1 clip 0, 1, C1 . Diff( x 0 0 , n ) Diff( x , n) 1 1 where C1 is smaller than C2 ( 0.0625 and 0.3125 respectively in the experiments ). (16) There is a risk in filtering alternate lines differently, as it may introduce visible line structures. Although the advantage of high quality de-interlacing will be apparent in areas with vertical detail, there is no advantage in regions that lack such high vertical frequency components. Therefore, to prevent the introduction of line structure in image parts that never profit from individual filtering, k1 is made equal to k2 if: MSE & MTI per sequence of the three algorithms 160 Legend 140 Ref. A1 A2 120 100 80 60 I rT o E S M 40 Diff( x 20 0 , n ) Diff( x 1 0 calendar mobile kiel tokyo calendar mobile kiel (17) tokyo MSE (left) and MTI (right) for individual sequences Figure 3 Comparison of algorithm performance indicators, averaged over 12 frames. 2. 5. Evaluation To evaluate the proposals resulting from the previous section, we selected a set of four critical sequences. Individual images from each of these four sequences are shown in the Figures 4-7. The sequences contain ample vertical detail, and motion with various, also sub-pixel, values in many directions. Using the algorithm of , in the version described in section 2 as a reference (REF), we calculated MSEs and MTIs for another two algorithms, resulting from the proposals in this paper and illustrated in Figure 2. The first algorithm (A1) has the control of the recursive loop realized according to eqs. (10-17). The second algorithm (A2) has k2 = 1, i.e. no recursive filtering of the original pixels, but is further identical to A1. Figure 4 Image from the "calendar" sequence. The vertically detailed image is moving upwards with a slowly increasing velocity. Figure 5 Image from the "Kiel-harbour" sequence. The sequence shows a relatively fast zoom of the harbour. Figure 7 Image from the "Mobile" sequence, containing horizontal as well as vertical motion. Critical vertical velocities do occur. Figure 6 Image from the "Tokyo" sequence, containing mainly slow horizontal motion and very high vertical frequencies. Figure 3 shows the average result of the comparison per sequence. Compared to REF both A1 and A2 yield a clearly improved MSE figure, which is expected to be mainly due to the elimination of median defects in the high spatial frequencies. The MTI figures, have also dropped considerably, particularly for the algorithm A1, i.e. the algorithm that also filters the original pixels. MSE figures of algorithm A1 and A2 differ only little. The additional recursive filtering of the original pixels mainly improves the MTI figure (with little or no disadvantage for the MSE figure). MSE(n) MTI(n) averaged over 4 sequences 110 averaged over 4 sequences 100 100 80 90 60 80 E S M I T M 40 70 60 20 Legend A1 Legend A2 Ref A1 50 A2 Ref 0 8 9 10 11 12 13 14 15 Frame number n 16 17 18 19 20 Figure 8 MSE(n) scores of the tested algorithms. 8 9 10 11 12 13 14 15 Frame number n 16 17 18 19 20 Figure 9 MTI(n) scores of the tested algorithms. Figures 8 and 9 show for twelve successive frames the MSE(n) and MTI(n) averaged over the four sequences for the three algorithms under test. The first processed output fields are omitted to allow for convergence of the recursive motion estimator and the recursive de-interlacing filter. 6. Conclusion An improvement generally applicable to existing de-interlacing algorithms, and a new evaluation measure for such algorithms have been proposed in this paper. The improvement is characterized by using motion compensated temporal recursive filtering. A typical feature of the proposal is that this filtering is not limited to the interpolated pixels only, but is extended to the filtering of the pixels existing in the interlaced input signal. This somewhat contra-intuitive action followed from the assumption that in order to have the first vertical repeat spectrum of original pixels compensated by that of the interpolated ones, it is essential that the two have identical vertical frequency response. As it cannot be prevented that the frequency spectrum of the interpolated pixels may be distorted by the limited accuracy of the applied motion vectors, a similar distortion can best be applied to that of the original pixels in order to maximally suppress alias. The application of the improvement in time-recursive de-interlacing was elaborated and the improved performance was verified. The verification has to be understood in a sense that the classical MSE performance measure had not suffered (even improved somewhat), whereas the new proposed criterion "inconsistency along the motion trajectory" (MTI) had greatly improved. 7. References         F.M. Wang, D. Anastassiou and A.N. Netravali, "Time-Recursive Deinterlacing for IDTV and Pyramid Coding", Signal processing: Image Communication 2, Elsevier 1990, pp.365-374. Kwon, Seo, Kim, and Kim, "A Motion Adaptive De-Interlacing Method", IEEE Tr. on Consumer Electronics, Vol. 38, No.3, 1992. G. de Haan and G.F.M. De Poortere, "Method and apparatus for processing a picture signal", European Patent Application no. EP-A 0 474 287. G. de Haan and P.W.A.C. Biezen, "Sub-pixel motion estimation with 3-D recursive search blockmatching", Signal Processing: Image Communication 6 (1994), pp.229-239. P. Delogne, L. Cuvelier, B. Maison, B. Van Caillie and L. Vandendorpe, "Improved Interpolation, Motion Estimation, and Compensation for Interlaced Pictures", IEEE Tr. on Image Processing, Vol. 3, No. 5, September 1994. A.W.M. van den Enden and N.A.M. Verhoeckx, Discrete-time signal processing, Prentice Hall (ISBN 0-13-216763-8), pp. 233-. J.W. Woods and S.C. Han, "Hierarchical Motion Compensated De-interlacing", Proc. SPIE Visual Communication and Image Processing VI, Boston, November 1991. G. de Haan, P.W.A.C. Biezen, H. Huijgen and O.A. Ojo, "True Motion Estimation with 3-D Recursive Search Block-Matching", IEEE Tr. on Circuits & Systems for Video Technology, Vol. 3, October 1993, pp. 368-388.
* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project