User manual | Time-recursive de-interlacing for high

Time-recursive de-interlacing for high-quality television receivers
G. de Haan and P.W.A.C. Biezen.
Philips Research Laboratories, Television Systems Group,
Prof. Holstlaan 4, WO01, 5656 AA Eindhoven, The Netherlands
Abstract
Abstract - A new de-interlacing algorithm is proposed, which is suitable for high quality flicker-free display
of television images and (as a basis) for line-rate conversions (e.g. vertical zoom). The algorithm includes
motion estimation and compensation techniques to achieve a high performance sequential scan conversion for
moving and stationary signals. The paper provides the algorithms and a novel evaluation method that shows
the improvement in an objective score.
1. Introduction
De-interlacing is a basic operation required for most video scanning format conversions. Vertical and temporal
interpolation of image data cause practical and fundamental difficulties, as the conditions of the sampling
theorem are generally not met in video signals. Linear methods, based on sampling rate conversion theory,
therefore negatively influence the resolution and/or the motion portrayal. The more advanced algorithms can
be characterized by their common attempt to interpolate the 3-D image data in the direction with the highest
correlation. To this end, they have either an explicit or an implicit detector to find this direction. In case of (1D)
temporal interpolation the explicit detector is usually called a motion detector. For 2D spatial interpolation
it is called an edge detector. The most advanced device estimating the optimal spatio-temporal (3D)
interpolation direction is a motion estimator. The interpolation filter can be either recursive or transversal,
but the number of taps in the temporal direction is preferably small, because of cost constraints.
Recently, papers have been published proposing a recursive scheme for motion compensated sequential scan
conversion [1,2]. Experiments suggest that the recursiveness significantly improves the motion compensated
median filter [3], on which these algorithms are based, if good quality motion vectors are available. In this
paper an improved recursive sequential scan conversion algorithm is introduced that suppresses remaining
artifacts of the prior art further. The proposal is not limited to a combination with a time-recursive deinterlacing algorithm. It can in fact be used to improve any known de-interlacing method. The paper provides
the algorithms and a novel evaluation method that shows the improvement in an objective score. For high
quality, sub-pixel accurate, motion estimation the algorithm of [4] is used.
2. De-interlacing techniques
In general the samples required for the motion compensated de-interlacing do not exist in the time discrete input
signal, e.g. due to non-integer velocities. In the horizontal domain this problem can be solved with linear
Sampling Rate Conversion (SRC) theory, see [6], but not in the vertical domain, as the constraints of the
sampling theorem are not met. Three solutions have been proposed recently:
1)
A straight extension of the motion vector into earlier pictures until it points to an existing pixel [7]
2)
The application of a generalized sampling theorem (GST) [5]
3)
Recursive de-interlacing of the signal [1,2]
Solution 1) is valid only if we assume the velocity constant over a larger temporal interval. This assumption is
often violated which makes the method practically useless.
The implication of GST, in method 2), is the possibility to perfectly reconstruct a signal sampled at 1/n times
the Nyquist rate with n independent sets of samples that describe the signal. For the de-interlacing problem n
= 2, and the required two sets are the current field and the motion compensated previous field. If the two do not
coincide, i.e. the object does not have an odd vertical motion vector component, the independency constraint
is fulfilled, and the problem can be solved. Practical problems are:
a)
A perfect reconstruction requires pixels for which the velocity need not be constant
b)
To apply GST in the motion estimator, three successive fields are required, and the velocity has to be
assumed constant over these three fields, which is a drawback in accelerating scene parts
c)
It is unclear how to cope with unreliable motion vectors, e.g. at picture parts where the motion model
is invalid
Solution 3) is based on the assumption that it is possible at some time to have a perfectly de-interlaced picture
in a memory. Once this is true, this picture is used to de-interlace the next input field. With motion
compensation this solution can be perfect as the de-interlaced picture in the memory allows the use of SRCtheory in the vertical domain. If this new de-interlaced field is written in the memory it can be used to deinterlace the next incoming field etc... Limitations of this method are:
I)
Propagation of errors due to motion vector inaccuracy and interpolation inaccuracies. The proposed
protection measures [1] again introduce alias in the output signal
II)
Even a perfectly de-interlaced picture can contain alias in the vertical frequency domain, assuming the
common case of a camera without an optical prefilter
It seems that recursive de-interlacing and de-interlacing based on GST are the best, though still imperfect,
methods presently known. In this paper an algorithm is presented that can be applied in combination with both
methods to suppress the remaining artifacts in the de-interlaced output signal. In fact this algorithm can be used
to improve any de-interlacing algorithm. To show the advantages, we combined the algorithm with the recursive
de-interlacing algorithm of [1], as the implementation requires only minor adaptations to accommodate the
improvement. The algorithm of [1] also served as a reference in our evaluation.
3. Description of the applied algorithms
In [1] a time-recursive de-interlacing algorithm is proposed in which the interpolated pixels are found by motion
compensating the previously found de-interlaced output frame:
F( x, n ) ,
Fout( x, n ) ( original pixels )
Fout (x D( x, n ), n 1 ) , ( interpolated pixels )
(1)
where F(x, t) is the interlaced input signal, Fout(x, n) the de-interlaced output, n the field number, D(x, n) the
displacement or motion vector, and x is the spatial position.
To prevent errors from propagating, several measures are described in [1] to protect the interpolated pixels.
Particularly, the median filter is recommended to realize this protection, which modifies eq. (1) to:
F( x, n ) ,
( original pixels )
0
, n),
1
Fout( x, n ) median
, ( interpolated pixels )
0
F( x , n) ,
1
Fout( x D( x, n ), n 1 )
F( x (2)
Although refinements are available in [1], we use this algorithm as the basis for our comparison. As the quality
of the resulting algorithm depends heavily on the performance of the motion estimator, we applied the high
quality motion estimation method of [5]. This algorithm, yields a quarter pel accuracy, and a close to truemotion vector field, which is relevant for scan rate conversion. Rather than calculating all possible candidate
vectors, the recursive search block-matcher takes spatial and/or temporal "prediction vectors" from a 3-D
neighbourhood, and a single updated prediction vector. This implicitly assumes spatial and/or temporal
consistency. The updating process involves updates added to either of the spatial predictions. We applied a
candidate set CS(X, n), from which the block-matcher selects its result vector D, defined by:
CS( X, n ) X
X
, n) U a( X, n ) œ C D( X ,
Y
Y
X
X
, n) , D( X , n) , C D( X
A D( X Y
Y
C CS max C D( X (3)
where the update vectors Ua(X, n) and Ub(X, n) are alternatingly (on block basis) available, and are taken from
a limited fixed integer update vector set US, in our case:
USi 0
,
0
0
,
1
0
,
1
0
,
2
0
,
2
1
,
0
1
3
,
,
0
0
3
!
0
(4)
To realize sub-pixel accuracy, the update set of eq. (4) is extended with fractional update values. We realized
a quarter pel resolution by adding the following fractional vectors to the update set:
USf 0
,
0.25
0
,
0.25
0.25
,
0
0.25
!
0
(5)
Because of the small number of candidate vectors, the method is very efficient. Furthermore, due to the inherent
smoothness constraint, it yields very coherent vector fields that closely correspond to the true-motion of objects.
This particularly, makes it suitable for scanning format conversion.
4. Improved recursive de-interlacing algorithm
The main imperfection of the recursive de-interlacing algorithm is remaining alias in the output signal. Although
this imperfection is worse for most alternative methods, further improvement seems attractive. Difficulty with
this defect is, that it is hardly visible in single images but mainly in moving sequences. This makes illustration
in a paper difficult, while also quantitative measures to show the improvement seem to lack. A common method
to evaluate the de-interlacing quality, e.g. applied in [5], compares an original sequentially scanned image with
a de-interlaced result using, what we will call here, an MSEs -criterion:
MSEs( n ) 1/N M
x field
Forig( x, n ) Fout( x, n ) 2
(6)
This criterion is not exclusively sensitive for alias, as it sums differences without discriminating for different
origin, e.g. resolution losses, noise, vector errors, etc. An additional inconvenience is that the criterion cannot
be used to check the performance of the algorithm on originally interlaced source signals.
A perfectly de-interlaced sequence is stationary along the motion trajectory in parts where the motion model
is valid. Applying this characteristic, we suggested an alternative in [5]; we measured how well the current
interlaced input field n was predicted by the motion compensated previously de-interlaced picture:
MSEi( n ) 1/N M
x field
F( x, n ) Fout( x D(x, n ), n 1 )
2
(7)
This measure has the advantage that it can be applied to judge the performance in absence of an original
sequentially scanned sequence. However, it has limited value in case of critical velocities, as in that case the
quality of the interpolated pixels is not reflected in the figure.
In an attempt to improve on this aspect, we suggest here to measure the "Motion Trajectory Inconsistency",
MTI(n), for all output lines in field n, defined as:
MTI( n ) 1/N M
x field
Fout( x, n ) Fout( x D( x, n ), n 1 )
2
(8)
A problem with this measure is that a good score is a necessary but insufficient constraint. It is possible, e.g.
applying a strong temporal filter, to force the MTI(n) to very low values, while the picture quality is seriously
degraded. However, a lower MTI(n) coupled to a hardly increased MSEi(n) score is a strong indication for
quality improvement. There is a clear analogy with the motion vector smoothness constraint [8], where motion
estimation techniques have been improved by adding a smoothness term in the match criterion which yields a
significant consistency improvement accepting a slight MSE-degradation. Similarly we introduce a Cost-figure
here defined as:
Cost( n ) [ MSEi( n ) MTI( n ) ] 1/N M
x field
F( x, n ) Fout( x D( x, n ), n 1 ) 2 /N M
x field
Fout( x, n )
(9)
The parameter allows tuning of the cost function to match the subjective impression. After quantifying
"remaining alias" applying eq. (9), evidently an improvement of the MTI figure could be realized by suppressing
A
nonstationarities along the motion trajectory. As long as this
does not seriously degrade the MSE-score, it brings
an improved Cost-score and, with tuned correctly, also an
improved subjective performance.
Baseband spectrum
Repeat spectrum
A
Original lines
As nonstationarities reside on interpolated as well as on
original lines, the implication is that both have to be
A
temporally filtered. That filtering of original pixels, which is
somewhat contra-intuitive, helps to suppress alias, can also
B
be understood from the frequency spectrum of the (signal
Interpolated lines
on) original and interpolated lines. In the original recursive
de-interlacing algorithm1, the pixels existing in the input
F
Fs /2
field are directly fed to the output and never modified. As
Figure 1 As the spectrum of the interpolated the first repeat spectrum of the interpolated lines will usually
lines is degraded (B) and that of the original suffer from inaccuracies in the motion vector estimates and
lines not (A), alias will remain in the result. The the protection features, this spectrum cannot fully
repeat spectra of A and B are in opposite phase compensate for the (anti-phase) repeat spectrum resulting
due to the interlace.
from the original lines. Figure 1 illustrates this matter.
F
Fs /2
Consequently, the recursive de-interlacing method that we propose interpolates not only the new lines, but, in
an attempt to minimize the motion trajectory inconsistency on both type of lines, also the lines existing in the
interlaced signal:
Fout( x, n ) ( 1 k( R ) ) . ( Fout( x D( x, n ), n 1 ) k( R ) . ( Fi( x, n )
(10)
where k(R) is a parameter that reflects the local reliability R of the motion vectors. The initial result Fi(x, n) can
equal Fout (x, n) as it results from eq. (2) but also, with little disadvantage as we will show later, using a simpler
intra-field interpolation:
F( x, n )
Fi( x, n ) F( x ,( original pixels )
0
0
, n ) F( x , n ) / 2, ,( interpolated pixels )
1
1
(11)
The filtering of pixels on existing lines of the input signal is not necessarily the same as that of the interpolated
ones. To reflect that more clearly we modify eq. (10) to:
Fout( x, n ) ( 1 k1 ) . ( Fout( x D( x, n ), n 1 ) k1 . ( Fi( x, n ) ) , interpolated
( 1 k2 ) . ( Fout( x D( x, n ), n 1 ) k2 . ( Fi( x, n ) )
(12)
,
Figure 2 shows the resulting architecture of the proposed de-interlacing algorithm. Using a limiter function
"clip" defined as:
1
In fact, in all alternative de-interlacing algorithms.
0,
(a < 0)
clip( 0, 1, a ) 1,
(a > 1)
(13)
a, ( 0 a 1 )
k2 is calculated as:
k2 clip 0, 1, C2 . Diff (14)
with:
Diff( x, n ) Fout( x D( x, n ), n 1 ) Fi( x, n )
(15)
Consequently, k2 depends on the prediction error, Diff, in the motion compensated recursive loop. As Diff is
related to the MSE(n) of eq. (7) it is possible by appropriately choosing C2, to tune the relative importance of
MSE(n) and MTI(n), as defined by eqs. (7, 8).
.
Compress
&
multiplex
1-k 1
Interlaced
input
Initial
Interpolated
k1
lines
sequential
scan
Original
convertor
lines
k2
De-interlaced
output
Field
memory
Motion
compensation
stage
Field
1-k 2
memory
Motion
estimator
Figure 2 Architecture for proposed sequential scan conversion algorithm. Parameters k are controlled by the
vector reliability.
As the reliability of interpolated pixels, resulting from the initial de-interlacing method, is less than that of the
original pixels, the filtering of these pixels should be stronger. As furthermore, Diff(x, n), on interpolated lines,
has little value to determine the quality of the motion compensated prediction we propose to derive k1 from the
quality of the motion compensated prediction at the vertically neighbouring original pixels using:
k1 clip 0, 1, C1 .
Diff( x 0
0
, n ) Diff( x , n)
1
1
where C1 is smaller than C2 ( 0.0625 and 0.3125 respectively in the experiments ).
(16)
There is a risk in filtering alternate lines
differently, as it may introduce visible line
structures. Although the advantage of high
quality de-interlacing will be apparent in areas
with vertical detail, there is no advantage in
regions that lack such high vertical frequency
components. Therefore, to prevent the
introduction of line structure in image parts that
never profit from individual filtering, k1 is made
equal to k2 if:
MSE & MTI per sequence
of the three algorithms
160
Legend
140
Ref.
A1
A2
120
100
80
60
I
rT
o
E
S
M
40
Diff( x 20
0
, n ) Diff( x 1
0
calendar
mobile
kiel
tokyo
calendar
mobile
kiel
(17)
tokyo
MSE (left) and MTI (right) for individual sequences
Figure 3 Comparison of algorithm performance indicators,
averaged over 12 frames.
2.
5. Evaluation
To evaluate the proposals resulting from the previous section, we selected a set of four critical sequences.
Individual images from each of these four sequences are shown in the Figures 4-7. The sequences contain ample
vertical detail, and motion with various, also sub-pixel, values in many directions. Using the algorithm of [1],
in the version described in section 2 as a reference (REF), we calculated MSEs and MTIs for another two
algorithms, resulting from the proposals in this paper and illustrated in Figure 2. The first algorithm (A1) has
the control of the recursive loop realized according to eqs. (10-17). The second algorithm (A2) has k2 = 1, i.e.
no recursive filtering of the original pixels, but is further identical to A1.
Figure 4 Image from the "calendar" sequence. The
vertically detailed image is moving upwards with a
slowly increasing velocity.
Figure 5 Image from the "Kiel-harbour" sequence.
The sequence shows a relatively fast zoom of the
harbour.
Figure 7 Image from the "Mobile" sequence,
containing horizontal as well as vertical motion.
Critical vertical velocities do occur.
Figure 6 Image from the "Tokyo" sequence,
containing mainly slow horizontal motion and very
high vertical frequencies.
Figure 3 shows the average result of the comparison per sequence. Compared to REF both A1 and A2 yield a
clearly improved MSE figure, which is expected to be mainly due to the elimination of median defects in the
high spatial frequencies. The MTI figures, have also dropped considerably, particularly for the algorithm A1,
i.e. the algorithm that also filters the original pixels. MSE figures of algorithm A1 and A2 differ only little. The
additional recursive filtering of the original pixels mainly improves the MTI figure (with little or no
disadvantage for the MSE figure).
MSE(n)
MTI(n)
averaged over 4 sequences
110
averaged over 4 sequences
100
100
80
90
60
80
E
S
M
I
T
M
40
70
60
20
Legend
A1
Legend
A2
Ref
A1
50
A2
Ref
0
8
9
10
11
12
13
14
15
Frame number n
16
17
18
19
20
Figure 8 MSE(n) scores of the tested algorithms.
8
9
10
11
12
13
14
15
Frame number n
16
17
18
19
20
Figure 9 MTI(n) scores of the tested algorithms.
Figures 8 and 9 show for twelve successive frames the MSE(n) and MTI(n) averaged over the four sequences
for the three algorithms under test. The first processed output fields are omitted to allow for convergence of the
recursive motion estimator and the recursive de-interlacing filter.
6. Conclusion
An improvement generally applicable to existing de-interlacing algorithms, and a new evaluation measure for
such algorithms have been proposed in this paper. The improvement is characterized by using motion
compensated temporal recursive filtering. A typical feature of the proposal is that this filtering is not limited to
the interpolated pixels only, but is extended to the filtering of the pixels existing in the interlaced input signal.
This somewhat contra-intuitive action followed from the assumption that in order to have the first vertical repeat
spectrum of original pixels compensated by that of the interpolated ones, it is essential that the two have
identical vertical frequency response. As it cannot be prevented that the frequency spectrum of the interpolated
pixels may be distorted by the limited accuracy of the applied motion vectors, a similar distortion can best be
applied to that of the original pixels in order to maximally suppress alias. The application of the improvement
in time-recursive de-interlacing was elaborated and the improved performance was verified. The verification
has to be understood in a sense that the classical MSE performance measure had not suffered (even improved
somewhat), whereas the new proposed criterion "inconsistency along the motion trajectory" (MTI) had greatly
improved.
7. References
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
F.M. Wang, D. Anastassiou and A.N. Netravali, "Time-Recursive Deinterlacing for IDTV and Pyramid
Coding", Signal processing: Image Communication 2, Elsevier 1990, pp.365-374.
Kwon, Seo, Kim, and Kim, "A Motion Adaptive De-Interlacing Method", IEEE Tr. on Consumer
Electronics, Vol. 38, No.3, 1992.
G. de Haan and G.F.M. De Poortere, "Method and apparatus for processing a picture signal", European
Patent Application no. EP-A 0 474 287.
G. de Haan and P.W.A.C. Biezen, "Sub-pixel motion estimation with 3-D recursive search blockmatching", Signal Processing: Image Communication 6 (1994), pp.229-239.
P. Delogne, L. Cuvelier, B. Maison, B. Van Caillie and L. Vandendorpe, "Improved Interpolation,
Motion Estimation, and Compensation for Interlaced Pictures", IEEE Tr. on Image Processing, Vol. 3,
No. 5, September 1994.
A.W.M. van den Enden and N.A.M. Verhoeckx, Discrete-time signal processing, Prentice Hall (ISBN
0-13-216763-8), pp. 233-.
J.W. Woods and S.C. Han, "Hierarchical Motion Compensated De-interlacing", Proc. SPIE Visual
Communication and Image Processing VI, Boston, November 1991.
G. de Haan, P.W.A.C. Biezen, H. Huijgen and O.A. Ojo, "True Motion Estimation with 3-D Recursive
Search Block-Matching", IEEE Tr. on Circuits & Systems for Video Technology, Vol. 3, October 1993,
pp. 368-388.

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project